Markdown can affect performance

Markdown is a very popular input format for static site generators, that's why Nikola has supported it since very early in life, even if I prefer reSt most of the time.

One thing that has surprised me a while ago is that, considering how minimalistic markdown is, and how little it does, it can be pretty slow to process. But it turns out this is not completely markdown's fault, but that the python markdown implementation is really, really, really slow.

How slow? Let's check it out. All benchmarks in this post were done in my notebook, a nice Asus Zenbook UX305, with a fast SSD and plenty of RAM. In all cases the builds were done using no parallelization.

The test site is an empty site to which I added 1000 copies of a simple, not too large markdown file (our theming guide, converted from reSt using pandoc).

As a baseline, a site with 1000 copies of the original reSt theming.txt builds in 126 seconds.

The markdown version of that site? It gets progressively slower as files build, so I suspect it leaks something between builds, and finishes in ... 1584 seconds. That is over 12 times slower than the reSt compiler.

But you don't want to switch to reSt? No problem! We support more markdown compilers than we probably should, so you can just choose which one you prefer.

Here is a chart showing the performance of each (HTML and reSt added as reference):

Seconds to build (bigger is worse) 00100100200200300300400400500500600600700700800800900900100010001100110012001200130013001400140015001500158472.67350427350428267.00000000000006194127.80512820512821492.28773310023314126182.93675213675215503.3090034965035110238.06837606837607505.9022435897436564293.20000000000005513.357808857808934348.331623931624518.220134032634132403.46324786324783518.544289044289125458.5948717948718519.678831585081614513.7264957264957521.4616841491842Seconds to build (bigger is worse)markdownkramdownrestpandoccommonmarkmistunemistune+cythonmisakahtml

There are just a few disadvantages to using any of the alternative Markdown compilers:

  • They are less tested, because fewer people use them (and if you use them, that changes!)
  • They may lack a specific feature of the "standard" markdown compiler, such as extension support (in which case, file a bug with us!)
  • The produced output may differ from the "standard", but that happens :-)

Nikola v7.8.1 is out!

On behalf of the Nikola team, I am pleased to announce the immediate availability of Nikola v7.8.1. It fixes some bugs and adds new features.

What is Nikola?

Nikola is a static site and blog generator, written in Python. It can use Mako and Jinja2 templates, and input in many popular markup formats, such as reStructuredText and Markdown — and can even turn Jupyter (IPython) Notebooks into blog posts! It also supports image galleries, and is multilingual. Nikola is flexible, and page builds are extremely fast, courtesy of doit (which is rebuilding only what has been changed).

Find out more at the website: https://getnikola.com/

Downloads

Install using pip install Nikola or download tarballs on GitHub and PyPI.

Changes

Bugfixes

  • Rename stories to pages in most of the codebase (Issue #1891)
  • Report missing/unknown dates in posts with filenames (Issues #2505, #2506)
  • Report filename of post that raised exception when scanning (Issue #2505)
  • Accept page and pages for the type field in post lists
  • Don’t crash if a post list has no posts (Issue #2489)

Features

  • Add Hebrew translation by Yitzhak Bar Geva
  • Add Chinese (Traditional) zh_TW translation by Po-Wen Chi
  • Add nikola theme --list-installed (Issue #2504)
  • Add KATEX_AUTO_RENDER setting to configure math delimiters
  • Update KaTeX version to 0.6.0: support aligned math display
  • Better support for a tree of files in data/

Really Static Comments with Staticman

Static sites have many advantages, we always say. But they also have at least one large inherent disadvantage: they are not dynamic. Not being dynamic means that forms that alter the content of the site itself are not possible without going outside the static side of things.

That's why for comments we all end up using something like Disqus or some other alternative service, or Isso if you prefer the annoyance of self-hosting instead of the annoyance of having your data be owned by someone else.

So... how about a service that lets you do forms, but:

  • You keep full control of your data
  • The service doesn't limit what you can do with the data
  • You can use it for most things a form is used for?
  • It's free!

That service is called Staticman and it's awesome.

This is how it works:

And Staticman will take the form's data, convert it to JSON, YAML, or whatever, and put it back in your site's GitHub repo. You can choose whether it uses "moderation", and do that via a pull request, or not, in which case it just commits directly.

Nikola can then use those data files from its templates, like in this example which displays a form to post comments, and a list of previous comments.

Then, if you combine that with an autobuild system, like using Travis or Netlify or Gitlab that means a few seconds/minutes after the form is submitted, the data appears back in the site.

So, what do we end up with? A site with comments, where you own all the data, and not even need to use Javascript to display the comments!

You can see it in action in this demo site: staticman-demo.netlify.com

TODO: submit the form via Javascript instead, so it does it in the background instead of dumping JSON results on you.

Have fun, leave a comment :-)

From Zero to Nikola in One Minute Without Installing Anything

Maybe slightly more than one minute, but it's close to that. Thanks to netlify and GitHub (or GitLab or Bitbucket) you can create, edit and publish a Nikola-based blog without installing anything on your own computer.

This gives you about 75% of the convenience of a hosted blogging solution while keeping all the advantages of a static site, like data ownership, versioning and avoiding vendor lockin.

How to do it: you start by forking this repo and follow the steps in this video:

Keep in mind that this is not in any way a crippled version of Nikola, it has all the same features as you would get installing it on your own device, you can add themes, plugins, image galleries, etc by just adding things in your GitHub repository.

You should be able to do this with your own site, as long as it's stored in a GitHub repository and you add a requirements.txt like this one to your root directory.

So, a big round of applause for netlify for an awesome service!

Nikola v7.8.0 is out!

On behalf of the Nikola team, I am pleased to announce the immediate availability of Nikola v7.8.0. It fixes some bugs and adds new features.

What is Nikola?

Nikola is a static site and blog generator, written in Python. It can use Mako and Jinja2 templates, and input in many popular markup formats, such as reStructuredText and Markdown — and can even turn Jupyter (IPython) Notebooks into blog posts! It also supports image galleries, and is multilingual. Nikola is flexible, and page builds are extremely fast, courtesy of doit (which is rebuilding only what has been changed).

Find out more at the website: https://getnikola.com/

Downloads

Install using pip install Nikola or download tarballs on GitHub and PyPI.

Changes

Features

  • Exposed gist as a shortcode (Issue #2459)
  • Always copy source files for listings (Issue #2473)
  • Detect dependencies in template strings (Issue #2455)
  • RSS feeds for sections (Issue #2068)
  • New data metadata that loads data from external files (Issue #2450)
  • Shortcode to escape to the template language (Issue #1227)
  • Added link to raw file in listings (Issue #1995)
  • New NO_DOCUTILS_TITLE_TRANSFORM (Issue #2382)
  • Update options of chart directive to Pygal 2.2.3
  • Pass global context to template shortcodes (Issue #2424)
  • Added new options --html2text and --transform-to-markdown to WordPress importer (Issue #2261)
  • Listing: guess the lexer if cannot be determined from the file name.
  • Read files from data/ and insert data in global context (Issue #2477)

Bugfixes

  • Added link to sources in the output of listing directive (Issue #2472)
  • Fix Windows crash with calendar locales (Issue #2332)
  • Remove the (useless) <base> tag by default; change conf.py to opt in (Issue #2471)
  • Show tag descriptions when TAG_PAGES_ARE_INDEXES is True (Issue #2444)
  • Record template dependencies for post-list shortcut (Issue #2451)
  • Default to English for docutils messages if no translations exist (Issues #2422, #2437)

Automating Nikola rebuilds with Travis CI

In this guide, we’ll set up Travis CI to rebuild a Nikola website and host it on GitHub Pages.

Why?

By using Travis CI to build your site, you can easily blog from anywhere you can edit text files. Which means you can blog with only a web browser and GitHub.com. You also won’t need to install Nikola and Python to write. You won’t need a real computer either — a mobile phone could probably access GitHub.com and write something.

Caveats

  • The build might take a couple minutes to finish (1:30 for the demo site; YMMV)
  • When you commit and push to GitHub, the site will be published unconditionally. If you don’t have a copy of Nikola for local use, there is no way to preview your site.

What you need

  • A computer for the initial setup that can run Nikola and the Travis CI command-line tool (written in Ruby) — you need a Unix-like system (Linux, macOS, *BSD, etc.); Windows users should try Bash on Ubuntu on Windows (available in Windows 10 starting with Anniversary Update) or a Linux virtual machine.
  • A GitHub account (free)
  • A Travis CI account linked to your GitHub account (free)

Setting up Nikola

Start by creating a new Nikola site and customizing it to your liking. Follow the Getting Started guide. You might also want to add support for other input formats, namely Markdown, but this is not a requirement.

After you’re done, you must configure deploying to GitHub in Nikola. There are a few important things you need to take care of:

  • Make your first deployment from your local computer and make sure your site works right. Don’t forget to set up .gitignore (We’ll add two very important entries later.)
  • You must set GITHUB_COMMIT_SOURCE = False — otherwise, Travis CI will go into an infinite loop.
  • We assume your source branch is src and you deploy to master. Any other configuration requires editing .travis.yml.
  • We enable builds only for the src branch by default. Older versions of the script did not include this provision, and thus committing to master (which you should not do, as your changes will be overwritten on next Travis rebuild) used to cause Rakefile errors.

If everything works, you can make some change to your site (so you see that rebuilding works), but don’t commit it just yet.

Setting up Travis CI

Next, we need to set up Travis CI. To do that, make sure you have the ruby and gem tools installed on your system. If you don’t have them, install them from your OS package manager.

First, download/copy the .travis.yml file (note the dot in the beginning; the downloaded file doesn’t have it!) and adjust the real name, e-mail (used for commits; line 15/16), and the username/repo name on line 24. If you want to render your site in another language besides English, add the appropriate Ubuntu language pack to the list in this file. Likewise, if you need any other Python/apt packages to build your site, add them to your config.

travis.yml (Source)

# Travis CI config for automated Nikola blog deployments
language: python
cache: apt
sudo: false
addons:
  apt:
    packages:
    - language-pack-en-base
branches:
  only:
  - src
python:
- 3.6
before_install:
- git config --global user.name 'Travis CI'
- git config --global user.email '[email protected]'
- git config --global push.default 'simple'
- pip install --upgrade pip wheel
- echo -e 'Host github.com\n    StrictHostKeyChecking no' >> ~/.ssh/config
- eval "$(ssh-agent -s)"
- chmod 600 id_rsa
- ssh-add id_rsa
- git remote rm origin
- git remote add origin git@github.com:USERNAME/REPO.git
- git fetch origin master
- git branch master FETCH_HEAD
install:
- pip install 'Nikola[extras]'
script:
- nikola build && nikola github_deploy -m 'Nikola auto deploy [ci skip]'
notifications:
    email:
        on_success: change
        on_failure: always

Next, we need to generate a SSH key for Travis CI.

echo id_rsa >> .gitignore
echo id_rsa.pub >> .gitignore
ssh-keygen -C TravisCI -f id_rsa -N ''

Open the id_rsa.pub file and copy its contents. Go to GitHub → your page repository → Settings → Deploy keys and add it there. Make sure Allow write access is checked.

And now, time for our venture into the Ruby world. Install the travis gem:

gem install --user-install travis

You can then use the travis command if you have configured your $PATH for RubyGems; if you haven’t, the tool will output a path to use on the first lines (eg. ~/.gem/ruby/2.0.0/bin/travis)

We’ll use the Travis CI command-line client to log in (using your GitHub password), enable the repository and encrypt our SSH key. Run the following three commands, one at a time (they are interactive):

travis login
travis enable
travis encrypt-file id_rsa --add

Commit everything to GitHub:

git add .
git commit -am "Automate builds with Travis CI"

Hopefully, Travis CI will build your site and deploy. Check the Travis CI website or your e-mail for a notification. If there are any errors, make sure you followed this guide to the letter.

(Revision 3, 2017-03-17: added master/src branching information, blocked non-src builds, clarified some things)

Nikola v7.7.12 is out!

On behalf of the Nikola team, I am pleased to announce the immediate availability of Nikola v7.7.12. It fixes some bugs and adds new features.

What is Nikola?

Nikola is a static site and blog generator, written in Python. It can use Mako and Jinja2 templates, and input in many popular markup formats, such as reStructuredText and Markdown — and can even turn Jupyter (IPython) Notebooks into blog posts! It also supports image galleries, and is multilingual. Nikola is flexible, and page builds are extremely fast, courtesy of doit (which is rebuilding only what has been changed).

Find out more at the website: https://getnikola.com/

Downloads

Install using pip install Nikola or download tarballs on GitHub and PyPI.

Changes

Features

  • New Telugu translation by Chillar Anand
  • Add sections filtering in the post list directive (Issue #2409)
  • Update Bootstrap to v3.3.7

Bugfixes

  • Added code documentation to ReadTheDocs (Issue #2124)
  • Pass language to docutils so admonitions are translated (Issue #2422)
  • Put 2-file post metadata in the same place as the text file when specifying a path in new_post, new_page (Issue #2420)
  • Register dependencies in post-list shortcode (Issue #2412)
  • Support post-list shortcode better (Issue #2408)
  • Fix gallery links in base theme (Issue #2416)
  • Respect DEPLOY_DRAFTS and DEPLOY_FUTURE in github_deploy (Issue #2406)
  • Don’t remove <!DOCTYPE html> if typogrify filters are in use
  • Avoid infinite loop if bootstrap3 can't be loaded (Issue #2402)
  • Fixed image rotation to update image size correctly (Issue #2418)

Nikola v7.7.9 is out!

On behalf of the Nikola team, I am pleased to announce the immediate availability of Nikola v7.7.9. It fixes some bugs and adds new features.

For this release, we also passed a milestone — Nikola has got over 1000 stars on GitHub. Thanks for the continued support!

What is Nikola?

Nikola is a static site and blog generator, written in Python. It can use Mako and Jinja2 templates, and input in many popular markup formats, such as reStructuredText and Markdown — and can even turn Jupyter (IPython) Notebooks into blog posts! It also supports image galleries, and is multilingual. Nikola is flexible, and page builds are extremely fast, courtesy of doit (which is rebuilding only what has been changed).

Find out more at the website: https://getnikola.com/

Downloads

Install using pip install Nikola or download tarballs on GitHub and PyPI.

Changes

Features

  • New sc or html reST role, which passes content unaltered (Issue #1854)
  • Point the user to our users site after 1st successful deploy (Issue #2096)
  • Option to disable color output using NIKOLA_MONO environment variable (Issue #2360)
  • Improved locale detection in Windows (Issue #2343)
  • Added enclosure_length meta field for better interop (Issue #2338)
  • New Lithuanian translation by Antanas Lasys

Bugfixes

  • Avoid conflicts caused by multiple copies of the same plugin (#2362)
  • Fix handling of some wordpress dumps (Issue #2340)
  • When using the plugin command, load ALL plugins (Issue #2359)
  • Fix plugin removal for plugins that are a package (Issue #2356)
  • Reload English messages for every theme to prevent caching (Issue #2347)
  • Cache theme messages after loading once (Issue #2344)
  • Add <link> tags to other languages even if the post is untranslated (Google complained otherwise)
  • Don't call sys.exit() from plugins if possible (Issue #1839)
  • Create Persistor directories only if site is configured (Issue #2334)
  • Remove newlines in imported WordPress titles (Issue #2330)