Nikola 3 is out and it is good.

I just released version 3 of my static site generator, Nikola

It's a major release, there is hardly any code from the previous version that was not moved, prodded, broken or fixed!

The main features of Nikola:

  • Blogs, with tags, feeds, archives, comments, etc.
  • Themable
  • Fast builds, thanks to doit
  • Flexible
  • Small codebase (programmers can understand all of Nikola in a few hours)
  • reStructuredText and Markdown as input languages
  • Easy image galleries (just drop files in a folder!)
  • Syntax highlighting for almost any programming language or markup
  • Multilingual sites
  • Doesn't reinvent wheels, leverages existing tools.

Changes for this release (not exhaustive!):

  • New optional template argument for "nikola init"
  • New "install_theme" task
  • Optional address option for the "serve" task
  • Better looking codeblocks
  • Russian translation
  • Use markdown/reSt compiler based on post extension
  • Don't fail when there are no posts/stories/galleries/tags
  • Use configuration options as dependencies
  • Use more relative links for easier site rellocation
  • Syntax highlight for markdown
  • Better multicore builds (make the -n 2 or -n 4 options work)
  • Configurable output folder
  • Don't fail on posts with periods in the name
  • Different page names for different languages
  • Recognize (some) Mako template dependencies
  • Is now a more "normal" python package.

A port of blog.txt to Nikola

If you want a very minimalistic theme for a Nikola-based site, I just did a quick and dirty port of Scott Wallick's blog.txt theme

If there is anything nice here, he did it. If there is something wrong or broken, I did it instead.

I did it basically to see if it was possible to port wordpress themes to Nikola. And it is, but it involves reading php files and loosely reinterpreting them into Mako templates.

While the port is far from perfect, it's a reasonable starting point for someone who is really interested in it.

Here is how it looks:

http://lateral.netmanagers.com.ar/galleries/random/blogtxt.png

And here is the download. To use it, just unzip it in your themes folder, set THEME to "blogtxt" in your dodo.py file, and rebuild the site.

At least the CSS files are easily adapted.

This theme is under a LGPL license (see included license.txt), enjoy!

Smiljan, a Small Planet Generator

I maintain a couple of small "planet" sites. If you are not familiar with planets, they are sites that aggregate RSS/Atom feeds for a group of people related somehow. It makes for a nice, single, thematic feed.

Recently, when changing them from one server to another, everything broke. Old posts were new, feeds that had not been updated in 2 years were always with all its posts on top... a disaster.

I could have gone to the old server, and started debugging why rawdog was doing that, or switch to planet, or look for other software, or use an online aggregator.

Instead, I started thinking... I had written a few RSS aggregators in the past... Feedparser is again under active development... rawdog and planet seem to be pretty much abandoned... how hard could it be to implement the minimal planet software?

Well, not all that hard, that's how hard it was. Like it took me 4 hours, and was not even difficult.

One reason why this was easier than what planet and rawdog achieved is that I am not doing a static site generator, because I already have one so all I need this program (I called it Smiljan) to do is:

  • Parse a list of feeds and store it in a database if needed.
  • Download those feeds (respecting etag and modified-since).
  • Parse those feeds looking for entries (feedparser does that).
  • Load those entries (or rather, a tiny subset of their data) in the database.
  • Use the entries to generate a set of files to feed Nikola
  • Use nikola to generate and deploy the site.

So, here is the final result: http://planeta.python.org.ar which still needs theming and a lot of other stuff, but works.

I implemented Smiljan as 3 doit tasks, which makes it very easy to integrate with Nikola (if you know Nikola: add "from smiljan import *" in your dodo.py and a feeds file with the feed list in rawdog format) and voilá, running this updates the planet:

doit load_feeds update_feeds generate_posts deploy

Here is the code for smiljan.py, currently at the "gross hack that kinda works" stage. Enjoy!

# -*- coding: utf-8 -*-
import codecs
import datetime
import glob
import os
import sys

import feedparser
import peewee

class Feed(peewee.Model):
    name = peewee.CharField()
    url = peewee.CharField(max_length = 200)
    last_status = peewee.CharField()
    etag = peewee.CharField(max_length = 200)
    last_modified = peewee.DateTimeField()

class Entry(peewee.Model):
    date = peewee.DateTimeField()
    feed = peewee.ForeignKeyField(Feed)
    content = peewee.TextField(max_length = 20000)
    link = peewee.CharField(max_length = 200)
    title = peewee.CharField(max_length = 200)
    guid = peewee.CharField(max_length = 200)

Feed.create_table(fail_silently=True)
Entry.create_table(fail_silently=True)

def task_load_feeds():

    def add_feed(name, url):
        f = Feed.create(
            name=name,
            url=url,
            etag='caca',
            last_modified=datetime.datetime(1970,1,1),
            )
        f.save()

    feed = name = None
    for line in open('feeds'):
        line = line.strip()
        if line.startswith('feed'):
            feed = line.split(' ')[2]
        if line.startswith('define_name'):
            name = ' '.join(line.split(' ')[1:])
        if feed and name:
            f = Feed.select().where(name=name, url=feed)
            if not list(f):
                yield {
                    'name': name,
                    'actions': ((add_feed,(name, feed)),),
                    'file_dep': ['feeds'],
                    }
                name = feed = None

def task_update_feeds():

    def update_feed(feed):
        modified = feed.last_modified.timetuple()
        etag = feed.etag
        parsed = feedparser.parse(feed.url,
            etag=etag,
            modified=modified
        )
        try:
            feed.last_status = str(parsed.status)
        except:  # Probably a timeout
            # TODO: log failure
            return
        if parsed.feed.get('title'):
            print parsed.feed.title
        else:
            print feed.url
        feed.etag = parsed.get('etag', 'caca')
        modified = tuple(parsed.get('date_parsed', (1970,1,1)))[:6]
        print "==========>", modified
        modified = datetime.datetime(*modified)
        feed.last_modified = modified
        feed.save()
        # No point in adding items from missinfg feeds
        if parsed.status > 400:
            # TODO log failure
            return
        for entry_data in parsed.entries:
            print "========================================="
            date = entry_data.get('updated_parsed', None)
            if date is None:
                date = entry_data.get('published_parsed', None)
            if date is None:
                print "Can't parse date from:"
                print entry_data
                return False
            date = datetime.datetime(*(date[:6]))
            title = "%s: %s" %(feed.name, entry_data.get('title', 'Sin título'))
            content = entry_data.get('description',
                    entry_data.get('summary', 'Sin contenido'))
            guid = entry_data.get('guid', entry_data.link)
            link = entry_data.link
            print repr([date, title])
            entry = Entry.get_or_create(
                date = date,
                title = title,
                content = content,
                guid=guid,
                feed=feed,
                link=link,
            )
            entry.save()
    for feed in Feed.select():
        yield {
            'name': feed.name.encode('utf8'),
            'actions': ((update_feed,(feed,)),),
            }

def task_generate_posts():

    def generate_post(entry):
        meta_path = os.path.join('posts',str(entry.id)+'.meta')
        post_path = os.path.join('posts',str(entry.id)+'.txt')
        with codecs.open(meta_path, 'wb+', 'utf8') as fd:
            fd.write(u'%s\n' % entry.title.replace('\n', ' '))
            fd.write(u'%s\n' % entry.id)
            fd.write(u'%s\n' % entry.date.strftime('%Y/%m/%d %H:%M'))
            fd.write(u'\n')
            fd.write(u'%s\n' % entry.link)
        with codecs.open(post_path, 'wb+', 'utf8') as fd:
            fd.write(u'.. raw:: html\n\n')
            content = entry.content
            if not content:
                content = 'Sin contenido'
            for line in content.splitlines():
                fd.write(u'    %s' % line)

    for entry in Entry.select().order_by(('date', 'desc')):
        yield {
            'name': entry.id,
            'actions': ((generate_post, (entry,)),),
            }

Version 2.1 Released

Update

This version was missing 2 files in the site theme, please get 2.1.1 instead.

The new release is available at GoogleCode. It contains a number of bugfixes, and some new features:

  • The input format for your site is now modular. This version includes support for Markdown
  • The theme support is much cleaner. It even comes with two themes! To learn more, there is a theming guide
  • All links in pages generated by Nikola are relative, which means you can deploy to anywhere in your server and all links will work.
  • Translation support has been greatly improved, and translations are now welcome!
  • There is now better support to add your own files into the output folder (just put them in "files" and they will be copied as needed).

Nikola 1.2 is out!

Version 1.2 of Nikola, my static site generator and the software behind this very site, is out!

Why build static sites? Because they are light in resources, they are future-proof, because it's easy, because they are safe, and because you avoid lockin.

New Features:

  • Image gallery (just drop pics on a folder)
  • Built-in webserver for previews (doit -a serve)
  • Helper commands to create new posts (doit -a new_post)
  • Google Sitemap support
  • A Handbook!
  • Full demo site included
  • Support for automatic deployment (doit -a deploy)
  • Client-side redirections

And of course the old features:

  • Write your posts in reStructured text
  • Clean, customizable page design (via bootstrap)
  • Comments via Disqus
  • Support any analytics you want
  • Build blogs with tags, feeds, feeds for your tags, indexes, and more
  • Works like a simple CMS for things outside your blog
  • Clean customizable templates using Mako
  • Pure python, and not a lot of it (about 600 lines)
  • Smart builds (doit only rebuilds changed pages)
  • Easy to extend and improve
  • Code displayed with syntax highlighting

Right now Nikola does literally everything I need, so if you try it and need something else... it's a good time to ask!

More info at http://nikola-generator.googlecode.com

Nikola 1.1 is out!

A simple yet powerful and flexible static website and blog generator, based on doit, mako, docutils and bootstrap.

I built this to power this very site you are reading, but decided it may be useful to others. The main goals of Nikola are:

  • Small codebase: because I don't want to maintain a big thing for my blog
  • Fast page generation: Adding a post should not take more that 5 seconds to build.
  • Static output: Deployment using rsync is smooth.
  • Flexible page generation: you can decide where everything goes in the final site.
  • Powerful templates: Uses Mako
  • Clean markup for posts: Uses Docutils
  • Don't do stupid builds: Uses doit
  • Clean HTML output by default: Uses bootstrap
  • Comments out of the box: Uses Disqus
  • Tags, with their own RSS feeds
  • Easy way to do a blog
  • Static pages outside the blog
  • Multilingual blog support (my own blog is english + spanish)

I think this initial version achieves all of those goals, but of course, it can be improved. Feedback is very welcome!

Nikola's home page is currently http://nikola-generator.googlecode.com

How it's done

I added a very minor feature to the site. Up here ^ you should be able to see a link that says "reSt". If you click on it, it will show you the "source code" for the page.

I did this for a few reasons:

  1. Because a comment seemed to suggest it ;-)
  2. Because it seems like a nice thing to do. Since I so like reSt, I would like others to use it, too. And showing how easy it is to write using it, is cool.
  3. It's the "free software-y" thing to do. I am providing you the preferred way to modify my posts.
  4. It was ridiculously easy to add.

Also, if you see something missing, or something you would like to have on the site, please comment, I will try to add it.

Nikola is Near

I managed to do some minor work today on Nikola, the static website generator used to generate ... well, this static website.

  • Implemented tags (including per-tag RSS feeds)
  • Simplified templates
  • Separated code and configuration.

The last one was the trickiest. And as a teaser, here is the full configuration file to create this site, except HTML bits for analytics, google custom search and whatever that would make no sense on other sites. I hope it's somewhat clear.

# -*- coding: utf-8 -*-

# post_pages contains (wildcard, destination, template) tuples.
#
# The wildcard is used to generate a list of reSt source files (whatever/thing.txt)
# That fragment must have an associated metadata file (whatever/thing.meta),
# and opcionally translated files (example for spanish, with code "es"):
#     whatever/thing.txt.es and whatever/thing.meta.es
#
# From those files, a set of HTML fragment files will be generated:
# whatever/thing.html (and maybe whatever/thing.html.es)
#
# These files are combinated with the template to produce rendered
# pages, which will be placed at
# output / TRANSLATIONS[lang] / destination / pagename.html
#
# where "pagename" is specified in the metadata file.
#

post_pages = (
    ("posts/*.txt", "weblog/posts", "post.tmpl"),
    ("stories/*.txt", "stories", "post.tmpl"),
)

# What is the default language?

DEFAULT_LANG = "en"

# What languages do you have?
# If a specific post is not translated to a language, then the version
# in the default language will be shown instead.
# The format is {"translationcode" : "path/to/translation" }
# the path will be used as a prefix for the generated pages location

TRANSLATIONS = {
    "en": "",
    "es": "tr/es",
    }

# Data about this site
BLOG_TITLE = "Lateral Opinion"
BLOG_URL = "http://lateral.netmanagers.com.ar"
BLOG_EMAIL = "[email protected]"
BLOG_DESCRIPTION = "I write free software. I have an opinion on almost "\
    "everything. I write quickly. A weblog was inevitable."

# Paths for different autogenerated bits. These are combined with the translation
# paths.

# Final locations are:
# output / TRANSLATION[lang] / TAG_PATH / index.html (list of tags)
# output / TRANSLATION[lang] / TAG_PATH / tag.html (list of posts for a tag)
# output / TRANSLATION[lang] / TAG_PATH / tag.xml (RSS feed for a tag)
TAG_PATH = "categories"
# Final location is output / TRANSLATION[lang] / INDEX_PATH / index-*.html
INDEX_PATH = "weblog"
# Final locations for the archives are:
# output / TRANSLATION[lang] / ARCHIVE_PATH / archive.html
# output / TRANSLATION[lang] / ARCHIVE_PATH / YEAR / index.html
ARCHIVE_PATH = "weblog"
# Final locations are:
# output / TRANSLATION[lang] / RSS_PATH / rss.xml
RSS_PATH = "weblog"

# A HTML fragment describing the license, for the sidebar.
LICENSE = """
    <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/2.5/ar/">
    <img alt="Creative Commons License" style="border-width:0; margin-bottom:12px;"
    src="http://i.creativecommons.org/l/by-nc-sa/2.5/ar/88x31.png"></a>
"""

# A search form to search this site, for the sidebar. Has to be a <li>
# for the default template (base.tmpl).
SEARCH_FORM = """
    <!-- google custom search -->
    <!-- End of google custom search -->
"""

# Google analytics or whatever else you use. Added to the bottom of <body>
# in the default template (base.tmpl).
ANALYTICS = """
        <!-- Start of StatCounter Code -->
        <!-- End of StatCounter Code -->
        <!-- Start of Google Analytics -->
        <!-- End of Google Analytics -->
"""

# Put in global_context things you want available on all your templates.
# It can be anything, data, functions, modules, etc.
GLOBAL_CONTEXT = {
    'analytics': ANALYTICS,
    'blog_title': BLOG_TITLE,
    'blog_url': BLOG_URL,
    'translations': TRANSLATIONS,
    'license': LICENSE,
    'search_form': SEARCH_FORM,
    # Locale-dependent links
    'archives_link': {
        'es': '<a href="/tr/es/weblog/archive.html">Archivo</a>',
        'en': '<a href="/weblog/archive.html">Archives</a>',
        },
    'tags_link': {
        'es': '<a href="/tr/es/categories/index.html">Tags</a>',
        'en': '<a href="/categories/index.html">Tags</a>',
        },
    }

execfile("nikola/nikola.py")