Steady as a rock

A very important feature in the beta 3 release of falcon (which will happen when I finish the documentation) is automated package building. Yes, falcon is now becoming a full fledged repository manager where you can dput your package to and it will build your package and install it in its database.

Here’s what it looks like when run manually, only one package in the queue:

dennis@blackbird:/data1/src/falcon$ bin/falcon-build-queue
Falcon repository manager 2.0.0~beta3 (C)2005-2007 Dennis Kaarsemaker
*  Building casper_1.87.dsc on Blackbird, Starfreighter
*  Trying to build on Blackbird
*  Executing the build command, logging to buildlog_Seveas-feisty-i386.casper_1.87_BUILDING.txt
*  Downloading ubiquity-casper_1.87_all.deb
*  Downloading casper_1.87_i386.deb
*  Succeeded building casper 1.87 on i386 buildd Blackbird
*  Uploaded casper 1.87 for building on amd64 buildd Starfreighter
*  Build result for casper 1.87
*  ============================
*    amd64 Starfreighter        PENDING
*     i386 Blackbird            OK
*  Run falcon-build-queue to check for finished builds
dennis@blackbird:/data1/src/falcon$ ssh starfreighter ./falcon-build-local-queue
* Building casper_1.87.dsc, log will be written to /pbuilder/result/buildlog_Seveas-feisty-amd64.casper_1.87_BUILDING.txt
dennis@blackbird:/data1/src/falcon$ bin/falcon-build-queue
Falcon repository manager 2.0.0~beta3 (C)2005-2007 Dennis Kaarsemaker
*  Downloading casper_1.87_amd64.deb
*  Succeeded building casper 1.87 on amd64 buildd Starfreighter
*  Build result for casper 1.87
*  ============================
*     i386 Blackbird            OK
*    amd64 Starfreighter        OK

When this is run via cron, it will be fully automatic.

Almost a year after I released the last version (1.5.3) of the falcon repository manager, beta 1 of version 2 is finally here! In the past months I’ve completely rewritten it, so if you need to manage a repository of .deb files, here’s a good application for you

New features in this release:

• Scanning code no longer uses apt-ftparchive
• All internal data is modeled using django, making it very flexible
• The templates for html indices are als django-based
• It is translatable!
• Configuration is now even easier, using an interactive config editor
• Numerous fixes and improvements
• The beginning of a plugin system is created (completion in beta 2)
• Same for an automatic building system (completion in beta 2)
• You can now easily install single source packages and binaries into the archive
• Having all data internally cached makes the code much less fragile
• Support for sha1/sha256 checksums in Packages and Sources files
• Compis with the new python policy, so compatible with edgy/feisty

Of course all the original features are still there:

• Support for multiple releases & components
• GPG signed repository
• Themable HTML indices
• No hassle with incoming if you don’t want to use it (but now you can easily use it!)
• Quick and easy creation of .iso images

Grab a deb at my repo or download the source with bzr from http://blackbird.kaarsemaker.net/code/falcon

OpenSSH is an awesome implementation of the ssh protocol and I use it a lot. However, to connect to hosts at work I sometimes need to traverse many firewalls, using insane ssh configurations such as:

Host foo
ProxyCommand ssh -A host1 ssh -A host2 ssh -A host3 nc %h %p

Which means that connecting to host foo takes a lot of time. Using ControlMaster connections around helps, but then you have to remember to close the shell which is the ControlMaster last (~& helps though, but that’s cheating).

After perusing the ssh(1) and ssh_config(5) manpages (and parts of the OpenSSH source), I whipped up this simple python script that takes away this limitation and also leaves the ControlMaster connection around for future re-use. It also runs ssh-add for you if your agent has no key yet.

It saves me quite a bit of hassle on a daily basis so it could be useful for you as well if you depend on ssh a lot.

You’ll want this in your ~/.ssh/config to enjoy it most:

ForwardAgent yes

ControlMaster auto
ControlPath ~/.ssh/ssh_control_%h_%p_%r

Django is of course an excellent framework for building web applications. But since it’s quit modular, it is also possible to use only parts of it in non-web products. Being fed up with the cheetah templating engine (which basically reimplements python, and is no fun to use) and sqlobject (where’s the documentation?), I decided that for falcon I would use django’s excellent object-relational mapper and templating engine.

Since I made that switch, hacking on falcon finally became fun again and the next release (albeit beta) is imminent!

But no matter how nice django is, there are some caveats when you want to use and extend django the way I did.

• You need to call django.conf.settings.configure manually, so I ran into an interesting bootstrapping problem.
• Changing configuration later needs setattr()
• Using your own templatetags without actually having what django calls an application is actually simple, but you do need to know how it works: you need a folder foo/templatetags, where foo is in sys.path and in django’s INSTALLED_APPS, then put your tags in foo/templatetags/bar.py
• Automatically initializing database tables is not a feature. But since getting the creation statements is a feature and access to the database connection is another one, that problem can be solved by looking at the django code, this is how I did it:

from django.core import management
management.disable_termcolors()
all_models = (pocket.Component, pocket.MetaComponent, package.SourcePackage, package.BinaryPackage)
all_sql = []
for model in all_models:
    sql, ref = management._get_sql_model_create(model, all_models)
    all_sql += sql
cursor = connection.cursor()
for s in all_sql:
    cursor.execute(s.replace('CREATE TABLE', 'CREATE TABLE IF NOT EXISTS'))

• Adding a new field type is surprisingly hard. Sqlobject has a PickleColumn, for which data is automatically pickled. I wanted to add this to django, without modifying django itself. After creating the field type (a subclass of django.db.models.Field), I only got errors. Adding a type mapping to what django calls the creation module, those errors were gone. Automatic pickling is also possible, if you override the get_db_prep_save function. But then it broke, since automatic unpickling is not possible, there is no ‘data thawing’ functionality in django. So I ended up letting the constructors of the models that use the PickleField do this.

After those things were solved, the excellent django documentation helped me write the rest of the surrounding code and falcon now is faster and better than ever. Now to fix the remaining bugs and then release it, it’s been too long since I made a release.

One thing I don’t like in django is that there is no from input firld that’s automatically set to the current logged in user (like the auto_now datetimefield). Problem with such a field is that there isn’t always a current user, since you can also manipulate django objects on the command line (which is great!). But given that there sometimes is a user, why not take advantage of that and inspect the request? Problem is that you don’t have access to the current request from everywhere, but it’s somewhere on the call stack. Fortunately, you don’t have to do ASM hacking to access the python callstack, python has a neat little gem called sys._getframe which works real pythonic and doesn’t even run off the stack but raises a ValueError if you try and access a non-existing frame. Don’t forget to del(frame) though, otherwise python (and in the case of mod_python: apache) segfaults.

def get_user():
    i=1
    user = None
    while True:
        try:
            frame = sys._getframe(i)
        except ValueError: # Stack isn't this deep
            break
        if 'request' in frame.f_locals and
           issubclass(type(frame.f_locals['request']) , HttpRequest):
            user = frame.f_locals['request'].user
            del(frame)
            break
        del(frame)
        i += 1
    return user

Pygments is probably the coolest syntax highlighter since sliced bread

The new paste.ubuntu-nl.org is powered by django, pygments and a hashcash based anti-spam technique. Now finally all of ubuntu-nl.org is migrated and I can start making the code publish-able.

If you didn’t know it yet: django rocks. Very, very much

Right now, almost all of ubuntu-nl.org is django-powered, even the forum (punbb) and planet templates (and planet config!) are generated by django for maximum style consistency. It took a while to perfect^Wfix all the bugs, and an SQL view (Yay, MySQL 5!) for integration of forum data and django data, but the dutch Ubuntu website is now better than ever.

For more python goodness:

• Planet is also python
• It’s all managed in bzr , which is written in python
• The code will be published on launchpad, which is also written in python
• Python also rocked for the migration scripts

Sinds vandaag (stiekem eigenlijk al sinds gister) ondersteunt Falcon meerdere releases. Dus ben ik al een dag lang bezig met het maken van pakketjes voor Ubuntu dapper

Hoe misbruik je docstrings:

dennis@mirage:~$ cat foo.py
"""python" "$0" "$@"; "exit"""
import sys

print "Ik ben een python programma"
print sys.version

dennis@mirage:~$ python foo.py
Ik ben een python programma
2.4.3 (#2, Mar 30 2006, 14:45:01)
[GCC 4.0.3 (Ubuntu 4.0.3-1ubuntu3)]

dennis@mirage:~$ sh foo.py
Ik ben een python programma
2.4.3 (#2, Mar 30 2006, 14:45:01)
[GCC 4.0.3 (Ubuntu 4.0.3-1ubuntu3)]

Woorden zijn overbodig - ik weet het, ik ben eng

import sys
is_palindrome = lambda w: len(w) < 2 or w[0] == w[-1] and is_palindrome(w[1:-1])
map(lambda w: is_palindrome(w[:-1]) and sys.stdout.write(w), sys.stdin)

Palindroomchecker in 3 regels

• seveas.ubuntulinux.nl naar twigs migreren
• gnome-vfs module voor FUSE afmaken
• python-notification gebruiken/volledig maken en het Evolution DBUS signaal zien op te vangen
• Meer vertalen aan ubuntulinux.nl
• Yars afmaken en netjes in een debje stoppen (en een nieuwe naam geven)
• Fluxx kopen en eens kijken of daar een python versie van gemaakt kan worden
• Freenode patches voor xchat afmaken
• Django leren en gebruiken
• Iets met afstuderen ofzo…

Gister (nou ja afgelopen nacht) heb ik Django ontdekt, ge-wel-dig! Nou kan ik eindelijk php en MySQL de deur uit gooien (moet ik wel eerst ‘even’ m’n CMS herschrijven in Django, maar aangezien Django al een CMS is wordt dat vrij makkelijk. Exit PHP — woehoe!!

Ik gebruik al python (Ja Django is geschreven in Python - m’n favoriete taaltje) voor een paar MoinMoin wiki’s, waaronder ook wiki.kaarsemaker.net waar ik laatst voor Dynamite alle specs aan het opschrijven ben geweest. Vandaag een stukje geschreven over het migreren van de ld.so/rtld.c hacks naar een simpele, nette library die met LD_PRELOAD toch voor main() nog code uitvoert. Alleen nog even uitzoeken of ik daarmee een equivalent van een elf PT_LOAD sectie kan maken. Als dat kan hoeft een checkpoint geen ELF binary meer weg te schrijven (wat inderdaad heel fijn is).

Ook het single process migration protocol begint vorm te krijgen, alleen nog even netjes opschrijven hoe simultane migraties afgehandeld worden. Ja, wikis zijn leuk als kladblok, ik kan er over een paar maanden gewoon m’n hele scriptie vanaf halen.

Het backlight van m’n laptop tft heeft het begeven. Dat wordt dus een nieuwe laptop kopen. Uiteraard verhindert dit mij niet om flink door te nerden: met dank aan urllib2 heb ik nu een scriptje wat pastebin posts kan maken!