Skip to content

HDMI2USB – Production Board Bring Up – Snippets – Prep Work

Two weeks ago I decided that I was going to take a week of work and spend it full time working on bringing up the HDMI2USB Production board that I’m working on creating with Numato. Since I’m making all our GSoC students do daily snippets, I thought it would be good for me to do snippets too! Will be good to look back on the week and see what I ended up actually doing.

The following snippets are for the preparation work I did before the week;


Friday – 18th July


Saturday – 19th July

  • Day off. Spent day playing DOTA 2 and StarCraft 2 with Joel’s mates.


Sunday – 20th

  • Concentrated on getting slidelint website up and running, targeting announcement at PyCon AU.
  • Created DNS configuration and deployed it. Also updated the domain tracking spreadsheet.
  • Rewrote the setup documentation (and ported it to Markdown).
  • Set up a nginx frontend.
  • Attempted to get circus-web interface working.
    • Turns out it doesn’t work under Python 3 as torandio2 is unmaintained.
    • Ported circus-web to sockjs-torando.
    • Found the tests don’t work.


Connecting to a Fritz!Box under Linux using vpnc

I have two Fritz!Box 7390 (one at my place and one at my parents primary residence) and one Fritz!Box 7270 devices (the house they are building). They are pretty reasonable “high end” ADSL routers and a cool feature is they integrated VPN support. I use this functionality to connect the three networks securely together (but that is not what this post is about). This VPN functionality can also be used to connect to your home network while not at home, but information on how to do this from Linux is very sparse (specially if you only want to use FOSS tools to do the connection), so here is how I did it.

Configuring your Fritz!Box

To set up your Fritz!Box you need a configuration file, there is no GUI available in the web interface to support create new VPN configurations. While a number of sites have pre-built .cfg files that look like they should work, my Fritz!Boxes only accept encrypted VPN configuration files (see end of post for more information) and no open source tool exists to encrypt them.

I instead needed to use the FRITZ!Box VPN Connection tool (while it is a Windows program but runs fine under Wine) to create the configuration. It can be downloaded from the AVM website (This is different to the VPN client that AVM also provide). Annoyingly the tool doesn’t just read existing .cfg configuration files, it instead reads it’s own vpnadmin.cfg found in c:/users/your username/Application Data/AVM/FRITZ!VPN and then generates a .cfg file and encrypts it.

Using the FRITZ!Box VPN Connection tool to create a configuration that is compatible with vpnc you must:

  • Select “Configure VPN for one User” type connection
  • On the “Select Device” screen, select “iPhone, iPod touch or iPad” option
  • On the “Enter the user’s email address” screen, despite the admin tool calling this field the user email, just enter a username. I recommend not having any special characters like @ or . in it.
  • On the “Enter IP address of the User” screen, be careful about what IP address you use (the default should be okay).
    • Don’t use the same IP address that the computer uses when connected via wireless/wired. While it seems like a good idea, as the computer would have the same IP address even when remote, it does not work and will mean the device is unable to access the internet when connected to wireless/wired.
    • The “Send all data over the VPN tunnel” option on this page does not seem to affect vpnc, it will always route all your data over the vpn connection. See later for how to fix this problem.
  • On the “Key for the connection” screen enter a password. Copy down the shared secret key, you’ll want it when creating the vpnc config. I recommend also keeping the default shared secret key it generates unless your super paranoid about entropy.

You can check that you have done this correctly in two ways;

  • The vpnadmin.cfg will have the iphone=1 and xauth_key="your password" options. See below for a partial example;
      user {
        nameoremail = "xxxx";
        key = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx";
        ip =;
        internetaccess = 0;
        iphone = 1;
        xauth_key = "xxxxx";

  • If you export an unencrypted config file (remember this can’t be loaded onto the Fritz!Box – so make sure you also export an encrypted version!), the remoteid section will have a key field and not a fqdn field, use_xauth will be set to yes and an xauth section will exist. See below for a partial example;
      keepalive_ip =;
      remoteid {
        key_id = "qnap";
      mode = phase1_mode_aggressive;
      phase1ss = "all/all/all";
      keytype = connkeytype_pre_shared;
      key = "f3e0hcca19ca2a3gaafbI.{1dGe3q8f84";
      cert_do_server_auth = no;
      use_nat_t = yes;
      use_xauth = yes;
      xauth {
        valid = yes;
        username = "qnap";
        passwd = "qnappassword";
      use_cfgmode = yes;

Configuring vpnc on your Linux box

The Fritz!Box needs both draft-ietf-ipsec-nat-t-ike-03 support and the client to adopt the server suggested lifetime (which for the Fritz!Box is 3600 seconds). These features where only added to vpnc recently, so make sure your vpnc is newer than subversion revision 511. Both the version in Debian Unstable and any Ubuntu newer than Precise (12.04) have vpnc 0.5.3r512, which is new enough version and you can just apt-get install vpnc. For Fedora 17 x86 systems you can find RPMs at

Create the vpnc config in /etc/vpnc/fritzbox.conf using the following as a template (replace the parts in bold/brackets):

IPSec gateway ip address or DNS name of your FritzBox

IKE DH Group dh2
Perfect Forward Secrecy nopfs

IPSec ID [username entered into the "Enter the user's email address" screen]
# "key" from the Fritz!Box VPN configuration
IPSec secret [shared secret key from the "Key for the connection" screen]

NAT Traversal Mode force-natt

Xauth username [username entered into the "Enter the user's email address" screen]
Xauth password [password entered into the "Key for the connection" screen - Not the password use to encrypt the vpnc configuration!]

As this file contains usernames and password, the config file should be owned by root and only readable by the owner.

sudo chown root /etc/vpnc/fritzbox.conf
sudo chmod 0600 /etc/vpnc/fritzbox.conf

You should now be able to connect to your home internet using:

sudo vpnc-connect fritzbox

When you are finished, use:

sudo vpnc-disconnect fritzbox

Internet Access while using vpnc

When vpnc connects it will change your default route to go via the tunnel. This will send all your local internet traffic over the VPN connection. If you are setting up the VPN to secure your internet browsing while on a hostile network such as a public WiFi hotspot (such as at a coffee shop) this is what you want.

However, if you just want to be able to access the hosts on your home network, you don’t want this as it will make your internet browsing extraordinarily slow. These routes are set up by /etc/vpn/vpnc-script script, so you will need custom one. Normally this script is provided a lot of information when connecting to other VPN concentrators (such as a Cisco EasyVPN device) but when connecting to the Fritz!Box you only seem to get limited information.

I created the following script in /etc/vpnc/fritzbox-script, marked it as executable (chmod a+x /etc/vpnc/fritzbox-script) and then added “Script /etc/vpnc/fritzbox-script” to my /etc/vpnc/fritzbox.conf file.



case "$reason" in
    /etc/vpnc/vpnc-script pre-init
    INTERNAL_IP4_PREFIX=$(echo $INTERNAL_IP4_ADDRESS | sed -e's/\.[0-9]\+$//')
    $IPROUTE link set dev "$TUNDEV" up mtu 1024
    $IPROUTE route replace "$INTERNAL_IP4_PREFIX.0/" dev "$TUNDEV"
    $IPROUTE route flush cache
    $IPROUTE link set dev "$TUNDEV" down
    echo "unknown reason '$reason'. Maybe vpnc-script is out of date" 1>&2
    exit 1
exit 0

Fritz!Box encrypted VPN configuration files

Fritz!Boxes will only accept VPN configurations which are encrypted. Otherwise when try to import the file you will get the cryptic "Error: Import of the VPN settings failed." as seen below;
Error: Import of the VPN settings failed.

To create an encrypted file you must use the FRITZ!Box VPN Connection tool. Then when exporting, select "Save VPN settings in a file under" option, check the "Encrypt VPN settings" check box (the file type should change to .eff), and enter your chosen password twice.

Export VPN settings

Lightning Timer website

When doing lightning talks it is very useful to have a timer so that people don’t over time. There use to be a very simple website called but it appears to have gone AWOL. I have set up a mirrored the code at and

By default it does 5 minutes, but if you can set the hash value to get different times;

Code found at

FastComplete, making bash completion fast on remote file systems

Here at Google we have quite a few remote file systems which contain various tools we use in our day-to-day work. As typing sucks we generally want the tools in our $PATH. When you try to tab complete Bash needs to stat a couple of thousand files and even on fast remote file systems this takes a drastically long time.

I wrote FastComplete as a solution to this problem. The tool creates a local cache of links on your hard drive to everything in your $PATH. It uses a couple of tricks to make sure all the stats remain locally, while still allowing the remote file to change without needing to update the cache. Linux should also keep this information in memory disk cache making tab completion almost instant again. Yay!

You can find FastComplete at It is a stand alone python program which shouldn’t have any non-core dependencies. The usage documentation is as follows;

Fast complete creates a local disk cache of your path.
It’s specifically designed to make bash tab complete run much faster. The correct fix would be to add caching to bash, but it was to hard to do so.

To find out what path fastcomplete is currently using:

> ~tansell/bin/fastcomplete
# Found 3977 commands
export PATH=/home/tansell/bin: ... :/home/build/google3/googledata/validators:/home/build/google3/ads/db

To get fastcomplete to rebuild it’s cache:

> ~tansell/bin/fastcomplete --rebuild
# Using path of '/home/tansell/bin: ... :/home/build/google3/googledata/validators:/home/build/google3/ads/db'
# Cache /usr/local/google/users//tansell/tabcache/d7e5fb63454ae33b4a171b6437be904a did not exist! Rebuilding....
# Looking in: /home/tansell/bin (execv)
# Looking in: /usr/bin (symlink)
# Looking in: /home/build/google3/ads/db (execv)
# Found 3977 commands
export PATH=/usr/local/google/users//tansell/tabcache/d7e5fb63454ae33b4a171b6437be904a

To use fastcomplete all the time add the following as the *LAST* line in your ~/.bashrc file. Fastcomplete will echo some output to stderr so you can see what is happening.

# Create a cache of the command
eval `~tansell/bin/fastcomplete $PATH`

Can haz kernel patches?

As it’s only a couple of weeks till another is underway, I thought I better post this draft from the last one! This year LCA is in Wellington, New Zealand, sadly I didn’t get my act together enough to make a proper holiday of it. I’m definitely looking forward to catching up with all the cool hackers at the conference.

One of the best talks I went to last year was the Ksplice talk. This is a wonderful tool which allows people to develop “hot patches” so that you never need to reboot again. The developers have done some very cool work which means for 88% of patches a hot fix can be generated automatically. The presenter was a really great speaker too, during the talk he explains some advanced concepts (like hot to fix-up memory structures) I was able to easily understand it all. I can’t watch until I never have to reboot my Linux machines again!

I also attended Rusty‘s tutorial on hacking lguest. Two years ago I submitted a very important patch to lguest which is now included in the kernel. This year I didn’t get as far as I had wanted mostly because I had not gotten enough sleep the night before. I did however submit one patch which was accepted. I guess that makes me a kernel hacker :).

Sadly, Rusty won’t be running another lguest tutorial this year, so it’s unlikely I’ll submit a third patch.

Tagged , , , , , , , , ,

Starhunter, Firefly’s little known older cousin.

Tulip - The ship from Starhunter

I have recently been watching and really enjoying a little known series called Starhunter. At first I thought it was an enjoyable rip off of Firefly, so i was surprised to find out that it was made first.

It has so many of the same elements, the quirky ship mechanic, the captain who doesn’t play by the rules with a darker background, raiders, etc. I have no doubt that Joss Whedon stole many the ideas for Firefly from this series and in the process improving on them in many ways. Sadly, like Firefly this series was canceled way before it reached a wide audience.

You can watch the “second season” (Starhunter 2300), which is closer to a spin off  rather then a second season (most of the characters have changed), on Youtube. I would highly recommend people who enjoyed Firefly check it out. Since its free, what have you got to loose?

While the first season is available on Joost, it is apparently horrible mangled. Sadly, the DVD’s are not available here in Australia so I’m going I’m going to have to buy them from Amazon.

Tagged , , ,

Programmer art, it’s deadly!

So one of the other things I worked on at was a project Joel Stanley started, “Open Velociraptor Per Child, the Game!”.

The game comes out of the Open Velociraptor Per Child project’s ideas. The OVPC project aims to give every child in the world their very own Velociraptor. The game simulates what would happen if this project was ever to achieve it’s wonderful but lofty goal.

My main contribution was adding support for (very bad) collision detection, scoring and difficult levels. However, what I spent most of my time was spent making bad programmer art, that of the particularly gruesome child death animation. I like to think it was this artwork that meant we won top spot at the Gaming Miniconf competition, beating the “Shave Bdale” game.

You can download the game from the git repository. Have fun!

Tagged , , , , ,

Compiling for Windows using Cygwin on Linux….

So for the past week while I have been at the best conference in the world I have been trying to compile tpserver-cpp for Windows. I had done the hard work and gotten it to compile (as documented here, here and here) on Windows previously. However, as I was in Hobart at a Linux conference I didn’t really have access to Windows computer. That was not going to stop me, so I attempted to cross compile the binaries under Linux. This has a number of advantages as it would mean when someone finally gets around to creating a autobuilder, we can produce Windows binaries too.

Ubuntu provides the mingw32 compilers in the repository so I didn’t think it would be all that hard to get working. The problem is that tpserver-cpp does not have a “native” Windows support but cygwin comes to the rescue and provides a compatibly layer. Using cygwin turned out to not be as simple as using mingw32 compiler with the cygwin headers.

I ended up using crosstool to build my own cygwin compiler. I battled for a long while with the fact that Ubuntu now enables “fortify source” by default. This breaks many versions of things like binutils and gcc (which often do naughty things which fortify source does not like). After I figured out how to disable it, I was still was only able to get an ancient version of gcc to compile (3.3.6) which meant I had to fix a lot of problems in the tpserver-cpp code. I guess someone had to do it eventually, but it was annoying that I was forced too.

I then manually downloaded a bunch of cygwin packages to build a tree for the dependencies (such as boost and guile). This was much faster then trying to compile them on my own. Finally, I was able to build tperver-cpp and create a Windows binary! I can confirm it runs fine under Wine and am now getting friends who are still shacked to Windows to test it there.

It sounds much simpler now, but it took me over a week of work to boil it down to these steps. It was like a constant game of wack-a-mole, once I had solved one problem another popped up.

So what now in this area? I want to get a recent version of the compiler working and preferably build all the dependencies ourselves (rather then rely on the cygwin compiled versions). I would ultimately like to see the cygwin compilers being packaged with Ubuntu/Debian in the same way that the mingw32 compilers are. I don’t know if any of that is likely to happen however as I never seem to have enought time. For now I have uploaded a copy of my cross compiler (It needs to be extracted so it is found in /opt/crosstool).

I hope this helps someone!

Tagged , , , , , , ,

OSDC & orbital death, better late then never…

So last year at the end of November I spoke at OSDC. I had been meaning to blog about the conference but never gotten around to doing so. The conference was great, but like previous OSDC’s I came down with something. Sadly, this meant that missed Andrew Tridgell’s talk about the EU-Microsoft agreement. Joel, who had been camped out in my lounge room for the week, said it was really worth listening too. I really wish OSDC recorded all the talks like does.

As part of speaking at the conference I had to produce a paper. This paper puts into writing a lot of what I have been talking about. I wasn’t going to post it, but after getting a email out of the blue about the topic, I’ve decided to put a copy here on my blog. I might as well also upload the presentation I gave, but it won’t be very interesting by itself. Both are released under a CC-BY-SA.

If there is one thing that any budding game developer (open source or otherwise) should take away from this talk, it is the following:

When a person is looking for normal software, they have an issue to solve.
When looking for computer games, they are looking to be entertained.

This fundamental difference in mindset should drive every aspect of your game, website and release process.

Edit: How could I forget? A big thanks goes out to Leslie Hawthorn who helped proof read the paper and making it much easier to read!

Tagged , , , ,

Reading cookies from most Firefox versions…

Yesterday, I wrote about how to reading the cookies from Firefox 3.0 from Python. This code snippet extends the previous example by adding code which finds the cookie file on various different operating systems (Windows, Linux and Mac OS X). Hope this helps people who need to do this.

#! /usr/bin/env python
# Reading the cookie's from Firefox/Mozilla. Supports Firefox 3.0 and Firefox 2.x
# Author: Noah Fontes <nfontes AT cynigram DOT com>, 
#         Tim Ansell <mithro AT mithis DOT com>
# License: MIT
def sqlite2cookie(filename):
    from cStringIO import StringIO
    from pysqlite2 import dbapi2 as sqlite
    con = sqlite.connect(filename)
    cur = con.cursor()
    cur.execute("select host, path, isSecure, expiry, name, value from moz_cookies")
    ftstr = ["FALSE","TRUE"]
    s = StringIO()
# Netscape HTTP Cookie File
# This is a generated file!  Do not edit.
    for item in cur.fetchall():
        s.write("%s\t%s\t%s\t%s\t%s\t%s\t%s\n" % (
            item[0], ftstr[item[0].startswith('.')], item[1],
            ftstr[item[2]], item[3], item[4], item[5]))
    cookie_jar = cookielib.MozillaCookieJar()
    cookie_jar._really_load(s, '', True, True)
    return cookie_jar
import cookielib
import os
import sys
import logging
import ConfigParser
# Set up cookie jar paths
def _get_firefox_cookie_jar (path):
    profiles_ini = os.path.join(path, 'profiles.ini')
    if not os.path.exists(path) or not os.path.exists(profiles_ini):
        return None
    # Open profiles.ini and read the path for the first profile
    profiles_ini_reader = ConfigParser.ConfigParser();
    profile_name = profiles_ini_reader.get('Profile0', 'Path', True)
    profile_path = os.path.join(path, profile_name)
    if not os.path.exists(profile_path):
        return None
        if os.path.join(profile_path, 'cookies.sqlite'):
            return os.path.join(profile_path, 'cookies.sqlite')
        elif os.path.join(profile_path, 'cookies.txt'):
            return os.path.join(profile_path, 'cookies.txt')
def _get_firefox_nt_cookie_jar ():
    # See
        import _winreg
        import win32api
    except ImportError:
        logging.error('Cannot load winreg -- running windows and win32api loaded?')
    key = _winreg.OpenKey(_winreg.HKEY_CURRENT_USER, r'Software\Microsoft\Windows\CurrentVersion\Explorer\Shell Folders')
        result = _winreg.QueryValueEx(key, 'AppData')
    except WindowsError:
        return None
        if ret[1] == _winreg.REG_EXPAND_SZ:
            result = win32api.ExpandEnvironmentStrings(ret[0])
            result = ret[0]
    return _get_firefox_cookie_jar(os.path.join(result, r'Mozilla\Firefox\Profiles'))
def _get_firefox_posix_cookie_jar ():
    return _get_firefox_cookie_jar(os.path.expanduser(r'~/.mozilla/firefox'))
def _get_firefox_mac_cookie_jar ():
    # First of all...
    result = _get_firefox_cookie_jar(os.path.expanduser(r'~/Library/Mozilla/Firefox/Profiles'))
    if result == None:
        result = _get_firefox_cookie_jar(os.path.expanduser(r'~/Library/Application Support/Firefox/Profiles'))
    return result
    'nt': _get_firefox_nt_cookie_jar,
    'posix': _get_firefox_posix_cookie_jar,
    'mac': _get_firefox_mac_cookie_jar
cookie_jar = None
    cookie_jar = FIREFOX_COOKIE_JARS[]()
except KeyError:
    cookie_jar = None
path = raw_input('Path to cookie jar file [%s]: ' % cookie_jar)
if path.strip():
    # Some input specified, set it
    cookie_jar = os.path.realpath(os.path.expanduser(path.strip()))
if cookie_jar.endswith('.sqlite'):
    cookie_jar = sqlite2cookie(cookie_jar)
    cookie_jar = cookielib.MozillaCookieJar(cookie_jar)

Edit: The latest version of this code can be found at and includes numerous fixes and updates.

Tagged , , , ,