Skip to content

HDMI2USB – Production Board Bring Up – Day 1 (21st July 2014)

  • Wrote snippets for the last 3 days.
  • slidelint work
  • Got set up in Joel and Rusty’s coworking space.
  • Explained to Ajit the idea of routing the debug output to multiple locations. Added a diagram to the Debug Design document.
  • Fixed command line processing in hdl-pretty code. Started down the path of auto-generating help from the customize stuff and then realised I had more important things to do.
  • Got most of my way through a Python program to generate UCF files from the HDMI2USB Variants spreadsheet.
  • Xilinx “Compatible” JTAG programmer turned up at Joel’s house. Yet to test to see if it works.

HDMI2USB – Production Board Bring Up – Snippets – Prep Work

Two weeks ago I decided that I was going to take a week of work and spend it full time working on bringing up the HDMI2USB Production board that I’m working on creating with Numato. Since I’m making all our GSoC students do daily snippets, I thought it would be good for me to do snippets too! Will be good to look back on the week and see what I ended up actually doing.

The following snippets are for the preparation work I did before the week;

 

Friday – 18th July

 

Saturday – 19th July

  • Day off. Spent day playing DOTA 2 and StarCraft 2 with Joel’s mates.

 

Sunday – 20th

  • Concentrated on getting slidelint website up and running, targeting announcement at PyCon AU.
  • Created DNS configuration and deployed it. Also updated the domain tracking spreadsheet.
  • Rewrote the setup documentation (and ported it to Markdown).
  • Set up a nginx frontend.
  • Attempted to get circus-web interface working.
    • Turns out it doesn’t work under Python 3 as torandio2 is unmaintained.
    • Ported circus-web to sockjs-torando.
    • Found the tests don’t work.

 

Connecting to a Fritz!Box under Linux using vpnc

I have two Fritz!Box 7390 (one at my place and one at my parents primary residence) and one Fritz!Box 7270 devices (the house they are building). They are pretty reasonable “high end” ADSL routers and a cool feature is they integrated VPN support. I use this functionality to connect the three networks securely together (but that is not what this post is about). This VPN functionality can also be used to connect to your home network while not at home, but information on how to do this from Linux is very sparse (specially if you only want to use FOSS tools to do the connection), so here is how I did it.

Configuring your Fritz!Box

To set up your Fritz!Box you need a configuration file, there is no GUI available in the web interface to support create new VPN configurations. While a number of sites have pre-built .cfg files that look like they should work, my Fritz!Boxes only accept encrypted VPN configuration files (see end of post for more information) and no open source tool exists to encrypt them.

I instead needed to use the FRITZ!Box VPN Connection tool (while it is a Windows program but runs fine under Wine) to create the configuration. It can be downloaded from the AVM website (This is different to the VPN client that AVM also provide). Annoyingly the tool doesn’t just read existing .cfg configuration files, it instead reads it’s own vpnadmin.cfg found in c:/users/your username/Application Data/AVM/FRITZ!VPN and then generates a .cfg file and encrypts it.

Using the FRITZ!Box VPN Connection tool to create a configuration that is compatible with vpnc you must:

  • Select “Configure VPN for one User” type connection
  • On the “Select Device” screen, select “iPhone, iPod touch or iPad” option
  • On the “Enter the user’s email address” screen, despite the admin tool calling this field the user email, just enter a username. I recommend not having any special characters like @ or . in it.
  • On the “Enter IP address of the User” screen, be careful about what IP address you use (the default should be okay).
    • Don’t use the same IP address that the computer uses when connected via wireless/wired. While it seems like a good idea, as the computer would have the same IP address even when remote, it does not work and will mean the device is unable to access the internet when connected to wireless/wired.
    • The “Send all data over the VPN tunnel” option on this page does not seem to affect vpnc, it will always route all your data over the vpn connection. See later for how to fix this problem.
  • On the “Key for the connection” screen enter a password. Copy down the shared secret key, you’ll want it when creating the vpnc config. I recommend also keeping the default shared secret key it generates unless your super paranoid about entropy.

You can check that you have done this correctly in two ways;

  • The vpnadmin.cfg will have the iphone=1 and xauth_key="your password" options. See below for a partial example;
    ...
      user {
        nameoremail = "xxxx";
        key = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx";
        ip = 192.168.179.201;
        internetaccess = 0;
        iphone = 1;
        xauth_key = "xxxxx";
      }
    ...
    

  • If you export an unencrypted config file (remember this can’t be loaded onto the Fritz!Box – so make sure you also export an encrypted version!), the remoteid section will have a key field and not a fqdn field, use_xauth will be set to yes and an xauth section will exist. See below for a partial example;
    ...
      keepalive_ip = 0.0.0.0;
      remoteid {
        key_id = "qnap";
      }
      mode = phase1_mode_aggressive;
      phase1ss = "all/all/all";
      keytype = connkeytype_pre_shared;
      key = "f3e0hcca19ca2a3gaafbI.{1dGe3q8f84";
      cert_do_server_auth = no;
      use_nat_t = yes;
      use_xauth = yes;
      xauth {
        valid = yes;
        username = "qnap";
        passwd = "qnappassword";
      }
      use_cfgmode = yes;
    ...

Configuring vpnc on your Linux box

The Fritz!Box needs both draft-ietf-ipsec-nat-t-ike-03 support and the client to adopt the server suggested lifetime (which for the Fritz!Box is 3600 seconds). These features where only added to vpnc recently, so make sure your vpnc is newer than subversion revision 511. Both the version in Debian Unstable and any Ubuntu newer than Precise (12.04) have vpnc 0.5.3r512, which is new enough version and you can just apt-get install vpnc. For Fedora 17 x86 systems you can find RPMs at http://www.pabloendres.com/2013/02/27/vpnc-and-fritzbox/

Create the vpnc config in /etc/vpnc/fritzbox.conf using the following as a template (replace the parts in bold/brackets):

IPSec gateway ip address or DNS name of your FritzBox

IKE DH Group dh2
Perfect Forward Secrecy nopfs

IPSec ID [username entered into the "Enter the user's email address" screen]
# "key" from the Fritz!Box VPN configuration
IPSec secret [shared secret key from the "Key for the connection" screen]

NAT Traversal Mode force-natt

Xauth username [username entered into the "Enter the user's email address" screen]
Xauth password [password entered into the "Key for the connection" screen - Not the password use to encrypt the vpnc configuration!]

As this file contains usernames and password, the config file should be owned by root and only readable by the owner.

sudo chown root /etc/vpnc/fritzbox.conf
sudo chmod 0600 /etc/vpnc/fritzbox.conf

You should now be able to connect to your home internet using:

sudo vpnc-connect fritzbox

When you are finished, use:

sudo vpnc-disconnect fritzbox

Internet Access while using vpnc

When vpnc connects it will change your default route to go via the tunnel. This will send all your local internet traffic over the VPN connection. If you are setting up the VPN to secure your internet browsing while on a hostile network such as a public WiFi hotspot (such as at a coffee shop) this is what you want.

However, if you just want to be able to access the hosts on your home network, you don’t want this as it will make your internet browsing extraordinarily slow. These routes are set up by /etc/vpn/vpnc-script script, so you will need custom one. Normally this script is provided a lot of information when connecting to other VPN concentrators (such as a Cisco EasyVPN device) but when connecting to the Fritz!Box you only seem to get limited information.

I created the following script in /etc/vpnc/fritzbox-script, marked it as executable (chmod a+x /etc/vpnc/fritzbox-script) and then added “Script /etc/vpnc/fritzbox-script” to my /etc/vpnc/fritzbox.conf file.

#!/bin/sh

IPROUTE=/sbin/ip

case "$reason" in
  pre-init)
    /etc/vpnc/vpnc-script pre-init
    ;;
  connect)
    INTERNAL_IP4_PREFIX=$(echo $INTERNAL_IP4_ADDRESS | sed -e's/\.[0-9]\+$//')
    $IPROUTE link set dev "$TUNDEV" up mtu 1024
    $IPROUTE addr add "$INTERNAL_IP4_ADDRESS/255.255.255.0" peer "$INTERNAL_IP4_ADDRESS" dev "$TUNDEV"
    $IPROUTE route replace "$INTERNAL_IP4_PREFIX.0/255.255.255.0" dev "$TUNDEV"
    $IPROUTE route flush cache
    ;;
  disconnect)
    $IPROUTE link set dev "$TUNDEV" down
    ;;
  *)
    echo "unknown reason '$reason'. Maybe vpnc-script is out of date" 1>&2
    exit 1
    ;;
esac
exit 0

Fritz!Box encrypted VPN configuration files

Fritz!Boxes will only accept VPN configurations which are encrypted. Otherwise when try to import the file you will get the cryptic "Error: Import of the VPN settings failed." as seen below;
Error: Import of the VPN settings failed.

To create an encrypted file you must use the FRITZ!Box VPN Connection tool. Then when exporting, select "Save VPN settings in a file under" option, check the "Encrypt VPN settings" check box (the file type should change to .eff), and enter your chosen password twice.

Export VPN settings

Lightning Timer website

When doing lightning talks it is very useful to have a timer so that people don’t over time. There use to be a very simple website called lightningtimer.net but it appears to have gone AWOL. I have set up a mirrored the code at lightningtimer.cc and lightningtimer.info

By default it does 5 minutes, but if you can set the hash value to get different times;

Code found at https://github.com/simonw/lightningtimer.net

FastComplete, making bash completion fast on remote file systems

Here at Google we have quite a few remote file systems which contain various tools we use in our day-to-day work. As typing sucks we generally want the tools in our $PATH. When you try to tab complete Bash needs to stat a couple of thousand files and even on fast remote file systems this takes a drastically long time.

I wrote FastComplete as a solution to this problem. The tool creates a local cache of links on your hard drive to everything in your $PATH. It uses a couple of tricks to make sure all the stats remain locally, while still allowing the remote file to change without needing to update the cache. Linux should also keep this information in memory disk cache making tab completion almost instant again. Yay!

You can find FastComplete at https://github.com/mithro/rcfiles/blob/master/bin/fastcomplete It is a stand alone python program which shouldn’t have any non-core dependencies. The usage documentation is as follows;

Fast complete creates a local disk cache of your path.
It’s specifically designed to make bash tab complete run much faster. The correct fix would be to add caching to bash, but it was to hard to do so.

To find out what path fastcomplete is currently using:

> ~tansell/bin/fastcomplete
# Found 3977 commands
export PATH=/home/tansell/bin: ... :/home/build/google3/googledata/validators:/home/build/google3/ads/db

To get fastcomplete to rebuild it’s cache:

> ~tansell/bin/fastcomplete --rebuild
# Using path of '/home/tansell/bin: ... :/home/build/google3/googledata/validators:/home/build/google3/ads/db'
# Cache /usr/local/google/users//tansell/tabcache/d7e5fb63454ae33b4a171b6437be904a did not exist! Rebuilding....
# Looking in: /home/tansell/bin (execv)
...
# Looking in: /usr/bin (symlink)
...
# Looking in: /home/build/google3/ads/db (execv)
# Found 3977 commands
export PATH=/usr/local/google/users//tansell/tabcache/d7e5fb63454ae33b4a171b6437be904a

To use fastcomplete all the time add the following as the *LAST* line in your ~/.bashrc file. Fastcomplete will echo some output to stderr so you can see what is happening.

# Create a cache of the command
eval `~tansell/bin/fastcomplete $PATH`

Can haz kernel patches?

As it’s only a couple of weeks till another Linux.conf.au is underway, I thought I better post this draft from the last one! This year LCA is in Wellington, New Zealand, sadly I didn’t get my act together enough to make a proper holiday of it. I’m definitely looking forward to catching up with all the cool hackers at the conference.

One of the best talks I went to Linux.conf.au last year was the Ksplice talk. This is a wonderful tool which allows people to develop “hot patches” so that you never need to reboot again. The developers have done some very cool work which means for 88% of patches a hot fix can be generated automatically. The presenter was a really great speaker too, during the talk he explains some advanced concepts (like hot to fix-up memory structures) I was able to easily understand it all. I can’t watch until I never have to reboot my Linux machines again!

I also attended Rusty‘s tutorial on hacking lguest. Two years ago I submitted a very important patch to lguest which is now included in the kernel. This year I didn’t get as far as I had wanted mostly because I had not gotten enough sleep the night before. I did however submit one patch which was accepted. I guess that makes me a kernel hacker :).

Sadly, Rusty won’t be running another lguest tutorial this year, so it’s unlikely I’ll submit a third patch.

Tagged , , , , , , , , ,

Starhunter, Firefly’s little known older cousin.

Tulip - The ship from Starhunter

I have recently been watching and really enjoying a little known series called Starhunter. At first I thought it was an enjoyable rip off of Firefly, so i was surprised to find out that it was made first.

It has so many of the same elements, the quirky ship mechanic, the captain who doesn’t play by the rules with a darker background, raiders, etc. I have no doubt that Joss Whedon stole many the ideas for Firefly from this series and in the process improving on them in many ways. Sadly, like Firefly this series was canceled way before it reached a wide audience.

You can watch the “second season” (Starhunter 2300), which is closer to a spin off  rather then a second season (most of the characters have changed), on Youtube. I would highly recommend people who enjoyed Firefly check it out. Since its free, what have you got to loose?

While the first season is available on Joost, it is apparently horrible mangled. Sadly, the DVD’s are not available here in Australia so I’m going I’m going to have to buy them from Amazon.

Tagged , , ,

Programmer art, it’s deadly!

So one of the other things I worked on at Linux.conf.au was a project Joel Stanley started, “Open Velociraptor Per Child, the Game!”.

The game comes out of the Open Velociraptor Per Child project’s ideas. The OVPC project aims to give every child in the world their very own Velociraptor. The game simulates what would happen if this project was ever to achieve it’s wonderful but lofty goal.

My main contribution was adding support for (very bad) collision detection, scoring and difficult levels. However, what I spent most of my time was spent making bad programmer art, that of the particularly gruesome child death animation. I like to think it was this artwork that meant we won top spot at the Gaming Miniconf competition, beating the “Shave Bdale” game.

You can download the game from the git repository. Have fun!

Tagged , , , , ,

Compiling for Windows using Cygwin on Linux….

So for the past week while I have been at the best conference in the world I have been trying to compile tpserver-cpp for Windows. I had done the hard work and gotten it to compile (as documented here, here and here) on Windows previously. However, as I was in Hobart at a Linux conference I didn’t really have access to Windows computer. That was not going to stop me, so I attempted to cross compile the binaries under Linux. This has a number of advantages as it would mean when someone finally gets around to creating a autobuilder, we can produce Windows binaries too.

Ubuntu provides the mingw32 compilers in the repository so I didn’t think it would be all that hard to get working. The problem is that tpserver-cpp does not have a “native” Windows support but cygwin comes to the rescue and provides a compatibly layer. Using cygwin turned out to not be as simple as using mingw32 compiler with the cygwin headers.

I ended up using crosstool to build my own cygwin compiler. I battled for a long while with the fact that Ubuntu now enables “fortify source” by default. This breaks many versions of things like binutils and gcc (which often do naughty things which fortify source does not like). After I figured out how to disable it, I was still was only able to get an ancient version of gcc to compile (3.3.6) which meant I had to fix a lot of problems in the tpserver-cpp code. I guess someone had to do it eventually, but it was annoying that I was forced too.

I then manually downloaded a bunch of cygwin packages to build a tree for the dependencies (such as boost and guile). This was much faster then trying to compile them on my own. Finally, I was able to build tperver-cpp and create a Windows binary! I can confirm it runs fine under Wine and am now getting friends who are still shacked to Windows to test it there.

It sounds much simpler now, but it took me over a week of work to boil it down to these steps. It was like a constant game of wack-a-mole, once I had solved one problem another popped up.

So what now in this area? I want to get a recent version of the compiler working and preferably build all the dependencies ourselves (rather then rely on the cygwin compiled versions). I would ultimately like to see the cygwin compilers being packaged with Ubuntu/Debian in the same way that the mingw32 compilers are. I don’t know if any of that is likely to happen however as I never seem to have enought time. For now I have uploaded a copy of my cross compiler (It needs to be extracted so it is found in /opt/crosstool).

I hope this helps someone!

Tagged , , , , , , ,

OSDC & orbital death, better late then never…

So last year at the end of November I spoke at OSDC. I had been meaning to blog about the conference but never gotten around to doing so. The conference was great, but like previous OSDC’s I came down with something. Sadly, this meant that missed Andrew Tridgell’s talk about the EU-Microsoft agreement. Joel, who had been camped out in my lounge room for the week, said it was really worth listening too. I really wish OSDC recorded all the talks like Linux.conf.au does.

As part of speaking at the conference I had to produce a paper. This paper puts into writing a lot of what I have been talking about. I wasn’t going to post it, but after getting a email out of the blue about the topic, I’ve decided to put a copy here on my blog. I might as well also upload the presentation I gave, but it won’t be very interesting by itself. Both are released under a CC-BY-SA.

If there is one thing that any budding game developer (open source or otherwise) should take away from this talk, it is the following:

When a person is looking for normal software, they have an issue to solve.
When looking for computer games, they are looking to be entertained.

This fundamental difference in mindset should drive every aspect of your game, website and release process.

Edit: How could I forget? A big thanks goes out to Leslie Hawthorn who helped proof read the paper and making it much easier to read!

Tagged , , , ,