Skip to content

Reading cookies from most Firefox versions…

Yesterday, I wrote about how to reading the cookies from Firefox 3.0 from Python. This code snippet extends the previous example by adding code which finds the cookie file on various different operating systems (Windows, Linux and Mac OS X). Hope this helps people who need to do this.

#! /usr/bin/env python
# Reading the cookie's from Firefox/Mozilla. Supports Firefox 3.0 and Firefox 2.x
#
# Author: Noah Fontes <nfontes AT cynigram DOT com>, 
#         Tim Ansell <mithro AT mithis DOT com>
# License: MIT
 
def sqlite2cookie(filename):
    from cStringIO import StringIO
    from pysqlite2 import dbapi2 as sqlite
 
    con = sqlite.connect(filename)
 
    cur = con.cursor()
    cur.execute("select host, path, isSecure, expiry, name, value from moz_cookies")
 
    ftstr = ["FALSE","TRUE"]
 
    s = StringIO()
    s.write("""\
# Netscape HTTP Cookie File
# http://www.netscape.com/newsref/std/cookie_spec.html
# This is a generated file!  Do not edit.
""")
    for item in cur.fetchall():
        s.write("%s\t%s\t%s\t%s\t%s\t%s\t%s\n" % (
            item[0], ftstr[item[0].startswith('.')], item[1],
            ftstr[item[2]], item[3], item[4], item[5]))
 
    s.seek(0)
 
    cookie_jar = cookielib.MozillaCookieJar()
    cookie_jar._really_load(s, '', True, True)
    return cookie_jar
 
import cookielib
import os
import sys
import logging
import ConfigParser
 
# Set up cookie jar paths
def _get_firefox_cookie_jar (path):
    profiles_ini = os.path.join(path, 'profiles.ini')
    if not os.path.exists(path) or not os.path.exists(profiles_ini):
        return None
 
    # Open profiles.ini and read the path for the first profile
    profiles_ini_reader = ConfigParser.ConfigParser();
    profiles_ini_reader.readfp(open(profiles_ini))
    profile_name = profiles_ini_reader.get('Profile0', 'Path', True)
 
    profile_path = os.path.join(path, profile_name)
    if not os.path.exists(profile_path):
        return None
    else:
        if os.path.join(profile_path, 'cookies.sqlite'):
            return os.path.join(profile_path, 'cookies.sqlite')
        elif os.path.join(profile_path, 'cookies.txt'):
            return os.path.join(profile_path, 'cookies.txt')
 
def _get_firefox_nt_cookie_jar ():
    # See http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/473846
    try:
        import _winreg
        import win32api
    except ImportError:
        logging.error('Cannot load winreg -- running windows and win32api loaded?')
    key = _winreg.OpenKey(_winreg.HKEY_CURRENT_USER, r'Software\Microsoft\Windows\CurrentVersion\Explorer\Shell Folders')
    try:
        result = _winreg.QueryValueEx(key, 'AppData')
    except WindowsError:
        return None
    else:
        key.Close()
        if ret[1] == _winreg.REG_EXPAND_SZ:
            result = win32api.ExpandEnvironmentStrings(ret[0])
        else:
            result = ret[0]
 
    return _get_firefox_cookie_jar(os.path.join(result, r'Mozilla\Firefox\Profiles'))
 
def _get_firefox_posix_cookie_jar ():
    return _get_firefox_cookie_jar(os.path.expanduser(r'~/.mozilla/firefox'))
 
def _get_firefox_mac_cookie_jar ():
    # First of all...
    result = _get_firefox_cookie_jar(os.path.expanduser(r'~/Library/Mozilla/Firefox/Profiles'))
    if result == None:
        result = _get_firefox_cookie_jar(os.path.expanduser(r'~/Library/Application Support/Firefox/Profiles'))
    return result
 
FIREFOX_COOKIE_JARS = {
    'nt': _get_firefox_nt_cookie_jar,
    'posix': _get_firefox_posix_cookie_jar,
    'mac': _get_firefox_mac_cookie_jar
}
 
cookie_jar = None
try:
    cookie_jar = FIREFOX_COOKIE_JARS[os.name]()
except KeyError:
    cookie_jar = None
 
path = raw_input('Path to cookie jar file [%s]: ' % cookie_jar)
if path.strip():
    # Some input specified, set it
    cookie_jar = os.path.realpath(os.path.expanduser(path.strip()))
 
if cookie_jar.endswith('.sqlite'):
    cookie_jar = sqlite2cookie(cookie_jar)
else:
    cookie_jar = cookielib.MozillaCookieJar(cookie_jar)

Edit: The latest version of this code can be found at http://blog.mithis.com/cgi-bin/gitweb.cgi and includes numerous fixes and updates.

Tagged , , , ,

WTF power scripts went in Intrepid….

On previous versions of Ubuntu, the scripts which are called after a resume from suspend have been found in /etc/acpi/resume.d directory. I used this functionality to turn off some of the hardware in my Vaio which I don’t use (such as the bluetooth and the cdrom drive).

This stopped working when I upgraded to Ubuntu Intrepid. Even more strangely while the scripts are still installed, even they are never called.

It appears that thanks to moving towards HAL (which is probably a “Good Thing”) these scripts are no longer used. The scripts which are used can be found in /etc/pm/. Not only has the location changed, but the script format has too.

Previously, my script was found in /etc/acpi/resume.d/99-custom.sh looked like the following,

#! /bin/sh
# Turn off the CD drive and the bluetooth device
echo 1 > /sys/devices/platform/sony-laptop/cdpower
echo 0 > /sys/devices/platform/sony-laptop/cdpower
 
echo 1 > /sys/devices/platform/sony-laptop/bluetoothpower
echo 0 > /sys/devices/platform/sony-laptop/bluetoothpower

Now my script script must be found in /etc/pm/sleep.d/10-custom and looks like the following,

#!/bin/sh -e
case "$1" in
	resume)
		# Turn off the CD drive and the bluetooth device
		echo 1 > /sys/devices/platform/sony-laptop/cdpower
		echo 0 > /sys/devices/platform/sony-laptop/cdpower
 
		echo 1 > /sys/devices/platform/sony-laptop/bluetoothpower
		echo 0 > /sys/devices/platform/sony-laptop/bluetoothpower
	;;
esac

The main reason I’m posting this on my blog is that this change does not seem to be documented anywhere. Searching on Google for things like “resume script intrepid” or “/etc/acpi/resume.d intrepid” does not come up with anything useful. Hopefully some people will find this helpful.

Tagged , , , , , , ,

$#%#! UTF-8 in Python

This is not a post about using UTF-8 properly in Python, but doing evil, evil things.

Python dutifully respects the $LANG environment variable on the terminal. It turns out that a lot of the time this variable is totally wrong, it’s set to something like C even though the terminal is UTF-8 encoding.

The problem is that there is no easy way to change a file’s encoding after it’s open, well until this horrible hack! The following code will force the output encoding of stdout to UTF-8 even if started with LANG=C.

# License: MIT
try:
    print u"\u263A"
except Exception, e:
    print e
 
import sys
print sys.stdout.encoding
 
from ctypes import pythonapi, py_object, c_char_p
PyFile_SetEncoding = pythonapi.PyFile_SetEncoding
PyFile_SetEncoding.argtypes = (py_object, c_char_p)
if not PyFile_SetEncoding(sys.stdout, "UTF-8"):
    raise ValueError
 
try:
    print u"\u263A"
except Exception, e:
    print e
Tagged , , , , ,

Reading Firefox 3.x cookies in Python

I found the following code snippet on my hard drive today. It allows you to access Firefox 3.x cookies in Python. Firefox 3.x moved away from the older text file format to a sqlite database.

This code is useful if you want to access something behind an authentication gateway and you also access the page through your web browser. You can also use this code to convert a sqlite database into a cookie file CURL can read.

I didn’t write this code, it was written by Noah Fontes when we where doing some scraping of the Google Summer of Code website (before I joined Google).

#! /usr/bin/env python
# Protocol implementation for handling gsocmentors.com transactions
# Author: Noah Fontes nfontes AT cynigram DOT com
# License: MIT
 
def sqlite2cookie(filename):
    from cStringIO import StringIO
    from pysqlite2 import dbapi2 as sqlite
 
    con = sqlite.connect(filename)
 
    cur = con.cursor()
    cur.execute("select host, path, isSecure, expiry, name, value from moz_cookies")
 
    ftstr = ["FALSE","TRUE"]
 
    s = StringIO()
    s.write("""\
# Netscape HTTP Cookie File
# http://www.netscape.com/newsref/std/cookie_spec.html
# This is a generated file!  Do not edit.
""")
    for item in cur.fetchall():
        s.write("%s\t%s\t%s\t%s\t%s\t%s\t%s\n" % (
            item[0], ftstr[item[0].startswith('.')], item[1],
            ftstr[item[2]], item[3], item[4], item[5]))
 
    s.seek(0)
 
    cookie_jar = cookielib.MozillaCookieJar()
    cookie_jar._really_load(s, '', True, True)
    return cookie_jar
Tagged , , , ,

In the land of the sheep…

I wrote this post while in New Zealand but never posted it, now I’m at Linux.conf.au I have time to finish it up.

Well its been a long time since I have posted on my blog. As I lasted mentioned I now work at Google, which has been going well but keeping me fairly busy. For the last month (October, 2009) I have been back in Mountain View, California. While I was there for mainly work purposes, I did get the chance to go to both the Summer of Code Mentor Summit and the GitTogether. Both where a lot of fun but tiering.

It was good to see the BZFlag guys again – they even had cool t-shirts this year. Not as cool as our Thousand Parsec shirts, however. 🙂 I was finally able to meet kblin who I had know through the WorldForge project for many years. As always he looked nothing like I expected.

At the GitTogther I was mainly interested in trying to make git usable with large media repositories. This is one area which Subversion still has an advantage. After much discussion we came up with a solution to the problem which I gave a short presentation.

It also gave me a chance to catch up with the Open Source Progams Office. It was great to catch up with Leslie Hawthorn and her fabulous crew.

No sooner had I gotten back from the states, I headed of to New Zealand. Lee Begg who I also first met through the WorldForge project and was the co-founder of the Thousand Parsec project, is getting married and I will be a grooms man.

Tagged , , , , ,

But…

they can fall to bits from too much love!

Tagged ,

Babylon 5, the adventure with DVD copy protection

I still have not found an apartment so as I have no internet to entertain me on the weekend I got a copy of the first season of Babylon 5 (it was only $20.00 AUD for the whole season). I had been working my way through Andromeda but no where seems to have the third season.

Anyway when I put the DVD video in the drive in my Sony Vaio Laptop running Ubuntu Hardy all I got where I/O errors, some examples are below;

[ 2283.614887] end_request: I/O error, dev sr0, sector 418256
[ 2283.620351] end_request: I/O error, dev sr0, sector 418264
[ 2283.626273] end_request: I/O error, dev sr0, sector 418264
[ 2283.631766] end_request: I/O error, dev sr0, sector 418272
[ 2283.637013] end_request: I/O error, dev sr0, sector 418272
[ 2283.642384] end_request: I/O error, dev sr0, sector 418280

The Vaio’s DVD drive is connected via the USB bus. This is done so that drive can be completely powered down. The device turned out to be a MATSHITA DVD-RAM drive as shown via the dmesg output below;

[ 2909.596251] scsi11 : SCSI emulation for USB Mass Storage devices
[ 2909.596944] usb-storage: device found at 24
[ 2909.596952] usb-storage: waiting for device to settle before scanning
[ 2911.901103] usb-storage: device scan complete
[ 2911.903506] scsi 11:0:0:0: CD-ROM            MATSHITA DVD-RAM UJ-852S  1.31 PQ: 0 ANSI: 0
[ 2911.948247] sr1: scsi3-mmc drive: 24x/24x writer dvd-ram cd/rw xa/form2 cdda tray
[ 2911.948372] sr 11:0:0:0: Attached scsi CD-ROM sr1
[ 2911.948460] sr 11:0:0:0: Attached scsi generic sg1 type 5

It took me forever to figure out what was going on. I had seen similar problems on my desktop before when the disk was scratched but these where brand new disks. So I took the disks into work and tested it out on a friends Mac, it played perfectly. There happened to be a Steve Irwin DVD video disk lying around, so I popped it in the Vaio, it also played perfectly! What was going on?

After much searching I came across some reference to problems with region coding. It turns out that  MATSHITA drives won’t let you read a dvd unless they they have a region set. As I had never played a DVD video before the region on the drive had never been set.

There is a tool in Linux which can be used to do the region setting, it is helpfully called regionset. After setting the region to “Region 4” I am now able to play my new DVDs! I wonder if I will be able to read my discs from the US and the UK. The libdvdcss2 should be able to decode the data if it can be read, hopefully the drive will still let that occurring. I will report back in comments here when I find out for sure.

It has been repetitively found that region encoding is anticompetitive and hence un-unenforceable in Australia. I have included a quote from the Stevens v Kabushiki Kaisha Sony Computer Entertainment case from 2005.

Ordinary principles of statutory construction, observed by this Court since its earliest days, have construed legislation, where there is doubt, to protect the fundamental rights of the individual. The right of the individual to enjoy lawfully acquired private property (a CD ROM game or a PlayStation console purchased in another region of the world or possibly to make a backup copy of the CD ROM) would ordinarily be a right inherent in Australian law upon the acquisition of such a chattel. This is a further reason why s 116A of the Copyright Act and the definition of TPM in s 10(1) of that Act should be read strictly. Doing so avoids an interpretation that would deprive the property owner of an incident of that person’s ordinary legal rights.

I thought I would just log my info here as there was very little information in Google about this problem. Hopefully I help some poor fool which brought a Sony Vaio like me.

Tagged , , , , , ,

My three weeks on a Mac

As everyone knows, I recently started at Google. When I started I was given a MacBook Pro to use as the company laptop before I had a chance to change it, I had to head off to Mountain View for training. This meant I ended up using a Mac for 3 and half weeks.

Now I am back in Australia I have decided to trade in my Mac for a nice PC running Linux. People have continually told me that Macs are the epitome for polished UI and once you get use to them, there is no going back. When I suggested that this might not be the case, I was told “but you have never used Mac” – well now I have and I have specific examples of why Apple’s are less usable then Linux.

My first bone to pick is with the unlock screen. As I work at Google and might have the codes for the orbital space laser on my laptop, I need to lock my screen anytime I walk away from my desk. In gnome on Linux I can just walk back to my computer and start typing my password, it makes sure that all the keys end up in the password box – no so on a Mac. When I get back, I first have to move the mouse or hit a key, I then have to wait for the twirling multi-color ball and then I get to type my password. If I just start typing I loose the first 3 or more characters of the password.

Next is the useless wireless indicator that Mac has. On Linux I can clearly see if I am connected, trying to connect or waiting, I can also see if I am on wireless or wired network. This is all thanks to Network Manager which is very, very cool. On Mac, you can’t tell if you are connected or the Mac is having a shit and still trying to connect. Often, I had to bring up a ping program to see if the wireless bars meant I was actually connected or not. If I plug in the ethernet, without specifically disabling the Airport how do I know where my packets are going?

The twirling ball of doom. Normally programs either lock hard or work. Not on Mac, instead you get a ball which twirls forever. After waiting for 15 minutes I just hard reset my computer. At least if I knew the computer was locked up I wouldn’t have to wait that 15 minutes.

I have often gotten this error “You cannot move any item to the Trash because it is being emptied” when doing a secure empty of my trash bin. How hard is it to put things in the trash while emptying it?

Alt Tab doesn’t work. It doesn’t change between windows, only applications. Often I have multiple windows open in one application. I first have to “alt tab” to the correct application, then I have to “command tab” to the correct window. How annoying!

So that was just a few issues I have had. Overall, I am much happier with Ubuntu and it keeps getting even better.

Tagged , , , ,

Tech Talk at Google – Gaming for Freedom

Last week on friday, I gave a Tech Talk about Open Source Gaming as part of Leslie Hawthorn’sOpen Source Developers @ Google” talk series. For those who were silly enough to miss it, it should be soon coming to a YouTube near you.

I had given a similar talk at my local LUG only recently, I think that version went a little better but it was quite a different crowd. The first part of my talk came across way more preachy then I had hoped. I also see now how I can better use Thousand Parsec as examples of the tips I came up for releasing FOSS games. I guess practice makes perfect, maybe I’ll get it right to one day be able to give it at Linux.conf.au.

At the beginning this time I tried some of the “one word per slide quick succession” talk which Anthony Baxter had suggested. I think however think I ended up just insulting every American! I don’t think I speak fast enough to make this type of talk successful, but I’ll keep experimenting.

If you have any feedback on the talk, please do send me an email!

The real reason for this post is to upload the slides for the tech talk so they can be linked from the YouTube video.

Edit: The talk has now been uploaded, you can access it via the following link or see it below,

Tagged , , , ,

I will survive…

It’s been a hectic week, but I have managed to survive my first week as a Noogler. As everyone knows, I started at Google last week. The first week has been quite hectic as I started at the Sydney office and then in the middle of week flew to the Googleplex in Mountain View, California (if anyone is located in this area and wants to have lunch, feel free to look me up!).

The week has been a little bit of a mess for a number of reasons. I’m joining the PSO group at Google which is only just started to establish a presence at the Sydney office, this has meant not a lot of people know what I should be doing. To make matters worse, I managed to arrive for the Memorial Day weekend (public holiday on Monday) which means the normal schedule for training (which starts on a Monday) is all changed. It’s still be very demanding and interesting anyway.

I arrived here in San Francisco on a Thursday (before I actually left according to wall time, Yay timezones!)  and after sleeping for about 18 hours straight managed to make it into the office on the Friday. The jet lag didn’t really end up hitting me until the weekend, so I’ve spent the last three days trying to reset my body clock instead of doing anything interesting.

I’m very pleased with the transparency inside of Google. I have yet to find a door which my keycard won’t open or a website which I can not access with my login. Of course I can’t tell anyone outside of Google what I have found out which is annoying, there is a lot of cool stuff going on.

Having worked mainly for small companies in the past, it’s nice to be able to walk into TechStop (Google’s tech support department) and get both hardware and support. It’s nice having small things like beverages (such as that all important coffee) and food provided. The food in the Sydney office is not bad, but it pales in comparison to the variety and quality that you get here at the Googleplex.

I have heard that people worry that with all the food at Google you just end up going to lunch with your work mates, I don’t think that is something having food provided creates. At the my previous work I ended up in that situation, always having lunch with the same people at the same locations, the only difference is that I was paying for it! Google is liberal about letting you invite friends to have lunch at their free cafeteria’s too.

Over all, it has been a very busy and interesting week. I’m looking forward to what tomorrow (and the rest of this week) holds.

Tagged , , ,