Planet London Python

Links for 2010-09-02 [del.icio.us]

2010-09-03T07:00:00+00:00

Battle of the Android Home Screen Launchers

IMAPClient 0.6.1 released

2010-09-02T21:56:00+00:00

I've just released IMAPClient 0.6.1.

The only functional change in the release is that it now automatically patches imaplib's IMAP4_SSL class to fix Python Issue 5949. This is a bug that's been fixed in later Python 2.6 versions and 2.7 but still exists in Python versions that are in common use. Without fix this you may experience hangs when using SSL.

The patch is only applied if the running Python version is known to be one of the affected versions. It is applied when IMAPClient is imported.

The only other change in this release is that I've now marked IMAPClient as "production ready" on PyPI and have updated the README to match. This was prompted by a request to clarify the current status of the project and seeing that all current functionality is solid and, I don't plan to change the existing APIs in backwards-incompatible ways, I've decided to indicate the project as suitable for production use.

As always, IMAPClient can be installed from PyPI (pip install imapclient) or downloaded from the IMAPClient site. Feedback, bug reports and patches are most welcome.

links for 2010-09-02

2010-09-02T21:01:38+00:00

Links for 2010-09-01 [del.icio.us]

2010-09-02T07:00:00+00:00

Ask the indie professor: Are gender stereotypes still present in indie music?

links for 2010-09-01

2010-09-01T21:03:12+00:00

“Editing” trees in Clojure with clojure.zip

(tags: clojure zip tree)
Graham King » Setting up Munin on Ubuntu

(tags: ubuntu munin monitoring sysadmin devops)
Munin – Trac

A Linux tool for monitoring resource trends

(tags: apache resource analysis logging monitoring sysadmin devops munin)
Lanyrd | the social conference directory

(tags: twitter conference simonwillison)
透明思考 | Migrating To A Decent SCM

How to move several thousand devs from clearcase to SVN in 7 days. ish.

(tags: scm devops clearcase svn)
Home – Graylog2 – Free Open Source remote TCP/UDP Syslog daemon with Web Interface

An open-source syslog storage and viewing system using MongoDB and rails

(tags: syslog syadmin logging devops rails mongodb)

Links for 2010-08-30 [del.icio.us]

2010-08-31T07:00:00+00:00

Preparedness, Privilege and Discrimination

2010-08-29T13:23:26+00:00

Coming very late to the party I noticed a blog entry from June about the JavaScript community's response to Google's financial support to allow more women to attend JSConf Europe. It sounds like it was indeed the usual real can of worms, and many of the comments show the usual lack of appreciation of the problems from the privileged side of the issue (in this case men, since it is a gender issue and women appear to be about as well represented in the JavaScript community as they are in the Python world, which is to say hardly at all).

The reason I found it interesting was that this year, for the first time, PyCon also used Google's support to fund the attendance for more women, who came from as far afield as Roumania and India. I was gratified to be told by several of those who would otherwise not have been at the conference how happy they were to have had the chance to attend. (In truth I had little to do with it: Google should get the credit for the funding, and the actual hard work was done by Peter Kropf and Gloria Willedsen). In the Python community's case there was one short period of adverse comment on the #python IRC channel, which resulted in my exchanging emails with one person to explain why I thought it was a good idea to encourage more women to be at PyCon. End of story, except that in 2010 women represented 11% of PyCon attendance, up from 2% the previous year. I count that as one of the better results of my time as chairman.

Now I am not saying this to be smug, but because I believe there was a reason for the difference in the reactions. Last year the PSF, at Guido van Rossum's urging, started a diversity mailing list which discussed the questions of race, gender and other discrimination extensively and sometimes acrimoniously. Eventually this led to a proposal for a "diversity statement", which was referred to the membership where it triggered another round of extensive and sometimes acrimonious discussions, leading to a referral back to the diversity list and a further proposal which was finally accepted by the membership more or less unchanged and adopted by the Board:

The Python Software Foundation and the global Python community welcome and encourage participation by everyone. Our community is based on mutual respect, tolerance, and encouragement, and we are working to help each other live up to these principles. We want our community to be more diverse: whoever you are, and whatever your background, we welcome you.

This may not be the best statement ever, but if anyone bothers to look it does make it clear that these issues have been addressed. Thus anyone who feels discriminated against can decide that at least they would have a chance of a fair hearing should they choose to complain (which, sadly, I imagine most don't, instead choosing to vote with their feet). Similarly, anyone about to indulge in discriminatory behavior might think twice before doing so.

The hidden benefit of this long-drawn-out process was the creation, on the diversity list, of a corpus of varied individuals who had discussed these issues and hammered out a shared approach to the problems that included a refusal to punish individuals for things done out of ignorance. It also meant that when one speaker used a slightly ill-advised graphic in a presentation the issue was dealt with then and there in a very direct manner without any recriminations, and I didn't even get to hear about it until much later that day. The speaker was advised that the material was inappropriate and that therefore the slides and the video of the talk wouldn't be published, and hopefully left without feeling that they weren't welcome at next year's conference.

I hope that the JavaScript community manages to develop its own understanding of diversity issues and its own process for dealing with them. I know it took up a lot of my time as PSF chairman and gave me some uncomfortable moments (and does not exempt me from the results of my own stupidity in the future), but I am glad it led to a tolerant community process that nevertheless has made it clear that discrimination is not acceptable.

The Hiring Smiley Curve

2010-08-28T22:20:21+00:00

I recently commented on how hi-tech companies seem to advertise for "rock star" developers. This appears to me to be a little shortsighted. Maybe Google have enough money to do it, but the average company looking for a singer wouldn't be able to go out and hire Mick Jagger or Van Morrison (yes, I know, I am showing my age - it will become obvious this isn't an accident). So why do they think they can afford the software equivalent?

As I mature (that sounds so much less pejorative than "grow older", doesn't it?) I have realized that my decision to eschew the corporate world and plough my own furrow wasn't entirely disastrous. I have been self-employed, or an employee of a company which I owned, for over twenty years now. The last large company I worked for was Sun Microsystems, and I left in 1988 after realizing that I was a misfit for the corporate environment. Since then I can honestly say I have never worked for a more charming or delightful person. Nowadays my boss never hesitates to take my interests into account.

Just occasionally, I have toyed with the idea of third-party employment, most recently with Google (twice). The first time I explained before going for interview that I was not prepared to relocate to the West coast . At the time I was moving back from the UK to the USA, and had just bought a house in Virginia, so I was a little surprised to learn three weeks after my interview that "we aren't prepared to make a remote hire, but would consider you for Mountain View". Large company mentality: ignore what the potential recruit wants, and try to hire them anyway. No sale. About a month ago they called me to ask if I would consider employment in the DC area, but it turned out they were only hiring for Java projects. I used to write Java but I'm all right now, so the recruiter and I agreed to part company amicably after ten minutes on the 'phone.

None of this amounts to a hill of beans but I was encouraged about a year ago, when I was talking to someone in New York City about experience levels and consulting opportunities, to learn about the "smiley curve" theory of work rewards. In brief, this theory says that you hire young raw recruits because they are full of energy and don't cost much. As people become more experienced they expect more money but their productivity doesn't go up commensurately, so they are less desirable but you need them because there aren't enough young turks to do all the work. Then, as you mature, your desirability goes up again because (direct quote, as far as I can remember) "you have seen everything and you know everything".

If true, this is quite encouraging. There must be lots of companies looking for someone as experienced as me. The only question now is whether they can afford me!

Links for 2010-08-27 [del.icio.us]

2010-08-28T07:00:00+00:00

Musings about django.contrib.auth.models.User

2010-08-27T20:15:00+00:00

Dawned on me that the Django auth user model that ships with Django is like the string built-in of a high level programming language. With the string built-in it's oh so tempting to add custom functionality to it like a fancy captialization method or some other function that automatically strips whitespace or what not. Yes, I'm looking at you Prototype for example.

By NOT doing that, and leaving it as it is, you automatically manage to Keep It Simple Stupid and your application code makes sense to the next developer who joins your project.

I'm not a smart programmer but I'm a smart developer in that I'm good at keeping things pure and simple. It means I can't show off any fancy generators, monads or metaclasses but it does mean that fellow coders who follow my steps can more quickly hit the ground running.

My colleagues and I now have more than ten Django projects that rely on, without overriding, the django.contrib.auth.models.User class and there has been many times where I've been tempted to use it as a base class or something instead but in retrospect I'm wholeheartedly happy I didn't. The benefit isn't technical; it's a matter of teamwork and holistic productivity.

Apple Going Over the Top?

2010-08-27T18:24:25+00:00

Yet again I rejoice that I am not an iPhone user. Indeed, given this latest news I might well just chuck my Mac mini away in protest (no, you probably don't want it - it's an aging obsolete PPC mini from about six years ago).

In news from the Electronic Frontier Foundation I learned today that Apple has applied for patents on method of spying intrusively on the users of their devices. Here's a partial list of possible applications:

The system can take a picture of the user's face, "without a flash, any noise, or any indication that a picture is being taken to prevent the current user from knowing he is being photographed";
The system can record the user's voice, whether or not a phone call is even being made;
The system can determine the user's unique individual heartbeat "signature";
To determine if the device has been hacked, the device can watch for "a sudden increase in memory usage of the electronic device";
The user's "Internet activity can be monitored or any communication packets that are served to the electronic device can be recorded"; and
The device can take a photograph of the surrounding location to determine where it is being used.

So enjoy your iPhones. I won;t be doing business with a company that thinks this way. Sigh. I suppose that means the new laptop will have to be PC based.

Links for 2010-08-26 [del.icio.us]

2010-08-27T07:00:00+00:00

Links for 2010-08-25 [del.icio.us]

2010-08-26T07:00:00+00:00

Git - SVN Crash Course
Git for SVN users.

links for 2010-08-25

2010-08-25T21:01:42+00:00

Design Patterns Library | Fellowship Technologies

interesting collection of design patterns for Web UIs.

(tags: web designpatterns ux)
Hosting Maven Repos on Github | cemerick

So basically, it's a level of redirection on just checking the dependencies in in the first place.

(tags: maven dependencymanagement git github)

Links for 2010-08-24 [del.icio.us]

2010-08-25T07:00:00+00:00

Cloud Services
If Grig Gheorghiu says this guy knows what he's talking about, he probably does

Links for 2010-08-24 [del.icio.us]

2010-08-25T07:00:00+00:00

Lifehacker Pack for Android: Our List of the Best Android Apps
BuildNotify
CCMenu equivalent for Ubuntu.
AlexZeitler's gitcheatsheet
TechCheatSheets.com
What it says on the tin.

links for 2010-08-23

2010-08-23T21:03:11+00:00

hugoduncan's clj-ssh at master – GitHub

A clojure wrapper over jsch

(tags: ssh clojure sysadmin)
BLDGBLOG: Pallet House

Several kinds of awesome

(tags: architecture reuse recycling building temporary)

Running CherryPy on Android with SL4A

2010-08-23T20:05:12+00:00

CherryPy runs on Android thanks to the SL4A project. So if you feel like running Python and your own web server on your Android device, well you can just do so. You’ve probably not heard something that awesome since the pizza delivery guy rung the door.

How to get on about it? Well that’s the surprise, CherryPy in itself doesn’t need to be patched. Granted I haven’t tried all the various tools provided by CherryPy but the server and the dispatching works just fine.

First, you need get the CherryPy source code, build and copy the resulting cherrypy package into the SL4A scripts directory.

Once you’ve plugged your phone to your machine through USB, run the next commands:

$ svn co http://svn.cherrypy.org/trunk cp3-trunk
$ cd cp3-trunk
$ python setup.py build
$ cp -r build/lib.linux-i686-2.6/cherrypy/ /media/usb0/sl4a/scripts/

Just change the path to match your environment. That’s it.

Now you can copy your own script, let’s assume you use something like below:

# -*- coding: utf-8 -*-
import logging
# The multiprocessing package isn't
# part of the ASE installation so
# we must disable multiprocessing logging
logging.logMultiprocessing = 0
 
import android
import cherrypy
 
class Root(object):
    def __init__(self):
        self.droid = android.Android()
 
    @cherrypy.expose
    def index(self):
        self.droid.vibrate()
        return "Hello from my phone"
 
    @cherrypy.expose
    def location(self):
        location = self.droid.getLastKnownLocation().result
        location = location.get('network', location.get('gps'))
        return "LAT: %s, LON: %s" % (location['latitude'],
                                     location['longitude'])
 
def run():
    cherrypy.config.update({'server.socket_host': '0.0.0.0'})
    cherrypy.quickstart(Root(), '/')
 
if __name__ == '__main__':
    run()

As you can see we must disable the multiprocessing logging since the multiprocessing package isn’t included with SL4A.

Save that script on your computer as cpdroid.py for example. Copy that file into the scripts directory of SL4A.

$ cp cpdroid.py /media/usb0/sl4a/scripts/

Unplug your phone and go to the SL4A application. Click on the cpdroid.py script, it should start fine. Then from your browser, go to http://phone_IP:8080/ and tada! You can also go to the /location path to get the geoloc of your phone.

Windows Vista Mystery Shares

2010-08-22T22:41:41+00:00

For reasons best known to Microsoft, when I try to delete a folder which has been shared (through the Explorer interface) it takes forever to complete. This would not be so bad if there were just one or two shares, but sadly (for reasons best known to Microsoft) a large number of folders randomly appear to have become shares (see the screen dump of a portion of my home directory at the right). I have no idea how these folders became shared. It certainly wasn't any intentional act of mine, and heaven alone knows what this does to performance.

Now, you are probably wondering why I don't just switch off sharing before I delete the directory. The answer to that is that although Windows is displaying the folders as shared, it doesn't really seem to believe that they are shared. So there doesn't appear to be an easy way to switch this sharing off.

If I had some idea how it had been switched on in the first place that might help, but as with so many other aspects of Windows performance this remains a mystery. If someone cold offer some insight I'd be happy to find out what's going on here.

Weekly python tips (WeeklyPyTips) mailing list

2010-08-21T23:20:47+00:00

I’ve just written the first few weeks worth of Python Tips for my new list: weeklypytips AT aweber.com or use the form below.

The list is aimed at beginner and intermediate Pythonistas, if you join then each week you’ll receive a Python tip (probably 3-6 paragraphs with source code). I originally announced it in the Python Tutor list.

The first batch of tips cover:

Downloading files from the Internet in just 2 lines
Scraping web sites
IPython introduction
Iteration
Controlling hardware (including serial ports, cameras and robots)

I have a long list of topics to cover, if the feedback is positive then I’ll keep adding the tips. To join the list just mail the email address above (the email can be empty), you’ll be asked to confirm your subscription, after that you’ll get one tip per week.

To make it really easy just use the form below:

Ian applies Artificial Intelligence for companies (Mor Consulting), programs Python, produces professional screencasts (ProCasts), writes The Screencasting Handbook and is also a sea-side dweller and consumer of fine coffees.

links for 2010-08-20

2010-08-20T21:02:02+00:00

Command-Line REDbot – RED news and information

A command-line interface for the cache header checking redbot. Very cool.

(tags: redbot http cache python)
RED

The Resource Expert Droid is an online app which validates cache headers for HTTP ans gives advice on them. Mundo coolio.

(tags: http cache validation onlineapps)

links for 2010-08-19

2010-08-19T21:02:44+00:00

Structure and Interpretation of Computer Programs, Video Lectures

Video lectures from the SICP authors

(tags: video sicp lisp mit)

links for 2010-08-18

2010-08-18T21:01:32+00:00

African Safari Photography & Safari Photography Tips

(tags: camera photography safari tips)
Digital Safari Equipment Tips

(tags: safari digital photography camera tips)
Visualising the Wikileak's war logs using Clojure

In a similar vein to my recent Clojure & Incanter work, albeit with slightly more important data

(tags: incanter clojure war afghanistan guardian)

links for 2010-08-17

2010-08-17T21:02:40+00:00

An implausibly illustrated introduction to HTML5 Web Workers

(tags: browser javascript multithreaded)
What Exactly Is a Doctorate?

Fantastic pictorial representation of a PHD

(tags: phd visualization visualrepresentations)

Schmidt Foot-In-Mouth Attack Continues

2010-08-16T22:52:36+00:00

Wow, two consecutive Eric Schmidt posts. But this really is quite newsworthy. An interview with Wall Street Journal published on Saturday claims

He predicts, apparently seriously, that every young person one day will be entitled automatically to change his or her name on reaching adulthood in order to disown youthful hijinks stored on their friends' social media sites.

This really is the most patent hogwash. Surely someone like Schmidt, with a brain the size of a planet, could foresee that if such name changing were to become commonplace it would inevitably lead to the creation of services that mapped between past and present-day identities? Given the ability to identify images recently demonstrated by sites like tineye.com it's only a matter of time before changing your name will no longer be a way to erase the records of your misdeeds.

This makes it all the more important that privacy and basic information security become high school subjects, but alas the corporate overlords that have the ear of government in most developed (and many less-developed) countries will be attempting to make sure that isn't a priority, because it isn't in their interests.

links for 2010-08-16

2010-08-16T21:02:52+00:00

Clj-http. A new http client for clojure.

(tags: http clojure)
Saving Incanter charts as PDF documents | Data Sorcery with Clojure

(tags: incanter pdf clojure)

Graphing Unique Users With Incanter

2010-08-16T19:02:13+00:00

In a previous post, I showed how we could use Clojure and specifically Incanter to process access logs to graph hits on our site. Now, we’re going to adapt our solution to allow us to to show the number of unique users over time.

We’re going to change the previous solution to pull out the core dataset representing the raw data we’re interested in from the access log – records-from-access-log remains unchanged from before:

(defn access-log-to-dataset
  [filename]
  (col-names (to-dataset (records-from-access-log filename)) ["Date" "User"]))

The raw dataset retrieved from this call looks like this:

Date	User
11/Aug/2010:00:00:30 +0100	Bob
11/Aug/2010:00:00:31 +0100	Frank
11/Aug/2010:00:00:34 +0100	Frank

Now, we need to work out the number of unique users in a given time period. Like before, we’re going to use $rollup to group multiple records by minute, but we need to work out how to summarise the user column. To do this, we create a custom summarise function which calculates the number of unique users:

(defn num-unique-items
  [seq]
  (count (set seq)))

Then use that to modify the raw dataset and graph the resulting dataset:

(defn access-log-to-unique-user-dataset
  [access-log-dataset]
    ($rollup num-unique-items "User" "Date"
      (col-names (conj-cols ($map #(round-ms-down-to-nearest-min (as-millis %)) "Date" access-log-dataset) ($ "User" access-log-dataset)) ["Date" "Unique Users"])))

(defn concurrent-users-graph
  [dataset]
  (time-series-plot :Date :User
                             :x-label "Date"
                             :y-label "User"
                             :title "Users Per Min"
                             :data (access-log-to-unique-user-dataset dataset)))

(def access-log-dataset
  (access-log-to-dataset "/path/to/access.log"))

(save (concurrent-users-graph access-log-dataset) "unique-users.png")

You can see the full source code listing here.

I’m a Windows Person. How can I contribute to Python? [Part I]

2010-08-16T13:24:00+00:00

If I might essay a generalisation: Windows-based users of Python are less likely to have become involved in the development of Python itself. This might be for the same reasons that other people don’t get involved: a lack of time, of inclination, or of confidence in their ability to help. But it might be because Windows users on the whole aren’t so accustomed to delving into the source code of the things they use, nor are they used to checking out from source control, testing fixes and rebuilding. It’s not so much part of the culture.

I’m keen to see more Windows users & developers active in core CPython development. Python has been supported on Windows for a long time by the core development team but it can sometimes feel a little bit like a poor cousin. The majority of Python users (and of key Python developers) are Unix-based: traditionally Linux & BSD, these days often OS X. Even if they’re not unsympathetic to the needs of Windows users, they may not be be in a position to help. But Python is a volunteer effort and if Windows-savvy developers aren’t available, Windows issues will remain unfixed and untested and better support for Windows-specific issues will not be forthcoming.

There are practicalities involved: Subversion or Mercurial, Visual Studio, being able to build Python from scratch and so on, but let’s start from the other end: you’re well-intentioned and motivated, but what should you do? I’m focusing on Windows-specific issues, but some of what I say here applies to anyone who’s interested in helping Python’s core development.

How should I start?

You don’t have to be an über-hacker or even a developer at all to contribute to Python. (Although you can do more if you are). You can start by submitting bug reports or requests for improvements to the issue tracker:

Good: Submit a bug report: send an email to bugs@python.org detailing what version of Python you’re using and on which version of Windows, what you did and what went wrong. Or what you think should happen. If you’re able to re-test with a newer version of Python that will help, in case the bug’s already been identified and fixed. If it’s a documentation issue you’re describing, be clear what alternative or additional text you’re proposing.
Better: Submit a bug report or feature request and a failing test case: send an email as above, but attach the simplest possible self-contained script which will cause the failure you’re describing or which will illustrate what should happen.
Best: Submit a bug report with a failing test and add a patch which you believe will solve the issue plus a patch to the relevant Python test module plus a doc patch if applicable. If this is a documentation issue, the ideal is a patch against the reStructuredText docs; at the least, a clear plain text version of what you’re proposing.

Things to bear in mind:

Don’t be disappointed if no-one takes any action. Ideally – and often – someone quickly notices the new issue, does a little categorisation on it and assigns it to a developer who’s known to be the owner of the area where the problem occurred. At the least, one or more developers may be added to the nosy list, and one of them may take ownership. Even if that doesn’t happen, you’ve definitely done the right thing by reporting the issue and offering a patch. If it’s Windows-related, feel free to add me to the nosy list. But even if a developer is assigned, developers are volunteers with their own availability, pressures and interests.
Don’t be surprised if the issue isn’t quite as straightforward as it appears at first blush, particularly if backwards compatibility is at stake. There may be some history of which you’re unaware, or some ramifications in other areas of the codebase which aren’t obvious.
Don’t give up in despair if the issue is rejected. This isn’t a hard-hearted manoeuvre by the core developers to reject all outside help. It just means that things aren’t always as simple as they might appear and some bigger-picture issue might have put the kibosh on your bright idea. If you feel that a valid point has genuinely been misunderstood you’re free to ask that the issue be reopened; but please don’t do this frivolously.
Not all versions of Python are under active development. At the time of writing, the 2.x series is at the end of its life with 2.7 the final release. Clear bugs may be fixed in that branch, but little else. It’s unlikely that anything categorised as a new feature will make it in there. The same goes for 3.1. The latest development branch is version 3.2 where new features will be targetted. Any earlier releases will either see no changes whatsoever or only critical security fixes.

I’m a Windows developer, but where are the problems?

Being a core Python developer on Windows can mean several things: doing any kind of Python core development when Windows is your primary platform; working on Windows-specific modules, such as msvcrt or winreg, or the Windows-specific aspects of other modules, such as os and subprocess; or assessing and maintaining platform-specific subtleties such as compiler-sensitive datatype sizes or line-ending issues.

A lot depends on your own inclinations, motivations and areas of expertise. If you’re an experienced Visual Studio user you might focus more on some of the compiler-specific issues or build file configurations. If you have experience of sysadmin-type work you might want to look at the os module which wraps some of the Win32 API in posix-like calls or the winreg module which exposes the registry functions. If you’re a general purpose C coder who just uses Windows as a platform then you could look at the core Python code or the parts of the stdlib written in C. If you’re an MSI guru, look at some of the outstanding issues in Python’s own MSI and the bdist_msi distutils subsystem.

Ultimately involvement probably comes from one of three things:

Scratching your own itch: is there something in the docs which you’ve never really understood because it relies on knowledge of Unix-specifics? Or just because it’s confusing? Have you been tripped up by a stdlib function which assumes that Windows paths always act like Unix paths? Have you tried to use os.access only to discover that it’s essentially useless on Windows? Do you think you could reimplement the signal module using the Win32 API? Or sponsor the merge of a 3rd-party Windows port of the curses module?
Scratching someone else’s itch: have a look at bugs.python.org and choose an issue which is unresolved. The issue tracker has recently had new quick-searches added to it down the left and you can easily build your own searches. This search, for example, looks for unclosed issues which include Windows in the component list. It can be a useful exercise if you simply confirm that an unclear issue still obtains on current versions of Python and, possibly, on recent versions of Windows. Or if you produce a failing test case for a merely textual or vague report. (Something which can take quite some time). But be wary of producing “me-too” noise as people following that particular issue will receive notification emails which don’t add usefully to the known state of affairs.
Finding an itch to scratch: if you’re keen to contribute but haven’t experienced any problems which need fixing, and haven’t found any existing bugs which take your fancy, then why not just browse the source code seeing how things are done and looking for ways to improve. Be careful here since cosmetic, trivial or frivolous changes are generally frowned upon. But, for example, the subprocess and multiprocessing modules both involve some reasonably serious Windows-specific plumbing. The module maintainers aren’t necessarily Windows experts and even if they are, the more eyes the better. There are other modules which have Windows-specific elements if only in small ways.

Where should I be helping?

The issue tracker: bugs.python.org hosts the Python issue tracker, an instance of roundup which keeps track of all bug reports and feature requests. It’s broadly followed by core developers, but a small team keeps track of incoming bugs and tries to categorise them appropriately and to assign them to a known developer if possible, or to add to the issue’s nosy list developers known to be interested in that area of the system. By helping out in this way, you get a fairly good idea of the issues being reported and the work being done on them. And on which core developers have expertise in which areas of the system. It’s also relatively easy to get higher privileges on the tracker than he default logged-in user has so that you can help classify and manage open issues.
Documentation: The documentation is maintained as reStructuredText files within Python’s version control repository and is published using Sphinx. It’s very easy to change and patch and there’s a batch file to rebuild it on Windows although you may need to tweak it a little. Doc patches are welcome, whether for small fry such as typos or slight clarifications, or for weightier rewrite.
stdlib Python: The Python-only parts of the standard library. A lot of the stdlib is written in Python itself; that’s what makes the whole package so portable. Even if you’re not a C developer, you can still debug and test issues in the stdlib itself. It’s especially possible to do this without a C compiler or even a Subversion checkout: you only need to put an altered version of the module in a test directory from which you run the Python interpreter and the interpreter will see the local version first.
stdlib Python & C: Some of the standard library relies partly or wholly on extension modules, written in C against Python’s C-API. The C is reasonably readable and is often concerned only with moving Python objects to and from some system library, which makes it a fairly restricted subset of C. To do anything useful with this, you will need a C compiler and, ideally, a version-controlled checkout.
Core Python: The core of CPython is written in C: the objects, the interpreter, the underlying data structures, key built-in modules. You’ll need to be a fairly good C programmer to work here, not least because the impact is high if a mistake is made.
Infrastructure: CPython needs a certain amount of scaffolding to build, especially on Windows where the standard autoconf/make tools don’t apply. This scaffolding tends to be under-supported, not least because it’s not very public-facing and doesn’t solve most people’s problems. But it still needs to be done. This includes Visual Studio solution and proj files, msi build mechanisms, buildbot scripts and documentation make files.

This is Part I; Part II should include information about checkout out the source, building it, creating & applying patches and working within the development community.

links for 2010-08-15

2010-08-15T21:00:57+00:00

Creating Executable Jars For Your Clojure Application

(tags: clojure jar executable)

Graphing Hits Per Minute Using Clojure & Incanter

2010-08-15T18:10:59+00:00

Continuing in a re-occurring series of posts showing my limited understanding of Clojure, today we’re using Clojure for log processing. This example is culled from some work I’m doing right now in the day job – we needed to extract usage information to better understand how the system is performing.

The Problem

We have an Apache-style access log showing hits our site. We want to process this information to extract information like peak hits per minute, and perhaps eventually more detailed information like the nature of the request, response time etc.

The log looks like this:

43.124.137.100 - username 05/Aug/2010:17:27:24 +0100 "GET /some/url HTTP/1.1" 200 24 "http://some.refering.domain/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.0.9) Gecko/2009040821 Firefox/3.0.9 (.NET CLR 3.5.30729)"

Extracting The Information

We want to use Incanter to help us process the data & graph it. Incanter likes its data as a sequence of sequences – so that’s what we’ll create.

First up – processing a single line. I TDD’d this solution, but have excluded the tests from the source listing for brevity.

user=> (use 'clojure.contrib.str-utils)
nil
user=> (use '[clojure.contrib.duck-streams :only (read-lines)])
nil

user=> (defn extract-records-from-line
  [line-from-access-log]
  (let [[_ ip username date] (re-find #"^(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}) - (\w+) (.+? .+?) " line-from-access-log)]
    [date username]))
#'user/extract-records-from-line

user=> (defn as-dataseries
  [access-log-lines]
  (map extract-records-from-line access-log-lines))
#'user/as-dataseries

user=> (defn records-from-access-log
  [filename]
  (as-dataseries (read-lines filename)))
#'user/records-from-access-log

A few things to note. extract-records-from-line is matching more than strictly needed – I just wanted to indicate the use of destructing for matching parts of the log line. I’m pulling in the username & date – the username is not strictly needed for what follows. Secondly, note the use of read-lines from clojure.contrib.duck-streams – rather than slurp, read-lines is lazy. We’ll have to process the whole file at some point, but it’s a good idea to look to use lazy functions where possible.

At this point, running records-from-access-log gives us our sequence of sequences – next up, pulling it into Incanter.

Getting The Data Into Incanter

We can check that our code is working properly by firing up Incanter. Given a sample log:

56.24.137.230 - fred 05/Aug/2010:17:27:24 +0100 "GET /some/url HTTP/1.1" 200 24 "http://some.refering.domain/" "SomeUserAgent"
12.14.137.140 - norman 05/Aug/2010:17:27:24 +0100 "GET /some/url HTTP/1.1" 200 24 "http://some.refering.domain/" "SomeUserAgent"
42.1.137.110 - bob 05/Aug/2010:17:28:24 +0100 "GET /some/url HTTP/1.1" 200 24 "http://some.refering.domain/" "SomeUserAgent"
143.124.1.50 - clare 05/Aug/2010:17:29:24 +0100 "GET /some/url HTTP/1.1" 200 24 "http://some.refering.domain/" "SomeUserAgent"

Let’s create a dataset from it, and view the resulting records:

user=> (use 'incanter.core)
nil
user=> (def access-log-to-dataset
(to-dataset (records-from-access-log "/path/to/example-access.log")))
#'user/access-log-dataset
user=> (view access-log-dataset)

The result of the view command:

Unfortunately, no column names – but that is easy to fix using col-names:

user=> (def access-log-dataset
(col-names (to-dataset (records-from-access-log "/path/to/example-access.log")) ["Date" "User"]))
#'user/access-log-dataset
user=> (view access-log-dataset)

At this point you can see that it would be easy for us to pull in the URL, response code or other data rather than the username from the log – all we’d need to do is change extract-records-from-line and update the column names.

Graphing The Data

To graph the data, we need to get Incanter to register the date column as what it is – time. Currently it is in string format, so we need to fix that. Culling the basics from Jake McCray’s post, here is what I ended up with (note use of Joda-Time for date handling – you could use the standard SimpleDateFormat if you preferred):

user=> (import 'org.joda.time.format.DateTimeFormat)
nil

user=> (defn as-millis
  [date-as-str]
  (.getMillis (.parseDateTime (DateTimeFormat/forPattern "dd/MMM/yyyy:HH:mm:ss Z") date-as-str)))
#'user/as-millis

user=> (defn access-log-to-dataset
  [filename]
  (let [unmod-data (col-names (to-dataset (records-from-access-log filename)) ["Date" "User"])]
    (col-names (conj-cols ($map as-millis "Date" unmod-dataset) ($ "User" unmod-dataset)) ["Date Time In Ms" "User"])))
#'user/access-log-to-dataset

While the date parsing should be pretty straightforward to understand, there are a few interesting things going on with the Incanter code that we should dwell on briefly.

The $ function extracts a named column, whereas the $map function runs another function over the named column from the dataset, returning the modified column (pretty familiar if you’ve used map). conj-cols then takes these resulting sequences to create our final dataset.

We’re not quite done yet though. We have our time-series records – representing one hit on our webserver – but don’t actually have values to graph. We also need to work out how we group hits to the nearest minute. What we’re going to do is replace our as-millis function to one that rounds to the nearest minute. Then, we’re going to use Incater to group those rows together – summing the hits it finds per minute. But before that, we need to tell Incanter that each row represents a hit, by adding a ‘Hits’ column. We’re also going to ditch the user column, as it isn’t going to help us here:

user=> (defn access-log-to-dataset
  [filename]
  (let [unmod-dataset (col-names (to-dataset (records-from-access-log filename)) ["Date" "User"])]
    (col-names (conj-cols ($map as-millis "Date" unmod-dataset) (repeat 1)) ["Date" "Hits"])))
#'user/access-log-to-dataset

Next, we need to create a new function to round our date & time to the nearest minute.

Update: The earlier version of this post screwed up, and the presented round-ms-down-to-nearest-min actually rounded to the nearest second. This is a corrected version:

(defn round-ms-down-to-nearest-min
  [millis]
  (* 60000 (quot millis 60000)))

If you wanted hits per second, here is the function:

(defn round-ms-down-to-nearest-sec
  [millis]
  (* 1000 (quot millis 1000)))

And one more tweak to access-log-to-dataset to use the new function:

(defn access-log-to-dataset
  [filename]
  (let [unmod-dataset (col-names (to-dataset (records-from-access-log filename)) ["Date" "User"])]
    (col-names (conj-cols ($map #(round-ms-down-to-nearest-min (as-millis %)) "Date" unmod-dataset) (repeat 1)) ["Date" "Hits"])))

Finally, we need to roll our data up, summing the hits per minute – all this done using $rollup:

(defn access-log-to-dataset
  [filename]
  (let [unmod-dataset (col-names (to-dataset (records-from-access-log filename)) ["Date" "User"])]
    ($rollup :sum "Hits" "Date"
      (col-names (conj-cols ($map #(round-ms-down-to-nearest-min (as-millis %)) "Date" unmod-dataset) (repeat 1)) ["Date" "Hits"]))))

$rollup applies a summary function to a given column (in our case “Hits”), using another function to determine the parameters for that function (“Date” in our case). :sum here is a built-in Incanter function, but we could provide our own.

And the resulting dataset:

Now we have our dataset, let’s graph it:

user=> (defn hit-graph
  [dataset]
  (time-series-plot :Date :Hits
                             :x-label "Date"
                             :y-label "Hits"
                             :title "Hits"
                             :data dataset))

user=> (view (hit-graph (access-log-to-dataset "/path/to/example-access.log")))

This is deeply unexciting – what about if we try a bigger dataset? Then we get things like this:

Conclusion

You can grab the final code here.

Incanter is much more than simply a way of graphing data. This (somewhat) brief example shows you how to get log data into an Incanter-frendly format – what you want to do with it then is up to you. I may well explore other aspects of Incanter in further posts.

links for 2010-08-13

2010-08-13T21:01:23+00:00

Reading Files in Clojure @ Irrational Exuberance

How to read a file without blowing your stack

(tags: file io clojure tips)
HTML5 Boilerplate – A rock-solid default for HTML5 awesome.

(tags: css html javascript)

Eric Schmidt Looks Forward to Big Brother

2010-08-12T16:33:44+00:00

"You only need to give up
just a little bit of freedom"

Well, Google's senior management are really keeping us guessing this month. I am not really sure any longer whether the company actually has any coherent point of view on individual privacy. Nowadays it seems you can't even expect consistency. In a startling repetition of his prior assertions that not only is Internet anonymity dead but that he wants to dance on its grave. According to a Read Write Web report Eric Schmidt, CEO of Google, included the following comments in his remarks to the Techonomy conference:

"The only way to manage this is true transparency and no anonymity. In a world of asynchronous threats, it is too dangerous for there not to be some way to identify you. We need a [verified] name service for people. Governments will demand it."

Any government that requires individuals to give up their rights to anonymity is a government past its sell-by date. To my mind this reveals further insight into Schmidt's "what's good for business is good for the people" view exemplified by the much-discussed recent joint statement with Verizon on network neutrality. It's fairly obvious that Schmidt sees government's role as paving the way for corporations to increase their profits, not preserving the freedoms of the citizenry who elected it. This is so far from "government of the people, for the people and by the people" that it's apparently time "do no evil" was replaced by "make more money".

"But that would hurt Google's
stock price!"

Yet it was only in May (yes, three months ago) that Schmidt was publicly suggesting that as far as Google was concerned "privacy is paramount". Does he really know what the priorities are any longer? It seems like we can dismiss any further utterances as the self-serving flip-floppery of the 129th richest man in the world. What does he really think? Apparently it depends on which way the financial wind is blowing. The really depressing thing is it's transparently obvious that here we have a man who will do well in politics. "The Best Democracy Money Can Buy" indeed.

links for 2010-08-11

2010-08-11T21:02:31+00:00

Linux.com :: Using screen for remote interaction

(tags: screen unix productivity pairing pairprograming linux)
Smart HTTP Transport

An efficient read/write HTTP transfer protocol for Git

(tags: git http transport sysadmin devops)

Creating Sparse Tabular Data With Clojure

2010-08-10T21:11:49+00:00

In the spirit of making my mistakes in public – something which I have a long history of on this blog – I thought I’d post up a solution I came up with for a relatively simple problem. I’m not unhappy with the solution – it works – but I can’t help thinking I’m missing something and there is a more elegant solution out there.

The Problem

For our input we have a set of time series records. Each record contains a list of name value pairs. For each record, the keys are not fixed – they may vary. Let’s imagine that we’re recording viewing figures for the major UK soap operas – some soaps are on every day, some are only on a few days a week.

In Clojure, we’re got our data in the following form:

(("Monday" {:eastenders 6.5, :thearchers 2.3, :corrinationstreet 5.6})
 ("Tuesday" {:eastenders 6.8, :thearchers 1.4})
 ...)

We want to convert this into a single table of data, with the keys from the source data representing the columns, with each row representing a different timestamp, so that we can visualise the data with something like gnuplot, Incanter or just plain old excel. So we want to get to something like this (yes, I know The Archers is on at the weekend too, but this is just an example):

Day	Eastenders	The Archers	Coronation Street
Monday	6.5	2.3	5.6
Tuesday	6.8	1.4
Wednesday		2.3	7
Thursday	6.7	2.8
Friday	9.8	2.1	7

The challenge here (such as it is) is that we want a sparse table, and that our code cannot know beforehand the total universe of soap names (what if a new soap launched?).

The Header Row

The first part of this problem as I saw it was to determine which soaps our records represented to create a header row. The solution I came up with was to stick the keys for all records into a set:

user=> (def soaps
'(("Monday" {"eastenders" 6.5, "thearchers" 2.3, "corrinationstreet" 5.6})
("Tuesday" {"eastenders" 6.8, "thearchers" 1.4})))

#'user/records
user=> (defn as-columns [records]
(apply (partial conj #{}) (mapcat keys (map second records)))))
#'user/columns

user=> (as-columns soaps)
#{"corrinationstreet" "eastenders" "thearchers"}

Assuming we want a CSV file to store our content, a function to create the header row becomes:

user=> (str "Date," (apply str (interpose "," (as-columns soaps))))
"Date,corrinationstreet,eastenders,thearchers"

The Data

Now we have a list of all possible keys (in this example, the names of the soaps), we can use this to extract data from the records for each row. Getting data from a map is straightforward – even handling the not-there case is simple enough:

user=> (def some-map
{"eastenders" 6.5, "thearchers" 2.3, "corrinationstreet" 5.6})
#'user/some-map

user=> (get some-map "eastenders" "-")
6.5

user=> (get some-map "hollyoaks" "-")
-

Given a list of known columns, we can use a list comprehension to extract the data in a consistent order:

user=> (for [col #{"eastenders" "thearchers" "corrinationstreet"}]
(get {"eastenders" 6.8, "thearchers" 1.4} col "-"))
(["-"] [6.8] [1.4])

Pulling It All Together

Taking those various strands, we end up with the following solution:

(defn as-columns
  [records]
  (apply (partial conj #{}) (mapcat keys (map second records))))

(defn header-row
  [records]
  (str "Date," (apply str (interpose "," (as-columns records)))))

(defn values-for-record
  [columns values]
  (for [col columns] (get values col "-")))

(defn as-row
  [record columns]
  (let [day (first record)
        values (second record)]
   (str day ","
    (apply str (interpose "," (values-for-record columns values))))))

(defn as-data
  [records]
  (apply str
    (interpose "\n"
      (for [record records]
          (as-row record (as-columns records))))))

(defn as-table
  [records]
    (apply str (header-row records) "\n" (as-data records)))

Things I like with the solution: it works.

Things I don’t like with the solution:

Duplicated call to as-columns, in both as-data and header-row
Still don’t think I’ve got the indentation right
Still worried I’m creating functions which are too large, or that are not readable
This solution was arrived at by hacking on the code in a REPL – not TDD. My Clojure skills are still lacking, so I have to embark on the occasional hack-a-thon to learn some things – this being one such exercise. I plan to re-implement this using TDD with what I’ve learnt to see what I end up with. It will be interesting to see if TDDing this allays my fears about function size.
Lots of (apply str (interpose… duplication going around – I should factor that out
Not sure if the list comprehension here is needed – have I missed something obvious?

links for 2010-08-10

2010-08-10T21:02:43+00:00

Introducing The NEW Sharpie LIQUID PENCIL

Very cool – writes like a pencil, erasable, but becomes permanent after three days. It's like we're living in the future!

(tags: pen pencil stationary sharpie)

Paypal Pending Balances

2010-08-08T23:12:44+00:00

According to the PayPal user agreement:

10.7 Reserves. If you receive Purchase Payments, PayPal, in its sole discretion, may place a Reserve on funds held in your Premier or Business Account when PayPal believes there may be a high level of risk associated with your Account. If PayPal places a Reserve on funds in your Account, they will be shown as “pending” in your PayPal Balance. If your Account is subject to a Reserve, PayPal will provide you with notice specifying the terms of the Reserve. The terms may require that a certain percentage of the amounts received into your Account are held for a certain period of time, or that a certain amount of money is held in reserve, or anything else that PayPal determines is necessary to protect against the risk associated with your Account. PayPal may change the terms of the Reserve at any time by providing you with notice of the new terms.

What this doesn't say is that PayPal will also deny you the ability to earn interest on those funds while they are held in reserve. Because Holden Web is organizing DjangoCon we currently have a substantial balance (which will soon disappear once the outgoing starts). So PayPal have decided that they can place $20,000 on reserve, thereby refusing me the right to earn interest on that money.

Naturally the "notice" I received just said that the funds were being held on reserve until further notice, and when I inquired about this the "explanation" I received didn't explain anything at all: it simply pointed me to the user agreement and suggested that I could find out what information they had used in the decision-making process by taking out a subpoena. Definitely not cool. In terms of customer service PayPal are scoring about 1 out of 10 here.

links for 2010-08-06

2010-08-06T21:02:21+00:00

iPhone Windowed HTTP Live Streaming Using Amazon S3 and Cloudfront Proof of Concept

(tags: cloudfront aws s3 iphone ffmpeg video)

Tests That Test Your Tests

2010-08-06T01:46:37+00:00

I just had an interesting experience with Steve Miller, the technical editor of the Python classes I am writing for O'Reilly School of Technology. We are just getting to the end of the second of four courses, and I like to think that we are moving right along: in the final lesson students write a simple GUI-based program that searches for and displays e-mail messages stored in a MySQL database.

I introduced test-driven development in the second course. Not only does this encourage good habits in the students, it also makes it somewhat easier to test some of their exercises (though I still do not have a good approach to testing Tkinter-based GUI applications). In time-honored fashion we start with tests and a program full of stubs, and then expand the stubs to pass the tests. For the email database the initial API is very simple: there is one function to store messages and two others to retrieve them, by primary key and Message-Id.

In the final chapter I start out with a very simple database table that stores the body of the message as a LONGTEXT column. The only other columns in the table (to start with—it gets more complex later) are the automatically-generated primary key and the Message-Id Header. The tests use a setUp() method that completely re-creates the database table and populates it from messages stored in a bunch of files:

 FILESPEC = "V:/Python2/Lesson12/MailData/*.eml"  
 class testRealEmail_traffic(unittest.TestCase):  
   def setUp(self):  
     """  
     Reads an arbitrary number of mail messages and  
     stores them in a brand new messages table.  
     DANGER: Any existing message table WILL be lost.  
     """  
     curs.execute("DROP TABLE IF EXISTS message")  
     conn.commit()  
     curs.execute(TBLDEF)  
     conn.commit()  
     files = glob(FILESPEC)  
     self.msgids = {} # Keyed by message_id  
     self.message_ids = {} # keyed by id  
     for f in files:  
       ff = open(f)  
       text = ff.read()  
       msg = message_from_string(text)  
       id = self.msgids[msg['message-id']] = maildb.store(msg)  
       self.message_ids[id] = msg['message-id']

There were two relatively simple tests, one to test each of the retrieval functions in a fairly simplistic way by verifying the correspondence between the primary key values and the Message-Id headers using the msgids and message_ids dicts created during the set-up. The initial stub under test only implemented the store() function, so these tests were initially expected to fail:

   def test_message_ids(self):  
     """  
     Verify that items retrieved by id have the correct Message-ID.  
     """   
     for message_id in self.msgids.keys():  
       pk, msg = maildb.msg_by_id(self.msgids[message_id])   
       self.assertEqual(msg['message-id'], message_id)  
   def test_ids(self):  
     """  
     Verify that items retrieved by message_id have the correct Message-ID.  
     """  
     for id in self.message_ids.keys():  
       pk, msg = maildb.msg_by_message_id(self.message_ids[id])  
       self.assertEqual(msg['message-id'], self.message_ids[id])

Steve and I had both run this code under much the same conditions as the students would, and verified that the tests did indeed fail due to the AttributeError exceptions raised by the missing msg_by_id() and msg_by_message_id() functions. Later steps have the student implement these functions, which makes the tests pass.

We were somewhat surprised to find when Steve ran his final checks that the tests were now passing, even though he had reverted to the original module with no implementations of the message retrieval functions! It took me the best part of an hour chatting with Steve to eliminate everything I could think of that might be wrong: no .pyc files left lying around, no odd path settings that allowed import from other copies of the code, and so on.

We finally tracked the issue down to something stupidly simple, as is often the case with bugs that have you scratching your head for an extended period: for production purposes the data files had been moved to an area where all students could share them (each student has their own V: directory). This meant that the setUp() method was not inserting any rows into the newly-created message table. The empty table in turn meant that neither test_message_ids() nor test_ids() was running the the body of the for loop, and consequently no AttributeError exceptions were being raised. The tests were passing even though the functions they were supposed to test had not been implemented!

My solution to this was to add a further test to verify that the table was not empty. That way, even if the other tests passed, this one would fail:

   def test_not_empty(self):  
     """  
     Verify that the setUp method actually created some messages.  
     If it finds no files there will be no messages in the table,  
     the loop bodies in the other tests will never run, and potential  
     errors will never be discovered.  
     """  
     curs.execute("SELECT COUNT(*) FROM message")  
     messagect = curs.fetchone()[0]  
     self.assertGreater(messagect, 0, "Database message table is empty")

The check could have been added to one of the other tests but it seemed to make more sense to keep it separate, since several tests relied on the table having been populated.

In this case the error condition was that the test were passing! I am happy about this because it clearly demonstrates the value of test-driven development, even though the result I was getting was normally the desired goal of testing. It has also taught me to be more careful about tests in loops: if there is no guarantee that the loop body will execute then the tests inside it can be completely useless.

Wave Goodbye

2010-08-05T21:48:37+00:00

So Google's blog announced today that development of Google Wave "as a standalone product" will end because "Wave has not seen the adoption we would like". It's kind of a shame, because Wave was intriguing, but as an infrequent Wave user I found several issues that made it less than user-friendly. So here are a few points that developers might like to take home from the train-wreck that is Wave (100 developers for two years is a substantial investment, even for Google). The servers will continue to be available "at least until the end of the year".

Don't try to replace standard GUI components with inferior and non-intuitive substitutes. The Wave scrollbar was a user interface disaster, and a source of frustration to many of the users I interacted with.
Don't promote technologies that depend heavily on high-bandwidth connectivity, or at least not for Internet use. Many times I was left frustrated, not knowing whether the Wave had crashed or whether it was simply waiting for a server response.
Realize that even the best technologies need marketing and publicity. 80%+ of desktop computer users don't use Windows because it's the best system, they use it because it's the best alternative they know about. If people don't know about your technology they won't use it, and techies alone probably aren't the right user base to make a product viral.

In some ways I am sorry that the Wave didn't succeed as so many techies apparently thought it would. In other ways I am disappointed that the technology didn't really deliver on its promise. It's one of those things that needs to be ubiquitous to succeed.

Google's misstep with Buzz earlier this year probably didn't help either - it led to distrust about Google's intentions with regard to (or, worse, competence at securing) users' personal data.

So whatever the next big thing on the Web is going to be, it isn't going to be Google Wave. RIP.

DjangoCon US is International

2010-08-05T18:58:38+00:00

As registrations for DjangoCon US grow it's been interesting to see where people are coming from. I originally thought that it would be US-only. While US delegates dominate the lists as you might expect, we have people coming from all over the world. Here's a graphic showing the distribution of delegates across the globe.

I think it's a measure of Django's excellence that DjangoCon attracts people from so far away. I well remember when Jacob Kaplan-Moss and Adrian Holovaty first came to PyCon in Washington DC to describe the system they were putting together. At that stage Django wasn't open source, and the encouragement of the enthusiastic PyCon audience was a major factor in its becoming so. The software has come a huge distance in a relatively short time, and is now a major Python success story.

There are still places left at the conference if you would like to register. Portland is an intriguing city that offers a warm welcome to visitors, and the Doubletree is a green hotel with excellent accommodation. The conference room rate is only guaranteed until August 13, so make sure to book your accommodation soon.

Wanted: Win32-savvy developers for core Python

2010-08-05T09:44:54+00:00

A python-dev post from Steve Holden gave rise to a wider discussion about the effectiveness of the buildbot system. But also to the question of having more people involved in core Python development who are knowledgeable about the Windows aspects of the core and the stdlib.

To get involved you certainly don’t need to be a Windows guru, nor a Python guru. You don’t need to have oodles of time on your hands. (Although all of the above would help). You do need a willingness to chase down occasionally obscure failing code paths and propose and implement patches. Or workarounds. Or documentation improvements. At present there is a very small number of us watching and fixing win32 issues and, speaking for myself, I have very little time available to help. The more people there are, each of which may have very little time, the more time is available overall to help Python develop and improve.

If you’d like to get involved but haven’t bothered previously, please bother now.

If you’d like to get involved but aren’t sure what to do, ping me or drop into #python-dev on IRC or just go to the Python bug tracker and look for Windows-related problems. (That search isn’t exhaustive as it relies on someone setting the Component field, but it certainly brings some results back).

Guido’s recently expressed a wish to give more people commit privileges, and MSDN have in the past generously provided free licences to allow core developers to develop and test on different Windows and Visual Studio versions.

There’s documentation on python.org for developers, and a new in-progress document by Brian Curtin on the Python sprints site.

Even if your contribution is to confirm that a bug still applies, or has been fixed, or to add tests to an undertested module which you use yourself, or to clarify or add some documentation, it will still be a very useful addition.

links for 2010-08-04

2010-08-04T21:03:43+00:00

Open Source Toolchains for Linux Systems Administrators

(tags: devops agile toolchain linux)

links for 2010-08-03

2010-08-03T21:01:32+00:00

Halo for the 2600 Released

An interesting description in the rationale and mechanics of creating Halo for the 2600

(tags: atari 2600 halo fps games)

Flying High: OpenGL from Python, Part 2

2010-08-03T09:04:43+00:00

This is second in a series of articles about algorithmically generating geometry to drive OpenGL from Python.

<< Back to part 1

Last time we got as far as creating some instances of our super-simple Shape class, and having Glyph and Render classes convert those to arrays for OpenGL and render them. This time, we start using that infrastructure to create some more interesting geometries, which means there’s less code, and more pretty pictures.

Composite Shapes

In order to create more complex shapes by composing instances of existing ones, we need a simple composite shape:

class MultiShape(object):

    def __init__(self):
        self.children = []
        self.matrices = []

    def add(self, child, pos=None, orientation=None):
        self.children.append(child)
        self.matrices.append(Matrix(pos, orientation))

A MultiShape contains a list of child Shapes, and a matrix for each child, indicating the child’s position and orientation relative to the MultiShape’s front-and-center.

This is probably as good a point as any to confess that for the purposes of this demo, I ended up writing my own Matrix class, along with my own Orientation class. Even my Vec3, which earlier I showed you defined as a named tuple, gradually started to sprout some methods, until it became a fully-formed custom vector class. This was pretty silly – it easily doubled the size of my code-base, and while it felt like rewarding and productive work, it was actually a waste of time. With hindsight, I should have predicted this would happen, and started out using an existing library for things like this, such as Euclid or Numpy. Way it goes.

Anyhow, if a Multishape is going to be usable wherever a Shape is currently used, it needs to provide the same interface, which luckily is very simple – it just needs to provide iterables of vertices, faces and face_colors. Here is how MultiShape provides a sequence of vertices, by chaining the vertices of all its children, each vertex transformed by the matrix of the relevant child shape:

@property
def vertices(self):
    return (
        matrix.transform(vertex)
        for child, matrix in zip(self.children, self.matrices)
        for vertex in child.vertices
    )

There is an inefficiency to this. When MultiShapes are nested, I’m transforming each vertex once for every branch in the tree. It would be cheaper to just multiply the matrices of nested MultiShapes, and then have the top-level MultiShape apply the resulting transforms to the vertices of each of its children. However, we’re only performing this work at application start-up, not every frame, so I’m choosing to eat it for the sake of simple-as-possible code.

Similar properties are defined on MultiShape to provide sequences of face indices and face_colors, by aggregating those of its children.

Using MultiShape, we can now easily compose groups of our basic Shapes. A new factory function composes a bunch of cubes into the same MultiShape:

def CubeCorners(edge, color1, color2):
    multi = MultiShape()
    multi.add(
        Cube(edge, repeat(color1)),
        position=Origin,
    )
    for pos in list(product(*repeat([-1, +1], 3))):
        multi.add(
            Cube(edge/2, repeat(color2)),
            position=Vec3(*pos) * (edge / 2),
        )
    return multi

A cluster of cubes, rendered in one glDrawElements call

Another new factory function, RingOf:

def RingOf(child, radius, number):
    multi = MultiShape()
    angle = 0
    orientation = Orientation()
    delta_angle = 2 * pi / number
    while angle < 2 * pi:
        angle += delta_angle
        pos = Vec3(0, radius * sin(angle), radius * cos(angle))
        orientation.pitch(delta_angle)
        multi.add(child, pos, orientation)
    return multi

returns copies of a given child shape, arranged in a ring:

A ring of cubes

A ring of truncated cubes

A ring of interpenetrated tetrahedrons

If we can compose basic shapes into rings, we can also compose rings into… um… tri-axis-rings:

def TriRing(edge, radius, number, colors):
    multi = MultiShape()
    ring = RingOf(Cube(edge, colors), radius, number)
    multi.add(ring, orientation=Orientation(XAxis))
    multi.add(ring, orientation=Orientation(YAxis))
    multi.add(ring, orientation=Orientation(ZAxis, XAxis))
    return multi

Tri-axis rings

Because we’re drawing each MultiShape using a single iteration of the Render.draw() loop, we’ve massively reduced the overhead in drawing each Shape, so we can easily add all of these at once into the world at 60fps, although it does form a bit of a visual cacophony:

All the rings, plus some other stuff

I wonder how much stuff we can add into a MultiShape before it starts to affect the framerate? Let’s investigate… How about a spherical glob of red blood cubes:

A glob of red blood cubes

It turns out I can get about 14,000 cubes (168,000 triangles) [1] into a single MultiShape like this before the framerate starts to drop. I’m still rendering these as regular ctypes arrays, not OpenGL vertex buffers (I don’t think my hardware supports that), so all the geometry is being sent needlessly over the bus to the GPU every frame.

How about an alternative, the RgbCubeCluster:

def RgbCubeCluster(edge, cluster_edge, cube_count):
    cluster = MultiShape()
    for _ in xrange(cube_count):
        color = Color.Random()
        pos = Vec3(
            color.r - 128,
            color.g - 128,
            color.b - 128,
        ) * (cluster_edge / 256)
        cluster.add(
            shape=Cube(edge, repeat(color)),
            position=pos,
        )
        return cluster

This creates a cluster of cubes, each one colored by its position in RGB space.

An RGB cube cluster

We still have enough oomph left over to dive the camera right into the midst of the RgbCubeCluster and reveal that all the previous stuff is still in the world too:

Whirling machinery at the center of an RgbCluster

Recursively Generated Geometry

Can we make any more interesting recursively-defined geometry? The first thing I thought of (no doubt this has been done many times before) was the 3D equivalent of a Koch curve: Take a tetrahedron, and for each face, embed a new, smaller tetrahedron sticking out of it. Recursively repeat this for each of the new smaller triangles that have been formed.

The first time I coded this, successive iterations literally replaced every new surface triangle that was formed by the process, with an arbitrary break after eight or so iterations. I was quite surprised by the result, which turned out to look like a slightly corrugated cube. At first I naturally assumed that a bug in my code was the cause, but after a period of contemplation, I realised this was the correct geometric result. The reason for it can be seen in this Wikimedia diagram of the first three iterations of forming a Koch surface:

The first iteration replaces every triangle by sticking a new tetrahedron out of it – exactly as I had done for every face of my original. The next iteration sticks smaller tetrahedrons onto every new surface, and the edges of these new, smaller tetrahedrons all line up with each other, to form long, contiguous straight seams in the resulting shape. By the third iteration (the final one shown here) the end result is becoming apparent. Each successive iteration merely reduces the size of the ridges – the overall shape of the surface is unchanged.

I modified my algorithm to only replace the triangular faces of the newly-formed smaller tetrahedrons, rather than replacing every triangular surface, and the result is this more pleasing snowflake shape.

A Koch tetrahedron

This algorithm is about 60 lines of code. A similar operation can be done on a cube, by poking a new, smaller cube out of each of its faces:

A Koch cube

The deeper red parts are the original cube and the early generations of children. The lighter yellow parts are the later generations.

The final and best example in this section was supplied by Oscar Lindberg, who was interested enough on my old blog post about this to download the code and produce some shapes of his own. Screenshots can’t do it justice, but the full stately geometry becomes wonderfully apparent when it’s in motion. The tetrix, aka the Sierpinski tetrahedron:

The tetrix, aka Siepinski Tetrahedron

Odds and Ends

That’s about all I’ve got to show you. Overall I’m really pleased by this, and excited to do some more of the same going forward.

You may have noticed I’ve cheated a little in the demo / screenshots – some of them show clear evidence of the rudimentary lighting shader I added (e.g. topmost faces are slightly lighter in color than other faces.) It would be simple enough to fake this, by providing slightly varying colors for each face of our shapes, but for those of you looking at the code: I didn’t do that. Instead, I had Glyph generate arrays of surface normals, which is done by Glyph.get_glnormals(), which works pretty much just like all the other Glyph methods we examined in part 1. I was getting tired of explaining how Glyph worked, so I figured you were probably getting tired of it too, and wouldn’t mind if I skipped a little which wasn’t strictly necessary.

I was initially a little disappointed by the performance at rendering many independently positioned and oriented objects, but now it’s picked up (see footnote [1]), it’s perfectly acceptable: a little over 450 separately moving cubes at 60fps. The OpenGL bindings in PyOpenGL wisely choose to prefer correctness and robustness over performance by default, so as a result, calling OpenGL from Python is not fast out of the box. The PyOpenGL documentation suggests many ways in which this performance can be regained once your program is working and debugged. I’m not yet using any of these suggestions, so hopefully my sluggish performance could be improved substantially.

In addition, Richard Jones suggested that the innermost Render.draw() loop could possibly benefit from Cython (optional static typing to be added to code written in a less-dynamic subset of Python.) This would not just improve the general performance of the code in that loop, by actually compiling it to C, but in doing so, it would eliminate the Python / C boundary performance penalties, and this is something I’m excited to try out in the near future.

[1] Update: A couple of hours after hitting publish on this, I discover that switching from the PyOpenGL bindings to those built into pyglet gives me two to four times the frame rate, for zero code change except the imports. Clearly I don’t understand how to get the best performance out of PyOpenGL. I’ve been back and updated the performance stats in this post, and hope to make another post about this at some point when I understand what I was doing wrong.

The demonstrated code is available via Mercurial, from http://code.google.com/p/flyinghigh-opengl-from-python

links for 2010-08-02

2010-08-02T21:02:36+00:00

BLDGBLOG: Gunnery Pagodas / Manhattan Niagara / The University of War

Some rather stagering photmontages. I want this book.

(tags: photography art montage photomontage japanese outofprint books)

Integrating SQLAlchemy into a CherryPy application

2010-08-01T08:20:50+00:00

Quite often, people come on the CherryPy IRC channel asking about the way to use SQLAlchemy with CherryPy. There are a couple of good recipes on the tools wiki but I find them a little complex to begin with. Not to the recipes’ fault, many people don’t necessarily know about CherryPy tools and plugins at that stage.

The following recipe will try to make the example complete whilst as simple as possible to allow folks to start up with SQLAlchemy and CherryPy.

# -*- coding: utf-8 -*-
import os, os.path
 
import cherrypy
from cherrypy.process import wspbus, plugins
 
from sqlalchemy import create_engine
from sqlalchemy.orm import scoped_session, sessionmaker
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Column
from sqlalchemy.types import String, Integer
 
# Helper to map and register a Python class a db table
Base = declarative_base()
 
class Message(Base):
    __tablename__ = 'message'
    id = Column(Integer, primary_key=True)
    value =  Column(String)
 
    def __init__(self, message):
        Base.__init__(self)
        self.value = message
 
    def __str__(self):
        return self.value.encode('utf-8')
 
    def __unicode__(self):
        return self.value
 
    @staticmethod
    def list(session):
        return session.query(Message).all()
 
 
class SAEnginePlugin(plugins.SimplePlugin):
    def __init__(self, bus):
        """
        The plugin is registered to the CherryPy engine and therefore
        is part of the bus (the engine *is* a bus) registery.
 
        We use this plugin to create the SA engine. At the same time,
        when the plugin starts we create the tables into the database
        using the mapped class of the global metadata.
 
        Finally we create a new 'bind' channel that the SA tool
        will use to map a session to the SA engine at request time.
        """
        plugins.SimplePlugin.__init__(self, bus)
        self.sa_engine = None
        self.bus.subscribe("bind", self.bind)
 
    def start(self):
        db_path = os.path.abspath(os.path.join(os.curdir, 'my.db'))
        self.sa_engine = create_engine('sqlite:///%s' % db_path, echo=True)
        Base.metadata.create_all(self.sa_engine)
 
    def stop(self):
        if self.sa_engine:
            self.sa_engine.dispose()
            self.sa_engine = None
 
    def bind(self, session):
        session.configure(bind=self.sa_engine)
 
class SATool(cherrypy.Tool):
    def __init__(self):
        """
        The SA tool is responsible for associating a SA session
        to the SA engine and attaching it to the current request.
        Since we are running in a multithreaded application,
        we use the scoped_session that will create a session
        on a per thread basis so that you don't worry about
        concurrency on the session object itself.
 
        This tools binds a session to the engine each time
        a requests starts and commits/rollbacks whenever
        the request terminates.
        """
        cherrypy.Tool.__init__(self, 'on_start_resource',
                               self.bind_session,
                               priority=20)
 
        self.session = scoped_session(sessionmaker(autoflush=True,
                                                  autocommit=False))
 
    def _setup(self):
        cherrypy.Tool._setup(self)
        cherrypy.request.hooks.attach('on_end_resource',
                                      self.commit_transaction,
                                      priority=80)
 
    def bind_session(self):
        cherrypy.engine.publish('bind', self.session)
        cherrypy.request.db = self.session
 
    def commit_transaction(self):
        cherrypy.request.db = None
        try:
            self.session.commit()
        except:
            self.session.rollback()  
            raise
        finally:
            self.session.remove()
 
 
 
 
class Root(object):
    @cherrypy.expose
    def index(self):
        # print all the recorded messages so far
        msgs = [str(msg) for msg in Message.list(cherrypy.request.db)]
        cherrypy.response.headers['content-type'] = 'text/plain'
        return "Here are your list of messages: %s" % '\n'.join(msgs)
 
    @cherrypy.expose
    def record(self, msg):
        # go to /record?msg=hello world to record a "hello world" message
        m = Message(msg)
        cherrypy.request.db.add(m)
        cherrypy.response.headers['content-type'] = 'text/plain'
        return "Recorded: %s" % m
 
if __name__ == '__main__':
    SAEnginePlugin(cherrypy.engine).subscribe()
    cherrypy.tools.db = SATool()
    cherrypy.tree.mount(Root(), '/', {'/': {'tools.db.on': True}})
    cherrypy.engine.start()
    cherrypy.engine.block()

The general idea is to use the plugin mechanism to register functions on an engine basis and enable a tool that will provide an access to the SQLAlchemy session at request time.

Flying High: Hobbyist OpenGL from Python

2010-07-31T13:58:30+00:00

This is a transcript-from-memory (what I wish I’d said) of the talk I just gave at EuroPython 2010, for which I owe a debt of gratitude to Richard Jones for his last-minute moral support while wrestling with projectors and refresh rates; and to the whole team of hard-working volunteers, especially John Pinner & Richard Taylor, who gave so much time and effort to make EuroPython in the UK brilliant once again.

The demonstrated code is available via Mercurial, from http://code.google.com/p/flyinghigh-opengl-from-python

With this talk I want to give an overview of creating 3D graphics in OpenGL from Python. Instead of covering topics already covered by a thousand OpenGL tutorials, I want to shift attention towards some ideas of how to generate the input to your renderer – how to algorithmically create geometry. I’ll show that with just a paltry few hundred lines of relatively simple code, you can generate some interestingly chunky shapes – virtual sculptures, if you will. Since this talk has the word hobbyist in the title, I want to emphasise how easy this is, and I want to have some fun with the pretty pictures.

Out of interest, how many people here are already expert OpenGL users (a few hands hesitantly go up, then some think about it and go down again) err, I mean how many have already used OpenGL to do anything at all (about half the people raise their hand.) Alright, well, I want you all to leave here enthused to go generate your own images or animations or games.

Inspirations

As the field of computer graphics advances, there’s an understandable tendency for more photorealism, This is laudable, but I also feel that the effort expended on achieving this technical goal is often undertaken without considering whether photorealism is the best aesthetic choice for a particular project.

In the past, games and other applications adopted particular visual styles out of technical necessity. As is often the case, these restrictions resulted in a diverse blossoming of creative ideas, producing an enormous set of distinctive visual styles and experiences.

Non-photo-realistic Quake

Crucially, the most successful and memorable examples of these were projects that found ways to work in harmony with the restrictions of the medium, rather than attempting to gloss over them.

Rez HD

Advances in computing power and technique provide modern games and applications with a far wider range of options in how to present themselves visually, and yet the greater proportion of them seem content with a conventional and unimaginative ‘near-photorealistic’ appearance. This disappoints me, because I feel that projects that opt for a more highly stylised look, when appropriately chosen, can create a vastly more striking and memorable artistic experiences. This is true in movies and all kinds of art.

Waking Life

As an amateur graphics programmer, I don’t have large resources nor much experience to throw at the problem, so my options and my abilities are limited. But, like a good artist, I believe it should still be possible to create things that are both strikingly beautiful and highly functional, either by working with the restrictions of the medium, or by finding creative ways to exploit or extend them.

Love

In particular, the kind of minimal, clean-lined aesthetic that amateur OpenGL programs take on by default are useful for their crisp precision, as charting and visualisation tools. But above that, I love them for their stark minimalism, their clean lines and homogeneous fields of colour.

Tron Legacy

I wish more professional game developers had an incentive to aim for less conventional aesthetics – whether they be deliberately retro, or else striking out in some new direction of their own. It’s that brave minority of projects which do this which form my inspiration.

Starting Point

I’m assuming we already have a minimal OpenGL application, that:

Opens a window
Provides an OpenGL context for us to render to
Sets appropriate 3D projection matrix
Sets the initial modelview matrix state based on the position and orientation of a ‘camera’ object
Calls our empty ‘draw’ function once per monitor refresh.

This results in a blank screen, at 60fps. Here’s a screenshot, so you can see exactly what it’s doing:

A blank screen

I’m using pyglet & PyOpenGL for this, but this isn’t important. Any framework that provides the above abilities, such as PyGame, along with bindings to OpenGL, will be just fine. Whichever framework you use, this minimal application might take on the order of about 150 lines of code, and is covered in countless tutorials all over the web.

From here on in I plan to show (or at least describe) pretty much all of the code that I add on top of this minimal OpenGL loop.

Goal

To begin with, I’m going to lead you as quickly as I can through a Shape class, that model 3D shapes, in a way useful for the creation of geometry, and then a Glyph class that converts these geometries into arrays for OpenGL. Finally these arrays get passed into a Render class, which simply calls glDrawElements to render them.

Our Goal

Once the above infrastructure is in place, we can have some fun generating interesting shapes to make pretty pictures with. The conventional way to provide geometry to your OpenGL code is by loading your data from files. Today though, I want to stick with generating geometry from code, to see where that leads.

Modelling Polyhedra

A polyhedron is a 3D shape with flat faces and straight edges. We can model coloured polyhedra using a simple Shape class:

Vec3 = namedtuple('Vec3', 'x y z')
Color = namedtuple('Color', 'r g b a')

class Shape(object):

    def __init__(self, vertices, faces, face_colors):
        # list of Vec3s
        self.vertices = vertices

        # list of faces, each face is a list of indices into 'vertices'
        self.faces = faces

        # List of colors, one per face
        self.face_colors = face_colors

An instance of this class, for example, might represent a yellow cube, or a tetrahedron with green and black faces, or any other coloured polyhedron we can imagine.

To demonstrate how classes Shape, Glyph and Render hang together, let’s examine an even simpler example, a yellow triangle joined to a red square:

Red Triangle & Yellow Square

You can see this geometry features five vertices (v0 to v4), which are used by the two faces. This might be represented by an instance of Shape:

v0 = Vec3( 1,  1, 0)
v1 = Vec3( 1, -1, 0)
v2 = Vec3(-1, -1, 0)
v3 = Vec3(-1   1, 0)
v4 = Vec3( 1,  0, 2)

red = Color(255, 0, 0, 255)
yellow = Color(255, 255, 0, 255)

shape = Shape(
    vertices=[v0, v1, v2, v3, v4],
    faces=[
        [2, 3, 4],    # f0, triangle
        [0, 1, 2, 3], # f1, square
    ],
    face_colors=[red, yellow],
)

The integers in the ‘faces’ member are indices into the vertices list. So the triangular face, for example, is formed by linking vertices 2, 3 and 4.

Step 1. Creating a Ctypes Vertex array

In order to render our Shape, we need to convert it to some ctypes arrays that OpenGL will eat:

glvertices – an array of GLfloats (three for each vertex)
glindices – an array of GLubytes (one for each index of each face)
glcolors – an array of GLubytes (four for each vertex)

To generate glvertices, we need to dereference the indices in Shape.faces, to produce a new list of vertices, rearranged into the order they are going to be drawn:

Step 1. Dereference indices

The most visible aspect of this change is that the vertices are re-ordered, such that the indices now simply read ’0, 1, 2, 3, 4, 5…’. However that isn’t actually necessary. The important part of this transformation is that vertices which are re-used are now duplicated in the vertex list. For example v0 now occurs twice. As a result of this vertex duplication, one the two instances of ’0′ in the faces lists now instead reads ’3′ (referencing the new second copy of v0).

This duplication of vertices is required, because when v0 is used for the first time, it is as part of the red triangle, and when it is used the second time it is as part of the yellow square. The color of the vertex changes from one occurrence to the next. All the attributes of a vertex (position, color, texture co-ords, normals, etc) are treated as an atomic unit, so whenever any attribute changes, as the color is changing here, the vertex position needs to be redundantly specified again, so as to create a new unique vertex with its own unique attribute values. Even if the color of v0 in our example was identical for each use, we will see later that other vertex attributes such as surface normals will still differ. Don’t sweat trying to eliminate these redundancies, they are necessary, unless every single attribute of the re-used vertex (including surface normals) are identical.

The code in Glyph.get_glverts() performs this dereferencing step:

class Glyph(object):

    def get_glverts(self, shape, num_glverts):
        glverts = chain.from_iterable(
            shape.vertices[index]
            for face in shape.faces
            for index in face
        )
        ArrayType = GLfloat * (num_glverts * 3)
        return ArrayType(*glverts)

This uses a generator to produce the vertices in the order that we need them. ‘ArrayType’ shows the standard idiom to create a ctypes array – we take the datatype of the array elements, in this case GLfloat since our vertex positions consist of three floats, and multiply it by the required length of the array. This yields a new array type. The final return statement instantiates this array type using the data supplied by the glverts generator.

Step 2. Creating Ctypes Index Arrays

The second job Glyph has to do is create a ctypes indices array, which is derived from the Shape’s faces. In doing this, it has to break the Shape’s faces down into individual triangles.

Step 2. Tessellate indices

The vertex list is unchanged by this step, and the first face – the triangle – is also unchanged. The second face, the square, has been broken into two triangles.

There are well-known algorithms for breaking an arbitrary polygon down into individual triangles. Using the utility functions found in the GLU library, this can be done in about 150 lines of Python. But in the interests of keeping it simple, I decided to restrict our code to just handling convex faces. Tessellating these faces can be done using a considerably simpler algorithm:

def tessellate(face):
    '''
    Break the given face into triangles.
    e.g. [0, 1, 2, 3, 4] ->
    [[0, 1, 2], [0, 2, 3], [0, 3, 4]]
    Does not work on concave faces.
    '''
    return (
        [face[0], face[i], face[i + 1]]
        for i in xrange(1, len(face) - 1)
    )

We again use a generator, to simply join up the face’s first vertex with all the other vertices, like this:

Tessellation of convex faces

Now we have our tessellate function, Glyph can now create the glindices array in much the same way as it generated the glvertices. I wasn’t smart enough to write this as a generator first time around, I presume it would require more than one generator to do it (anyone?), so I’m needlessly creating an in-memory copy of the sequence, but it turns out I need to take its length right afterwards anyway, so what the heck:

class Glyph(object):

    def get_glindices(self, faces):
        glindices = []
        face_offset = 0
        for face in faces:
            indices = xrange(face_offset, face_offset + len(face))
            glindices.extend(chain(*tessellate(indices)))
            face_offset += len(face)
        ArrayType = GLubyte * len(glindices)
        return ArrayType(glindices)

This is more complex than get_glvertices because it is performing both of the transformations described in steps 1 and 2. But it’s still pretty straightforward. Note that the type of the index array will have change from GLubytes to GLushorts (or GLuints) if the number of vertices rises above 256 (or 65,536.)

Step 3. Creating Ctypes Color Arrays

Finally, we need an array of vertex colors. This is the simplest of the lot, generated by repeating the face_color for each face, once per vertex:

class Glyph(object):

    def get_glcolors(self, faces, face_colors, num_glvertices):
        glcolors = chain.from_iterable(
            repeat(color, len(face))
            for face, color in izip(faces, face_colors)
        )
        ArrayType = GLubyte * (num_glvertices * 4)
        return ArrayType(chain(*glcolors))

First Light

It’s might seem like a teensy bit of a slog to get here, but it hasn’t been more than sixty lines of code, and now we’re in a position to pass our ctypes arrays into OpenGL’s drawElements. This happens in our Render.draw() method:

class Render(object):

    def draw(self, world):
        for item in world:
            glVertexPointer(3, GL_FLOAT, 0, item.glyph.glvertices)
            glColorPointer(4, GL_UNSIGNED_BYTE, 0, item.glyph.glcolors)

            # TODO: handle the item's position and orientation

            glDrawElements(
                GL_TRIANGLES,
                len(item.glyph.glindices),
                GL_UNSIGNED_BYTE,
                item.glyph.glindices
            )

This is canonical OpenGL render code, so I’m not going to dissect it, but now we get some actual visible output:

Red triangle, yellow square

Hooray! \o/ We can move our camera position around, and view this 3D object from different angles.

There’s a minor wrinkle here that I’m glossing over. I’ve turned on backface culling, so the triangle and square aren’t visible if we view them from the back. For all our future examples I plan on using closed polyhedra, so we won’t be able to see the ‘backs’ of the faces – those will be on the inside of the polyhedron.

The Fun Stuff

So now we’ve got all our infrastructure in place, we can start creating factory functions to churn out some Shapes. Let’s start with something straightforward, a tetrahedron (triangle-based pyramid):

def Tetrahedron(edge, face_colors=None):
    size = edge / sqrt(2)/2
    vertices = [
        (+size, +size, +size),
        (-size, -size, +size),
        (-size, +size, -size),
        (+size, -size, -size),
    ]
    faces = [ [0, 2, 1], [1, 3, 0], [2, 3, 1], [0, 3, 2] ]
    return Shape(vertices, faces, face_colors)

Which produces:

A tetrahedron

Then a cube factory:

def Cube(edge, face_colors=None):
    e2 = edge / 2
    verts = list(itertools.product(*repeat([-e2, +e2], 3)))
    faces = [
        [0, 1, 3, 2], # left
        [4, 6, 7, 5], # right
        [7, 3, 1, 5], # front
        [0, 2, 6, 4], # back
        [3, 7, 6, 2], # top
        [1, 0, 4, 5], # bottom
    ]
    return Shape(verts, faces, face_colors)

The six faces are quite evident, but the use of itertools.product to produce the list of vertices perhaps deserves a bit of exposition. It’s an inspired tip from ΤΖΩΤΖΙΟΥ. Just to spell it out in longhand:

>>> from itertools import repeat, product
>>> list(product(*repeat([-1, +1], 3)))
[(-1, -1, -1), (-1, -1, 1), (-1, 1, -1), (-1, 1, 1),
 (1, -1, -1), (1, -1, 1), (1, 1, -1), (1, 1, 1)]

So there are the eight vertices of the cube, and that gets us the following:

A cube

We can add a few more vertices and faces, to make ourselves a truncated cube:

A truncated cube

Once we’ve got truncated cubes, we might as well add one last face to form the entrance:

A truncated cube with entrance

There’s nothing to stop us adding several of these shapes into the world at once, but since we haven’t yet moved any of them away from the origin, they just sit there, embedded within one another:

A cube and tetrahedron interpenetrate

A truncated cube with two tetrahedrons

Moving objects around

In our earlier Render.draw() method, we left a ‘TODO’ comment in place, to note that we weren’t yet handling item positions and orientations. Here’s what Render.draw looks like when we fill that code in:

class Render(object):

    def draw(self, world):
        for item in world:
            glVertexPointer(3, GL_FLOAT, 0, item.glyph.glvertices)
            glColorPointer(4, GL_UNSIGNED_BYTE, 0, item.glyph.glcolors)

            glPushMatrix()
            glTranslatef(*item.position)
            glMultMatrixf(item.orientation.matrix)

            glDrawElements(
                GL_TRIANGLES,
                len(item.glyph.glindices),
                GL_UNSIGNED_BYTE,
                item.glyph.glindices
            )
            glPopMatrix()

Again, this is very standard OpenGL usage. To set an item’s position attribute, I’m going to use a bit of code that I already snuck into the demo without telling you about. It’s the code that moves the camera around in space. A simplified version is here, class Orbit, which will return a new position each time it gets called. The locus of this position is an orbit around the origin:

class Orbit(object):
    def __init__(self, distance, speed, phase=None):
        self.distance = distance
        self.speed = speed
        if phase is None:
            phase = random.uniform(0, 2 * pi)
        self.phase = phase

    def __call__(self, time):
        bearing = time * self.speed + self.phase
        x = self.distance * math.sin(bearing)
        z = self.distance * math.cos(bearing)
        return Vec3(x, 0, z)

The actual camera uses a slightly longer version I call WobblyOrbit (not shown), which operates in exactly the same way. Any ‘mover’ class, i.e. one that returns a Vec3 position when called, can be used to move the camera, or any other item, around in space:

class GameItem(object):
    def __init__(self, ** kwargs):
       self.__dict__.update(** kwargs)

world.add( GameItem(
    shape=Cube(1, repeat(red)),
    mover=Orbit(distance=20, speed=4),
) )

# then, in world.update():
for item in self.items:
    if item.mover:
        item.position = item.mover(self.time)

Similarly, we can spin items using ‘spinner’ classes, that tweak an item’s orientation as time goes by.

With these all in place, we can now add many Shapes to the world, each moving and rotating independently:

Several independently positioned and oriented shapes

Next week: Composite Shapes…

This is all great as far as it goes, but it turns out we have a performance problem. Adding more than about 450 shapes at once starts to slow down below 60fps (This is all on my trusty 2005-era Thinkpad T60 laptop.) The bottleneck turns out to be in our Render.draw() loop. Each of those OpenGL functions are from (wrappers around) the OpenGL C library, and calling across the Python / C boundary like this incurs a per-function call overhead. Also, a second looming problem is that creating more interesting shapes is going to become more onerous and error-prone, as we create longer and more complex lists of vertices and faces in our code.

One partial solution to both these problems is to use composite shapes, in which we can compose many copies of our basic shapes into one single, more complex shape. This will allow us to use algorithmic means to produce more fun geometry, and will also help us draw more complex shapes, composed of many simpler shapes, without requiring several separate OpenGL function calls for each of the simple shapes.

On to Part 2 >>

A plugin system for unittest(2)

2010-07-30T12:15:19+00:00

One of the big problems with unittest, which I have been maintaining for well over a year now, is how hard it is to extend. For a long time I've been saying that my "big task" with unittest was to implement a plugin system. ... [838 words]

links for 2010-07-28

2010-07-28T21:02:52+00:00

nottingham-http-cache-channels-01.txt

(tags: cache squid http ietf)
mnot’s Weblog: Cache Channels

(tags: cache squid http)

Links for 2010-07-27 [del.icio.us]

2010-07-28T07:00:00+00:00

billboard-in-the-beginning-man-created-god.png (PNG Image, 1920x960 pixels) - Scaled (67%)
Three Dimensions of a Software Programmer: How to get things done
Excellent post on the programming craft

links for 2010-07-27

2010-07-27T21:02:32+00:00

The worlds longest photo exposure

(tags: photography longexposure pinhole)

rnalexander

2010-07-24T18:28:37+00:00

In your online registration, if you’re only going to accept a specific subset of characters for the password be very clear what they are.
If the person registering should get that wrong tell them that in the error message, instead of telling them that you can’t process their application repeated times, and then let them find out about said restrictions when they go through the call centre.
Also, in your online registration, if you’re going to offer people ‘extra keys’ for your system, please be clear that for each ‘extra key’ a person orders, they will also be charged for an ‘extra period charge’ as well. This will prevent your potential clients from being horribly surprised when they see a charge for nearly £200 going through instead of about £50.
Speaking of charging, give people a confirmation screen before you charge them. In other words, the first time they see the actual total of what they are about to be charged should not be on the ‘Verified by VISA screen’
Further to the speaking of charging, when you call people back from a call centre that shows up on their phone as a blocked number, just use the security data they provided, do not also ask the people you call back to give you their credit card number, and their expiration date.
Further again to the speaking of credit card numbers, do not have someone from the call centre read back a persons credit card number and expiration date to attempt to reassure them. This is not reassuring, it is, in fact, the opposite of reassuring.

But hey, it’s early days, and a few mistakes are to be expected. I’d only really worry if this kind of behaviour was in some way associated with a bank, I mean, that would really make me uncomfortable.

links for 2010-07-22

2010-07-22T21:01:28+00:00

links for 2010-07-21

2010-07-21T21:02:52+00:00

Christophe Heyman Photography – Nikon Pages

Modifying Nikon teleconvertors to take a wider variety of lenses

(tags: nikon teleconvertors diy)
Special Lenses For Nikon 'F' Mount

A greta selection of the more exotic Nikon F Mount lens. Including an f/075…

(tags: nikon photography)
Teleconverters for Nikon 'F' Mount

(tags: nikon teleconvertors review photography)

AWS: S3 vs EBS-backed instances

2010-07-19T20:09:20+00:00

As part of the day job at ThoughtWorks, I’m involved with training courses we run in partnership with Amazon, giving people an overview of the AWS service offerings. One of the things we cover that can often cause some confusion is the different types of images – specifically the difference between S3 and EBS-backed images. I tend to give specific advice over what type of instance I prefer, but before then some detail is in order.

S3-hosted Images

Instances launched from S3-hosted images use the local disk for the root partition. Once spun up, if you shut down machine down the state of the operating system is lost. The only way to persist data is to store it off-instance – for example on an attached EBS volume, on S3 or elsewhere. Aside from the lack of persistence of the state of the OS, these images are limited to being 10GB in size – not a problem for most Linux distros I know of, but a non-starter for some Microsoft images.

There is also a time delaying spinning these images up, due to the time taken to copy the Image from S3 to the physical machine – the whole image has to be downloaded before the instance can start the boot up process. In practice, popular images seem to be pretty well cached and so that the delay may not be an issue for you. As with everything in AWS, the barrier to entry is so low it is worth testing this yourself to understand if this is going to be a deal breaker.

EBS-backed Images

An EBS-backed instance is an EC2 instance which uses such a device as its root partition, rather than local disk. EBS-backed images can boot in seconds (as opposed to minutes for less-well cached S3 images) as they just need to stream enough of the image to start the boot process. They also can be up to 1TB in size – in other words the max size of any EBS volume. EBS volumes also have different performance characteristics than local drives, which may need to be taken into account depending on your particular use.

The key benefit is that EBS-backed instances can be ‘suspended’ – by stopping an EBS-backed instance, you can resume it later, with all EBS-backed state (e.g. the OS state) being maintained. In this way it is more akin to the slices at slicehost, or a real, physical machine.

There is one out and out downside when compared to S3-hosted images – your charges will be higher. You will be charged for IO, which could mount up depending on your particular usage patterns. You’re also charged for the allocated size of the EBS volume too.

So which is best?

Aside from the increased cost, and the differing benefits of a block-storage rather than local disk in terms of performance characteristics, there is one key downside to an EBS. It encourages you to think of EC2 instances as persistent, stable entities which are around forever. In general, I believe you’ll gain far more from embracing the ‘everything fails, deal with it’ ethos – that is, have your machines configure themselves afresh atop vanilla gold S3-backed AMIs on boot (e.g. using EC2 instance data to download & install your software). This means you cannot fall back into the old thought patterns of constantly patching, tweaking an OS, to the point where attempting to recreate it would be akin to an act of sysadmin archeology.

There will remain scenarios where EBS-backed instances make sense (and those of you using Windows Server 2008 have no choice), but I always recommend that people moving to AWS limit their use and instead adopt their practices to use the S3 – thereby also embracing more of the promise of the AWS stack as a whole.

Loving EuroPython Tutorials

2010-07-18T23:08:53+00:00

I’ve been loving the two days of tutorials preceding the EuroPython conference. This morning I attended Richard Jones‘ splendid Introduction to Game Programming. It was an absolute pleasure to be walked through the creation of an example game like this, using Python game libraries like pyglet and Cocos, by someone who really knows what he’s doing. Also, it’s nice to have something visible to show after a morning’s work:

My asteroids

The code is based very heavily on samples that Richard provided and talked us through in great detail, so although I now understand it pretty thoroughly, I can’t take much credit. Excepting, that is, for a handful of minor tweaks I couldn’t resist making along the way, like the flickery animated engine thrust, that made me gurgle with delight.

For the artwork, I stole the shuttle icon (sssshhhhhh!), added the engine thrust, and created the asteroids themselves entirely from scratch in Gimp. They are just rotatable bitmaps, albeit designed in homage of vector graphics of yore, complete with simulated CRT glow. Brilliant!

EuroPython 2010

2010-07-17T22:17:08+00:00

I’m hugely looking forward to EuroPython in Birmingham from Monday. I’m driving up Monday very early (I wish I’d booked the hotel room for Sunday night too…). Browsing through the abstracts I’d say all the following look darned interesting!

C++ integration
concurrent sequential processes
Arduino hacking
javascript
OpenData
aerodynamics
PyPy and Unladen Swallow
game programming
OpenGL
Pyjamas
idiomatic Python
MediaCore
Twisted and gevent
science and maths
SHOGUN machine learning

I’ll bring Headroid along and I hope to organise a Birds of a Feather session on Artificial Intelligence and robotics. If you’re interested in these topics, I’d love to say hi!

links for 2010-07-14

2010-07-14T21:01:21+00:00

Ignite Realtime: Openfire Server

An OS "Real Time Collaboration" server – so Jabber with some cool features and plugins

(tags: jabber irc im team collaboration communication opensource)

22,937* faster Python math using pyCUDA

2010-07-14T11:52:10+00:00

I’ve just uploaded a new Mandelbrot.py demo for pyCUDA, it adds a new calculation routine that straddles the numpy (C based math) and the pure-CUDA implementations. In total there are 4 variants to choose from. The speed differences are huge!

Update – this Reddit thread has more details including real-world timings for two client problems (showing 10-3,677* speed-ups over a C task).

This post builds upon my earlier pyCUDA on Windows and Mac for super-fast Python math using CUDA.

You’ll need CUDA 3.1 and pyCUDA installed with a compatible NVIDIA graphics card. This version of the Mandelbrot code forces single precision math – this means it’ll work on all CUDA cards (even the older ones – full list). It runs on my MacBook (Leopard) and Windows, the Windows machines use a 9800 GT and GTX 480. Here’s what it generates:

The big-beast graphics card for my physics client is a GTX 480 – this is NVIDIA’s top of the line consumer card (costing £420GBP in the UK a few weeks back). It is huge – it covers two slots, uses one PCIe 2.0×16 slot and has a requirement for 300-400W of power (I’m using a 750W PSU to be safe on a Gigabyte GA H55M S2H motherboard):

The mandelbrot.py demo has four options (e.g. ‘python mandelbrot.py gpu’):

‘gpu’ is a pure CUDA solution on the GPU
‘gpuarray’ uses a numpy-like CUDA wrapper in Python on the GPU
‘numpy’ is a pure Numpy (C-based) solution on the CPU
‘python’ is a pure Python solution on the CPU with numpy arrays

The default problem is a 1000*1000 Mandelbrot plot with 1000 max iterations. I’m running this on a 2.9GHz dual core Windows XP SP3 with Python 2.6 (only 1 thread is used for all CPU tests). The timings:

‘gpu’ – 0.07 seconds
‘gpuarray’ – 3.45 seconds – 49* slower than GPU version
‘numpy’ – 43.4 seconds – 620* slower than GPU version
‘python’ – 1605.6 seconds – 22,937* slower than GPU version
‘python’ with psyco.full() – 1428.3 seconds – 20,404* slower than GPU version

By default mandelbrot.py forces single precision for all the math. Interestingly on my box if I let numpy default to numpy.complex128 (two double precision floating point numbers rather than numpy.complex64 with two single precision floats) then the Python result is faster:

‘numpy’ – 34.0 seconds (double precision)
‘python’ – 627 seconds (double precision) – 2.5* faster than the single precision version

The ’22,937*’ figure is a little unfair in light of the 627 second result (which is 8,957* slower) but I wanted to use only single precision math for consistency and compatibility across all CUDA cards (the older cards can only do single precision math).

On my older dual core 2.66GHz machine with a 9800 GT I get:

‘gpu’ – 1.5 seconds
‘gpuarray’ – 7.1 seconds – 4.7* slower than GPU version
‘numpy’ – 51 seconds – 34* slower than GPU version
‘python’ – 1994.3 seconds – 1,329* slower than GPU version

If we compare the 0.07 seconds for the GTX 480 against the 1.5 seconds for the 9800 GT (albeit on different machines but the runtime is just measuring the GPU work) then the GTX 480 is 21* faster than the 9800 GT. That’s not a bad speed-up for a couple of years difference in architectures.

If you take a look at the source code you’ll see that the ‘gpu’ option uses a lump of C-like CUDA code, behind the scenes all pyCUDA code is converted into this C-like code and then down to PTX via their compiler. This is the way to go if you understand the memory model and you want to write very fast code.

The gpuarray option uses a numpy-like interface to pyCUDA which, behind the scenes, is converted into CUDA code. Because it is compiled from Python code the resulting CUDA code isn’t as efficient – the compiler can’t make the same assumptions about memory usage as I can make when hand-crafting CUDA code (at least – that’s my best understanding at present!).

The numpy version uses C-based math running on the CPU – generally it is regarded as being ‘pretty darned fast’. The python version uses numpy arrays with straight Python arithmetic, this makes it awfully slow. Psyco 2.0.0 makes it a bit faster.

Feedback and extensions are welcomed via the wiki!

If you want to get started then make sure you have a compatible CUDA card, get pyCUDA (installation instructions), compile pyCUDA (takes 30 minutes from scratch if you’re on a well-known system), try the examples and run mandelbrot.py. The mailing list is helpful.

It’d be nice to see some comparisons with PyPy, ShedSkin and other Python implementations. You’ll find links in my older ShedSkin post. It’ll also be interesting to tie this in to some of the A.I. projects in the A.I. Cookbook, I’ll have to ponder some of the problems that might be tackled.

Books:

The following two books will be useful if you’re new to CUDA. The first is very friendly, I’m still finding it very useful.

Undocumented feature of the week: $PIP_DOWNLOAD_CACHE

2010-07-13T11:46:15+00:00

Use Python? You should be using Pip. A replacement for easy_install, that supports uninstalling and plays nice with virtualenv. An apt-get for Python packages, if you will.

It has a marvellous undocumented feature. Set the environment variable PIP_DOWNLOAD_CACHE to prevent re-downloading the same packages repeatedly when setting up environments on the same machine:

> set | grep PIP
PIP_DOWNLOAD_CACHE=C:\Documents and Settings\jhartley\.pip_download_cache

> pip install mock
Downloading/unpacking mock
Creating supposed download cache at C:\Documents and Settings\jhartley\.pip_download_cache
 Downloading mock-0.7.0b2.zip (242Kb): 242Kb downloaded
 Storing download in cache at c:\documents and settings\jhartley\.pip_download_cache\http%3a%2f%2fpypi.python.org%2fpackages%2fsource%2fm%2fmock%2fmock-0.7.0b2.zip
[snip]
Successfully installed mock

> pip uninstall mock
[snip]
 Successfully uninstalled mock

> pip install mock
Downloading/unpacking mock
 Using download cache from C:\Documents and Settings\jhartley\.pip_download_cache\http%3A%2F%2Fpypi.python.org%2Fpackages%2Fsource%2Fm%2Fmock%2Fmock-0.7.0b2.zip
[snip]
Successfully installed mock

(This text is copied from my unholy bastardised shell of choice at work, Windows CMD shell with Cygwin binaries foremost on the PATH.)

Using the download cache like this is substantially faster. Exactly what you need if you’re continually setting up environments under various version of Python for testing or what-have-you.

The directory is created if it doesn’t exist. Network access is still required when installing like this, presumably for the version checks.

Thanks to the irrepressible fuzzyman for bringing this to my attention.