Planet London Python

Links for 2010-03-09 [del.icio.us]

2010-03-10T08:00:00+00:00

Court Allows Torture Suit Against Rumsfeld
Perhaps this will make public officials think twice about putting themselves above the law

Links for 2010-03-09 [del.icio.us]

2010-03-10T08:00:00+00:00

The Panic Status Board
A nice Big Visible status display
How to build a superluminal computer
Faster than light computing. Awesome. Might even run Ruby at an acceptable speed. ;-)
Globe, Covent Garden
BBC News - UK Skynet military satellite system extended
Skynet? What's wrong with these people?
First Camelsdale Cubs
Steve's Django/Pinax site.
Setting up a Mercurial VCS on Ubuntu
Vagrant
"Automated creation and provisioning of virtual machines using VirtualBox." Nice.
How Django processes a request

Encoding json on Silverlight with System.Json

2010-03-10T00:01:49+00:00

At the backend of last I year I wrote a blog entry on decoding json on Silverlight. Well, the time has finally come and we're now encoding json to post back to our Django application. ... [292 words]

Speed test between django_mongokit and postgresql_psycopg2

2010-03-09T14:38:00+00:00

Following on from yesterday's blog about How and why to use django-mongokit I extended the exampleproject which is inside the django-mongokit project with another app called exampleapp_sql which does the same thing as the exampleapp but does it with SQL instead. Then I added a very simple benchmarker app in the same project and wrote three functions:

One to create 10/100/500/1000 instances of my class
One to edit one field of all 10/100/500/1000 instances
One to delete each of the 10/100/500/1000 instances

[325 more words]

Links for 2010-03-08 [del.icio.us]

2010-03-09T08:00:00+00:00

15 Reasons Why We Need a Revolt in This Country
Quite.

Links for 2010-03-08 [del.icio.us]

2010-03-09T08:00:00+00:00

How and why to use django-mongokit (aka. Django to MongoDB)

2010-03-08T11:00:00+00:00

Here I'm going to explain how to combine Django and MongoDB using MongoKit and django-mongokit.

MongoDB is a document store built for high speed and high concurrency with a very good redundancy story. It's an alternative to relational databases (e.g. MySQL) that is what Django is tightly coupled with in it's ORM (Object Relation Mapping) and what it's called now is ODM (Object Document Mapping) in lack of a better acronym. That's where MongoKit comes in. It's written in Python and it connects to the MongoDB database using a library called pymongo and it turns data from the MongoDB and turns it into instances of classes you have defined. MongoKit has nothing to do with Django. That's where django-mongokit comes in. Written by yours truly.

[1551 more words]

Links for 2010-03-07 [del.icio.us]

2010-03-08T08:00:00+00:00

Ten things you don’t know about the Earth
Interesting stuff
The Top 10 jQuery Plugins for the JavaScript-Weary
More good jQuery stuff
Greetings from Idiot America
Truer by the day
Classic Jazz on JazzRadio.com

Links for 2010-03-06 [del.icio.us]

2010-03-07T08:00:00+00:00

New York Isn’t Silicon Valley, and That’s Why They Like It
Interesting discussion of the high-tech startup scene in New York
Zombie Invasion! - Bibletastic.com
Well, this strikes me as funny though I regret it may offend some of my friends

Links for 2010-03-05 [del.icio.us]

2010-03-06T08:00:00+00:00

Children’s Drawings Painted Realistically
Eeeek!

Fun and (adventure) games at the London Python Dojo

2010-03-05T13:16:26+00:00

Just saw a tweet from Rene saying that he’d enjoyed last night’s Dojo at Fry-IT. I did, too, and for much the same reasons: the small group format makes for a more engaged, friendlier evening. We were carrying on with our not-so-spectacular text adventure game built in previous weeks. Altho’ there had been discussion about different groups working on separate pieces which would then come together, I think our eventual choice for all groups to work on the same thing was the right one. As Nicholas — Dojo organiser and former teacher :) — pointed out (correctly): if you’ve all been working on the same piece of code and the same structures, it’s much easier to follow the show-and-tell at the end.

In the spirit of previous Dojos, which had been very much led by TDD-aware people, I’d got all test-y in our group and we spent way more time in generating meaningful tests than launching into functional code. (As well as reworking the crufty parser which everyone had to cope with). As far as I can tell, *none* of the other groups were testing. Just goes to show… testing really does slow you down for no nett gain ;)

It was definitely interesting to see the different styles & approaches adopted by the different groups. As well as their attitude to the source material: most were “respectful” of the descriptions and objects supplied (by Bruce & John) but others simply hacked them about to suit their requirements. And one off-the-wall group simply made up their own thing, generating random monsters doing random things. As far as I could tell.

Although this format worked well, I think varying from time to time is good — as we have been doing — not least because different approaches suit different people and we want people to keep coming! Thanks as always to Nicholas and Fry-IT for organising / hosting / feeding. Pictures are up here. (Apparently that site’s Django driven, in case it makes you any more likely to click on the link…)

Links for 2010-03-04 [del.icio.us]

2010-03-05T08:00:00+00:00

Printable Sketch Templates and Sketch Books for Wireframing
Useful designer tools

Links for 2010-03-04 [del.icio.us]

2010-03-05T08:00:00+00:00

Space Invaders, Enterprise Edition
Funny. I mean, this is a joke, right?
Daring Fireball: This Apple-HTC Patent Thing
The promise pattern
Automated deployment systems: push vs. pull
JTOpen
Simple
XML serialization and configuration framework for Java
My Solar System
Fun!

What do we use Python for at work?

2010-03-04T16:44:52+00:00

Not uncommonly, I suspect, Python was introduced here at work in stealth mode: it wasn’t on the list of products starting with “MS” which we genreally use, but it got the job done and the management has been pragmatic enough to accept its use to the extent that it’s now installed on the baseline image for company PCs and laptops.

So what do we do with it? Well, a surprising amount when I start to enumerate it. As is often the case, quite a few of the uses are of the “glue” style: creating an easy bridge between two other pieces of software, one of which is often the operating system. As an example I years ago wrote a (tiny) piece of code to enumerate the installed printers and pipe them out to a file. Our in-house business app calls the Python and picks up the result to display to the user as a pick list. That’s just one example among many others, some of which are so small that I tend to forget that they exist until some bizarre corner case arises which means I have to revisit the code. They just work and go on working.

A by no means exhaustive list of Python Pieces off the top of my head:

* That list of printers
* The startup wrapper for our main business app
* sql2xl — provider of data to the masses (and indirectly responsible for a world of Frankensheets, I’m afraid).
* Sales Boards - our Pygame-driven availability-visualisation app
* reports - a compact module combining simple dialogs and sql2xl
* screengrabber - capture parts of the screen to save to the database
* imageviewer - simple pygame-based image display
* convert images provided by our customers to thumbnails and place them on a replicating database for our handheld scanners
* convert a batch of Word docs to PDF & PCL
* simple manipulation of DXF maps to add our internal site ids
* absolutely loads of occasional AD / filesystem / WMI scripts for the sysadmins
* the internal contacts / portraits webpage
* web-based password reset for our HR system
* web front-end, mail ingest and alerting (a la Roundup, I admit) for our Helpdesk system

.. and a whole slew of other stuff which pretty much exists to demonstrate just how versatile Python is :)

Breathing life into the active_directory module

2010-03-04T16:35:18+00:00

For a long time I’ve not been able to give my active_directory module the attention it’s needed, in spite of several people helpfully providing patches to, eg, escape slashes in monikers, bind to the Global Catalog and do other useful things. However… the need to do various things with it at work plus some questions from interested users has resulted in a flurry of housekeeping and — hopefully — improvements. It’s not done yet but, assuming I can sustain the effort, you should see the result in a few days time.

As with much of my stuff, the functionality I implement and the time I spend on it is largely a function of what the sysadmins here at work want me to for them (and how much else I can squeeze in at the same time). Just at the moment there are several initiatives to claw back wasted disc space and do other housekeeping exercises. So I get to allocate some time to my projects. Let’s see how long it lasts…

The all new and improved unittest: unittest2

2010-03-04T13:43:29+00:00

At PyCon I gave a talk on some of the new features that are coming to unittest. unittest is the Python standard library testing framework. ... [346 words]

New Release: ConfigObj 4.7.2

2010-03-04T12:28:07+00:00

A new version of ConfigObj has just been released: 4.7.2 This is a bugfix release with several important bugfixes. It is recommended that all users of ConfigObj 4.7 update. ... [447 words]

The promise pattern

2010-03-04T08:14:35+00:00

After reading my William Reade's post about a small path library he wrote to tame a build script, I thought I'd share a cool trick that has helped me greatly in the current project - the promise pattern.

The code itself is greatly simple:

class FilePromise(object):

    def __init__(self, filename, basepath=None):
        self.basepath = basepath
        self.filename = filename

    def fetch(self):
        filepath = self.filename
        if self.basepath is not None:
            filepath = os.path.join(self.basepath, filepath)
        return open(filepath)

The cool thing about this is not the code itself, but the amazing flexibility it offers your code when instead of working with paths and files, you work with promises. All you need to do to support arbitrary data sources is create a class with a filename attribute (which might not make sense in some cases, but is useful for debugging) and a fetch method that will return a file-like object.

I have so far implemented additional promises that:

Fetch files from a CouchDB database
Decode embedded images in SVG
Crop and resize images (well not yet really, but plan to)

Why not use a path element like William does? In my case, I really do not care about the file system, I just care about contents. A promise also can easily be replaced for testing reasons - just create a small MemoryPromise that will return data passed in the constructor as a StringIO.

Why not pass around a file-like object? Two reasons - it's hard to debug (how do you print that?) but mainly, the fetch method might be expensive, and should be run near the point the contents will be used, not at construction time. Also, the intent of the code becomes clearer this way, I think.

The idea is stolen from Cocoa's drag-and-drop system, where a drag source can either supply the drag data directly to the drop target, or provide a promise the target will follow-up, if it's expensive to provide the data up-front. Hopefully Apple will not sue me. In drag-and-drop a lot of times a drag is initiated but not completed (user changes his mind) so this way an expensive calculation is avoided.

Something I haven't yet considered is how to embed this in an async program. Presumably an AsyncPromise fetch will return a deferred that will fire when the data is ready. The client code needs to be expecting this though. I've seen twisted's maybeDeferred for this kind of pattern, I'll follow up if I can do this.

Comments

Links for 2010-03-03 [del.icio.us]

2010-03-04T08:00:00+00:00

25 Free Must Download Design Programs
Some goodies here

Links for 2010-03-03 [del.icio.us]

2010-03-04T08:00:00+00:00

National stereotypes are thriving – at least in fashion and figure skating

An Awesome PyCon

2010-03-04T00:07:07+00:00

As always PyCon was awesome, I had a great time catching up with old friends and making some new friends. The conference wasn't bad either... ... [876 words]

Links for 2010-03-02 [del.icio.us]

2010-03-03T08:00:00+00:00

To hell with web standards
"Secret W3C member lists? Anonymous holds? What is this, the Senate?

Some might say this is Adobe’s fault.

Bullshit: what possible purpose could secret lists and anonymous holds offer except to allow — no, to condone — this behavior. The process allows this kind of action; I blame the process."
Running Processes
"Run this program, and restart it if it falls over." Nice.
Why Elastic Wings Are The Key To Flapping Flight
Top Ten Java and XSLT Tips
XSLT Processing with Java - O'Reilly Media
Java and XSLT: Chapter 5: XSLT Processing with Java
Call Java methods from within XSLT

Links for 2010-03-01 [del.icio.us]

2010-03-02T08:00:00+00:00

Unit Test Achievements
Expecter Gadget
"Better expectations (assertions) for Python"
Johnny Cache
"Clever twist on ORM-level caching for Django."

Massive improvement on sorting a fat list

2010-02-28T15:00:00+00:00

IssueTrackerMassContainer is a simple Zope product that is used to put a bunch of IssueTrackerProduct instances into. It doesn't add much apart from a nice looking dashboard that lists all recent issues and then with an AJAX poll it keeps updating automatically.

But what it was doing was it recursively put together all issues across all issue trackers, sorting them and then returning only the first 20. Fine, but once the numbers start to add up it can become a vast sort operation to deal with.

In my local development copy of 814 issues, by the use of pympler and time() I was able to go from 7 Mb taking 2 seconds down to using only 8 Kb and taking 0.05 seconds.

[409 more words]

Synchronizing Slides & Audio For A Screencast

2010-02-27T15:28:51+00:00

At a recent ThoughtWorks techy meetup, we recorded a couple of videos for internal use. One of them involved someone speaking to slides. The problem was that we had no projector, and our flip camera wasn’t really up to doing a great job of filming the screen. So the problem I was faced with was the best way to synchronize the audio track from the video to the slides.

What follows is a far from efficient workflow that allowed me to sync the audio track with the slides. The only constraint was my skills, and the fact that I only had freeware software and whatever came with the mac (iLife 08).

0. Convert Flip AVI -> mp4

I’m not sure if I needed to do this, but I figured the OSX tools would play nicer with m4v than with the avi’s from the Flip. I didn’t have the Flip software installed. I used Handbrake to convert the video.

1. Import into GarageBand to rip out the audio track

I probably could of done this via Handbrake, but it was fairly easy to do in GarageBand. I imported the video, selected the video track, and removed it.

2. Convert the Keynote slides to Images

Given that I was in GarageBand, I figured I’d create a podcast, using the Keynote slides as artwork for the podcast. Exporting the slides as JPEG is a simple job – I chose to have Keynote create one image for each stage in a transition. I then imported the JPEGs into Aperture so I could access them via GarageBand’s media browser.

3. Add the artwork to the podcast

You’ll need to make sure the Podcast track and the media browser is visible. Then, it’s a simple matter of dragging in the relevant slide in sync with the audio track. The UI is pretty intuitive.

4. Export the podcast

Now it gets trickier. At this point, GarageBand thinks we have an audio, rather than video, podcast. I tried exporting into iTunes, but just got an audio track – I’m sure the artwork would show up on my iPhone, but I want a movie for uploading to our internal video hosting service (we have Google apps, so get Google Video).

What I ended up doing was exporting to iWeb, exporting out the resulting Podcast site, then grabbing the resulting quicktime movie. This quicktime movie was almost what I wanted – a video, with the audio track from the flip, with synchronized slides. But I wanted to do a couple more things – namely I wanted to split the presentation into two parts (5-10min videos are easier for people to digest than a single 20min presentation), and add some additional titles.

5. Split and add titles

Luckily, iMovie can handle this. I’m fairly sure it was able to directly import the quicktime movie. Adding the titles is a little fiddly, far worse is the simple task of splitting the movie itself. I hope to try iMove 09 soon to see if this is improved.

It was at this point I realised I had an issue – namely that the artwork you embed in podcasts via GarageBand is limited to a square, 300×300 pixel dimensions. This led to some clipping of the slides. If I have to use this workflow again, I’ll make sure to export square slides from keynote in the first place.

6. Export and upload

Which is finally where things got easy.

Notes

The GarageBand project was massive. The two resulting videos from iMovie were less than 15MB – the GarageBand project which just contained the slides and the audio track was over 450MB.
I’d love to find a better option for syncing the artwork – being limited to 300×300 pixels is a pain, especially when I’m uploading to a video hosting service capable of much higher quality widescreen playback.
I like the power of iMovie, but I dislike the UI intensely. I’d love to be able to increase/decrease clip selection using the keyboard for more control.
For some reason Google Video too ages to process the two small videos – significantly larger videos uploaded afterwards were available within a few minutes. It is possible that the small, squashed size of the movie gave the transcoding a problem, or it may just of been a transient glitch, but I ended up having to wait overnight for them to finally be available.
I’d love to know of a better workflow using free/cheap software, so please get in touch if you have any ideas!

Small Python from Synapse is a Big Deal

2010-02-25T06:49:13+00:00

The guys from Synapse didn't appear to be at PyCon this year, but they impressed me last year with their hacking of the "Easy Button" to tweet on Twitter when the button was pressed.

It seems they are still impressing people, as they have just been awarded a SmartGrid.TMCnet.com Product of the Year award. Congratulations to them - I am sure they will be a significant Python user in future.

Links for 2010-02-22 [del.icio.us]

2010-02-23T08:00:00+00:00

jacobian's django-deployment-workshop
Django Deployment Workshop

pycon as seen from across the Tweet

2010-02-22T10:20:42+00:00

That was an interesting experience: I (re)started using Twitter a couple of weeks ago, for no particular reason except that it seemed to be an increasingly significant backchannel of information around Python, meaning that useful information passes through it which is then assumed or referred to in other forums [*] such as the Python mailing lists or people’s blog posts.

But by fortunate coincidence that meant that I was able to watch the #pycon Twitter stream and get a more immediate flavour of the conference than I would have by following later blog posts. Obviously it was quite fun watching people try to get their questions on the screen for Guido’s keynote speech (took me a while to realise what was going on…).

Throughout the conference you get a mixture of the very immediate (”Has anyone got a Mac VGA Adapter for Room C?”), the impending (”BoF Session for Ruby Enthusiasts in the downstairs toilets”), the infinitely retweeted (”Unladen Swallow….”), the humorously overheard (”OH: OK; Yak shaved”), the running-joke (”Bring goat food to the Testing Sprint”), the contentious (”We were more diverse…”), the you-had-to-be-there (”Looks like @djangopony was left unattended”), the gratitude (”Thanks to everyone…”) and of course the plain exhausted (”Back home now after #pycon”).

I obviously didn’t follow everything in real time, not least because of the time difference: I do have other things to do with my time :) But it was enjoyable watching other people enjoy themselves and feeling at least a spectator to an Occasion in the Python year. Now all I’ve got to do is find time to catch up with slides, video presentations, and A Little Bit of Python.

[*] Note to pedants: seems to me that “forum” has become an English word by adoption which means that you can justifiably form its plural by adding an “s”. If I were talking about meeting places of the Ancient World I might argue for the more classical “fora”.

#PyCon on Twitter ... or wait for posts like these.

2010-02-20T02:37:47+00:00

John Graves, a New Zealand reader, wrote in to let me know:

Plug for PyConPads
http://bit.ly/dgBcpA

Plug for PyCon 2010 Slides
http://bit.ly/dASBF7

By the way, New Zealand had a very successful Kiwi PyCon last November:
http://nz.pycon.org/

Thanks, John. Hope this gives the links a little extra publicity.

Async needs syntax support

2010-02-19T20:51:06+00:00

OH: "async won't take hold until it sucks less." It cannot suck less because we are writing async code in languages designed to be procedural, given that you read them top-to-bottom.

I am starting to get more familiar with Twisted and it is indeed very powerful. Even if I don't like it, I will never do any network or async stuff using threads any more. Other libraries may be nicer for web but when you want a variety of protocols Twisted is the only game in town.

The thing is, Python is a procedural language. You write commands top-to-bottom, and they are executed as such. When you want to encapsulate things running in parallel, or at a later time, you put them in a function and register that. There are more things you want to do - block until all results have come in, or fail early if one result fails and so on.

The best way to visualise this is to use some kind of visual support - think arrows flowing around and whatnot - but it will be clunky and hard to implement, and mind you us Vim users will hate it (of course there's going to be an emacs plugin soon ;).

So I don't think async will start to suck less soon, because the basic way we get to interact with it is just the wrong approach. I've no idea what a better approach would be though.

Comments

A Python 3 Idiom?

2010-02-18T00:04:59+00:00

A while ago I wrote a short Python 3 snippet as an introduction to a post about factory functions. Second thoughts are proverbially the best, and I realised that in the following code:

def powers(a):
    return a, a*a, a*a*a

a, square, cube = powers(10)
print(a, square, cube)

the last two lines could be refactored as:

print(*powers(10))

This technique is ineffective in Python 2, where print is a statement and the syntax would be invalid.

Customer Hostility

2010-02-17T07:54:24+00:00

Seth Godin is right: the world is full of commercial dinosaurs, due to expire shortly. And I will be happy to dance on their graves.

I have written before about the rigors of modern day travel, and the travel industry (or a large part of it) appears to be doing its best to discourage people from traveling. Part of it, of course, is the TSA and its associated stupidities, but I will not again go into the irrationality of spending billions of dollars to make what is already the safest mode of transport in the world even safer (assuming that the TSA is indeed living up to that dubious claim, which I frankly doubt -- otherwise why aren't they shouting from the rooftops about the number of plots they have foiled?)

So today, for "convenience" I am flying to Atlanta. To minimize the airport hassle I have already checked in on-line, and paid $23 for the privilege of checking a bag. Assuming the charge on the way back is the same, this will amount to just over 24% of the price I have already paid for the ticket.

I have always disliked unbundling, ever since I argued in Sun against Eric Schmidt's proposal that Sun unbundle the C compiler from what was then SunOS. Since I was a minion and Eric was a big-shot you can guess how that one went (though Sun's eventual fate makes me glad that I sold my stock at $28). I suppose that's why I'm a socialist: while it may not give each user of a service the absolute lowest possible cost, it's much more efficient to spread the load around. If the rich weren't so mean, the poor could enjoy better lives. Sorry, where was I?

Oh, right. So, I have got through the security theater (doubly inconvenient this trip, since I am carrying two laptops) and I decide that, to conserve battery, I will find a venue that allows me to plug my power brick in. What do I find? Each and every establishment in the newly-refurbished Dulles airport appears to have been designed specifically to exclude any possibility that the on-line traveler such as myself might steal a few joules at their expense.

So, for the record, let me squeak this into the ears of any future airport Eric Schmidts who might be reading. If you provide power and wifi (assuming, of course, that your license from the airport doesn't force you to make your customers rely on the airport's crappy facilities) in your establishment, and advertise that fact, I will support your business in preference to the short-sighted idiots who insist on trying to milk me dry whenever I come into contact with them.

So now we come to Verizon Wireless. Having finally found a place quiet enough that I can plug into the power without creating a trip hazard (which, believe me, would be impossible when the airport was anything like busy), and to avoid whatever extortionate rates the airport authority might choose to charge, I plug in my cellular broadband card and fire up the Internet,only to be confronted by a dialog box telling me that I can upgrade. Upgrades are always good, right? Well, yes, but ... here's the dialog box, at the left.

Call me picky if you like, but this seems to me like the ultmate failure in quality assurance. I have to wonder how many pairs of eyeballs scrutinized this dialog box before it went into production, and how come none of the owners of said eyeballs thought to ask the question "What 'Update' button"? I am imagining that each scrutineer, supposing they notice any discrepancy at all between the text in the dialog box and the reality they were presented with, thought to themselves (consciously or otherwise) "Well, clearly they mean the 'Download' button". To which my immediate response is that they cannot possibly be geeks. I am a geek, and believe me I am intimately familiar with all the shit that can land on your head in short order when you make that kind of dumb-assed assumption. So being a geek, I decline to either "Download", or "Update" or whatever kind of trouble it is that Verizon would like to get into.

Next I am presented with an opportunity to "view my account". Now I actually have two accounts with Verizon Wireless, one of which seems to be uncomfortably and inextricably intertwined with my home-based FIOS account. So I am impressed to see both a "register" link and a pulldown that allows me to select "Business Account" rather than "Personal Account". Which I do before clicking the "register" link, whereupon I am presented with a form that asks me for quite a bit of information including both my Employer Identification Number (EIN) and my Dun and Bradstreet number.

Being in an airport, I figure I have done pretty well to produce the former, and I happily fill in the other fields, only to be presented with the message shown at the left. Now, this is telling me a number of things I don't particularly like.

First off, there is something "invalid" about my company name. But of course I am left to guess what; it would clearly be far too easy to provide any description of the rules that are declaring my company name to be invalid.

Next I am informed that the Tax ID is invalid. I discover by reading the form (not the error message) that this is because I have failed to put a dash between the second and third digits, but apparently the programming monkeys who have put this application together have never thought about maybe, hey, just throwing anything that isn't a digit away. Even though only the digits in this particular identifier carry any meaning. Give me a break.

There is a similar problem with the telephone number, though in this case the programmers have taken the trouble to tell me exactly how the number needs to be formatted (thank you!). I especially like the double period at the end of that line, which implies that not only do I have to put dashes after the third and sixth digits, I also have to put a period at the end of it. This reminds me of many interactions where people have asked me my telephone number and I have replied "eight hundred dash four nine four dash nine one one nine". Not. This is getting quite cussworthy, but I am restraining myself.

Now the message about the email address again throws me into confusion, since I have (as a look at the form clearly shows) entered my email address one hundred percent correctly. username@domain.com, just like I always type it. So again I am left to guess exactly what about my email address the Verizon system has chosen to declare "invalid". The f-bomb is now hanging in the air, poised like the sword of Damocles above a web site that has clearly been programmed by monkeys, and not even monkeys with brains.

Lastly, just in case I didn't understand that I should have put that oh-so-important dash between the second and third digits, I am presented with a message that my "Federal Tax ID" was incorrect, as though the form has requested two separate tax IDs (which, by the way, it hasn't).

At this point my almost-forty-three years of programming experience, integrally connected to my geek identity, makes me want to scream "WHAT IS WRONG WITH THESE PEOPLE? DON'T THEY WANT MY [expletive deleted] BUSINESS?". I swear, at this moment, if there was a button I could click with the label "Consign VerizonWireless and all its employees to hell" I would click it whether or not there was the slightest chance that the click would have any positive effect. It would just make me feel better. I am trying to interact with a corporate entity (which due to a strange and to my mind totally obsolescent notion of law has most of the legal rights of an individual) that doesn't give two hoots what its customers think about it or how infuriating its systems are to use.

At this stage I realize I am not complaining just as a programmer, or even just as a web systems designer, I am complaining as a human being. Where, I ask myself, is the VerizonWireless Quality Assurance department in all this? (They must have one: no corporation of that size can possibly have failed to consider the liability implications of not doing). Did anyone actually take the time to interact with this application and deliberately provide "invalid" answers to determine whether the systems response would be considered reasonable by a human being?

I swear, sometimes, I truly believe that top-level corporate managers' ultimate dream is a system that requires no human intervention at all. If this really is the case then I have to explain (and will do so, given access to the Verizon top level management and a sock full of wet sand, in explicit detail) that they are a long way from reaching their goal.

Sure, we all understand (even those of use who aren't lucky enough to hold stock in public corporations) that a publicly-owned business has a duty to be cost-efficient. We also understand that in a capitalist world we are likely to be forced to interact with entities that don't always choose to employ human labor to interact with its human customers. But in the final analysis, if I personally can choose to pay a few percent more to deal with a business that bothers to pay someone to listen to my problems and help me get my confusion sorted out, I will always take that option.

That is why, being the bloody-minded socialist that I am, I usually choose to stand in line at the supermarket checkout rather than take the option of the "self-service" checkout. I don't want to serve myself. I want the companies I deal with to pay someone to help me, and I want to be sure that my own business always presents a human face to its human customers. All Holden Web's customers are corporations, but it's their employees that I have to satisfy.

So, if you happen to be a Verizon Wireless stockholder, please ask some awkward questions at the next stockholders meeting. This laziness should be stomped on quickly, and hard.

If you happen to be in the IT industry, please make sure you don't phone it in by producing this kind of shabby excuse for a human interface.

And now I am in Atlanta, and things feel better.

Software QA Rant

2010-02-17T07:43:39+00:00

Don't want to publish the whole thing on the Python feed, but this post applies equally to Python users if you have the time to read it ...

QCon London

2010-02-16T21:12:20+00:00

I’ve been invited to speak on colleague Chris Read’s track at QCon London this March. The track itself is chock full of a number of experienced proffesionals (including two ex-colleagues) so I fully intend to raise my game accordingly. We’re lucky enough to have Michael T. Nygard speaking too, author of perhaps the best book written for software developers in years in the form of Release It!

The track – “Dev and Ops – a Single Team” – attempts to address many of the issues software professionals have in getting their software live. It will cover many aspects, both on the hardcore technical and on the softer people side. Hopefully it will provide lots of useful information you can take back to your own teams.

My talk – From Dev To Production- will be giving an overview of build pipelines, and how they can be used to get the whole delivery team focused on the end objective – shipping quality software as quickly as possible. It draws on some of my recent writing on build patterns, and a wealth of knowledge built up inside ThoughtWorks over the past few years.

My experience of QCon SF last year was excellent – I can thoroughly recommend it to any IT professional involved in shipping software. If you haven’t got your ticket already, go get them now before the prices go up!

Struggling with Test Driven Clojure

2010-02-16T20:56:50+00:00

I’ve recently been working on a Clojure application that I hope to open source soon. It’s been my first experience of using Clojure, and is almost certainly one of the most thought provking things I’ve done in a long while. One of the things that is still causing me issues is how to go about TDDing Clojure applications – or rather functional programs in general.

My natural inclination – for many reasons – is to use TDD as my process of choice for developing my code. Beyond its use as a design tool, it’s having a saftey net to catch me if I screw something up. It allows me to be a little more brave, and drastically reduces the cycle between changing some code and being happy that it works. I’m used to that saftey net – I feel lost without it.

Stuart Halloway said during his Clojure talk at Qcon SF that despite being a TDD fan he finds it hard to TDD in a new language, and I get exactly what he means. A big part of it is that you’re getting to grips with the idioms, capabilities, libraries and tools associated with your new language – and a lack of this knowledge is going to impact on your ability to write good tests, let alone worry about implementing them.

Typically, when learning a new language I try and write a small application that has a real world need. BigVisibleWall was my attempt to learn Scala – but it had a real goal. With BigVisibleWall, as with my current Clojure project, I started by implementing the system by just writing the production code. I’m pushing the limits of my knowledge constantly, attempting to understand the size and shape of the solution space that I find myself in with this new tool. Once I got BigVisibleWall working with a small set of features, I broke it down and rewrote it TDD style – at that point, I had enough Scala (and I mean *just* enough) to be able to do this without it feeling like I was wading through treacle.

I consciously decided to follow the same pattern with my Clojure project. Code the main logic, get it running, then break it down and rewrite it piece by piece using TDD. But then I hit a problem – Scala and Java are similar enough languages that my programming style didn’t have to change much from one to the other. Therefore the way I structured the code and thought about TDD didn’t have to shift much. In both cases I was driving the design of an Object Oriented system. With Clojure though it wasn’t just the language which was different, it was so many of the underlying concepts were different. Put simply, I really don’t know where to begin.

My first instinct is to start decomposing functions, passing in stubs to the functions under test. But this just feels like I’m trying to shoehorn IOC-type patterns into a functional program. But what am I left with – testing large combinations of functions? That feels wrong too.

So what about you lot out there in blogland? Any other OO types trying to make the switch and encountering the same issues? Or any FP practitioners for whom TDD is second nature? Or does TDD just not fit with FP after all?

Some questions about the "blocking" of HTML5

2010-02-16T09:11:21+00:00

When people say that the publication of HTML5 "blocked" by Larry Masinter's "formal objection", what exactly do they mean?
Why does the private w3c-archive mailing list exist? Why can't anyone reveal what happens on there? What are the consequences for doing so? Who gets to be on that list in the first place?
Can anyone raise a "formal objection"?
Is anyone calling for the HTML Working Group to be "rechartered"? If so, what does that involve?
If there are concerns about the inclusion of Canvas 2D in the specification, why were these not resolved earlier?

Some background reading. I was planning to fill in answers as they arrive, but I screwed up the moderation of the comments and got flooded with detailed responses - I strongly recommend reading the comments.

Clojure on App Engine – my take

2010-02-14T12:32:23+00:00

I’ve been working on a couple of spare time projects, both of which I hope to release more formally in the next few weeks. One of them involves development of a simple web application for deployment on Google App Engine. As part of the development, I had to modify an existing open source Clojure API – my changes are now available for all.

appengine-clj was written by ThoughtWorks colleague John Hume. It provides some Clojure-esque wrappers over Google App Engine’s user authentication and low level datastore API. John outlines his use of the library in a highly useful post on using Compojure & Clojure on the App Engine – it was this post which helped immensely in getting started myself.

There were a couple of minor issues with the latest version of John’s API which stopped me from being able to use it for my latest project – so I created a fork to make the changes I wanted. First, a general issue. I love projects which make checkout and build easy and bullet-proof. For me, that means check in the build tool & all dependencies. I know this is a contentious point – I may well write a post on it later. The other issue is that since the 1.2 version of the SDK some of the APIs have changed a little, so I updated the datastore testing macro accordingly.

My Clojure skills are highly limited, and the modest modifications are probably botched, but nonetheless it seems to work. My fork can be found on GitHub.

New Python Class Available

2010-02-12T17:32:30+00:00

I am happy to announce that the O'Reilly School of Technology have made the first of four Python classes available on their interactive teaching system. Students passing all four modules will be eligible to receive a Certificate for Professional Development from the University of Illinois' Office of Continuing Education.

The PSF has discussed certification schemes in the past, and the general feeling of the membership appeared to be against them. This may be true for high-level personnel and those with advanced degrees, but we should remember that Python will also be increasingly used by technicians and others whose formal educational background does not include a training in computer programming. I feel that a university-approved certification will be helpful at that level.

While many certifications aren't that meritorious, if a scheme requires proper practical experience and the training includes mentoring by live instructors or advisers rather than just the passing of a multiple-choice examination it can have some real value.

Time will tell, but I believe we will have passed a milestone once a university-accredited Python certification scheme is available. All I have to do now is write the other three courses ...

[NOTE: the author of this blog will receive royalties based on the number of students taking the classes mentioned in this article]

Choose Python

2010-02-12T10:27:05+00:00

Ethan Furman linked to this "Choose Python" poster on comp.lang.python. While some of the humor might not mean much to people who haven't chosen Python yet it certainly gave me a smile on a cold, grey morning.

[EDIT: as you can see in the comments design credit should go to Tim Lesher]

Ubuntu 9.10 Wireless on Dell Precision M6300

2010-02-12T10:16:49+00:00

For reasons too complex to go into (they involve a talking penguin) I have just installed Ubuntu 9.10 (Karmic) 32-bit on the laptop I was talking about when I wrote this post last June.

Whether it's the change to 32-bit or the change to Karmic I do not know, but I am happy to report that everything works like a dream: I just had to install the BroadCom proprietary driver (boo! - I wish companies like Nvidias and BroadCom would cooperate more fully with the open source world) and suddenly I could disconnect the wires and away I went. This post is being made from the very laptop about which I complained so bitterly eight months ago.

The future is tree shaped

2010-02-10T16:22:16+00:00

Have been thinking and reading more about parallelism recently. This set of slides from Guy Steele distilled a lot for me.

In order to realise parallel hardware performance we need to optimize our programs for computational bandwidth rather than latency. In terms of programming this means deprecating accumulation (cons, fold, streams, sequences) and favouring divide-and-conquer. This suggests a move to trees as the fundamental abstract building block for data.

Fix for ConceptNet error “Settings cannot be imported, because environment variable DJANGO_SETTINGS_MODULE is undefined”

2010-02-10T03:11:29+00:00

If you’re using ConceptNet and you see:

ImportError: Settings cannot be imported, because environment variable
DJANGO_SETTINGS_MODULE is undefined.

then the fix is simple (I’ve been hacking away at an idea whilst at IUI2010 – thanks Rob for the fix).

To replicate the error run:

from csc.nl import get_nl

en_nl = get_nl('en')

en_nl.is_stopword('the')

The fix is to run:

import csc.conceptnet.models

which sets up Django, the call is_stopword again and all is fine.

Ian produces professional screencasts (ProCasts), writes The Screencasting Handbook, programs Python, researches Artificial Intelligence (Mor Consulting) and is also a sea-side dweller and consumer of fine coffees.

One More Day to Register for PyCon on the Web

2010-02-09T19:47:23+00:00

Incredible though it seems, PyCon tutorials start a week tomorrow!

If you are planning to attend and want to register at pre-conference prices you have just over 24 hours to do so!

PyCon is the place to learn about new and existing Python features and meet Python personalities. Register now while you still have chance to save money!

A Little Bit of Python Episode 4: A Pre-PyCon Special

2010-02-08T00:13:48+00:00

A Little Bit of Python is an occasional podcast on Python related topics with myself, Brett Cannon, Jesse Noller, Steve Holden and Andrew Kuchling. The website is in progress and apparently nearly ready, thanks to Jesse and various other people who we will thank as soon as it is done. ... [233 words]

ConfigObj 4.7.1 (and how to test warnings)

2010-02-07T23:52:41+00:00

I hate doing releases. I haven't managed to automate the whole process (I should probably work on that), although setup.py sdist upload certainly helps. ... [290 words]

Discover 0.3.2 and the load_tests protocol

2010-02-07T23:25:47+00:00

discover is a test discovery module for the standard library unittest test framework. Test discovery is built into unittest in Python 2.7 and 3.2. ... [335 words]

Another Day, Another Dojo

2010-02-05T09:57:36+00:00

My first Dojo of 2010 yesterday, as I missed the January one (which is a shame as it sounded like fun). This was a return to the conventional Dojo style with pair-programming at the front, the task being to merge the Adventure Game efforts from the previous month’s team-by-team effort. But first, a commercial break… Jonathan Hartley advertised the upcoming PyWeek competition and demonstrated his team’s previous effort as well as the amusingly literal Murder of Crows entry.

Meanwhile, back at the Dojo… Nicholas started off partly to set things up for people (like me, his co-pilot) who hadn’t been there the month before. As we moved through the programming pairs, we did manage to get a working codeset together by merging the location-parsing code from one team with the cmd-based loop from another. But it was clear that things were moving slowly and that the audience was somewhat disengaged…

So we finished off with a broad discussion of ways ahead, of what might work better, and of whether the artificial nature of the Dojo setting meant that people couldn’t “show off”, so to speak, their natural coding style — and ability. Michael G and others made a few cogent suggestions to the effect that smaller teams would work better but with some kind of DVCS (git seemed to be the front-runner) to assist the teams in collaborating. And it seemed that at least one team should be creatives, rather than coders.

I enjoyed it, as ever, but it _is_ difficult to keep 20+ people in a medium-sized room totally engaged. Even with the best will in the world, it’s hard to read code on a screen from some distance back. This was one of the reasons why a team-based approach met with broad approval. Also it is a little dispiriting when you’re doing your best up front but half your “audience” is otherwise engaged.

The point about how “natural” you should be in the Dojo was interesting. Everyone has a different way of coding, of approaching a problem, of looking for information and so on. Dave[*], the last pilot, mentioned that he’d normally take much longer sizing a problem up but that having a time limit forced him into action sooner. But you’re up there with a limit of 10 minutes, probably on someone else’s machine, maybe on an “alien” operating system. You’re fumbling with running up a command prompt; you’re not sure how to bring up the Python docs; the keyboard shortcuts which your fingers remember don’t work — or worse, do something else entirely! We did discuss each person bringing up his own laptop, although that has obvious drawbacks of getting the projector to switch smoothly and so on. It was out of this discussion that the move towards a team-based approach next time arose.

TJG

[*] Dave, incidentally, was the programmer of the BBC version of the Graphics Adventure Creator — did you know that?

It Certification Considered Useless

2010-02-02T21:38:16+00:00

A few years ago, I was sharing a drink with a friend of mine. He was about to become a fully qualified architect. In the UK, one cannot call themselves an architect without having carried out the full, three part course, which takes at least seven years. Typically, as the course involves working in the industry, architects often took more than seven years to complete their ‘part three’ – my friend completing it in the minimum possible time made him one of the youngest qualified architects in the country. As he was about to be fully qualified, he was explaining the need to get indemnity insurance, as his opinion as a qualified architect made him liable for the quality of advice given, even advice given informally down at the pub.

There has been a short history of various individuals, companies and professional bodies within IT attempting to define and issue certification. By and large, they have not caught on. There is no belief that software delivered by ‘certified’ individuals is any better than that developed by uncertified individuals. Nor is there any evidence that in terms of getting jobs that certification counts for anything other than in specific, narrow (mostly vendor specific) technical domains – something that few serious software professional would consider worthwhile.

So, in general, why does certification exist? Societal pressure determines where certification is essential. It is important that key individuals in positions of power are properly vetted – and recognised – for the role they play in society. Which is why there are laws governing who can call themselves a lawyer, architect, engineer, surveyor. Which is why certified profesionals have responsibility placed upon them regarding the veracity of information and quality services they provide as a member of that profession.

In terms of architecture – in similar terms to medical doctors for example – society has deemed the roles they play as being important enough that certification carries with it legally enforceable expectations regarding their competency. With this responsibility, comes recognition – and a clear understanding as to how the profession is valued by society as a whole.

Certification in the land of IT is not being driven by a need for society to ensure that we are doing our jobs properly – to ensure that only competent individuals call themselves ‘programmers’, ’sysadmins’ or whatever. Nor is it being driven by a societal desire to recognise our contribution to society as as a whole. It is being driven by IT itself – at best as a misguided attempt to recognise an ability in a certain set of skill, at worst as a way of generating money. As such, certification in the world of IT is a toothless concept, lacking in any sense of legitimacy, and distracts us from the more worthy goal of understanding how we contribute to the world around us, and how we grow competency to the point where we can even consider ourselves a profession at all.

Interview with Pardus Linux

2010-02-02T10:28:52+00:00

I recently did an interview on Python with the Pardus Linux magazine. Pardus Linux is a distribution developed in Turkey (by the Turkish National Research Institute of Electronics and Cryptology) with the goal of being usable by "normal" people rather than just geeks. ... [3716 words]

Thief Level : Week 2

2010-02-01T13:20:31+00:00

This week I spent a little while considering the backstory of the level, and now have at least a rudimentary scenario: Garrett the thief is taking an opportunistic foray into the local Shope of Curiosities, having heard that their prize exhibit, the McGuffin of Antioc, has been removed from its high-security public display, in order to be cleaned or maintained somewhere on-site.

On top of this, I’ve been refining the layout of the museum building, starting with the two-storey entrance hall, complete with a balcony running round it:

Plus, I’ve been planning the possible routes a thief might take to get from one room to another. Generally, the conventional paths – in through the main entrance and up the stairs and down the corridor – will be blocked by guards. So the player has to clamber up the outside of the building, explore the roof, dangle from a rope, pick a lock, find a key in the janitor’s quarters, which opens all the windows, and ledges outside a couple of windows lead somewhere interesting, etc. I don’t want it to turn into a key fetch quest, but at the same time, I don’t want the player to be able to simply waltz all through the whole building. I’ve tried to engineer a single interesting primary route through the building, with the possibility of a few minor variations so players feel like they can exercise some freedom and decision making.

Having done all that, I’m now quite happy that my plans are sufficient to produce a small but adequate level. I’ll aim to get that complete, and any fancy window dressing I can layer on top will be a bonus.

I completed the modelling of all the rooms in the building, and doorframes inbetween them. I applied some quick floorboard textures to differentiate the floors and ceilings from the walls. Here you can see the view from the main entrance, looking into the two-storey entrance hall, with the balcony around it visible up on the next level:

And the view while approaching the top of the stairs, looking down over the balcony. There will be a railing when it’s done:

Thanks heaps to Qolelis for a comment with a tip about textures on stairways, to rotate the texture 90 degrees on each stair’s vertical rise. I only just saw that, but will definitely apply it this week.

Update: I feel a bit self-concious that I’m creating the bare minimum that could qualify to be a Thief level. There is not yet a lively, engaging backstory to the level, complete with colorful characters, cleverly intertwined with the canon of the original game.

Similarly, the mechanical contents of my level are as simple as possible. I haven’t stretched myself, thinking of imaginative locations or motives for Garrett to explore. I do not plan to have any clever special objects or custom scripting in my level, defining dramatic changing mission objectives as the player reveals new information. It’s a very straightforward ‘get into a building, steal the loot, and get out’.

Partly this is very deliberate – I want the level to be as minimal as it can possibly be, so as keep it achievable. But also, this is partly in response to my feeling that being creative is hard, especially when under pressure. Right now I feel as though I have enough to worry about just getting to grips with the minutia of the level editor. I almost feel as if I need to become comfortable with that before I can relax enough to get creative with it.

This isn’t entirely unexpected. Clearly one cannot do great work on one’s first attempt. But at the same time, I don’t want to just ‘give up’ on the creative aspects. I want to do as good a job as I can do, under the constraints of a small, straightforward ‘first time’ level done in a reasonable timeframe. So maybe I just need to keep iterating. Embelish the dramatic backstory little by little, see what occurs to me as I go on. Look for some flash of inspiration as I bury myself in the process. Fair enough. Baby steps.

Update 2: I created a quick TODO list, as a first approximation of how much work there is to be done. I ended up with a list of 67 mandatory items (eg. Add doors inside each door frame; First pass at lighting; Add balcony railing.) In addition I have 18 optional items (eg. Add carpets and rugs; Hide moss arrows in the garden; Entrance hall main exhibit.) The screenshots above represent about six completed items (eg. Dromed tutorials; first floor rooms; doorways and arches between rooms; staircase.) So at the current rate, it’s roughly 28 weeks of work, which is double or triple what I’d planned on. Hopefully my rate of completing items will increase substantially as I get into the groove. I’ll have to monitor this going forward, and slash scope if I can’t drastically accelerate.

A Little Bit of Python Episode 3

2010-01-30T21:00:06+00:00

A Little Bit of Python is an occasional podcast on Python related topics with myself, Brett Cannon, Jesse Noller, Steve Holden and Andrew Kuchling. We still don't have our own website although that is due to land any day now. ... [172 words]

TypeError: object.init() takes no parameters

2010-01-30T19:07:00+00:00

At my employer we are in the process of migrating from Python 2.4 to 2.6. When running some existing code under Python 2.6 we started getting DeprecationWarnings about "object.__new__() takes no parameters" and "object.__init__() takes no parameters".

A simple example that triggers the warning:

class MyClass(object):

    def __new__(cls, a, b):
        print 'MyClass.__new__', a, b
        return super(MyClass, cls).__new__(cls, a, b)

    def __init__(self, a, b):
        print 'MyClass.__init__', a, b
        super(MyClass, self).__init__(a, b)

obj = MyClass(6, 7)

This gives:

$ python2.4 simple-warning.py
MyClass.__new__ 6 7
MyClass.__init__ 6 7

$ python2.6 simple-warning.py
MyClass.__new__ 6 7
simple-warning.py:5: DeprecationWarning: object.__new__() takes no parameters
  return super(MyClass, cls).__new__(cls, a, b)
MyClass.__init__ 6 7
simple-warning.py:9: DeprecationWarning: object.__init__() takes no parameters
  super(MyClass, self).__init__(a, b)

It turns out that a change to Python for 2.6 (and 3) means that object.__new__ and object.__init__ no longer take arguments - a TypeError is raised when arguments are passed. To avoid breaking too much pre-existing code, there is a special case that will cause a DeprecationWarning instead of TypeError if both __init__ and __new__ are overridden. This is the case we were running into with our code at work.

The reason for this change seems to make enough sense: object doesn't do anything with arguments to __init__ and __new__ so it shouldn't accept them. Raising an error when arguments are passed highlights code where the code might be doing the wrong thing.

Unfortunately this change also breaks Python's multiple inheritance in a fairly serious way when cooperative super calls are used. Looking at the ticket for this change, this issue was thought about but perhaps the implications were not fully understood. Given that using super with multiple inheritance is common and "correct" practice, it would seem that this change to Python is a step backwards.

rst_break plugin for PyBlosxom

2010-01-30T15:29:00+00:00

I just scratched an itch by writing a small plugin for PyBlosxom that allows the rst (reStructured Text) and readmore plugins to work together [1]. It defines a reST "break" directive which gets transformed into the breakpoint string the readmore plugin looks out for. This allows for "Read more..." breaks to be inserted in for reST based articles.

For further information see the Code page here and at the top of the plugin itself.

[1]	Yes, the audience for this plugin is probably tiny!

Howto bundle binary dependancies with py2exe, et al.

2010-01-27T22:26:00+00:00

Hey. I notice that py2exe-users has a recent question that seems to be about how to correctly bundle the required Microsoft C runtime DLL with executables generated from Python scripts.

Last month I updated the py2exe wiki tutorial to cover this issue, as best as I was able. Since people are still asking questions about it, I figured I’d promote that change a little.

Check out the new, revamped py2exe tutorial page!

That is all.

iPad Name Already in Use

2010-01-27T14:41:24+00:00

One thing that Apple appears to have lost sight of in all the thrills of announcing their new tablet computer is that Widgetaria, presumably an Apple business partner, already promotes software called iPad. I wonder if they care?

pyCUDA on Windows and Mac for super-fast Python math using CUDA

2010-01-26T13:01:22+00:00

I’ve just started to play with pyCUDA which lets you run parallel math operations on a CUDA-compliant NVidia graphics card through Python.

CUDA stands for Compute Unified Device Architecture – it is an architecture that lets us program the Graphics Processing Unit (GPU) on a high powered graphics card to do scientific or graphical math calculations rather than the usual texture processing for games. In essence it is a mini supercomputer that is specialised just for fast math operations – if you can figure out how to use it.

The goal is to off-load the CPU-intensive calculations for two of my clients (a physics company and a flood modelling company) to achieve 10* to 100* speed-ups using commodity graphics cards.

pyCUDA makes it easy to interactively program a CUDA device rather than hitting C++ code with the slow write/compile/debug loop. Recent MacBooks (mine was bought in January 2009) have NVidia cards with CUDA-compatible devices built-in (mine is a 9400M). For my desktop computer I have a 9800 GT (costing £100).

It turns out that this is bleeding-edge stuff – getting pyCUDA compiled on my MacBook and Win XP machine took some time (forum posts for Mac and Windows issues) thankfully the group is helpful and the wiki has an installation section for Windows, Mac and Linux and some reasonable documentation.

Right now I’ve got as far as running some of the demo code on my MacBook (showing a 5* speed-up over the CPU) and my desktop (showing a 30* speed-up over the CPU). I’ll report more as I progress.

Update – pyCUDA works inside IPython too, lovely.

Update – I don’t have OpenGL working for gl_interop.py but as noted here you need “CUDA_ENABLE_GL = True” in siteconf.py and you need PyOpenGL installed. When rebuilding my MSVC threw a hissy fit, it isn’t essential to my work so I’m skipping this demo.

Update – I’ve submitted a patch and two examples to the wiki (SimpleSpeedTest, Mandelbrot). I get 200* speed-ups on the speed test (using a for loop on a sin() calculation) and 5 to 20* speed-up on Mandelbrots (it seems to scale very well vs numpy with increasing dimensions).

Update – There are lots of interesting papers for CUDA surfacing like this one showing a 3* speed-up for voice recognition tasks (using CPU and GPU together) and yet another way to improve fluid dynamic simulations. This Tom’s 3D article gives a great write-up (starting with the history of audio cards) on where 3D is right now and how NVidia is beating ATI for scientific computing.

Creating a Level for Thief 2

2010-01-25T13:50:26+00:00

After Sinister Ducks, my videogame creation mentor suggested that I create a mod for an existing game, in order to distance myself a little from the programming aspects of creating a game, and instead spend a little time considering the gameplay and the art and the music from the perspective of the user. Sounds like useful advice.

So, the last couple of weeks I’ve been working through the tutorials for DromEd, the notoriously cranky level editor for vintage sneak-em-up Thief: The Dark Project. (Specifically for the sequel, Thief 2: The Metal Age, which has a slightly improved engine and editor.) I chose this for three reasons:

Released in 1998, Thief is old enough that the assets are simple low-fidelity geometry and bitmaps. These are easy enough for me to create and edit, plus if I intersperse existing game assets with my own shoddy creations, there won’t be a tremendously jarring disparity in apparent quality.
Even though Thief is ten years old and the company that created it long gone, there’s still a thriving community of amateur afficionados, churning out new missions at the rate of several per month, many of which are of exceedingly high quality – in some cases exceeding that of the original game. I’ll be in good company, will have some meaningful feedback, and will have forums to turn to when I get into difficulties.
Last but not least, Thief is one of my favourite games of all time. The emphasis on sneaking around and avoiding confrontation suits my sensibilities. Your protagonist, Garrett, is a marvellous, mercenary character. Best of all, in Thief 1, it reveals unexpected depth halfway through – the player’s expectations of a succession of simple heists takes a strange twist when the powerful storyline reveals itself.

Having finished the tutorials, this weekend I broke ground on creating my own level, or ‘fan-mission’ (FM), in the parlance:

I’m using the DromEd Toolkit, which is DromEd with some third-party patches and bugfixes applied to it. My first impressions with DromEd are that it’s very clunky and ugly, and startlingly lacking in documentation. I’ve taken to dipping into the configuration files to see what keyboard commands exist to experiment with. There are a bewildering variety of binary patches to modify the executable in various exciting ways, and forum posts about it, although helpful and prolific, seem fragmentary and rife with broken links. I’ve still no idea whether I ought to be using Dromed Delux instead, nor where I should get that from. It’s a glorious chaotic riot, and it’s a little intimidating.

Still, having said that, the binary patches have all worked fine for me, and the more I use the editor, the more it’s starting to grow on me.

I’m setting my FM in a museum. There’s already an existing museum mission out there, but as opposed to its marble-halled austerity, I’m imagining this will be more like the cramped, cosy, wood-panneled chaotic collection of something like the wonderful Sir John Soane’s Museum in London.

So progress as of week 1 looks like this: I’ve carved out some very basic geometry to form a stocky museum building. Here you can just about make out a hole in the brick facade that will form the front entrance. This is not the entrance that the player will likely be using:

This is all very crude thus far – with repeating textures on large surfaces, and plain uniform lighting. There are a complete set of mostly rectangular ground-floor rooms, with interconnecting doorways. The highlight of my modelling to date is this stairway leading up to the (otherwise nonexistant) next floor.

I’m not so happy with the wood texture I chose – I’ll go back and look for something more uniform. But I am happy with the way the stairs flair out at the bottom. I realised in the process of creating this that this makes it possible to fit a flight of stairs into a smaller space than would otherwise be possible, by allowing the bottom few steps to gracefully project out into the corridor.

Having finished this last night, I then dreamed about geometric operations on three dimensional spaces, which I think is a good sign.

Build Pattern: Chained Continuous Build

2010-01-24T18:22:36+00:00

One of the problems quickly encountered when any new team adopts a Continuous Build is that builds become slow. Enforcing a Build Time Limit can help, but ultimately if all of your Continuous Build runs as one big monolithic block, there are limits to what you can do to decrease build times.

One of the main issues is that you don’t get fast feedback to tell you when there is an issue – by breaking up a monolithic build you can gain fast feedback without reducing code coverage, and often without any complex setup.

In a Chained Continuous Build, multiple build stages are chained together in a flow. The goal is for the earlier stages to provide the fastest feedback possible, so that build breakages can be detected early. For example, a simple flow might first compile the software and run the unit tests, with the next stage running the slower end-to-end tests.

With the chain, a downstream stage only runs if the previous stage passed – so in the above example, the end-to-end stage only runs if the build-and-test stage passes.

Handling Breaks

As with a Continuous Build you need to have a clear escalation process by which the whole team understands what to do in case of a break. Most teams I have worked with tend to stick to the rule of downing tools to fix the build if any part of the Continuous Build is red – which is strongly recommended. It is important that if you do decide to split your continuous build into a chain that you don’t let the team ignore builds that happen further along the chain.

Build Artifacts Once vs Rebuild

It is strongly suggested that you build the required artifacts once, and pass them along the chain. Rebuilding artifacts takes time – and the whole point of a chained build is to improve feedback. Additionally getting into the habit of building an artifact once, and once only, will help when you start considering creating a proper Build Pipeline (see below).

And Build Pipelines

Note that a chained build is not necessarily the same thing as a Build Pipeline. A Chained Continuous Build simply represents one or more Continuous Builds in series, whereas a Build Pipeline fully models all the stages a software artifact moves from development right through to production. One or more Chained Continuous Builds may form part of a Build Pipeline, and a simplistic Build Pipeline might not represent anything other than Chained Continuous Builds, but Build Pipelines will often incorporate activities more varied than compilation or test running.

Fast Feedback vs Fast Total Build Time

One thing to note is that by breaking a big build up into smaller sections to improve fast feedback, counterintuitively you may well end up increasing overall build time. The time to build and pass artifacts from one stage to another adds time, as does dispatching calls to build processes further down the chain. This balance has to be considered – consider being conservative in the splits you make, and always keep an eye on the total duration of your build chain.

Tool Support

Tooling can be complex. Simple straight-line chains can be relatively easily build using most continuous build systems. For example a common approach is to have one build check in some artifact which is the trigger point for another Continuos Build to run. Such approaches have the downside that the chain isn’t explicitly modelled, and reporting of the current state of the chain ends up having to be jury rigged, typically through custom dashboards. More complex still is dealing with branching chains.

Continuous Build systems have got more mature of late, with many of them supporting simple Chained Continuous Builds out of the box. TeamCity, Hudson and Cruise and others all have some form of (varying) support. Cruise probably has the best support for running stages in parallel (caveat: Cruise is written by ThoughtWorks, the company I currently work for), and has some of the better support for visualising the chains, but given the way all of these tools are moving expect support in this area to get much better over time.

IMAPClient 0.5.2

2010-01-24T17:58:00+00:00

IMAPClient 0.5.2 has just been released. This release fixes 2 bugs (#28 and #33). Much thanks to Fergal Daly and Mark Eichin for reporting these bugs.

Install from the tarball or zip or upgrade using easy_install or pip.

Register for PyCon, or the Kitten Gets It

2010-01-22T21:14:47+00:00

Just sayin', it might be better if you planned to attend the conference.

It's not that I like harming innocent small furry creatures, it's just that there are still thousands of Python users and potential Python users who still don't know what excellent value for money the conference is, or how much fun you can have there.

PyCon is in Atlanta this year, and despite the parlous state of the world economy there's a chance that it will be the biggest Python event ever. But hey, we all know that size isn't everything.

Although I was notionally on the program committee, and did at least register my opinions of the talks I was allocated by the submissions scheme, I (yet again) didn't manage to make a single meeting due to pressure of other business. I'm actually not too unhappy about that, given the incredibly difficult task that the committee had to perform in order to ... well, I was going to say "sort the wheat from the chaff", but in fact there was so little chaff that wouldn't really be an appropriate analogy.

A number of people have blogged already about Five PyCon Talks You Must Not Miss, but since there's still a few hours to get in at the early bird rate I thought I'd throw out my list of unmissables from the extensive list of talks. How's that for arrogance?

A Short Pinax Tutorial, Danny Greenfeld. I have heard Danny speak at local Python user groups. If I had time I would be attending the half day Pinax tutorial that he and James Tauber are giving, but this is the second best way to find out how to get started with Pinax.

What Every Developer Should Know About Database Scalability, Jonathan Ellis. This one will be straight from the horse's mouth - it's always worth hearing Jonathan's summaries of his immense practical experience.

IronPython Tooling, Dino E Viehland. As a Windows user who has hardly touched IronPython so far I am interested in finding out what my options are. Nobody is likely to know better than Dino, who is probably the most prominent member of the Microsoft development team.

Scaling Python Webapps from Zero to 50 Million Users - A Top-Down Approach, Jinal Jhaveri. Although I haven't heard Jinal speak before I can't resist the lure of hearing what someone with really high-volume Python web experience has to say.

Why Not Run All Your Tests All the Time? A Study of Continuous Integration Systems, Titus C Brown. Ever since I hear Titus' tutorial (with Grig Gheorghiu) on testing a couple of year's ago I've wanted to hear what he has to say about CI. This is my chance.

Alas, five talks isn't anywhere near enough to encompass everything I want to hear, and I am also keenly anticipating Turtles All The Way Down: Demystifying Deferreds, Decorators, and Declarations, Pay Only for What You Eat: A Tour of the Repoze.BFG Repository and Philosophy, On The Subject Of Source Code, Python's Dusty Corners, Debating 'til Dawn: Topics to Keep You Up All Night, Powerful Pythonic Patterns and Tests and Testability. This is going to be a knockout conference even before you consider all the amazing things that will be happening in Open Space.

You really have to be there. And you'll be helping to save a poor kitten that never did anyone any harm in its life!

Planet London Python

Links for 2010-03-09 [del.icio.us]

Links for 2010-03-09 [del.icio.us]

Encoding json on Silverlight with System.Json

Speed test between django_mongokit and postgresql_psycopg2

Links for 2010-03-08 [del.icio.us]

Links for 2010-03-08 [del.icio.us]

How and why to use django-mongokit (aka. Django to MongoDB)

Links for 2010-03-07 [del.icio.us]

Links for 2010-03-06 [del.icio.us]

Links for 2010-03-05 [del.icio.us]

Fun and (adventure) games at the London Python Dojo

Links for 2010-03-04 [del.icio.us]

Links for 2010-03-04 [del.icio.us]

What do we use Python for at work?

Breathing life into the active_directory module

The all new and improved unittest: unittest2

New Release: ConfigObj 4.7.2

The promise pattern

Links for 2010-03-03 [del.icio.us]

Links for 2010-03-03 [del.icio.us]

An Awesome PyCon

Links for 2010-03-02 [del.icio.us]

Links for 2010-03-01 [del.icio.us]

Massive improvement on sorting a fat list

Synchronizing Slides & Audio For A Screencast

0. Convert Flip AVI -> mp4

1. Import into GarageBand to rip out the audio track

2. Convert the Keynote slides to Images

3. Add the artwork to the podcast

4. Export the podcast

5. Split and add titles

6. Export and upload

Notes

Small Python from Synapse is a Big Deal

Links for 2010-02-22 [del.icio.us]

pycon as seen from across the Tweet

#PyCon on Twitter ... or wait for posts like these.

Async needs syntax support

A Python 3 Idiom?

Customer Hostility

Software QA Rant

QCon London

Struggling with Test Driven Clojure

Some questions about the "blocking" of HTML5

Clojure on App Engine – my take

New Python Class Available

Choose Python

Ubuntu 9.10 Wireless on Dell Precision M6300

The future is tree shaped

Fix for ConceptNet error “Settings cannot be imported, because environment variable DJANGO_SETTINGS_MODULE is undefined”

One More Day to Register for PyCon on the Web

A Little Bit of Python Episode 4: A Pre-PyCon Special

ConfigObj 4.7.1 (and how to test warnings)

Discover 0.3.2 and the load_tests protocol

Another Day, Another Dojo

It Certification Considered Useless

Interview with Pardus Linux

Thief Level : Week 2

A Little Bit of Python Episode 3

TypeError: object.__init__() takes no parameters

rst_break plugin for PyBlosxom

Howto bundle binary dependancies with py2exe, et al.

iPad Name Already in Use

pyCUDA on Windows and Mac for super-fast Python math using CUDA

Creating a Level for Thief 2

Build Pattern: Chained Continuous Build

Handling Breaks

Build Artifacts Once vs Rebuild

And Build Pipelines

Fast Feedback vs Fast Total Build Time

Tool Support

IMAPClient 0.5.2

Register for PyCon, or the Kitten Gets It

TypeError: object.init() takes no parameters