Planet London Python

A Little Bit of Python Episode 4: A Pre-PyCon Special

2010-02-08T00:13:48+00:00

A Little Bit of Python is an occasional podcast on Python related topics with myself, Brett Cannon, Jesse Noller, Steve Holden and Andrew Kuchling. The website is in progress and apparently nearly ready, thanks to Jesse and various other people who we will thank as soon as it is done. ... [233 words]

ConfigObj 4.7.1 (and how to test warnings)

2010-02-07T23:52:41+00:00

I hate doing releases. I haven't managed to automate the whole process (I should probably work on that), although setup.py sdist upload certainly helps. ... [290 words]

Discover 0.3.2 and the load_tests protocol

2010-02-07T23:25:47+00:00

discover is a test discovery module for the standard library unittest test framework. Test discovery is built into unittest in Python 2.7 and 3.2. ... [335 words]

Links for 2010-02-05 [del.icio.us]

2010-02-06T08:00:00+00:00

A new way to deploy web applications
Ian Bicking's new webapp deployment tool; toppcloud.
Become a Gmail Master Redux
Generating Sequences of Primes in Conway’s Game of Life
Movist
VLC alternative for Mac
What's hot? Introducing Zeitgeist
“When something appears on the Zeitgeist page, it’s because it performed better (got more attention) than the norm for that content type/section/day”. The application itself is written in Python and runs on Google App Engine.
Evolution of a Python programmer

Another Day, Another Dojo

2010-02-05T09:57:36+00:00

My first Dojo of 2010 yesterday, as I missed the January one (which is a shame as it sounded like fun). This was a return to the conventional Dojo style with pair-programming at the front, the task being to merge the Adventure Game efforts from the previous month’s team-by-team effort. But first, a commercial break… Jonathan Hartley advertised the upcoming PyWeek competition and demonstrated his team’s previous effort as well as the amusingly literal Murder of Crows entry.

Meanwhile, back at the Dojo… Nicholas started off partly to set things up for people (like me, his co-pilot) who hadn’t been there the month before. As we moved through the programming pairs, we did manage to get a working codeset together by merging the location-parsing code from one team with the cmd-based loop from another. But it was clear that things were moving slowly and that the audience was somewhat disengaged…

So we finished off with a broad discussion of ways ahead, of what might work better, and of whether the artificial nature of the Dojo setting meant that people couldn’t “show off”, so to speak, their natural coding style — and ability. Michael G and others made a few cogent suggestions to the effect that smaller teams would work better but with some kind of DVCS (git seemed to be the front-runner) to assist the teams in collaborating. And it seemed that at least one team should be creatives, rather than coders.

I enjoyed it, as ever, but it _is_ difficult to keep 20+ people in a medium-sized room totally engaged. Even with the best will in the world, it’s hard to read code on a screen from some distance back. This was one of the reasons why a team-based approach met with broad approval. Also it is a little dispiriting when you’re doing your best up front but half your “audience” is otherwise engaged.

The point about how “natural” you should be in the Dojo was interesting. Everyone has a different way of coding, of approaching a problem, of looking for information and so on. Dave[*], the last pilot, mentioned that he’d normally take much longer sizing a problem up but that having a time limit forced him into action sooner. But you’re up there with a limit of 10 minutes, probably on someone else’s machine, maybe on an “alien” operating system. You’re fumbling with running up a command prompt; you’re not sure how to bring up the Python docs; the keyboard shortcuts which your fingers remember don’t work — or worse, do something else entirely! We did discuss each person bringing up his own laptop, although that has obvious drawbacks of getting the projector to switch smoothly and so on. It was out of this discussion that the move towards a team-based approach next time arose.

TJG

[*] Dave, incidentally, was the programmer of the BBC version of the Graphics Adventure Creator — did you know that?

Links for 2010-02-04 [del.icio.us]

2010-02-05T08:00:00+00:00

Celebrities accessorise for Haiti
"To help raise funds for the victims of the Haiti earthquake," begins a press release, "international actress and face of Mango, Scarlett Johansson has designed an exclusive handbag." Ah.
Library of Dust
*In 1913... an Oregon state psychiatric institution began to cremate the remains of its unclaimed patients. Their ashes were then stored inside individual copper canisters and moved into a small room, where they were stacked onto pine shelves...Over time, however, the canisters have begun to react chemically with the human ashes held inside them; this has thus created mold-like mineral outgrowths on the exterior surfaces of these otherwise gleaming cylinders.*
The Curious Case of the Evolving Apostrophe
Last orders for pint glass as we know it?
What pregnant women won’t tell you. Ever.
Right, that's it - I'm *never* getting pregnant.
NASA: A Change of Orbit

Choose Python

2010-02-05T07:23:54+00:00

Ethan Furman linked to this "Choose Python" poster on comp.lang.python. While some of the humor might not mean much to people who haven't chosen Python yet it certainly gave me a smile on a cold, grey morning.

Links for 2010-02-03 [del.icio.us]

2010-02-04T08:00:00+00:00

Me and my homeopathic overdose
Hadley on Homeopathy
ShowMeWhatsWrong.com
"Generate a help URL so friends and family can send you instant screen recordings to help trouble shoot computer issues." Family tech-support - the bane of every IT person's existence.
Rich-Text Editors for 2010 and Beyond
IT Certification Considered Useless
NASA - Suspected Asteroid Collision Leaves Trailing Debris
Kim Jong-il's regime is even weirder and more despicable than you thought
CQRS à la Greg Young
The Command and Query Responsibility Segregation (CQRS) pattern
Our licence fees pay for climate denial | Sunny Hundal | Comment is free | guardian.co.uk
"After watching last night's Newsnight, I can only come to one conclusion: the BBC has become this country's most pernicious climate-change-denying media outlet in the UK."
Information, Freedom, Flame-bait
Information wants to be free, writers want to get paid.
Apache Thrift
"Thrift is a software framework for scalable cross-language services development. It combines a software stack with a code generation engine to build services that work efficiently and seamlessly between C++, Java, Python, PHP, Ruby, Erlang, Perl, Haskell, C#, Cocoa, Smalltalk, and OCaml."
Physicist discovers how to teleport energy
I wonder if they can teleport some into me.

Links for 2010-02-02 [del.icio.us]

2010-02-03T08:00:00+00:00

It Certification Considered Useless

2010-02-02T21:38:16+00:00

A few years ago, I was sharing a drink with a friend of mine. He was about to become a fully qualified architect. In the UK, one cannot call themselves an architect without having carried out the full, three part course, which takes at least seven years. Typically, as the course involves working in the industry, architects often took more than seven years to complete their ‘part three’ – my friend completing it in the minimum possible time made him one of the youngest qualified architects in the country. As he was about to be fully qualified, he was explaining the need to get indemnity insurance, as his opinion as a qualified architect made him liable for the quality of advice given, even advice given informally down at the pub.

There has been a short history of various individuals, companies and professional bodies within IT attempting to define and issue certification. By and large, they have not caught on. There is no belief that software delivered by ‘certified’ individuals is any better than that developed by uncertified individuals. Nor is there any evidence that in terms of getting jobs that certification counts for anything other than in specific, narrow (mostly vendor specific) technical domains – something that few serious software professional would consider worthwhile.

So, in general, why does certification exist? Societal pressure determines where certification is essential. It is important that key individuals in positions of power are properly vetted – and recognised – for the role they play in society. Which is why there are laws governing who can call themselves a lawyer, architect, engineer, surveyor. Which is why certified profesionals have responsibility placed upon them regarding the veracity of information and quality services they provide as a member of that profession.

In terms of architecture – in similar terms to medical doctors for example – society has deemed the roles they play as being important enough that certification carries with it legally enforceable expectations regarding their competency. With this responsibility, comes recognition – and a clear understanding as to how the profession is valued by society as a whole.

Certification in the land of IT is not being driven by a need for society to ensure that we are doing our jobs properly – to ensure that only competent individuals call themselves ‘programmers’, ’sysadmins’ or whatever. Nor is it being driven by a societal desire to recognise our contribution to society as as a whole. It is being driven by IT itself – at best as a misguided attempt to recognise an ability in a certain set of skill, at worst as a way of generating money. As such, certification in the world of IT is a toothless concept, lacking in any sense of legitimacy, and distracts us from the more worthy goal of understanding how we contribute to the world around us, and how we grow competency to the point where we can even consider ourselves a profession at all.

Interview with Pardus Linux

2010-02-02T10:28:52+00:00

I recently did an interview on Python with the Pardus Linux magazine. Pardus Linux is a distribution developed in Turkey (by the Turkish National Research Institute of Electronics and Cryptology) with the goal of being usable by "normal" people rather than just geeks. ... [3716 words]

Links for 2010-02-01 [del.icio.us]

2010-02-02T08:00:00+00:00

Acid-washed is an 'ironic' fashion, never meant to flatter
"Ironic fashion"? Good lord.
Amazon 'Capitulates' to Macmillan
"Just me, or does it seem like an Apple device that won’t ship for another 53 days already upended Amazon’s Kindle business? What kind of sense does it make to accuse a publisher of having a “monopoly” over its own titles?"
One day, all adverts will look like PowerPoint
Barack Obama to drop Nasa moon mission in budget cutbacks
Sad, but probably inevitable.
Twitter on Scala
Still mostly RoR.
San Francisco’s Answer to Westboro Baptist Church
Think iBooks Looks Familiar? You're Not The Only One.
DjangoGigs.com
Does what it says on the tin.

Thief Level : Week 2

2010-02-01T13:20:31+00:00

This week I spent a little while considering the backstory of the level, and now have at least a rudimentary scenario: Garrett the thief is taking an opportunistic foray into the local Shope of Curiosities, having heard that their prize exhibit, the McGuffin of Antioc, has been removed from its high-security public display, in order to be cleaned or maintained somewhere on-site.

On top of this, I’ve been refining the layout of the museum building, starting with the two-storey entrance hall, complete with a balcony running round it:

Plus, I’ve been planning the possible routes a thief might take to get from one room to another. Generally, the conventional paths – in through the main entrance and up the stairs and down the corridor – will be blocked by guards. So the player has to clamber up the outside of the building, explore the roof, dangle from a rope, pick a lock, find a key in the janitor’s quarters, which opens all the windows, and ledges outside a couple of windows lead somewhere interesting, etc. I don’t want it to turn into a key fetch quest, but at the same time, I don’t want the player to be able to simply waltz all through the whole building. I’ve tried to engineer a single interesting primary route through the building, with the possibility of a few minor variations so players feel like they can exercise some freedom and decision making.

Having done all that, I’m now quite happy that my plans are sufficient to produce a small but adequate level. I’ll aim to get that complete, and any fancy window dressing I can layer on top will be a bonus.

I completed the modelling of all the rooms in the building, and doorframes inbetween them. I applied some quick floorboard textures to differentiate the floors and ceilings from the walls. Here you can see the view from the main entrance, looking into the two-storey entrance hall, with the balcony around it visible up on the next level:

And the view while approaching the top of the stairs, looking down over the balcony. There will be a railing when it’s done:

Thanks heaps to Qolelis for a comment with a tip about textures on stairways, to rotate the texture 90 degrees on each stair’s vertical rise. I only just saw that, but will definitely apply it this week.

Update: I feel a bit self-concious that I’m creating the bare minimum that could qualify to be a Thief level. There is not yet a lively, engaging backstory to the level, complete with colorful characters, cleverly intertwined with the canon of the original game.

Similarly, the mechanical contents of my level are as simple as possible. I haven’t stretched myself, thinking of imaginative locations or motives for Garrett to explore. I do not plan to have any clever special objects or custom scripting in my level, defining dramatic changing mission objectives as the player reveals new information. It’s a very straightforward ‘get into a building, steal the loot, and get out’.

Partly this is very deliberate – I want the level to be as minimal as it can possibly be, so as keep it achievable. But also, this is partly in response to my feeling that being creative is hard, especially when under pressure. Right now I feel as though I have enough to worry about just getting to grips with the minutia of the level editor. I almost feel as if I need to become comfortable with that before I can relax enough to get creative with it.

This isn’t entirely unexpected. Clearly one cannot do great work on one’s first attempt. But at the same time, I don’t want to just ‘give up’ on the creative aspects. I want to do as good a job as I can do, under the constraints of a small, straightforward ‘first time’ level done in a reasonable timeframe. So maybe I just need to keep iterating. Embelish the dramatic backstory little by little, see what occurs to me as I go on. Look for some flash of inspiration as I bury myself in the process. Fair enough. Baby steps.

Update 2: I created a quick TODO list, as a first approximation of how much work there is to be done. I ended up with a list of 67 mandatory items (eg. Add doors inside each door frame; First pass at lighting; Add balcony railing.) In addition I have 18 optional items (eg. Add carpets and rugs; Hide moss arrows in the garden; Entrance hall main exhibit.) The screenshots above represent about six completed items (eg. Dromed tutorials; first floor rooms; doorways and arches between rooms; staircase.) So at the current rate, it’s roughly 28 weeks of work, which is double or triple what I’d planned on. Hopefully my rate of completing items will increase substantially as I get into the groove. I’ll have to monitor this going forward, and slash scope if I can’t drastically accelerate.

Links for 2010-01-31 [del.icio.us]

2010-02-01T08:00:00+00:00

Acheulean tools
I held one of these in my hand today.
Who Can Do Something About Those Blue Boxes?
Bye bye Flash.
Amazon, Macmillan: an outsider's guide to the fight

A Little Bit of Python Episode 3

2010-01-30T21:00:06+00:00

A Little Bit of Python is an occasional podcast on Python related topics with myself, Brett Cannon, Jesse Noller, Steve Holden and Andrew Kuchling. We still don't have our own website although that is due to land any day now. ... [172 words]

TypeError: object.init() takes no parameters

2010-01-30T19:07:00+00:00

At my employer we are in the process of migrating from Python 2.4 to 2.6. When running some existing code under Python 2.6 we started getting DeprecationWarnings about "object.__new__() takes no parameters" and "object.__init__() takes no parameters".

A simple example that triggers the warning:

class MyClass(object):

    def __new__(cls, a, b):
        print 'MyClass.__new__', a, b
        return super(MyClass, cls).__new__(cls, a, b)

    def __init__(self, a, b):
        print 'MyClass.__init__', a, b
        super(MyClass, self).__init__(a, b)

obj = MyClass(6, 7)

This gives:

$ python2.4 simple-warning.py
MyClass.__new__ 6 7
MyClass.__init__ 6 7

$ python2.6 simple-warning.py
MyClass.__new__ 6 7
simple-warning.py:5: DeprecationWarning: object.__new__() takes no parameters
  return super(MyClass, cls).__new__(cls, a, b)
MyClass.__init__ 6 7
simple-warning.py:9: DeprecationWarning: object.__init__() takes no parameters
  super(MyClass, self).__init__(a, b)

It turns out that a change to Python for 2.6 (and 3) means that object.__new__ and object.__init__ no longer take arguments - a TypeError is raised when arguments are passed. To avoid breaking too much pre-existing code, there is a special case that will cause a DeprecationWarning instead of TypeError if both __init__ and __new__ are overridden. This is the case we were running into with our code at work.

The reason for this change seems to make enough sense: object doesn't do anything with arguments to __init__ and __new__ so it shouldn't accept them. Raising an error when arguments are passed highlights code where the code might be doing the wrong thing.

Unfortunately this change also breaks Python's multiple inheritance in a fairly serious way when cooperative super calls are used. Looking at the ticket for this change, this issue was thought about but perhaps the implications were not fully understood. Given that using super with multiple inheritance is common and "correct" practice, it would seem that this change to Python is a step backwards.

rst_break plugin for PyBlosxom

2010-01-30T15:29:00+00:00

I just scratched an itch by writing a small plugin for PyBlosxom that allows the rst (reStructured Text) and readmore plugins to work together [1]. It defines a reST "break" directive which gets transformed into the breakpoint string the readmore plugin looks out for. This allows for "Read more..." breaks to be inserted in for reST based articles.

For further information see the Code page here and at the top of the plugin itself.

[1]	Yes, the audience for this plugin is probably tiny!

Links for 2010-01-29 [del.icio.us]

2010-01-30T08:00:00+00:00

Daring Fireball: Various and Assorted Thoughts and Observations Regarding the Just-Announced iPad
Some interesting stuff here, both pro and con.
Spirit
The little rover that could. :-(
Unwarranted Insults: JD Salinger
"Katie Price has published nine books in the last five years you know. Just saying."
Adobe speaks up about Flash on the iPad
Strangely enough, they aren't happy bunnies.
ThoughtWorks Sites
Laser fusion test results raise energy hopes
RFID Data Reveals London's Polycentric Pattern of Commuting
Mute button
bpython
Fancy interface to the Python interpreter
Tinkerer’s Sunset
When DVD Jon was arrested after breaking the CSS encryption algorithm, he was charged with “unauthorized computer trespassing.” That led his lawyers to ask the obvious question, “On whose computer did he trespass?” The prosecutor’s answer: “his own.” If that doesn’t make your heart skip a beat, you can stop reading now.

Howto bundle binary dependancies with py2exe, et al.

2010-01-27T22:26:00+00:00

Hey. I notice that py2exe-users has a recent question that seems to be about how to correctly bundle the required Microsoft C runtime DLL with executables generated from Python scripts.

Last month I updated the py2exe wiki tutorial to cover this issue, as best as I was able. Since people are still asking questions about it, I figured I’d promote that change a little.

Check out the new, revamped py2exe tutorial page!

That is all.

iPad Name Already in Use

2010-01-27T14:41:24+00:00

One thing that Apple appears to have lost sight of in all the thrills of announcing their new tablet computer is that Widgetaria, presumably an Apple business partner, already promotes software called iPad. I wonder if they care?

pyCUDA on Windows and Mac for super-fast Python math using CUDA

2010-01-26T13:01:22+00:00

I’ve just started to play with pyCUDA which lets you run parallel math operations on a CUDA-compliant NVidia graphics card through Python.

CUDA stands for Compute Unified Device Architecture – it is an architecture that lets us program the Graphics Processing Unit (GPU) on a high powered graphics card to do scientific or graphical math calculations rather than the usual texture processing for games. In essence it is a mini supercomputer that is specialised just for fast math operations – if you can figure out how to use it.

The goal is to off-load the CPU-intensive calculations for two of my clients (a physics company and a flood modelling company) to achieve 10* to 100* speed-ups using commodity graphics cards.

pyCUDA makes it easy to interactively program a CUDA device rather than hitting C++ code with the slow write/compile/debug loop. Recent MacBooks (mine was bought in January 2009) have NVidia cards with CUDA-compatible devices built-in (mine is a 9400M). For my desktop computer I have a 9800 GT (costing £100).

It turns out that this is bleeding-edge stuff – getting pyCUDA compiled on my MacBook and Win XP machine took some time (forum posts for Mac and Windows issues) thankfully the group is helpful and the wiki has an installation section for Windows, Mac and Linux and some reasonable documentation.

Right now I’ve got as far as running some of the demo code on my MacBook (showing a 5* speed-up over the CPU) and my desktop (showing a 30* speed-up over the CPU). I’ll report more as I progress.

Update – pyCUDA works inside IPython too, lovely.

Update – I don’t have OpenGL working for gl_interop.py but as noted here you need “CUDA_ENABLE_GL = True” in siteconf.py and you need PyOpenGL installed. When rebuilding my MSVC threw a hissy fit, it isn’t essential to my work so I’m skipping this demo.

Ian produces professional screencasts (ProCasts), writes The Screencasting Handbook, programs Python, researches Artificial Intelligence (Mor Consulting) and is also a sea-side dweller and consumer of fine coffees.

Creating a Level for Thief 2

2010-01-25T13:50:26+00:00

After Sinister Ducks, my videogame creation mentor suggested that I create a mod for an existing game, in order to distance myself a little from the programming aspects of creating a game, and instead spend a little time considering the gameplay and the art and the music from the perspective of the user. Sounds like useful advice.

So, the last couple of weeks I’ve been working through the tutorials for DromEd, the notoriously cranky level editor for vintage sneak-em-up Thief: The Dark Project. (Specifically for the sequel, Thief 2: The Metal Age, which has a slightly improved engine and editor.) I chose this for three reasons:

Released in 1998, Thief is old enough that the assets are simple low-fidelity geometry and bitmaps. These are easy enough for me to create and edit, plus if I intersperse existing game assets with my own shoddy creations, there won’t be a tremendously jarring disparity in apparent quality.
Even though Thief is ten years old and the company that created it long gone, there’s still a thriving community of amateur afficionados, churning out new missions at the rate of several per month, many of which are of exceedingly high quality – in some cases exceeding that of the original game. I’ll be in good company, will have some meaningful feedback, and will have forums to turn to when I get into difficulties.
Last but not least, Thief is one of my favourite games of all time. The emphasis on sneaking around and avoiding confrontation suits my sensibilities. Your protagonist, Garrett, is a marvellous, mercenary character. Best of all, in Thief 1, it reveals unexpected depth halfway through – the player’s expectations of a succession of simple heists takes a strange twist when the powerful storyline reveals itself.

Having finished the tutorials, this weekend I broke ground on creating my own level, or ‘fan-mission’ (FM), in the parlance:

I’m using the DromEd Toolkit, which is DromEd with some third-party patches and bugfixes applied to it. My first impressions with DromEd are that it’s very clunky and ugly, and startlingly lacking in documentation. I’ve taken to dipping into the configuration files to see what keyboard commands exist to experiment with. There are a bewildering variety of binary patches to modify the executable in various exciting ways, and forum posts about it, although helpful and prolific, seem fragmentary and rife with broken links. I’ve still no idea whether I ought to be using Dromed Delux instead, nor where I should get that from. It’s a glorious chaotic riot, and it’s a little intimidating.

Still, having said that, the binary patches have all worked fine for me, and the more I use the editor, the more it’s starting to grow on me.

I’m setting my FM in a museum. There’s already an existing museum mission out there, but as opposed to its marble-halled austerity, I’m imagining this will be more like the cramped, cosy, wood-panneled chaotic collection of something like the wonderful Sir John Soane’s Museum in London.

So progress as of week 1 looks like this: I’ve carved out some very basic geometry to form a stocky museum building. Here you can just about make out a hole in the brick facade that will form the front entrance. This is not the entrance that the player will likely be using:

This is all very crude thus far – with repeating textures on large surfaces, and plain uniform lighting. There are a complete set of mostly rectangular ground-floor rooms, with interconnecting doorways. The highlight of my modelling to date is this stairway leading up to the (otherwise nonexistant) next floor.

I’m not so happy with the wood texture I chose – I’ll go back and look for something more uniform. But I am happy with the way the stairs flair out at the bottom. I realised in the process of creating this that this makes it possible to fit a flight of stairs into a smaller space than would otherwise be possible, by allowing the bottom few steps to gracefully project out into the corridor.

Having finished this last night, I then dreamed about geometric operations on three dimensional spaces, which I think is a good sign.

Build Pattern: Chained Continuous Build

2010-01-24T18:22:36+00:00

One of the problems quickly encountered when any new team adopts a Continuous Build is that builds become slow. Enforcing a Build Time Limit can help, but ultimately if all of your Continuous Build runs as one big monolithic block, there are limits to what you can do to decrease build times.

One of the main issues is that you don’t get fast feedback to tell you when there is an issue – by breaking up a monolithic build you can gain fast feedback without reducing code coverage, and often without any complex setup.

In a Chained Continuous Build, multiple build stages are chained together in a flow. The goal is for the earlier stages to provide the fastest feedback possible, so that build breakages can be detected early. For example, a simple flow might first compile the software and run the unit tests, with the next stage running the slower end-to-end tests.

With the chain, a downstream stage only runs if the previous stage passed – so in the above example, the end-to-end stage only runs if the build-and-test stage passes.

Handling Breaks

As with a Continuous Build you need to have a clear escalation process by which the whole team understands what to do in case of a break. Most teams I have worked with tend to stick to the rule of downing tools to fix the build if any part of the Continuous Build is red – which is strongly recommended. It is important that if you do decide to split your continuous build into a chain that you don’t let the team ignore builds that happen further along the chain.

Build Artifacts Once vs Rebuild

It is strongly suggested that you build the required artifacts once, and pass them along the chain. Rebuilding artifacts takes time – and the whole point of a chained build is to improve feedback. Additionally getting into the habit of building an artifact once, and once only, will help when you start considering creating a proper Build Pipeline (see below).

And Build Pipelines

Note that a chained build is not necessarily the same thing as a Build Pipeline. A Chained Continuous Build simply represents one or more Continuous Builds in series, whereas a Build Pipeline fully models all the stages a software artifact moves from development right through to production. One or more Chained Continuous Builds may form part of a Build Pipeline, and a simplistic Build Pipeline might not represent anything other than Chained Continuous Builds, but Build Pipelines will often incorporate activities more varied than compilation or test running.

Fast Feedback vs Fast Total Build Time

One thing to note is that by breaking a big build up into smaller sections to improve fast feedback, counterintuitively you may well end up increasing overall build time. The time to build and pass artifacts from one stage to another adds time, as does dispatching calls to build processes further down the chain. This balance has to be considered – consider being conservative in the splits you make, and always keep an eye on the total duration of your build chain.

Tool Support

Tooling can be complex. Simple straight-line chains can be relatively easily build using most continuous build systems. For example a common approach is to have one build check in some artifact which is the trigger point for another Continuos Build to run. Such approaches have the downside that the chain isn’t explicitly modelled, and reporting of the current state of the chain ends up having to be jury rigged, typically through custom dashboards. More complex still is dealing with branching chains.

Continuous Build systems have got more mature of late, with many of them supporting simple Chained Continuous Builds out of the box. TeamCity, Hudson and Cruise and others all have some form of (varying) support. Cruise probably has the best support for running stages in parallel (caveat: Cruise is written by ThoughtWorks, the company I currently work for), and has some of the better support for visualising the chains, but given the way all of these tools are moving expect support in this area to get much better over time.

IMAPClient 0.5.2

2010-01-24T17:58:00+00:00

IMAPClient 0.5.2 has just been released. This release fixes 2 bugs (#28 and #33). Much thanks to Fergal Daly and Mark Eichin for reporting these bugs.

Install from the tarball or zip or upgrade using easy_install or pip.

Register for PyCon, or the Kitten Gets It

2010-01-22T21:14:47+00:00

Just sayin', it might be better if you planned to attend the conference.

It's not that I like harming innocent small furry creatures, it's just that there are still thousands of Python users and potential Python users who still don't know what excellent value for money the conference is, or how much fun you can have there.

PyCon is in Atlanta this year, and despite the parlous state of the world economy there's a chance that it will be the biggest Python event ever. But hey, we all know that size isn't everything.

Although I was notionally on the program committee, and did at least register my opinions of the talks I was allocated by the submissions scheme, I (yet again) didn't manage to make a single meeting due to pressure of other business. I'm actually not too unhappy about that, given the incredibly difficult task that the committee had to perform in order to ... well, I was going to say "sort the wheat from the chaff", but in fact there was so little chaff that wouldn't really be an appropriate analogy.

A number of people have blogged already about Five PyCon Talks You Must Not Miss, but since there's still a few hours to get in at the early bird rate I thought I'd throw out my list of unmissables from the extensive list of talks. How's that for arrogance?

A Short Pinax Tutorial, Danny Greenfeld. I have heard Danny speak at local Python user groups. If I had time I would be attending the half day Pinax tutorial that he and James Tauber are giving, but this is the second best way to find out how to get started with Pinax.

What Every Developer Should Know About Database Scalability, Jonathan Ellis. This one will be straight from the horse's mouth - it's always worth hearing Jonathan's summaries of his immense practical experience.

IronPython Tooling, Dino E Viehland. As a Windows user who has hardly touched IronPython so far I am interested in finding out what my options are. Nobody is likely to know better than Dino, who is probably the most prominent member of the Microsoft development team.

Scaling Python Webapps from Zero to 50 Million Users - A Top-Down Approach, Jinal Jhaveri. Although I haven't heard Jinal speak before I can't resist the lure of hearing what someone with really high-volume Python web experience has to say.

Why Not Run All Your Tests All the Time? A Study of Continuous Integration Systems, Titus C Brown. Ever since I hear Titus' tutorial (with Grig Gheorghiu) on testing a couple of year's ago I've wanted to hear what he has to say about CI. This is my chance.

Alas, five talks isn't anywhere near enough to encompass everything I want to hear, and I am also keenly anticipating Turtles All The Way Down: Demystifying Deferreds, Decorators, and Declarations, Pay Only for What You Eat: A Tour of the Repoze.BFG Repository and Philosophy, On The Subject Of Source Code, Python's Dusty Corners, Debating 'til Dawn: Topics to Keep You Up All Night, Powerful Pythonic Patterns and Tests and Testability. This is going to be a knockout conference even before you consider all the amazing things that will be happening in Open Space.

You really have to be there. And you'll be helping to save a poor kitten that never did anyone any harm in its life!

Musicians : wake up!

2010-01-22T09:43:24+00:00

I left this as a comment somewhere else, in response to someone complaining about how:

“…musicians are asked to just get over the fact that no one pays for music… [it is] my hope that someday more people make the same realization that unless people to help finance records, you’ll never get that fantastic song you can’t stop listening to all summer.”

Hey there. I really disagree that the onus is on the rest of society to solve all musicians’ problems. People have been copying my work on the internet for decades now. I’m a computer programmer. We’re in exactly the same boat as musicians – probably more so, and have been there for longer. But we manage to get by.

Some programmers attempt to stop copying of their work, but this is rarely successful. Others take a different tack. We’ve had to adapt. We acknowledge that our work will be routinely pirated on the internet, and we adopt our business model accordingly. We have to make and sell different sizes and types of tools than we were accustomed to making, and sell them through different channels. We have had to find other ways of charging for our work. My own company gives away our software for free. We embrace piracy as inevitable, and use it as a promotional tool.

This is difficult. It is counter-intuitive, grounded as we all are in the scarcity economy of the physical world. It requires us to change our attitudes and outlook and behaviour. Sometimes we fail. We are still figuring out the realities of our markets in a post-internet world. But goddammit we are trying and sometimes we try things and they work. The software business is more diverse and thriving than it ever has been – despite all our work being just as freely copyable (and freely copied) as music.

The financial hardship that musicians are feeling has always been felt by musicians. Copying music on the internet has not made this any worse than it always was. This is simply the cost of choosing to be a musician.

Musicians must adapt, just like programmers have. It saddens me to see so few musicians even try. Is it really true that computer programmers are more flexible and adaptable and imaginative than musicians. Who would have thunk it?

Embracing piracy as a promotional tool can allow bands to make more money on live shows than they have ever made selling records, but still, the majority of musicians sit around, complacent, thinking that the responsibility lies on the rest of society, to put things back the way they were before the internet.

No. I for one refuse. Adapt or die – if you can’t make it, someone else will take your place. By definition, that someone will be more adaptable, imaginative, risk-taking and dynamic. Isn’t that supposed to be what our creatives are good at?

People will doubtless think I’m being harsh – imposing my tyranny on struggling musicians. But I’m not. When I say “adapt or die”, I’m not imposing my will. I’m merely reminding you of reality. If we were talking about a guitarist who couldn’t play the guitar, we would all agree that they have put themselves in a bit of a paradoxical situation, and that they must either adapt (learn to play) or die (stop being a guitarist.)

This is how competition works. Competition might seem harsh, but it’s best for everyone in the long run. The poor guitarists are eventually persuaded by market realities to stop being guitarists and try being something else. The good guitarists get rewarded, and the standard of music available to the world is increased as a result.

Instead of untalented guitarists, today we are talking about musicians who would like to get paid, but who don’t want to expend any energy on figuring out how they will get paid. Can you see the paradoxical situation they have put themselves in? This attitude is entirely understandable and human – I also would enjoy locking myself in my room and making beautiful computer programs which nobody else need ever see. But if I expect to get paid for making computer programs, then I must interface with the reality of the world around me. It’s no use me simply wishing that High St stores will start selling my software in shrinkwrapped boxes again – those days are long gone. Instead, I must figure out what programs people want, and produce them in a timely manner at high quality, promote them somehow, and figure out my angle on how I’m going to get people to pay.

It is not too much for us to ask musicians to also have to conform to reality like this. If they want to make money, and the means of making money from music are changing, then they will have to expend a little energy figuring out how they are going to get people to pay. If they don’t want to think about that at all, then fine, they won’t get paid. Adapt or die, we’ll all be better off for it.

MSWindows Programming : Propogating child process exit values out of .bat scripts

2010-01-21T20:04:25+00:00

“God dammit. Why won’t you just DO what I WANT you hopeless pile of crap!”

So goes the refrain. I think you know where I’m coming from. Yet again, I have ended up learning far more about crappy DOS scripting than I ever wanted to know.

So I’m writing a program to automate some small task on Windows. One of the jobs of this tool is to modify the current environment. But I don’t know how a child process in Windows can modify the environment of it’s parent (namely the command-line shell that invoked it.) Can it be done?

So I hack a ghastly workaround: Wrap the script in a .bat file. A .bat file is invoked from the command-line in the same process as the shell, so any change it makes to the environment are made to the environment of the invoking shell itself. This also has the advantage that the tool can now be invoked by typing ‘toolname’, just like on other platforms, as opposed to ‘toolname.py’ or even ‘python toolname.py’. So I wrap my Python script ‘toolname.py’ with a new file, ‘toolname.bat’, living in the same directory:

:: first run our tool
python "%~dp0%~n0.py" %*

:: then make any changes to the environment
set THIS=THAT

The %~dp0 and %~n0 gobbledygook is a batch file way of referencing the same drive, path and filename (minus extension) as the current script, to which I add ‘.py’ to run toolname.py. Easy enough.

There’s a minor problem: The environment changes that need to be made depend on what goes on inside toolname.py. So I have that Python write a new batch file to the temp directory, containing all the ’set’ commands which will replace the hardcoded ’set THIS=THAT’ in the above script. Then we call that new temporary bat file from here:

:: first run our tool
python "%~dp0%~n0.py" %*

:: then make any changes to the environment
call %Temp%\%~n0-setvar.bat

Which is good enough. Presumably it will barf all over the place if run concurrently. But there’s a more pressing problem. I need the exit value of this tool to be equal to the exit value from toolname.py. Currently, the exit value of this .bat script is always zero, because the ‘call’ command at the end is always successful.

One solution I’ve seen used is to remember the exit value from toolname.py, and then use the DOS exit command to propagate this value out to our caller:

:: first run our tool
python "%~dp0%~n0.py" %*
set EXITVAL=%ERRORLEVEL%

:: then make any changes to the environment
call %Temp%\%~n0-setvar.bat

exit %EXITVAL%

The problem with this is that ‘exit’ doesn’t do what you think it does. It doesn’t just stop interpreting the current script, rather it terminates the current interpreter, ie. the shell that is running the script. If you run this from a command-line, since Windows doesn’t differentiate between a console and a shell, your window disappears. Sigh.

The exit command has a fix for this: It takes a switch ‘/B’, that causes it to just end the current script, rather than killing the shell. But now, it ignores any %EXITVAL% parameter you try to feed it, so the exit value of your batch file is always zero.

This is what I get for developing software on Windows. Nothing ever works the way it ought to. It’s as though everything were designed to oppose simple engineering idioms, like composing systems out of small, interchangeable parts.

So here’s what I finally did. The exit value of running a batch script can be set without using the hopelessly brain-dead ‘exit’ command. It is equal to the exit value of the last process the script invokes. So instead of exit, simply find a process that will exit with the value you need, and invoke it as the final command in your batch script:

:: first run our tool
python "%~dp0%~n0.py" %*
set EXITVAL=%ERRORLEVEL%

:: then make any changes to the environment
call %Temp%\%~n0-setvar.bat

:: and propagate the exit value to our invoker
python -c "import sys; sys.exit(%EXITVAL%)"

Bingo. I now have a Python process that can modify the environment of its invoking shell, and propagates the correct exit value out of the wrapping DOS script. [Short bow. Applause. Roses. etc.]

My Fonts Look Crap. I blame Windows and ATI.

2010-01-20T20:21:09+00:00

I know, I know. But I use Windows for 8 hours a day at work. So what do you do?

Here’s the poster-boy for programmer-friendly fonts, your friend and mine, everyone loves him, Inconsolata:

(Update: I evidently have some javascript or CSS trickery that squashes these images horizonally if your browser is narrow, to make them fit in the window. Since I’m banging on about microscopic differences between microscopic fonts, then if you’ve got this far you probably ought to make sure your browser window is wide enough.)

I don’t know about you, but to me that looks unutterably crap. Wonky and irregular. On the right of the orange line is what it looks like with ClearType turned off. Meh. Presumably I’m doing it wrong, somehow, but unless I figure out how, Inconsolata can fuck right off.

So then I fire up the trusty fallback, Consolas:

This is better, certainly, but the letters look cramped. For the number of visible lines in a window that size, the letters are awful small. Again, on the right of the line is ClearType turned off – which in this case distinctly worse.

So I continue peering at a succession of monospaced idiocy, eventually ending up on DejaVu Sans Mono:

This still isn’t perfect, but it’s the best I can find. Note that it bears out my impression of Consolas being cramped – even though the letters are substantially larger, we still manage to fit an extra line of text in. This time, turning ClearType off (right of the line) makes it a little crisper, and a little more wonky, but not much.

This isn’t just in Vim. It looks like this in all applications. I’ve tried running the ClearType Tuning Powertool, to no avail. Admittedly, all these fonts start to look a damn sight better when I increase the size a few notches. But that’s bugger all use really, isn’t it? Maybe I should be digging out some mono fonts designed especially to be viewed at small sizes? Oooh, now that’s actually not a bad idea: How about those Android fonts everyone’s banging on about? Droid Sans Mono:

This isn’t bad, but does contain all the old ‘zero vs upper-case O’ and ‘one versus lower-case L’ ambiguities. I think I’ll stick with DejaVu Sans Mono.

Does Inconsolata work OK on Windows for everyone else?

Update: Gerry suggested trying out Proggy Fonts (thanks!). These are bitmap fonts (unless you want to do without extended characters) so they won’t scale. Nevertheless, here’s how some representative fonts from there look. First up, Proggy Clean slashed zero:

I love the crisp clarity of a pixel-perfect bitmap font with no scaling or anti-aliasing. On the downside, I have a feeling that the characters’ shapes aren’t quite as well-formed and beautiful as DejaVu. Not sure whether or not this would bug me. Next up, Proggy Opti:

Color me impressed. Opti still has the wonderful pixel-perfect clarity of Proggy Clean, and manages to fit four extra lines of text into the same sized window. Obviously in order to achieve this, the characters are smaller than Deja Vu, but this is no bad thing. I would have liked to scale Deja Vu down a tad from the size you see it above, but if I try, the letters start to become a little distorted and indistinct. I might well give Opti a spin for a few days, see how it wears on me. Thanks for the suggestion Gerry!

Tip: creating a Xapian database in Python

2010-01-18T23:00:00+00:00

This cost me some hair-pulling today as I was trying to write a custom test runner for a Django project I'm working on that creates a test Xapian database just for running the tests. Basically, you can't do this:

os.mkdir(database_file_path)

Because if you do you end up getting these strange DatabaseOpeningError exceptions. So, here's how you do it:

import xapian
xapian.WritableDatabase(database_file_path,
xapian.DB_CREATE_OR_OPEN)

Hopefully by blogging about this some other poor coder will save some time.

Welcome BlueBream

2010-01-17T21:37:30+00:00

Finally, a chance to start afresh with a promising set of technologies that has consistently been under-utilized. I am not even going to mention the old name of the project, simply say that if you use Python you have likely heard of it. I have blogged in the past about the confusion of search results. Hopefully under the new name the vast preponderance of search results will be accurate and useful. So go meet BlueBream.

Organisation Antipattern: Chase The Ball

2010-01-17T14:57:58+00:00

Have you ever watched young children play football (of Soccer for our Atlantic cousins)? During the game, you can be certain of one thing – most of the team on both sides will be doing nothing but chasing the ball. There is no thought about the bigger picture, no tactical decision making (let alone anything as grand as strategy). they only thought on everyone’s minds is that “We need to get the ball”.

This thinking in children is understandable. Less clear can be the basis for this kind of behaviour within an organisation.

Typically, the ‘lets drop everything and Chase The Ball’ mentality comes from organisations which are primarily reactive, rather than proactive. Companies which exist almost always in crisis mode typically have a reactive attitude – working in teams like this can feel like you’re being buffeted by forces outside of your control, running from one disaster to another, never making progress towards where you need to go.

Small organisations that have got big often never get out of there reactive, firefighting mentality. All that happens is that more people are fighting the same fire.

To be clear I’m not talking about the problem being a single focus. I’m talking about a team or organisation fixating on a single focus which isn’t actually aligned with the longer term objectives. A bunch of kids chasing the ball need to think about winning the game, not getting the ball. Likewise an organisation needs to have clear line of sight to what its goals are, and always understand how what they are doing now gets them to where they want to go.

There will always be urgent, short term things that need to be addressed. The important thing is that they are dealt with in proportion, without loosing sight of the overall objectives. Ring-fence team members (making sure they aren’t always the same people) to Chase The Ball, and leave the larger organisation to focus on winning the game, or better yet winning the league.

Update: Fixed typo, thanks Ben and Julian!

So, One of Life's Little Frustrations ...

2010-01-17T07:31:23+00:00

... is the way Vista insists on taking the numbers in my filenames as numbers, thereby sorting them in precisely no convenient order. These files, however, are effectively named using a Hamming Code, so 62 is actually greater than 212, just like XP always knew. I just know this is going to turn out to be one of those awful tweaks that I should have put in three minutes after starting Vista.

I don't think we'll bother those nice Python people with that one. Which probably dooms this post to have me as the single reader.

Sudden pleasant thought. I will email the client to whom I first promised this work and then they will know that work is not only underway but almost complete. I have to finish this today or send an incomplete deliverable, since the client has paid for the classes, which have been delivered.

Happy Sunday. And people think that consulting is a part-time job ...

Not Really Python, But ...

2010-01-17T00:39:50+00:00

We interrupt the newly established Python-only regime with a message from your fellow human-beings in Haiti.

if YouGiveAShit:

visit("http://pyfound.blogspot.com/2010/01/haitis-problems.html")

'Nuff said. This is a long-term project.

Build Pattern: Build Time Limit

2010-01-16T21:31:15+00:00

Anyone who has worked in a team which uses a Continuous Build inevitably starts to learn about the cost of a long running build:

More time between checkin and a report of a failure
Higher chance of Continuous Build containing multiple checkins, increasing the chance of an integration break and complicating rollback
Fixing a build related to a checkin made much earlier decreases productivity, leading to a reduction in productivity

There are other ‘build’ times to be aware of. A long Checkin Gate build leads to an increased chance of someone else checking in before you, increasing the chance of an integration break when you do checkin. It also disrupts the developers normal flow – they cannot easily work on new code, so effectively have to down tools waiting for the Checkin Gate has finished. You also need to consider the time taken to run a single test – be it a small-scoped unit test, or a larger end to end test.

No matter what the build is, a long build interrupts programmer flow, decreasing focus, and therefore decreasing productivity.

Different Builds, Different Limits

As a team, you should decide on acceptable Build Time Limit for each ‘build’ – for example individual tests, Checkin Gates, and stages in your continuous build. You may even consider failing these builds if those time limits fail. Setting the Build Time Limit at the right level – and keeping it there – will help keep productivity high.

Different builds get run with different frequencies. The more often a build is run, the faster it needs to be. Experience suggests the following time limits:

Single small-scoped unit test – sub-second
End-to-end test – a few seconds
Checkin Gate – 30 seconds to a couple of minutes at most
Continuous Build – a handful of minutes

When your Continuous Build is part of a larger Build Pipeline, you may find it useful to set Build Time Limits for each stage in the pipeline. One might argue that enforcing Build Time Limits for each stage of a Build Pipeline – manual or automated – may be overkill, but having some reporting of when a limit is exceeded will help directly highlight bottlenecks in creating production deployable software.

Team Ownership

Teams must take ownership of ensuring that the Build Time Limit is enforced. Further, they should always look for opportunities to reduce them further. Any decision to increase any Build Time Limit should be taken by the whole team – likewise any decrease in a Built Time Limit with decreases test coverage should be agreed with all. Everyone should be empowered however to look for quick wins.

Some teams find the need for a Build Tzar/Build Cop role – someone who is in charge of the health of the build. I consider such roles as being short term measures only, and should certainly be considered an anti-pattern if they exist for any length of time. At the extreme end of this spectrum is the dedicated build team. Empowering the whole team is key.

Making Things Faster

There are a number of ways of making individual tests fast, which will depend both on the nature of the technology being used and the way it is being used. Consider making a Checkin Gate fast using a Movable Checkin Gate. Long Continuous Build times can be mitigated through the use of a Chained Continuous Build, perhaps as part of a larger Build Pipeline.

You may also want to simply remove tests that are slow but provide little coverage. Often, it may even be the case that slow running tests represent a performance issue in the system itself.

Some teams have also shown significant speed improvements by using the right hardware – such as faster CPUs, RAM disks or SSD drives. However simply throwing hardware at the problem can help speed a Continuous Build up, but this does little to affect the build time on local development machines – a situation where your continuous build is faster than your local development build is the opposite of what you want.

Definitely Not P*thon

2010-01-16T17:34:02+00:00

And, to complete my test of the feed reconfiguration I am going to post something that isn't about the P-language and ensure that it doesn't show up in Planet P*thon.

To make ad hominem remarks about the collective denizens of the planet would be like talking behind their backs - and besides, what if the test fails?

Just the Python, Please

2010-01-15T18:51:11+00:00

The webmasters are having a spring clean and suggested that I might like to provide a "pure Python" feed. Michael Foord pointed out that such feeds existed already. If you want to skip the irrelevant waffling and occasional foray into politics then from now on Planet Python should see such a feed.

Direct subscribers wanting only Python can follow
http://holdenweb.blogspot.com/search/label/python
http://holdenweb.blogspot.com/feeds/posts/default/-/python
if they prefer.

Other blogs:
A Yorkshireman in America where the rabid politics really live
On Your Desktop lame celebrity desktop blog
PyCon conferences - you probably get this via the Planet
Python Software Foundation News others contribute more than I do

Rewrites and New Year Resolutions

2010-01-14T09:23:40+00:00

The reason so many New Year Resolutions get dropped, is because people start doing something out of the ordinary (for them) in order to institute a change, but never make that change a habit. It’s the reason dieting does not work – you shouldn’t go on a diet, you should change your diet. The former implies a one-off activity that will somehow leave you better off – and it may for a short period of time. The later states that you will change your habits, so that now on you will do something differently, from this point forward.

When people embark on a Great Rewrite, they are undertaking the equivalent of a crash diet. Sure, you loose a few pounds, but when the old habits come back, and the diet has finished, all those pounds come back. Rather than kid yourself that by starting afresh you’ll learn from your old mistakes, start making changes in what you do now. In other words work to change the diet you have, the system you have – that will lead to habit change which will stick. It’ll also force you to deal with the problems you caused in the first place.

Working to change the system you have now has other benefits – other than being more likely to institute a change that sticks. It makes it easier for the whole team to be involved with the change, rather than leaving some people supporting the old system. It allows you to trade-off delivering new features and bug-fixes against architectural changes. But perhaps most importantly it helps create a team which understands that fixing a situation – improving it for the better – is possible and achievable.

I Am Not Sam Newman

2010-01-14T08:29:57+00:00

It has come to this. After many years of mis-directed mail, I have finally decided to put pen to paper (well, photon to monitor, but you get the idea) and state that I Am Not Sam Newman.

World, here me now. It is possible – nay likely – that more than one person has the same combination of first and last names as another individual. We know for example that there are at least 54 Dave Gormans in the world. That bloke went to the trouble of creating an entire TV series about the fact that the whole first name/surname thing doesn’t not guarantee a unique identifier for human beings. The Chinese, to their credit, have worked this out a while ago.

Now I’m trying to be nice about it. I have decided not to publish the emails from people asking me if I want to do documentary voice overs, well-wishers hoping my testicles get better soon (well, I think they meant prostate), or the offers to speak on the corporate circuit about my hilarious non-pc anecdotes about how I once called someone a monkey. Others, in a similar position to me, have very much gone on the offensive in this regard, but I’m not quite as funny as Tony Hawks (the comedian, not the skateboarder).

So, oh blogosphere, here my cry – I Am Not That Sam Newman – the controversial Australian sports personality. And for the record, despite the fact that I live in the UK, I’m not the other Sam Newman either – the Actor known for his voice over work, appearances in Holby City and the forthcoming lead role of Prince Andrei in War & Peace starring Brenda Blethyn and Malcolm McDowell.

I Am this Sam Newman.

But yes, I am related to Paul Newman. Feel free to forward on any royalty cheques my way.

Links for 2010-01-12 [del.icio.us]

2010-01-13T08:00:00+00:00

British police arrest people just for DNA samples
Institutionalized racism in the British police service
Improve your jQuery - 25 excellent tips
Good stuff from the jQuery world

WildlifeNearYou: It began on a fort...

2010-01-12T22:53:20+00:00

Back in October 2008, myself and 11 others set out on the first /dev/fort expedition. The idea was simple: gather a dozen geeks, rent a fort, take food and laptops and see what we could build in a week.

The fort was Fort Clonque on Alderney in the Channel Islands, managed by the Landmark Trust. We spent an incredibly entertaining week there exploring Nazi bunkers, cooking, eating and coding up a storm. It ended up taking slightly longer than a week to finish, but 14 months later the result of our combined efforts can finally be revealed: WildlifeNearYou.com!

WildlifeNearYou is a site for people who like to see animals. Have you ever wanted to know where your nearest Llama is? Search for "llamas near brighton" and you'll see that there's one 18 miles away at Ashdown Forest Llama Farm. Or you can see all the places we know about in France, or all the trips I've been on, or everywhere you can see a Red Panda.

The data comes from user contributions: you can use WildlifeNearYou to track your trips to wildlife places and list the animals that you see there. We can only tell you about animals that someone else has already spotted.

Once you've added some trips, you can import your Flickr photos and match them up with trips and species. We'll be adding a feature in the future that will push machine tags and other metadata back to Flickr for you, if you so choose.

You can read more about WildlifeNearYou on the site's about page and FAQ. Please don't hesitate to send us feedback!

What took so long?

So why did it take so long to finally launch it? A whole bunch of reasons. Week long marathon hacking sessions are an amazing way to generate a ton of interesting ideas and build a whole bunch of functionality, but it's very hard to get a single cohesive whole at the end of it. Tying up the loose ends is a pretty big job and is severely hampered by the fort residents returning to their real lives, where hacking for 5 hours straight on a cool easter egg suddenly doesn't seem quite so appealing. We also got stuck in a cycle of "just one more thing". On the fort we didn't have internet access, so internet-dependent features like Freebase integration, Google Maps, Flickr imports and OpenID had to be left until later ("they'll only take a few hours" no longer works once you're off /dev/fort time).

The biggest problem though was perfectionism. The longer a side-project drags on for, the more important it feels to make it "just perfect" before releasing it to the world. Finally, on New Year's Day, Nat and I decided we had had enough. Our resolution was to "ship the thing within a week, no matter what state it's in". We're a few days late, but it's finally live.

WildlifeNearYou is by far the most fun website I've ever worked on. To all twelve of my intrepid fort companions: congratulations - we made a thing!

A little thing about cron

2010-01-12T13:20:00+00:00

Here's something I just learned the hard way.

If you edit a crontab with "crontab -e", cron won't reload the updated crontab immediately. Changes will be read at 1 second past the next minute boundary. For example, if you change the crontab at 10:54:32, cron will reload it at 10:55:01. This means if you're trying to test how something runs under cron and you're impatient so you set that thing to run at the next minute, you won't see it run!

I spent a good half hour chasing my tail on this one. Set the test entry to run 2 minutes ahead instead.

Links for 2010-01-11 [del.icio.us]

2010-01-12T08:00:00+00:00

A war already lost
Been saying this all along
Programmer Competency Matrix
Maybe needs some tweaking, but a useful set of categorizations
Terror in Perspective
People have such a hard time being rational about terrorism

Psyco 2 Binaries for Windows and Python 2.4, 2.5 and 2.6

2010-01-12T00:16:10+00:00

Pysco is a specializing compiler (a kind of JIT) for Python written by Armin Rigo. The difficulty of maintaining and extending psyco was one of the motivating factors behind the inception of PyPy. ... [191 words]

Clojure editor/IDE options – IntelliJ v Emacs

2010-01-11T08:11:08+00:00

So all the cool Clojure kids keep wanting me to use Emacs. The problem is that I haven’t used Emacs for the last 10 years – since, in fact, I had to support a C application on about 7 different flavours of UNIX. As you can imagine, I’ve since expunged many of those past memories.

My IDE of choice – ever since I joined ThoughtWorks – has been IntelliJ. Yes, I had to spend my time in the wilderness with Eclipse, long enough that I feel well placed to compare the two and consider IntelliJ superior for the languages I use often. La Clojure now seems to play nicely with IntelliJ’s Community Edition, so I’m giving that a try.

Ultimately, I’m learning a new language, one which often requires my brain to work in a quite different fashion than it is used to. As such, I’m trying to limit the number of new things I have to deal with. If, however, I’m missing out on something by not using Emacs, I may be persuaded to give it a go. So can anyone out there tell me what I’m missing?

Links for 2010-01-10 [del.icio.us]

2010-01-11T08:00:00+00:00

Top 10 Cannabis Studies the Government Wished it Had Never Funded
Sigh. The insane "war on drugs" continues.
Natural Gas Conversions Could Cost a Couple Hundred : Gas 2.0
If this is even remotely close to true it is insane
Snow stories: rare self-rolling snow balls found in UK - Telegraph
Shows how unusual current weather conditions are
Artificial
Fair point?
The Luckiest Man EVER - Video, Clip - ROFL.TO
Somebody up there likes him
Origami Boulder Company
Have loved this site for ten years
Letter to the Patent Office from Professor Donald Knuth. Programming Freedom.N11.Feb95.
Exactly

A Rambling Recording on Member Lookup in Python (podcast)

2010-01-10T20:25:22+00:00

I was thinking about the Python object model, in part as a result of my post on The Python Class Statement. Python is a really easy language to learn, but it also has advanced features like its protocols, descriptors and metaclasses, that make the full object model pretty complex - and that's before you start looking at the corner cases. ... [539 words]

Build Pattern: Movable Checkin Gate

2010-01-10T19:48:08+00:00

The Checkin Gate defines a set of tests which need to pass before a developer checks in. Typically, the tests are a subset of the total test suite – selected to provide a good level of coverage, whilst running in a short space of time.

There is an inherent trade-off with a Checkin Gate though – you may end up having blank spots in your coverage of the gate itself, which can increase the frequency of build breakages in your Continuous Build. By applying a Movable Checkin Gate, you attempt to offset this shortcoming by changing what is in the Checkin Gate suite.

Selection Based On Planned Work

Periodically, you assess the kinds of work coming up. If you are using an iterative development process, you may do this at the beginning of each iteration. Based on the kinds of changes the team will be working on during the next period, select tests which cover these areas of code, removing others which cover functionality unlikely to change. The theory is that you are selecting tests that cover areas of code which are most likely to get broken. The tests should be selected such that they don’t exceed your Build Time Limit.

After each movement of the site driving the Checkin Gate, you can assess the success by looking at the failure rate of the Continuous Build.

The key is to have a series of well categorized tests – tagging could work well here.

Selection Based On Build Failure

An alternative technique for selecting the makeup of the Checkin Gate can be based on build failures. If tests not in the Checkin Gate start failing in your Continuous Build, put them into the Checkin Gate suite, swapping out other tests to keep you below your Build Time Limit.

Updates

Added link to the new Build Time Limit Pattern.

Data Transformation and Language Syntax

2010-01-10T18:33:28+00:00

I’m currently working on a personal project by way of learning Clojure – it’s actually a program to match up my itemised phone bill against my list of contacts to help me expense my calls. I find it best to have a real-world problem I need to solve to learn a new programming language. The problem itself is rather dull, but it did give me a chance to consider an issue I’ve hit with many other languages.

One of the core parts of my telephone expense program is the process of normalising phone numbers so I can match them up. What I am trying to do is something long the lines of:

Strip spaces, then add the missing area code, then internationalize it

So in Clojure there are a number of functions I’ve written, each of which take, and return, a string (the program is nowhere near finished, so consider this to be virtually pseudo code) :

(defn #^String normalize [str]
  (internationalize (add-missing-areacode (strip-spaces str))))

In Java, this would look like:

public String normalize(String str) {
  return internationalize(addMissingAreaCode(stripSpaces(str)));
}

The problem is that I, and most of the western world, read from left to right – with both Java and Clojure I’m having to read from right to left to determine what is being done. One system I use frequently has a construct which matches what I’m after – UNIX:

strip-spaces "44 1230 9183" | add-area-code | internationalize

So what other languages support this kind of construct? I suspect I could coax Scala into doing something like this, and it seems that it is right up Python’s alley (Django’s excellent templating system has filters which do exactly that). But if I want to use Clojure, am I stuck with this inside-out programming model? What other JVM-based languages would help me here – Ioke perhaps? It seems right up AINC’s alley, but that syntax makes me want to cry…

Update 11 Jan 2010: Thanks to Matt for pointing me towards Clojure’s ‘->‘ macro. This looks pretty close to what I’m after. So I *think* I should be able to do something along the lines of:

(-> phoneNumber stripSpaces addAreaCode internationalize)

Which is very cool.

The Great Rewrite

2010-01-10T15:29:28+00:00

There is a rustle in the posit-in notes. The water cooler ripples. USB-powered missile launchers inexplicably fire, whilst nerf guns jam mid-battle. There is the smell of sulfur in the air. The Great Rewrite Approaches.

The signs were there. Grumbling from the developers – sometimes new to the project. “This code is horrible!”, “Completely unfit for purpose!”, “If only we could start again…”.

Delays to new functionality are laid at the door of the code. The one and only solution now on offer is to rewrite the entire codebase – nothing short of this will help. Eventually, managers are won over, and The Great Rewrite begins.

It is an epic undertaking. Some poor fools have to stay behind and look after the existing system, whilst others forge ahead into a brave, new world, leaving the horrid, old, decrepit and so uncool system behind.

Morale soars – the developers have a spring in their step. The business, initially, is confident. “Don’t worry – the new version is right around the corner!” they are told. Meanwhile support for the existing system is suffering – the team maintaining the existing codebase is a fraction of the size it used to be, and most of the senior technical people have to be on the rewrite.

The natives grow restless – the system they use, day in, day out, isn’t moving on. Feature requests seem to disappear into a black hole. “Soon” they are promised. “Soon, all your dreams will come true! Once The New System is launched, what you want is top of the list!”.

Months pass. And still, the rewrite continues. But it is closer now – inching towards readiness. Finally, long overdue, The New System is ready. The users are excited – all the recent troubles are to cease, as The Great Rewrite is over.

And now, the launch day.

There are bugs. Things that used to work, don’t work any more. There are few, if any new features. The system is new, but doesn’t offer the users anything new – but they have to learn to get to grips with The New System. The disgruntled emails start.

“Don’t worry!” says the Project Manager. Now The Great Rewrite has finished, the new features will arrive any day now!

And some of them do. Initially, at least, new features are easier than before to create, and ship. But after time, the same problems with the code base emerge. It turns out that having the same group of people building the same old system without changing their approach or ideas doesn’t lead to a different type of system. They never had to deal with the old issues head-on, they just sidestepped them, pressing on into the greenfield.

More time passes. Features take longer to ship, the code is harder to deal with. And once again, talk turns to another Great Rewrite…

Notes on the Python Class Statement

2010-01-10T13:49:13+00:00

Python classes are created at runtime, usually when you execute a script, or import the module they are defined in. Class creation is done primarily with the class statement. ... [889 words]

Release: ConfigObj 4.7.0 and validate 1.0.1

2010-01-10T00:14:15+00:00

I've just released ConfigObj 4.7.0 and validate 1.0.1. ConfigObj is an easy to use configuration file reader and writer module. ... [576 words]

Sinister Redux

2010-01-09T15:57:47+00:00

A few months ago, Glenn and Christian and Menno and myself created Sinister Ducks, a game with some quacking, for PyWeek 9, a contest to write a game in Python in a week. It plays a little like a simplified version of the arcade classic Joust – press fire to flap. When birds collide, the highest one wins, while the lowest one sheds feathers and plummets to the ground.

Since the competition, I’ve been polishing and refactoring for my own edification in odd hours here and there, and I’m pleased to declare that process finished. Behold, Sinister Ducks 1.0.2:

Things I added since the competition include:

Fixed the game being so ridiculously easy that one could play it interminably
There are now lives, and an actual Game Over screen and everything
Completely reworked how the gameplay works in regard to attacking ducks and collecting feathers – our previous mechanic didn’t work very well. The feathers now temporarily boost a score multiplier, displayed top-left. Attacking other birds yields more points for successful chained consecutive attacks, without collecting any feathers along the way. But you can only get feathers in the first place by attacking birds. It’s your job to figure out the resulting best-scoring dynamic.
Understand and enjoying the scoring dynamic is aided with little floating numbers showing the value of each bird you defeat.
Enemy birds now come in larger waves as time goes on, and one or two of the sinister ducks are angry and fast.
The game speed is now scaled by the time between refreshes, so it now runs perfectly fine on slow machines or VMs that can’t make 60fps, or if your monitor has an unusual refresh rate, or even if you disabled vsync in your graphics drivers to yield hundreds of frames per second.
Refactoring and OpenGL performance fixes (those sprites are texture-mapped quads, under the covers.) so that it’s now massively faster than it should ever need to be. The refresh rate with vsync disabled on my own 5 year old lappy is about 400fps.

You can download a Windows binary to download, unzip and double-click. On Linux or Macs it should run from source code, if you can be bothered to install the dependencies first, as described in the README.

The game’s only website is its Google Code repository, from where you can download it.

PyCrypto 2.1.0 Binaries for Windows 32bit Python 2.6, 2.5 and 2.4

2010-01-09T15:57:29+00:00

PyCrypto is a Python cryptography package originally created by Andrew Kuchling and now maintained by Dwayne C. Litzenberger. For a while I've been hosting Windows binaries for version 2.0.1. ... [126 words]

Links for 2010-01-07 [del.icio.us]

2010-01-08T08:00:00+00:00

BBC iPlayer Console - Listen live - BBC Radio 4 FM
Brilliant!

Fun with Unicode, Latin-1 and a C1 Control Code

2010-01-07T12:42:27+00:00

Unicode is a rabbit-warren of complexity; almost fractal in nature, the more you learn about it the more complexity you discover. Anyway, all that aside you can have great fun (i.e. ... [554 words]

Links for 2010-01-06 [del.icio.us]

2010-01-07T08:00:00+00:00

Absent DB API Test Files Fixed

2010-01-07T01:41:59+00:00

I saw a somewhat annoying comment from a web site reader recently:

The following message was received via the web site.

From:       
Email:      
Subject:    /PyConTX2007/dbapi.tgz  Not Available
Telephone:

I am grateful to the anonymous commenter for letting me know that I'd failed to redirect certain essential links in the recent web site migration.

It would have been even better if I could have replied to the comment to let my reader know that the problem is fixed (which it now appears to be). Alas he or she left absolutely no contact information whatsoever.So all I can do is record the fix and hope that this file wasn't being urgently sought (which I doubt it was).

Current State of Unladen Swallow (Towards a Faster Python)

2010-01-07T01:35:39+00:00

I'm helping to organise the Python Language Summit that precedes the PyCon Conference this year. One of the first topics we'll be discussing is "Python 3 adoption and tools and the Python language moratorium" which will include representatives of the major Python implementations telling us the current state of their implementation and their plans or progress for supporting Python 3. ... [561 words]

Two Days Left to be a PyCon Early Bird

2010-01-05T12:39:25+00:00

Just a quick reminder that PyCon early bird registrations end on January 6, so you don't have long to take advantage of these rates. Quick! Before you forget!

Links for 2010-01-04 [del.icio.us]

2010-01-05T08:00:00+00:00

20 Forgotten Photos from the Apollo 11 Mission
Memories of a bolder age