Planet London Python

February 11, 2019

Harry Percival

Python Architecture Stuff: do we need more?

I've been learning lots of new stuff about application architecture recently, and I want to get a sense from you, dear reader, of whether these ideas are of interest, and whether we should try and build some more resources (blogs, talks, etc) around it.

How should we structure an application to get the most out of our tests?

For me it all started with this question. At the end of my book I concluded on a chapter discussing how to get the most out of your tests, on the tradeoffs between unit, integration and end-to-end tests, and made some vague, flailing gestures towards topics I didn't really understand: ports and adapters, hexagonal architecture, functional core imperative shell, the clean architecture, and so on.

Since then I've managed to fall in with a tech team that are actively implementing these sorts of patterns, in Python. And the thing is, these architectural patterns are nothing new, people have been exploring them for years in the world of Java and C#. They were just new to me... and I may be over-reaching from my own experience here (I'm interested in your reaction), but they are perhaps new to the Python community in general?

It does feel like, as we mature, more and more of what were once small projects and plucky startups turn into complex business and (whisper it) enterprise software, so this stuff is likely to get more and more salient.

I came to it initially from the angle of testing, and the right kind of architecture really can help you to get the most out of your tests, by separating out a core of business logic (the "domain model") and freeing it from all infrastructure dependencies, allowing it to be tested entirely through fast, flexible unit tests. At made it finally felt like the test pyramid was an achievable goal rather than an impossible aspiration.

Classic books on the topic (All Java.)

The classic books (Evans on DDD and Fowler on Architecture Patterns, are classic, and anyone with an interest in this stuff should read them, but if you're anything like me, then wading through all that public static void main AbstractFactoryManager gubbins is a bit wearing. Maybe some more lightweight, Pythonic intros would make it all feel a bit more plausible, a bit less enterprise-architecture-astronaut-ey?

Some existing resources in the Python world:

Made's chief architect, the Venerable Bob, has written a 4-part blog series on the way we do things here, which I really enjoyed when I first started. They're a quick, practical intro to the basic concepts of DDD, ports and adapters / dependency inversion, and to some extent, event-driven architecture. And all in Python. (trigger warning: type hints).

  1. Ports and Adapters with Command Handler pattern in Python
  2. Repository and Unit of Work Pattern in Python
  3. Commands and Queries, Handlers and Views
  4. Why use Domain Events?

There's more on but those are the main four. We'd love to get some feedback on them, what's covered well, what could do with more explanation, and so on...

And: a very timely release from last Christmas, check out Clean Architectures in Python by Leonardo Giordani. It's really two books in one, part one being an intro to TDD, but part 2 has four chapters introducing very similar patterns to the ones I'm talking about here.

I also enjoyed a talk from about a year ago by David Seddon The Rocky River, how to architect your Django monolith, showing someone else starting to think about how we go beyond the basic Django models/views/templates architecture.

There's lots more at this listing of DDD resources by Valentin Ignatev, which I came across recently on twitter. It feels like something is in the air.

Call to action: is this stuff interesting?

Bob's already had some good feedback to his blog posts, and Leonardo has had some good initial sales, so I'm sensing some interest from the Python community out there, but I'd like to sanity-check it.

  • Does this stuff sound interesting or relevant? Do you want to hear more?
  • Are you doing Python stuff that's getting beyond the bounds of "basic webapp development" or "data pipeline"? Are you finding it difficult to write fast unit tests? Are you starting to want to disentangle your business logic from whichever framework you use?
  • Are you doing DDD or using any of these classic patterns with Python already? Do you maybe have all the answers and want to tell me about it? Or maybe just some answers and things that have worked well for you?
  • Do you think this stuff all sounds hella abstract and pointless? Maybe is a bit of an outliner in the Python world, in that we're writing logistics/ERP/enterprisey software in Python, and it all feels very different from what you do day-to-day?
  • What do you think the Python / dynamic languages community would most benefit from in terms of new guides to these topics?

I'd love to hear from you. Comments are open, or hmu on twitter, @hjwp

Did I say read the classics? Read the classics.

by Harry at February 11, 2019 03:28 PM

February 05, 2019

Ian Ozsvald

New public course on Successfully Delivering Data Science Projects for March 1st

On Friday February 1st I ran my first Successfully Delivering Data Science Projects, this is a part of my new plan to give more training this year. This went really well and I got to both teach and learn a lot from my students. We talked through best practice, project design, derisking strategies, communication plans and we tried various new tools that’ll improve workflow. Conversation has continued in our private slack channel (which all attendees get access to).

The next iteration of Successfully Delivering Data Science Projects is online for March 1st, the course has half sold-out already. If you’d like to improve your confidence around the successful delivery of Python data science projects – you’ll want to get a ticket soon. The material I teach is based on years of helping clients from start-ups to corporates to successfully deliver data science projects.

I’m really happy that the discursive format gave room for students to raise their own issues and to add recommendations for tools and books in addition to my own. We continued our conversations in the pub after whilst decompressing – there we got to dig into some of the hard topics (such as mental health, imposter syndrome and running open source projects) in a more relaxed setting.

The topics covered in the next iteration will include:

  • Building a Project Plan that derisks uncertainties and identifies expected deliverables, based on a well-understood problem and data set (but starting from…we don’t know what we have or really what we want!) – you take the project plan template away for use in your own projects
  • Scenarios based on real-world (and sometimes very difficult) experience that have to be solved in small teams
  • Team best practice with practical exercises covering coding standards, code reviews, testing (during R&D and in production) and retrospectives using tools such as nbdime, pandas profiling and discover-feature-relationships – you take away the solutions and a guide to running code reviews to support relentless quality improvements in your team’s solutions
  • Group discussion around the problems everyone faces, to be solved or moved forwards by everyone in the group (the group will have more experience than any single teacher)
  • A slack channel that lives during and after the course for continued support and discussion among the attendees

You’re welcome to get in contact if you have questions. Further announces will be made on my low-volume training email list. I will also link to upcoming courses from my every-two-weeks data scientist jobs and thoughts email list.


Ian is a Chief Interim Data Scientist via his Mor Consulting. Sign-up for Data Science tutorials in London and to hear about his data science thoughts and jobs. He lives in London, is walked by his high energy Springer Spaniel and is a consumer of fine coffees.

The post New public course on Successfully Delivering Data Science Projects for March 1st appeared first on Entrepreneurial Geekiness.

by Ian at February 05, 2019 10:24 AM