Testing with nose, or why setuptools is annoying.

Created on Sep 19, 2008, 18:35 by Toby White, not updated since.

I use nose for my Python tests. It's not the only Python testing framework out there, but it seems to fit my needs.

Doctest discovery

I started using it mainly because test discovery for doctests seems to be far harder to arrange than it ought to be. There's no point in me evangelizing doctests more than has been adequately done before, but it's far too hard to arrange to run all doctests within a project, which rather defeats their purpose. nose seems to manage it perfectly well without trying.)

Anyway; so nose has this concept of plugins, which let you extend test discovery , or add extra fixtures, or whatever. Indeed, nose's core functionality is implemented by bundled plugins. It picks up all available plugins automatically by scanning the entrypoints from packages installed by setuptools. This has the irritating effect that

  • nose itself can't work without being installed, since it needs to find its own bundled plugins.

  • any additional plugins you write or use have to be installed as well.

Now I don't like this at the best of times; I get annoyed by software that insists it knows better than me where it should live, and I especially don't like blindly installing new software which might go & stomp all over existing installed software. Once you introduce versioning into the equation, I get even more annoyed; you end up with a python version of DLL Hell, with one application needing version 0.8 of a package, and another needing version 0.9.

Ruby gems

Incidentally - see also Tim Bray's take on a related issue from the Ruby point of view. I'm familiar enough with Python to work my way around these problems when they arise, but not so for Ruby - how do you install a gem-distributed package locally without interfering with your system Ruby libraries?)

But - since most of the rest of the world apparently shares none of my concerns with these issues, I struggle manfully onwards.

This week, there's been a thread on the testing-in-python mailing list "why you should distribute tests with your application / module". I agree 100% - tests should always be distributed with applications; I think almost all of the software I've ever written has had a bundled test-suite. (This was particularly useful for Fortran software, where there is such a wide range of compilers, but it still helps in tracking down system-specific issues even in Python).

Unfortunately, nose's setup.py requirements fly in the face of this; most users won't have nose installed. I could just about forgive nose for this, if I could rely on distributing custom plugins with my package, and being able to pick them up from the local path; but I can't even do that.

It's also worth noting that this is an issue even for software with a very limited distribution. If you're working collaboratively on a project, then all your colleagues need to be able to run tests too; and if you're working in a heterogeneous environment, then adding additional dependencies and installation requirements becomes rapidly onerous, and liable to piss off your co-developers.

Anyway. To round this story off with at least a moderately cheerful ending, I was happy enough with nose's usability not to abandon it, but pissed off with its requirements enough to try and fix them. There's a patch in the nose bug-tracker which at least partly fixes the issue, so that nose will pick up plugins from sys.path.

I suspect in the long term, though, the answer to most of these issues lies in the use of virtualenv. Enough people insist on requiring setuptools-based install, that it will probably be easier simply to isolate every app with its own dependencies in a virtualenv, and just distribute that instead.

In the meantime, for anyone actually reading this; REQUIRING SETUPTOOLS IS FUCKING ANNOYING, MMM'KAY? DON'T DO IT

Comments have been automatically disabled | digg this | reddit | del.icio.us