You are here: Home Articles Scrambled eggs
Navigation
OpenID Log in

 

Scrambled eggs

by Martin Aspeli last modified Jun 17, 2009 11:15 AM

Solutions for buildouts that misbehave after being left alone for a while and other egg version related issues

Eggs are great. They've facilitated a revolution in the way we manage packages and releases, and allowed us to evolve our systems in a much more dynamic manner.

But eggs can also cause a lot of pain, when your buildout decides to automatically upgrade some packages and you end up chasing dependency conflicts that "spiral": fix one by upgrading a particular package, and then you get a new one that requires you to upgrade another package.

Most Plone developers are blissfully unaware of this, but for some, it's become a problem. I'll try to summarise some important points here and give a few tips on what works and what doesn't.

What's in my Zope anyway?

It's important to realise that in Plone 3.x we are running on Zope 2.10. Zope 2.10 comes with (most of) the packages that make up Zope 3.3, the last monolithic (non-egg) release of Zope 3, before the Zope 3 community eggified their packages and started releasing them as individual eggs on PyPI. So, in the olden days of Plone 3.0-3.1 buildouts, you didn't have any egg metadata about any Zope 3 packages.

What does that mean? Well, if you installed a package that depends on "zope.component", then setuptools wouldn't realise that you already had this, and so it would try to download the latest version of zope.component form PyPI. That version would take precedence over the one that shipped with Zope 2.10, and may in turn pull in other dependencies.

Fake eggs?

To solve this problem, the plone.recipe.zope2install buildout recipe will by default now install what we call "fake eggs". This is basically just like a develop egg that points to the source code in parts/zope2/lib/python/* (or wherever your Zope software home is). setuptools now realises that it has a version of zope.component, zope.interface and all the other standard Zope packages, and so won't necessarily try to download new versions.

But which version? We said Zope 2.10 ships with Zope 3.3, but the packages didn't have version numbers then, and it all gets a bit blurry. So the fake eggs recipe installs them as version 0.0, unless you tell it to use a different version number for a particular recipe. That means that if you have a package that depends on "zope.component >= 3.1", setuptools will still try do download a new version, believing the version it has is too old.

Where do we manage version requirements?

Package requirements come from setup.py files (a package is depending on another package), or from an "eggs" line in buildout.cfg. They come in two flavours: those that depend on a package only ("I require zope.component") and those that require a specific version or range of versions ("I require zope.component >= 3.4"). The worst kind are those that require exact versions (zope.component == 3.4.1) in setup.py. Requiring a maximum version in setup.py is pretty bad too (zope.component >=3.4,<=3.4.999). You should never do an exact version pin in setup.py and rarely, if ever, do a max version specifier.

Sometimes, a minimum version requirement is necessary in setup.py, because you know that a package positively depends on a bug fix or feature that comes from a given minimum. Specifying exact versions and maxima, however, is pretty nasty, because you can't ever override those requirements without hacking the setup.py file or forking the package.

However, setuptools will always merrily download the latest version it can finds. And it's not always particluarly clever, either. If it stumbles upon a particular version early in the build, it'll stick with that, even if an old version would've been better in order to resolve other dependency constraints. This leads to two types of problems:

  • Setuptools makes a bad call and downloads a package that's too new, conflicting with a "maximum" requirement in a package that's processed later on. These types of problems are not too hard to fix.
  • Your buildout works fine, until one day someone uploads a newer version of a dependency to PyPI. Some time afterwards, you run a fresh buildout (or use the -n switch), only this time you get the newer package. This depends on other new packages, and suddenly you have a lot of new software in your build. If you're lucky, you get a version conflict somewhere. If you're unlucky, the build succeeds, but sometimes goes wrong at runtime.

Why is this a problem now?

This problem has started hitting some people recently.

Some packages (like Singing & Dancing, Dexterity and other things that depend on z3c.form) require you to upgrade core Zope packages to work around bugs or missing features in Zope 3.3 (which is pretty old by now). These often have a dependency on e.g. zope.compnent >= 3.4.0. They also typically come with installation instructions that require you to make sure you have the appropriate fake eggs and possibly pin a few packages in your [versions] block. More on that later.

This problem has recently gotten much worse, because the nice Zope 3 folks are trying to refactor Zope to have radically fewer dependencies, and are releasing lots of new packages.

As a real example, consider a package that wants to use the IContained interface. In Zope 3.3 and 3.4, this lives in zope.app.container. In the "Zope Toolkit" it lives in zope.container. zope.app.container is a smaller package that has some integration code and some backwards compatibility code. Now, let's say you depend on a package z3c.foo that was recently upgraded to depend on the new zope.container. You'll get that, but you'll also have the old zope.app.container. These two packages are meant to be in sync, but setuptools has no way to know that. In this case, you get two copies of the same interface: one in zope.container and one in zope.app.container. The results will be unpredictable and difficult to debug, I promise you.

Putting a blindfold on setuptools

Really, this is the world telling you that unless you really know what you're doing and you're willing to sink a lot of time into understanding what's in each package and how they fit together, you don't want that version of z3c.foo. It's not built for the platform you're on. Sorry!

However, in practice it's often useful to be able to upgrade certain things. Sometimes that's safe, and sometimes it isn't. You really need to analyse it on a case-by-case basis.

Hopefully, someone's done that analysis for you and come up with a "known good set" (KGS) of package that work together. There are two ways to manage that KGS and make sure it's used in your build:

  • Create your own index, similar to the PyPI one, that links to all the eggs you want to use in your project, and nothing more. Tell buildout to use that index instead of the default. This has the advantage of working with "straight" setuptools (e.g. easy_install), although it's harder to manage and you need to be careful you never use the standard index, or setuptools will happily ignore your index.
  • Pin the exact versions of every package you use in a buildout.cfg [versions] block. This is easier to manage, since you can do it all in buildout.cfg. However, it depends on this particular feature of buildout.

For most people in a Plone context, the latter is going to be easier. In fact, if you have a Plone 3.2 or later buildout, you're likely using this already. Your buildout.cfg may look something like this:

[buildout]
unzip = true
parts = instance zopepy
extends = 
    http://dist.plone.org/release/3.3rc2/versions.cfg
versions = versions

[versions]
zc.buildout = 1.2.1

Here, we use the buildout 'extends' feature to merge our own buildout.cfg with another configuration, loaded over the internet (you can also point to a local file via relative path). If you open the url to the versions.cfg block in your browser, you'll see that it's just a buildout.cfg like file that has a single section full of versions.

Also notice that we do "versions = versions" in our own buildout.cfg. This tells buildout that it should look for version pins in a section called [versions]. This will consist of the pins set in the external file, here containing the known good set for Plone 3.3rc2. We can augment and override this with any specific pins set in the current buildout.cfg file, as we've done for the zc.buildout egg above.

Also note that ti's possible to extend multiple configurations. Here's what a Dexterity buildout using Dexterity 1.0a1 might look like:

[buildout]
extensions = buildout.dumppickedversions
unzip = true
parts = instance
extends = 
    http://dist.plone.org/release/3.3rc3/versions.cfg
    http://good-py.appspot.com/release/dexterity/1.0a1
versions = versions
develop = 

[versions]
zc.buildout = 1.2.1

[instance]
recipe = plone.recipe.zope2instance
zope2-location = ${zope2:location}
user = admin:admin
http-address = 8080
debug-mode = on
verbose-security = on
eggs = 
    Plone
    plone.app.dexterity

[zope2]
recipe = plone.recipe.zope2install
url = ${versions:zope2-url}
fake-zope-eggs = true

In this case, you'll see that we're first getting the KGS for Plone 3.3rc3, and then we're getting the KGS for Dexterity 1.0a1. You'll notice that this is managed with Good-py, a web service that's designed to help people who build "platforms" create and manage multiple (possibly inter-related) KGS. Good-py also helps track "known bad versions" and resolve potential conflicts between platforms. You can read more about it here.

In this case, the Dexterity KGS will override and upgrade certain packages. But it's a carefully monitored set that is known to work with the platforms that Dexterity supports. A lot of time has gone into finding that set of packages so that you don't have to figure it out each time.

There are limits to this, of course. You could have yet another "platform" with another KGS that would be out of sync with the Dexterity one. In some cases it can work, but in some cases it might not. If both platforms are managed on good-py, it will be able to tell you this, at least, if you request them both together.

Prevention is the best cure

So, what can you do if you end up with dependency issues like this?

First of all, you need to understand what's in your build at all times. For that, you should use the buildout.dumppickedversions extension. By default, this will print to the console any packages that were downloaded and installed by buildout, excluding any that were already pinned.

Keep an eye on this list. If things appear in there you didn't know about, figure out what caused it. Once you're happy with your build, and especially once you deploy to production, pin your eggs to the known good versions. The extension can write a .cfg file with this information, which you can depend on. I commonly use a setup where I have a devel.cfg buildout I use for development that uses "newest = true" and write a kgs.cfg file with the picked versions. I check this into svn when I'm happy that I have a stable build, allowing me to trace back any problems later. I then have a production.cfg build which adds this generated kgs.cfg to the 'extends' line, meaning it pins virtually everything in "production" mode, but allows some flexibility in "development" mode.

Secondly, watch out for "too new" packages. The further you stray from Zope 3.3, the more likely you are to have problems. Packages from Zope 3.4 are relatively close to the 3.3. Once you get into 3.5 and 3.6, be careful! In general, if I positively have to upgrade something, I try to get the latest available 3.4.x package and leave it at that. It does require careful testing, though.

Third, if you get in a loop, go back to basics. Use the dumppickedversions extension to figure out which package caused the dependency avalanche and remove it. Re-build and make sure you're back to a sane set of dependencies. Then see if you can use an earlier version of the troublesome dependency. If you can't, check its installation instructions - maybe it wants you to use a particular KGS, or pin a particular set of dependencies? And if not, consider giving up! Even if you can get it to work, having a build that's half of Zope 3.3 and half of ZTK with version 3.7 of zope.this and 3.8 of zope.app.that is probably going to land you in trouble later. Sometimes it's just not worth it.

Finally, if you're sure that you already have something that's good enough, you can tell plone.recipe.zope2install to fake it for you, by listing additional fake eggs:

additional-fake-eggs =
    ZODB3 = 3.7.1
    zope.annotation = 3.3.2

This lists the exact versions to fake, rather than the default of 0.0. You can also add packages here that you have in your global environment, which are not part of Zope. But be careful. Outright lying here (like saying you have zope.component 3.5 installed) could mean that your build completes, but your application breaks.

Hopefully, most people will never have to worry about this stuff. Stick to the safe packages that are built for the version of Plone you're on, and you should never see these issues. But if you are hit with weird versions, I hope this article will help you debug and resolve your problems. And if you want to set up a system on Good-py, drop me an email. :)

Document Actions

New stories, old problems

Posted by http://roquette.myopenid.com/ at Jun 28, 2009 06:49 PM
I look into this and I can see very old stories of packages, dependences, conflicts and so on.
There are already some systems that resolve things as complicated (or more) as this. Look the Debian Package System.
Do you see a happy end for this python/eggs/buildout/etc complex problem in a near future?

(PS.: (#plone->f_krei) no hard feelings about the emule thing? I didn't say explicit, but my hard copy is original £24.99 :)
Plone Book
Professional Plone 4 Development

I am the author of a book called Professional Plone Development. You can read more about it here.

About this site

This Plone site is kindly hosted by: 

Six Feet Up