Too much speed at the Performance Sprint
Back with a sore back but a faster Plone
Thanks to many of you, I spent the weekend in Denmark, making Plone go faster. I was sorry to miss the first two days, but I had a blast nonetheless. Sleeping on Malthe's floor did my back in. I did, however, find out that Malthe is a mad genius who has a secret door in his bathroom and the same sense of humour as Daniel Nouri, that Copenhagen probably has more Norwegians than Danes, and that "åtte-og-halvtreds" is approximately 58Kr.
It was a sprint full of crazy ideas. An IRC bot was born and died, complete with a unit test framework and decorators. Sprint kudos was handed out in terms of who made something go twice as fast. Malthe decided to rewrite ZPT in Common Lisp and to introduce some kind of Markov chains into Plone. Matt and Sasha put MD5 hashes into text indexes, which somehow makes them faster, and produced some stress testing JMeter test plans to try and produce as many ConflictErrors as they could. Daniel wrote a decorator that would cause a function to only do something the second time it was called. This was briefly hailed as the solution to our things-get-reindexed-twice-in-Archetypes problem, but the idea was short-lived. Instead, Daniel looked into making it only happen the last time the method is called. To do that, Matt suggested we create an anti-object, which when combined with a real object produces pure energy, but it was deemed a little hard. Not to be outdone, Florian, Helge and Hanno theorised that we could "easily" (erm...) compile ZPT into C code and the idea of using PyPy for just about anything, including RestrictedPython, ZPT and zope.interface was floated repeatedly. Oh, and Florian and Martijn spent a day or so on an infinitely zooming animation. It was pretty fast.
Some work was also done. The Jarn guys have been great at blogging about their speedups - content creation and indexing can be made faster now, and the benefits of having a super-fast templating language were clearly illustrated with pretty graphs from Apple Numbers. Nevermind that Helge hand-compiled it in Python.
I paired with Andreas (and, at times, Florian and Martijn) and worked out that all our folders really should be BTree-based (you can hear the resounding d'uh, can't you?). So, we spent a few hours working out how exactly the fifteen (I was going to make that number up for hyperbole, but the number I came up with was nine) different folder implementations and base classes in Zope are all related. With pretty pictures. Then, we worked out where to insert a new one. :)
This is based on BTreeFolder2 from Zope but adds orderability. The idea is to have one folder implementation, but enable a UI switch to optimise for "large" and "small" use cases, much like we do in the user management UI now. A "large" folder would use a search-based UI and not support ordering, a "small" folder would use a batched-based UI and support explicit ordering.
Most of this is already done! There's a PLIP for it, complete with a buildout, branches for CMFPlone, Archetypes, ATContentTypes and a new package, plone.folder with the common implementation. Andreas also added some pretty graphs (can't have a performance sprint without them!) that show how much faster they are for common cases.
We have found a way to ensure that existing instances of Folder and Large Plone Folder continue to work, but have the Folder type be the new default. A little bit of work remains, but this should be a pretty good candidate for Plone 4.0 (it's not eligible for the 3.x series because it's pretty invasive).