Is Plone too hard?
I've been looking over the responses to the Plone Community Survey that Jon Stahl put out a couple of weeks back. There are lots of very interesting answers, and Jon and Alex have spent some time grouping them together and looking at trends.
Some of the trends are as expected. People generally like Plone's GUI and out of the box features, the installers, and the community. Some omissions are highlighted as expected - this or that feature is missing (blogs, forums, multimedia); this or that feature isn't configurable enough (HTML filtering, inline editing); this or that feature isn't advanced enough (commenting, categorisation). I won't dwell on those too much now, because they are largely tractable, well-understood problems that will be solved when enough people care or decide to pay for it.
The one thing that comes up again and again, though, is the issue of complexity. Is Plone too hard? Has Plone gotten harder? Harder for whom? This is a lot more difficult to address, since it quickly becomes a subjective discussion. However, I'm going to try to highlight a few themes and outline ways in which we can improve the situation. Some of themes are social or organisational. I'll leave those for another day and focus on the technical aspects now.
This posting is a follow-up to yesterday's entry, where I told a fictional story about two IT consultants trying to implement a Plone solution in two weeks. If you haven't read it yet, please do so before continuing here. In part, I wrote this story to promote some of the technologies I think we should adopt or build. However, it was also an experiment in trying to describe what an "easy" Plone looks and feels like, in order to see whether the story would be plausible.
Having written it (and thought about it quite a lot) I'm convinced that not only is it plausible - it's actually not that far off. Nothing I described would require unreasonable or incompatible changes in Plone, and nothing would be impossible to produce out of volunteer effort. The most important thing now is to get agreement on exactly what story we want to be able to tell and how we can get there. If everyone is pulling in the same direction, I think we're looking at 9-18 months until this is in the hands of real users.
Is it hard to whom?
When discussing complexity, we need to ask ourselves which audience we are catering to. For the purposes of this discussion, I'll consider five different (but somewhat overlapping) audiences.
- Plone end users and content managers
- Site administrators
- Integrators who want to customise and extend Plone (but still want their end product to be recognisable as Plone)
- Advanced developers who want to do complex integration with external systems or who want to change Plone significantly for their deployment
- Plone core developers
I think we have the first audience - end users and content managers - largely under control. Site administration is also relatively well thought out, although there are a few cases where more or better control panels would be useful. I also won't concern myself too much with Plone core developers since we pretty much deserve what we get. If anyone's going to swallow the complexity pill, it should be us. There is an argument that moving more of the "hit" further down the stack makes it harder to recruit new core developers, but I'll leave that to one side for now, since really this is just a question of stabilising the higher levels of the stack and then refactoring.
That leaves us with numbers 3 and 4 on the list. I think the complexity barrier hits us at two points: How to get started as a "light integrator" without having to scale a mountainous learning curve, and how to transition from "light integrator" to "proper developer" (or alternatively, how a "proper developer" works with a "light integrator/business power user" colleague to realise a shared design) without having to ditch everything you did and learned already.
Argh, it's too much!
I'm going to go out on on a limb here and say this: Plone is not hard. If you're a developer and you know how the pieces fit together, you can do things to Plone that much of our competition would just say is "unsupported". We have come to take infinite customisability (without forking!) for granted. Let's not forget what a huge benefit this is and how completely ingrained it is in Plone's culture and architecture.
Creating a content type with Archetypes, registering a new view, adding an adapter to customise navtree behaviour, adding a viewlet to show del.icio.us tags on a page and using an annotation to store trend statistics based on content access... all of this is easy. Each requires about 100 lines of code (liberally guesstimated). The hard part is knowing which 100 lines and knowing what technologies are relevant. Knowing where to start is the single, largest problem a new a new integrator/developer faces.
The good news is that I think this can be solved with about 100-200 print-pages worth of documentation. I wrote a 400 page book in five months. If we had four or five people writing this documentation, we could have it done by Easter. We won't, but we could. I think this documentation needs to be the most in-your-face thing on plone.org. It should be called "Where do I start?", and be organised by audience. For each audience, we need to explain the pre-requisite fundamental concepts (for an integrator it may be things like "how to think about hierarchical content models"; for a developer, that something like "what is an adapter") and then break down into specific topics. Each topic should have a task-focused title ("Managing additional member metadata"), a couple of paragraphs of background to clarify what we meant by that title, and then point to other, existing documentation. We should also make good use of diagrams and symbols to make it easier to understand how Plone's architecture fits together and where the various bits of functionality come from.
In the survey, we actually had a lot of people saying the documentation is great. I'm guessing those were the people who knew enough to know what to search for.
This exercise will also allow us to identify where there are gaps in the existing documentation and farm those out to be closed down. The largest problem, however, is resources. The Plone documentation team is perpetually limping from lack of (wo)man-power. Getting people excited about documentation is hard, and ensuring that people always write good documentation is equally difficult.
The Plone-can-do-anything fallacy
We are reasonably good at making strong statements about what Plone is and what it doesn't try to be. However, we need to be better. I think it's a fallacy to design a classic social networking tool on top of Plone, and yet people do it. In fact, I'd probably use Plone if someone came to me and said "build me a social network in a week".
Traditionally this fallacy has been a result of poor competition and/or poor symbiosis with our competition. People want to use Plone for its UI, its core feature set and because they like the Plone community. If they then want to add a "corporate facebook" to their Plone-based intranet, they'll start modelling Archetypes content types. Then they'll run into performance problems and start trying to use an SQL database to store relationships, and then they'll have to re-invent bits of Plone's UI for their non-Ploneish-within-Plone functionality. It can be done. Some of these tasks can and should be made easier, even. But by and large, we have plenty to do in the content management game, we don't need to play every other game too.
This comes down to the question of whether Plone is a framework or a product. I think that Plone is a product that provides a platform for content-centric web applications, a bit like WebSphere Portal Server is a product that provides a (more general) platform for business intranet portal solutions. It's not a framework like Zope 3 or Grok is.
Grok is rapidly getting ready for prime-time. It makes a lot of sense for Plone to have a symbiotic relationship with Grok, since Grok provides a nice, clean development model that uses many of the same Zope 3 component and concepts as Plone under the hood. The Grok and Plone communities intersect. Heck, grok.zope.org is a Plone site! People also want to start using Grok's configuration mechanism over ZCML in Plone.
We should tell people to use Grok (or Django or TurboGears/Pylons) for things that radically depart from Plone's core feature set. We are already seeing developers who ditch Plone in favour of these frameworks, when these are more appropriate for the problems they are trying to solve. Rather than getting alarmed and trying to frantically entice them back with more and more features, we should draw on their experiences and document integration success stories. Repoze already makes this possible, with sites that share themes (using Deliverance), navigation and authentication but use multiple Python-based back-ends. We should embrace Repoze and Deliverance as first-class citizens in the Plone world, help drive their development forward (including a push to eggify the remaining parts of Zope 2 and get the Zope 2 release process to a point where the Repoze project doesn't need to repack bits of Zope 2) and actively encourage complex integration between Plone and other Python-based applications.
In a way, this exacerbates the complexity problem, because it adds yet more layers to the stack and will result in deprecation of certain configurations. However, it also makes us less esoteric, by using tools (WSGI, eggs) that other Python frameworks use and makes certain things that are quite hard today (such as sharing a theme across hetrogenous systems) much easier.
Too much or too little configuration?
If I want to add a form to capture some data, is that a site configuration issue, or an integration/programming issue? What about a content type with a custom schema? What about complex navigation structures?
Largely, I think all of the above should be possible through-the-web. Basic forms, content schemata (which are really just an extension of forms) and navigation structures (including keywords and collections) belong in the site administrator/control panel domain. I'm working on some ideas for building content types using through-the-web or GUI tools (think ArchGenXML on steroids). I'll blog about those later and seek input. However, I think fundamentally, the notion that you need to be a programmer to add a new content type schema is wrong. Equally, you need to be able to do this in a programming way if you are a programmer and want or need the full flexibility of Python at your fingertips. Those need to be two sides of the same coin, not disparate techniques with no forward migration path.
However, it's also important to retain some of the "opinionated design" in Plone. I hate systems that try to give you every possible option on a huge configuration screen. Only options that are actually useful and which are easy to understand should be configurable, and we should always, always have sensible defaults. Making the hard choices is what the customer pays us for.
Themes are not themes, they are customisation
Building a new theme is probably the first brick wall that people hit when trying to deploy their initial Plone site. Every serious deployment requires some degree of branding, and public-facing sites require significant theming. It is fundamentally wrong that we expect people to use the same techniques to do arbitrary customisation of any and all of Plone's UI constructs that we use for building a simple theme that just changes the logo and colour scheme. The power and flexibility required to do the former is disproportionate to the skill set of the people who do the latter.
We should invest more time in understanding what kind of theming engines work in other systems. However, Paul Everitt has already invented a system that can theme more or less anything. It's called Deliverance, and it assumes you are more interested in HTML than in ZPT. Deliverance needs more real-world exposure, and a real champion in the Plone world. It probably won't replace theming-by-customisation in the most advanced deployments, but Deliverance should be the first tier in our theming/branding story.
If we deploy using Repoze, this is virtually feature-complete already. However, I would like to investigate ways in which Deliverance themes could be selectable and manageable from within Plone. If Plone could signal to Repoze when to invoke a given Deliverance theme in the WSGI pipeline, then we could easily support conditional theming such as "show the public branding only for non-logged-in users" or "use this theme in the /intranet part of the site, and this other theme everywhere else".
Arbitrary differences in customisation
One thing that is hard at the moment is the fact that there are two ways to customise visual elements: the "old" way using CMF skin layers and the "new" way using Zope 3 browser resources. Skin layers still have more powerful TTW customisability and are easier to customise on the filesystem since the don't require ZCML registrations. Zope 3 browser views/viewlets/resources and portlets are more powerful and easier for developers, and can be customised much more precisely (e.g. change a view for a given content type only, or invoke a viewlet only when a particular marker interface is applied to the current context).
I supported our move to embrace Zope 3 browser components. However, we need to unify the customisation story and remove the requirement to write any ZCML or Python for simple template-only (or image-only) customisation. I think z3c.jbot is a step in the right direction here, and it's on the shortlist for inclusion in Plone 3.1 or 3.2. However, there is still more thinking to be done here. We should start with a blank sheet of paper and dream up an ideal visual component customisation story, and then layer it on top of the two existing customisation mechanisms.
The TTW/Filesystem Schizophrenia Syndrome
I won't bore you with the old story of how we all used to do through-the-web (TTW) development and then realised this was really hard to deploy and control and jumped into filesystem products and then eggs and so on. However, sometimes it's easy to forget that one of the best things about Plone is that amount of configurability we have though-the-web. The ZMI is not alway ideal (I'd like to see the workflow tool become a graphical AJAXy thing in Plone...), but it's powerful and quick as hell. Formlib makes it easy to expose a bag of properties as a form, and we can turn those forms into pretty Plone control panels.
At some point, people need to go to the filesystem though. GenericSetup is all about exporting and importing state. It's nearly there, but there are gaps where handlers have not yet been written. We should make it a rule that our principal configuration GUI is through-the-web, and we have a way of synching all state to the filesystem. Ideally, this should include some basic content export/import feature that allows content to be exported independently of configuration.
GenericSetup is great, but not sufficient on its own. The portal_setup tool is cryptic and it's not that easy to synchronise profiles among multiple developers if you don't understand how to build profiles by hand. Furthermore, the most useful type of profile for most people is an extension profile that merely modify Plone (thus providing some hope that the same profile will work properly on future versions of Plone), whilst portal_setup is best at dumping out full base profiles that purge and then configure every setting. We should have an in-Plone GUI that make dumping, importing and synchronising configuration state easy and transparent. This would go a long way towards easing the migration from administrator/TTW integrator to filesystem integrator/developer and reduces some of the risk inherent in making changes TTW that may be hard to translate into a production environment.
Finally, I think we should consider that not all configuration state needs to be stored natively in the ZODB. If things are written to XML or ConfigParser files outside Zope and merely read in Plone, that makes this synching a non-issue. Sometimes, that's too much hassle, but new technologies we invent should consider this as a valid approach. To make this easier, I'd like to invent a way (using ZCML or entry points) that packages can say "I am a policy/configuration package, you can store a GS profile here" or "I am a theme product, you can dump customised templates here" or "I am a content type product, you can keep your content type definition here".
Grep is not your friend
As people start to become Plone programmers, they stare into the black abyss and wonder "where is the API?". Plone does not have an API. The aforementioned bliss of infinite customisability means that the whole code base is an API. Unfortunately, that's not terribly useful to new developers.
We certainly have aspects of Plone that are designed to be pluggable with a well-thought-out API. We need more of those, and we need to document them better. We should identify the most common extension points (new content types, new forms, need to load a configuration profile and so on) make sure they have the simplest possible APIs, and then document them properly.
With the adoption of Zope 3 technologies, we've also gotten a lot better at writing sensible interfaces with minimal documentation. If you've read the documentation to the point where you know what to look for, these - and the raw source code - probably provide enough detail to get you started. Better documentation is always desirable, but a comprehensive API reference with examples and background explanations everywhere is probably never going to materialise given the size of that task.
The usual answer here is to use grep - search the source code and look for the relevant docstrings... or read the Python code to figure out what's going on. Sometimes, that's actually a really useful approach, but it's only really useful if you know what to search for and roughly where to search. That all comes with experience that you wont' get if you can't find anything in the first place.
Now, imagine we had an all-powerful introspection tool that combined Clouseau, DocFinderTab, the @@manage-viewlets screen, Zope 3 APIDoc and a few more bits of information into one compact developer tool. Call it the Plone Introspector. With this tool installed, a developer could peel back the layers on any screen in Plone and get information about where the main view, the viewlets and portlets, and adapters and event handlers registered for the context, and whatever else all came from. With links to API documentation generated on-the-fly from the code of all installed modules and further links to canonical documentation on plone.org, this would make it much easier to see where to start. We could even provide an in-browser code browser for drilling into this information. The icing on the cake would be buttons to generate the relevant code that would be needed to customise any element found in the Introspector, dumping it straight into a filesystem product or a TTW customisation container. Raise your hand if you'd want one of these!
The good news is that this actually pretty doable. Plone and Zope are quite self-documenting. This tool alone would go a long way towards making Plone more approachable to new developers and could smoothen the learning curve significantly.
My environment floweth over
We currently have a few different ways of building a Plone environment - old-fashioned instances, buildouts, virtualenvs. Some of these overlap more than others. This causes a documentation headache, confuses people, and makes it harder to create in-Plone tools that allow some environment configuration (say a control panel to download and install third-party products).
We should have one configuration management system, and that system should be zc.buildout. I realise some people have other preferences, and we shouldn't make it impossible to use non-buildout options, but buildout should be the default, the one we document and the one people are assumed to be using. I think this is already happening, in fact.
One nice thing about buildout, is that we can hide specific policy behind recipes. Right now, we use plone.recipe.plone to configure a Plone instance using a combination of downloaded products and eggs. Once Plone is all-eggs, we'll change the recipe a bit, but users won't even notice. Buildout also makes it easier to have a single environment with options to switch between development and deployment configurations.
Installing third party products should also be easier. Ideally, it should be as possible though a Plone GUI, although that may be tricky in some cases. If not, dropping a file in a directory is a good approach, as is changing one line (not two) in a buildout.cfg configuration.
Oh, and ZCML slugs must die. That's pretty easy, actually, and we mostly have code for this. Any product that you explicitly add to your Plone environment that broadcasts that it's a Plone product should be loaded automatically by Plone.
Skeletons in my closet
ZopeSkel has transformed the way developers work with Plone. Like Daniel Nouri, I don't much like code generation - it's a symptom that your API requires too much boilerplate. However, I do like skeleton generation. By using Paste Script and ZopeSkel, we can ensure that we present a unified interface to getting a "good practice" code skeleton up and going quickly. By adding just enough example code and comments, we can make it easy for a new developer to get started.
At a minimum, I think we need skeletons for a "policy product" that contains GenericSetup configuration and site-specific customisations; a "theme product" that can house a theme; and a "content type product" that contains a simple content type. As I've hinted at before, these should include ZCML or entry points that broadcast to Plone that you've installed them, and Plone should then be able to use them intelligently, for example by supporting synchronisation of TTW state to these products.
Finally, it's crucial that the skeleton code is kept up to date at all times. Even if the documentation is a little bit out of date, the code you get should be up-to-the-minute accurate (and bug-free). If the core developers use these tools (as they should prefer if they are well-designed), this won't be a problem.
I'd like my database now, thankyouverymuch
A lot of thought has gone into whether or not Plone needs to use relational databases out of the box. Sometimes, that's purely for marketing reasons (Oracle is an easier sell than the ZODB). Those needs can be met by RelStorage (which replaces PGStorage) - a way of sticking the ZODB pickles in Oracle or PostgreSQL. That doesn't really give you and kind of relational data storage, though.
We could also consider having options for storing content data in relational schemata, either with or without a ZODB stub. I question how widely useful that is, though. Being able to talk to an RDBMS using some Plone-controlled schema is not the same as integrating with an existing database. Laurence Rowe's collective.tin is an interesting integration technology, but I suspect it's more a tool for advanced integrators than a must-have for Plone core.
If I were to suggest a policy, it'd be that Plone should not attempt to be a general-purpose SQL CRUD tool. For a large number of people, the SQL integration story will be "here's SQLAlchemy, here's the way you integrate it with Plone (collective.lead), here's an example tutorial, now go write some views and leave us alone". For people who need more general-purpose SQL CRUD forms, they should look to Grok, Django and others, and maybe integrate those with Plone using Repoze as outlined above.
Speed is addictive
Slow things are frustrating. I actually think Plone's performance is fine if you know how to set it up. We should make that easier by giving CacheFu an overhaul, both from an architecture and a UI perspective. It could make more sensible predictions about non-core content types and deal with authenticated users better, for example. Integration with Varnish as a preferred caching reverse proxy is pretty easy nowadays, and should be part of our standard configuration.
However, speed also applies to developers. Restarts suck. Slow restarts suck more. We should investigate doing auto-restarts when .py files are changed, the way paster serve --reload does. In general, though, development techniques should not require restarts. Being able to auto-refresh code - especially that in Zope 3 browser views, viewlets and portlets - would be a boon. Even if it only worked on 80% of cases, it'd be better than nothing.
Finally, we should make sure that tests run as fast as possible to encourage people to write more of them. Stefan Holek is king here, and there are already many improvements in Zope 2.11. However, we should make it easier to write lightening-fast tests by providing sensible mocks to cover unit test scenarios and encourage the separation of deeper integration tests into separate test files that can be ignored during normal test runs.
Deployment is tricky
One of the most common questions we get is "can I use Plone with my $5/mo web hosting". We tend to brush such people off - Plone is a serious system that requires a serious server. But sometimes that serious server is only required for content editing, and all anyone wants is a static website that goes on a fast server that hosts Apache only. We should have a static deployment story, even if it doesn't cater to every possible need. The architecture here needs to be pluggable so that people can transform and control the output for specialised deployments.
We should also make deployment using Repoze and mod_wsgi, straight into Apache, a realistic, if not preferred, option. This is much easier (and possibly more performant) than a Zope process on a different port.
Finally, enabling SSL support in your site should be as simple as ticking a box in Plone UI, provided you have set up Apache for SSL as well.
I think that's plenty to start with. Save for a couple of minor points, however, I think that most developers with a sufficiently deep understanding of Plone could see how most of this would work without too much of a stretch.
Each of these areas (and other ones) will need a "champion" in the community - someone to do the legwork and cheer leading and actually make it happen. If that excites you, please get in touch! Much of this is work that it's hard to find paying sponsors for, but that hasn't stopped us before and we can't afford for it to stop us now. Make no mistake - our competition is sharper than ever, and our users are better informed, with higher expectations. But they are loyal, they love Plone and the Plone community. Together, we can bring the complexity bar down a notch and put Plone at the forefront of flexibility and ease-of-use once again.