You are here: Home Articles An über-buildout for a production Plone server
Navigation
OpenID Log in

 

An über-buildout for a production Plone server

by Martin Aspeli last modified Jan 06, 2009 09:01 PM

This one, to be precise, and it includes nginx, Varnish, load-balanced ZEO clients, log rotation and more goodness.

This weekend, I decided to finally learn a bit more about nginx. I actually really like it. It's much easier to understand and configure than Apache is, and is reportedly also a lot faster, at least at high loads (I would love to have some equivalent of mod_svn, though).

Encouraged by this, I set about updating this server to use nginx. In doing so, I discovered the gocept.nginx recipe, which can configure nginx in a buildout. I decided to create a standalone über-buildout that would contain all the services I need on the server. You can find it at the end of this post.

Update #2 (supersedes #1): I've simplified the ZEO client setup a little using plone.recipe.zope2cluster. I think it could be made a bit more elegant with zc.recipe.macro, but that doesn't quite work as advertised when creating parts on the fly. I also removed buildout.eggtractor in favour of explicitly stating develop eggs (it's a production server, after all), and set up a specific versions block to pin zc.recipe.testrunner and zope.testing, which was causing some problems in some configurations.

The buildout sets up the following things:

  • Plone 3.2
  • Develop eggs go in src/, other eggs are listed in a separate section.
  • A ZEO storage server.
  • Two ZEO clients (it should be easy to add more).
  • A load balancer for these that also takes care of virtual hosting. This uses a standalone nginx instance. I could've used pound or HAProxy or something similar, of course, but nginx was nice and easy to set up.
  • A Varnish caching proxy.
  • An nginx instance to go in front, with two virtual hosts: one for Plone, and one for a simple file server.
  • A supervisor to control it all.

All you need is the buildout.cfg and the boilerplate bootstrap.py to get it going.

The buildout.cfg file is quite long, but I've tried to keep it well structured and commented. Furthermore, common settings such as hostnames, ports, usernames and passwords are kept in separate sections near the top.

On my server, I now have a small file called production.cfg that looks like this:

[buildout]
extends = buildout.cfg

[instance-settings]
user = zope-admin:password

[supervisor-settings]
user = supervisor-admin
password = password 

[plone-sites]
main = plone-instance-id

[hosts]
main = www.server.com
fileserver = files.server.com

[ports]
main = 80

[users]
main = www
cache = www
balancer = www
zope = www
supervisor = www

With this, I do:

$ python bootstrap.py # once
$ ./bin/buildout -c production.cfg
$ ./bin/supervisord

Actually, I lie slightly... I also had to do some chowning on the parts/, var/ and files/ directories so that they were owned by user www.

I can monitor all the servers using the very nice supervisor web GUI on 9001, and stop/start services there. To shut everything down in one go, I'd do:

$ ./bin/supervisorctl shutdown

I have a system startup script that basically does this.

The buildout also creates a logrotate file (parts/logrotate.conf) that can be used with the logrotate service by symlinking it to e.g. /etc/logrotate.d. This rotates the logs for all the services weekly (well, Varnish doesn't have a logfile, and supervisor does its own log rotation).

There are a few things that could be improved still:

  • I'd love to have a way to add more ZEO clients with a configuration parameter, but this would require some more clever recipes and may make everything a bit too obtuse.
  • Having all the configuration in buildout.cfg is nice. On the other hand, it means that changing the configuration means re-running buildout. This only takes a second, since the build-intensive operations are in separate parts that are not re-evaluated each time, but it's easy to forget. And of course, the recipes only go so far and may not allow you to edit all configuration options.
  • I could have used Varnish's load balancing capabilities and skipped the separate load balancer. I'm not sure if this would give better performance. In the event, I wanted to experiment with a separate configuration. Also plone.recipe.varnish does not support load balancing at the moment, and so I would've had to use a custom file.
  • Depending on whether the buildout is run as the effective user, you may need to do some chown'ing to get the right permissions on things. It'd probably be possible for buildout to do that, but I'm not sure it's a good idea to start messing with file permissions automatically like that. Perhaps for var/, parts/ and files/, though.

Here is the buildout. Feel free to use it if it's useful. I'd be grateful if anyone wants to turn this into a proper how-to on plone.org with a bit more background to explain what these pieces all do and how to use them.

# Deployment buildout
# ===================
# 
#  This buildout configures a number of servers:
#
#   - 'main', an nginx web server that may run on port 80
#   - 'cache', a varnish cache that is configured to proxy a Plone site
#   - 'balancer', an nginx instance used as a load balancer for ZEO clients
#   - 'zeoserver', a ZEO server
#   - 'instance1' and 'instance2', two ZEO clients
# 
# Log rotation is configured for all of these, except Varnish, which does not
# write a log file. The log rotation configuration file is in 
# parts/logrotate.conf and this needs to be symlinked into the main logrotate
# configuration.
# 
# In the 'main' nginx configuration, virtual hosting will be enabled for a
# single Plone site and a file server. This can be adjusted as required.
#
# Finally, a supervisor instance is set up. To start it all up, do:
#
#  $ ./bin/supervisord
#
# Go to http://localhost:9001 to see supervisor status.
#
# The configuration is fully contained within this buildout. Hostnames, ports
# and common options can be changed in the buildout sections at the top of
# of this file. More detailed configuration is contained in the relevant
# parts, further down the file.

[buildout]
newest = false
parts =
    zope2
    zeoserver
    
    instance1
    instance2
    
    nginx-build
    varnish-build
    
    fileserver-directory
    
    main
    cache
    balancer
    
    logrotate.conf
    
    supervisor
    zopepy
    omelette

develop = 
    src/Products.NuPlone

# Plone version - 3.2

find-links = 
extends = http://dist.plone.org/release/3.2/versions.cfg
versions = versions

# Version pins, above and beyond what the Plone 3.2 versions block says

[versions]
zc.recipe.testrunner = 1.1.0
zope.testing = 3.6.0

# Download urls - encapsulate Zope, Varnish and nginx versions

[downloads]
zope = http://www.zope.org/Products/Zope/2.10.6/Zope-2.10.6-final.tgz
varnish = http://downloads.sourceforge.net/varnish/varnish-2.0.2.tar.gz
nginx = http://sysoev.ru/nginx/nginx-0.7.30.tar.gz


# Default settings for ZEO clients. http-address is set with zc.recipe.macro

[instance-settings]
eggs = 
    Plone
    Products.CacheSetup
    Products.NuPlone
zcml =
products = 
user = admin:admin
zodb-cache-size = 5000
zeo-client-cache-size = 300MB
debug-mode = off
zope2-location = ${zope2:location}
zeo-client = true
zeo-address = ${zeoserver:zeo-address}
effective-user = ${users:zope}
http-address = $${:http-address}

# Default settings for supervisor

[supervisor-settings]
user = admin
password = admin

# Plone sites - used in VirtualHost configuration

[plone-sites]
main = blog

# Hostnames for various servers. 'main' is the public hostname.

[hosts]
main = www.example.com
fileserver = dist.example.com
cache = 127.0.0.1
supervisor = 127.0.0.1
balancer = 127.0.0.1
instance1 = 127.0.0.1
instance2 = 127.0.0.1

# Ports for various servers. 'main' is for the public hostname

[ports]
main = 8000
cache = 8101
balancer = 8201
zeo-server = 8301
instance1 = 8401
instance2 = 8402
supervisor = 9001

# OS users to drop to for various processes

[users]
main = optilude
cache = optilude
balancer = optilude
zope = optilude
supervisor = optilude

##############################################################################  
# 1. Zope - build, ZEO server, ZEO clients
##############################################################################

[zope2]
recipe = plone.recipe.zope2install
fake-zope-eggs = true
additional-fake-eggs = 
    ZConfig
    pytz
skip-fake-eggs =
    zope.testing
url = ${downloads:zope}

[zeoserver]
recipe = plone.recipe.zope2zeoserver
zope2-location = ${zope2:location}
zeo-address = ${ports:zeo-server}
effective-user = ${users:zope}

# Generate instances using zc.recipe.macro - we have one macro that
# contains the shared settings, 

[instance1]
recipe = collective.recipe.zope2cluster
instance-clone = instance-settings
http-address = ${hosts:instance1}:${ports:instance1}

[instance2]
recipe = collective.recipe.zope2cluster
instance-clone = instance-settings
http-address = ${hosts:instance2}:${ports:instance2}

##############################################################################  
# 2. Build nginx and varnish for later configuration
##############################################################################

[nginx-build]
recipe = zc.recipe.cmmi
url = ${downloads:nginx}

[varnish-build]
recipe = zc.recipe.cmmi
url = ${downloads:varnish}

##############################################################################  
# 3. Create directories
##############################################################################

[fileserver-directory]
recipe = ore.recipe.fs:mkdir
path = ${buildout:directory}/files

##############################################################################  
# 4. Configure front-end web server
##############################################################################

[main]
recipe = gocept.nginx
nginx = nginx-build
configuration = 
    user ${users:main};
    error_log ${buildout:directory}/var/log/main-error.log warn;
    worker_processes 1;
    daemon off; 
    events {
        worker_connections 1024;
    }
    http {
        # Proxy to Varnish cache
        upstream cache {
            server ${hosts:cache}:${ports:cache};
        }
        
        # Main server goes upstream to Varnish
        server {
            listen *:${ports:main};
            server_name ${hosts:main};
            access_log ${buildout:directory}/var/log/main-plone-access.log;
            location / {
                proxy_pass http://cache;
            }
        }
        
        # File server
        server {
            listen *:${ports:main};
            server_name ${hosts:fileserver};
            access_log ${buildout:directory}/var/log/main-dist-access.log;
            autoindex on;
            root ${buildout:directory}/files;
        }
        
    }

##############################################################################  
# 5. Configure Varnish for Plone
##############################################################################

[cache]
recipe = plone.recipe.varnish
daemon = ${buildout:directory}/parts/varnish-build/sbin/varnishd
bind = ${hosts:cache}:${ports:cache}
backends = ${hosts:balancer}:${ports:balancer}
cache-size = 1G
user = ${users:cache}
mode = foreground

##############################################################################  
# 6. Configure load balancer
##############################################################################

[balancer]
recipe = gocept.nginx
nginx = nginx-build
configuration = 
    user ${users:balancer};
    error_log ${buildout:directory}/var/log/balancer-error.log warn;
    worker_processes 1;
    daemon off; 
    events {
        worker_connections 1024;
    }
    http {
        upstream zope {
            server ${hosts:instance1}:${ports:instance1} max_fails=3 fail_timeout=30s;
            server ${hosts:instance2}:${ports:instance2} max_fails=3 fail_timeout=30s;
        }
        server {
            listen ${hosts:balancer}:${ports:balancer};
            server_name ${hosts:main};
            access_log off;
            rewrite ^/(.*)  /VirtualHostBase/http/${hosts:main}:${ports:main}/${plone-sites:main}/VirtualHostRoot/$1 last;
            location / {
                proxy_pass http://zope;
            }
        }
    }
    
##############################################################################  
# 7. Set up supervisor to run it all
##############################################################################

[supervisor]
recipe = collective.recipe.supervisor
port = ${ports:supervisor}
user = ${supervisor-settings:user}
password = ${supervisor-settings:password}
serverurl = http://${hosts:supervisor}:${ports:supervisor}
programs =
    10 zeo       ${zeoserver:location}/bin/runzeo                  true ${users:zope}
    20 instance1 ${buildout:directory}/parts/instance1/bin/runzope true ${users:zope}
    20 instance2 ${buildout:directory}/parts/instance2/bin/runzope true ${users:zope}
    30 cache     ${buildout:directory}/bin/cache                   true ${users:cache}
    40 balancer  ${nginx-build:location}/sbin/nginx [-c ${balancer:run-directory}/balancer.conf] true ${users:balancer}
    50 main      ${nginx-build:location}/sbin/nginx [-c ${main:run-directory}/main.conf]         true

##############################################################################  
# 8. Log rotation
##############################################################################

[logrotate.conf]
recipe = zc.recipe.deployment:configuration
text = 
    rotate 4
    weekly
    create
    compress
    delaycompress

    ${buildout:directory}/var/log/instance1*.log {
        sharedscripts
        postrotate
            /bin/kill -USR2 $(cat ${buildout:directory}/var/instance1.pid)
        endscript
    }
    
    ${buildout:directory}/var/log/instance2*.log {
        sharedscripts
        postrotate
            /bin/kill -USR2 $(cat ${buildout:directory}/var/instance2.pid)
        endscript
    }
    
    ${buildout:directory}/var/log/zeoserver.log {
        postrotate
            /bin/kill -USR2 $(cat ${buildout:directory}/var/zeoserver.pid)
        endscript
    }
    
    ${buildout:directory}/var/log/main*.log {
        sharedscripts
        postrotate
            /bin/kill -USR1 $(cat ${main:run-directory}/main.pid)
        endscript
    }
    
    ${buildout:directory}/var/log/balancer*.log {
        sharedscripts
        postrotate
            /bin/kill -USR1 $(cat ${balancer:run-directory}/balancer.pid)
        endscript
    }
    
##############################################################################  
# 9. Debugging tools - preconfigured python interpreter and omelette
##############################################################################

[zopepy]
recipe = zc.recipe.egg
eggs = ${instance-settings:eggs}
interpreter = zopepy
extra-paths = ${zope2:location}/lib/python
scripts = zopepy

[omelette]
recipe = collective.recipe.omelette
eggs = ${instance-settings:eggs}
products = ${instance-settings:products}
packages = ${zope2:location}/lib/python ./
Document Actions

varnish > nginx > zope

Posted by http://leman78.myopenid.com/ at Jan 05, 2009 07:13 AM
if I was right you have the following construction:
nginx(virtHosting) > varnish(Cache) > nginx(loadbalancing) > Zope's

why not as below:
varnish(cache) > nginx(virtHosting/loadbalancing) > Zope's

regards maik derstappen

varnish > nginx > zope

Posted by Martin Aspeli at Jan 05, 2009 08:53 AM
Because I want a file server on a different virtual host (files.server.com vs. www.server.com). Varnish can't act as a fileserver.

Also, the Varnish configuration from plone.recipe.varnish is quite Plone-specific. I wouldn't want it in front of anything other than a Plone site.

Martin

Session Affinity

Posted by http://laurencerowe.myopenid.com/ at Jan 05, 2009 09:14 AM
I don't think nginx does session affiinity (using a cookie to direct users consistently to the same client). If you have lots of logged in traffic this can be quite important. This is a good reason to use haproxy. In a big intranet we have:

haproxy -> varnishes -> haproxy -> zopes

Session Affinity

Posted by Martin Aspeli at Jan 05, 2009 09:24 AM
Indeed. Jarn have released a buildout recipe for haproxy, but I couldn't figure out how it worked. ;-)

For this site, I certainly don't need session affinity. I'm not sure if session affinity makes much difference when not using session data, but I suppose it might since the cache is more likely to be relevant if the user stays on the same node.

If we could make haproxy as easy to set up as the nginx balancer config above, that'd be really great.

Martin

zeo client config

Posted by http://claytron.myopenid.com/ at Jan 05, 2009 11:42 PM
I hadn't seen the zc.recipe.macro, that's pretty wicked. I wrote a similar recipe (around the same time) that does the same thing for the single purpose of creating a zeo cluster:

http://pypi.python.org/pypi/collective.recipe.zope2cluster/

I wonder how zc.recipe.macro would act in a multi config setup like the one I presented at the Plone conf (http://tr.im/2B5b). Say I wanted to extend the config you have and add "eggs += my.package". At first glance, I'm not sure how you'd do that.

zeo client config

Posted by Martin Aspeli at Jan 06, 2009 05:13 AM
Nice! I think I may use the zope2cluster recipe instead. zc.recipe.macro is nice, but I can't get it to work in the way I really want. It's supposed to be able to create parts on the fly, so that I could list only an 'instances' part in the buildout config and it would spawn several other things. However, I can't get that functionality to work.

Martin

zeo client config

Posted by Martin Aspeli at Jan 06, 2009 09:02 PM
I just updated the buildout (and this blog post) to use plone.recipe.zope2cluster. I can't quite get zc.recipe.macro to generate parts in the way it should. I need to test out something Aaron suggested, but for now I think plone.recipe.zope2cluster is the best bet.

Cheers,
Martin

Using the cmmi vs your systems package manager

Posted by Calvin Hendryx-Parker at Jan 08, 2009 10:19 AM
If you were to compromise a bit on portability, wouldn't it be better to let the system manage the installation of the packages? You are going to miss out on some of the default facilities such OS specific patches and the automated tools for checking if the packages are out-of-date or have security issues. For example FreeBSD has a port audit tools that will let you know (via a cronjob) nightly if there are any security issues with any installed packages.

Just my $0.02

Using the cmmi vs your systems package manager

Posted by Martin Aspeli at Jan 09, 2009 07:30 PM
This is certainly true. However, nginx and varnish are both quite new and many operating systems don't have up-to-date packages for them. Also, I quite like the idea of having everything happen in one buildout. I can put this on a new server and be pretty sure it's going to work reliably.

Still, this is the nice thing about buildout - the CMMI parts are separate, and you could easily disable them and use OS level packages.

Martin
Sponsor message

Many people new to computers tend to think that work at home is quite easy, however the first detail is choosing a catchy and good domain name for branding.

Plone Book
Professional Plone 4 Development

I am the author of a book called Professional Plone Development. You can read more about it here.

About this site

This Plone site is kindly hosted by: 

Six Feet Up