I’m working on a suite of buildout configuration files to use for different circumstances. Having a special buildout for development in a zeo cluster, and another for a single monolithic zope instance (plus useful debug tools already installed) was easy to do and is working well.

I then get to the notion of staging vs production. We stage our plone sites on a separate server, often using the staging instance for content development, debugging, etc. The IP addresses everything is listening on, the ports, etc, may need to vary between staging and production, but everything else has to be identical.

The idea of just running bin/buildout -c staging.cfg on the staging server seemed very elegant.

The production buildout.cfg does a lot. I’ve got homemade recipes that set up process control, multiple facets of apache (load balancer, ssl), etc. My first idea was to just duplicate the production buildout.cfg, and then modify it as needed, but I figured that would be hard to maintain. There had to be a better way.

So I read through the zc.buildout docs, and the Adding And Removing Options section caught my eye.

So say my production buildout.cfg file looks like this:

[buildout]
parts = apache
            ssl
            client1
            client2
            zeoserver
... 

[client1]
recipe = plone.recipe.zope2instance
zope2-location = ${zope2:location}
user = me:pass
http-address = 8080

[apache]
recipe = my.apache.recipe
listen = 243.25.21.2:80

[ssl]
recipe = my.ssl.recipe
cert_file = /blah/blah
key = /secure/server.key
httpd_part = apache

... etc, you get the idea

And my staging.cfg file looks like this:

[buildout]
extends = buildout.cfg

[client1]
http-address = 9090

[apache]
listen = 127.0.0.1:5500

So I’m able to selectively override options in arbitrary parts. This is great. I like the idea of making the config that represents reduced functionality subtract functionality out of the other. Worries of having to maintain two files, or worse, have a dissimilar staging environment are mitigated.

But there’s a problem. Let’s say that on staging, I don’t care about SSL. I need to disable this part somehow, and from what I can tell, at least from the docs (code spelunking pending) there’s no way to do this.

I tried using the -= syntax that’s useful for lists of option values:

[buildout]
extends = buildout.cfg
parts -= ssl

[client1]
http-address = 9090

[apache]
listen = 127.0.0.1:5500

If this had worked, it would have been EPIC. But alas, it did not (should it have worked?).

So I poked around some more and checked with some folks on IRC with no luck.

I sort of solved the problem, by just repeating all the parts and then commenting out the ones I didn’t want:

[buildout]
extends = buildout.cfg
parts = apache
#          ssl
            client1
            client2
            zeoserver

[client1]
http-address = 9090

[apache]
listen = 127.0.0.1:5500

This works, but I don’t like it :P

I wonder if this is intentional, a bug, or just an oversight in zc.buildout. I’ll dig further when I’ve got time.

I was recently took up the task of engineering my own plone infrastructure. I want to share what I did, and get any feedback I can about the scalability and usefulness of what I’ve done.

Requirements

• SSL – It must be secure while logged-in users are on the site*.
  • It must NOT be secure for public pages (unless the user is logged in)
  • It must force SSL for login pages.
• Hide /Plone – the infrastructure must present the user with URLs that don’t contain the plone instance id.
• Admin access – the infrastructure must provide SSL-encrypted access to the root ZMI.

* I understand that in recent versions of plone, this isn’t important to protect the user’s session cookies, however, I’m concerned about SSL encryption after login due to the content that will be stored on the site.

Approach

I’m very familiar with the Apache httpd server, so I wanted to leverage that experience as much as possible. I’m using the Apache 2.2.9 package that comes with Debian Lenny.

I used Plone 3.3.1 as I was developing this infrastructure, but I don’t think there’s anything about it that wouldn’t work with any version of Plone (as long as it’s got a virtual host monster).

We’ve settled on using one IP address (an alias) per Plone stack to keep the sites autonomous. This makes deployment and host registration easier.

I used IP-based Virtual Hosting and mod_proxy in conjunction with mod_proxy_balancer.

I’m logging to the local disk, and will be setting up a log rotation scheme separately.

Here’s what it looks like:

Production Plone Infrastructure (Existing Setup)

So I’ve got 3 Zope instances, acting as Zeo clients, on ports 1040, 1041, and 1042. I’ve heard more is just wasteful and fewer tends to not be enough.

I then have three virtual hosts. One is the load balancer, and back-ends to the Zope instances. It’s listening on port 8080 (the traditional proxy port, it really could be anything).

The other two virtual hosts are for "public" traffic (port 80), and SSL (port 443). They both back-end via a ProxyPass directive to the balancer virtual host. What’s different is what they tell the Virtual Host Monster to rewrite the URLs to and the RewriteRules that nudge the browser to use the right virtual host when doing certain actions, like logging in and logging out. Of course, the SSL virtual host is set up to do SSL using my SSL certificate (self-signed for now).

Configuration

I’ve got the config files sort of spread out because I’m using some buildout recipes I wrote to configure all this stuff, but here’s what it looks like all mashed into one file, pretending that you’re not using my recipes:

# httpd.conf for Apache2 - inspired by:  http://www.links.org/?p=264

ServerRoot  /etc/apache2

ServerName  localhost

Listen 127.0.0.1:8080
Listen 127.0.0.1:443
Listen 127.0.0.1:80

PidFile  /var/run/apache2.pid
LockFile /var/lock/apache2/accept.lock

LoadModule alias_module            /usr/lib/apache2/modules/mod_alias.so
LoadModule auth_basic_module       /usr/lib/apache2/modules/mod_auth_basic.so
LoadModule authn_file_module       /usr/lib/apache2/modules/mod_authn_file.so
LoadModule auth_digest_module      /usr/lib/apache2/modules/mod_auth_digest.so
LoadModule authz_user_module       /usr/lib/apache2/modules/mod_authz_user.so
LoadModule authz_host_module       /usr/lib/apache2/modules/mod_authz_host.so
LoadModule authz_groupfile_module  /usr/lib/apache2/modules/mod_authz_groupfile.so
LoadModule dir_module              /usr/lib/apache2/modules/mod_dir.so
LoadModule include_module          /usr/lib/apache2/modules/mod_include.so
LoadModule mime_module             /usr/lib/apache2/modules/mod_mime.so
LoadModule ssl_module              /usr/lib/apache2/modules/mod_ssl.so
LoadModule cgi_module              /usr/lib/apache2/modules/mod_cgi.so
LoadModule rewrite_module          /usr/lib/apache2/modules/mod_rewrite.so
LoadModule headers_module          /usr/lib/apache2/modules/mod_headers.so
LoadModule proxy_module            /usr/lib/apache2/modules/mod_proxy.so
LoadModule proxy_http_module       /usr/lib/apache2/modules/mod_proxy_http.so
LoadModule proxy_balancer_module   /usr/lib/apache2/modules/mod_proxy_balancer.so
#[BUILT-IN] LoadModule log_config_module  modules/mod_log_config.so
#[BUILT-IN] LoadModule logio_module       modules/mod_logio.so

# worker MPM - tuned
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
StartServers           1
MaxClients            16
MinSpareThreads        2
MaxSpareThreads        8
ThreadsPerChild        8
MaxRequestsPerChild    0

User www-data
Group www-data
ServerAdmin me@somehost.com

DirectoryIndex index.html

# The following lines prevent .htaccess and .htpasswd files from being
# viewed by Web clients.
<FilesMatch "^\.ht">
    Order allow,deny
    Deny from all
    Satisfy All
</FilesMatch>

DefaultType text/plain

<IfModule mime_module>
    TypesConfig /etc/mime.types
</IfModule>

# "combinedio" includes actual counts of actual bytes received (%I)
# and sent (%O) - requires the 'mod_logio' module

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

ErrorLog /var/log/apache2/error.log
CustomLog /var/log/apache2/access.log combinedio

### Zeo Cluster Virtual Host ###################################################
<VirtualHost 127.0.0.1:8080>
    ProxyPreserveHost On

    Header add Set-Cookie "balancer_zopes=route.%{BALANCER_WORKER_ROUTE}e; path=/;"

    ProxyPass / balancer://zeocluster/
    <Proxy balancer://zeocluster>
        BalancerMember http://127.0.0.1:1040 route=client1
        BalancerMember http://127.0.0.1:1041 route=client2
        BalancerMember http://127.0.0.1:1042 route=client3
    </Proxy>
</VirtualHost>

### SSL Virtual Host ###########################################################

# example from http://httpd.apache.org/docs/2.2/mod/mod_ssl.html#sslsessioncache
# I think this should be conservative
SSLSessionCache shm:/usr/local/apache/logs/ssl_gcache_data(512000)

<VirtualHost 127.0.0.1:443>
    ServerName localhost

    SSLEngine on

    # this didn't work...
    # SSLCACertificatePath ...

    SSLCertificateKeyFile /home/jj/localhost.key

    # SSLCertificateChainFile /home/jj/localhost.ca-bundle

    SSLCertificateFile /home/jj/localhost.crt

    # ProxyPassInterpolateEnv On
    RedirectMatch /.*logged_out$ http://localhost:80/

    ProxyPass /admin http://localhost:8080/VirtualHostBase/https/localhost:443/VirtualHostRoot/_vh_admin
    ProxyPass / http://localhost:8080/VirtualHostBase/https/localhost:443/plone/VirtualHostRoot/

</VirtualHost>

### Non-SSL Virtual Host #######################################################

<VirtualHost 127.0.0.1:80>
    ServerName localhost

    ProxyPass / http://localhost:8080/VirtualHostBase/http/localhost:80/plone/VirtualHostRoot/
    RewriteEngine On
    RewriteCond %{HTTP_COOKIE} "__ac="
    RewriteRule ^(.*) https://%{SERVER_NAME}:443$1 [L]
    RewriteRule (.*)/login_form https://%{SERVER_NAME}:443$1/login_form [R]
</VirtualHost>

I was able to start apache using this file on a fresh Debian Lenny VM (bear in mind you have to create a self-signed certificate and a private key first)

The command looks like this:

   $ sudo /usr/sbin/apache2 -f /path/to/above/httpd.conf &

This has to be done as root (using sudo here) since port 80 and 443 are "priveledged" ports.

I, of course, have a process management daemon (runit) that handles this for me (and buildout recipes to configure it), which I will blog about at some point

I initially had this set up for testing with SSH tunneling (ssh -L) for the ports, and it worked well. This did however affect the setup, primarily I had to change the VHM paths so the URLs would be re-written to the local side of the tunnels, so if I connected like this:

    $ ssh -L1400:localhost:80 -L1443:localhost:443 jj@vmhost

The ProxyPass directive for the SSL VH looks like this:

    ProxyPass /admin http://localhost:8080/VirtualHostBase/https/localhost:1443/VirtualHostRoot/_vh_admin
    ProxyPass / http://localhost:8080/VirtualHostBase/https/localhost:1443/plone/VirtualHostRoot/

And the ProxyPass directive for the other host:

    ProxyPass / http://localhost:8080/VirtualHostBase/http/localhost:1400/plone/VirtualHostRoot/

This way the urls are pointed back through the SSL tunnel.

Discussion

The general Apache configuration is an amalgam of the /etc/apache2.conf file that came with Debian and some website I found about optimization.

Cluster

The Zeo Cluster virtual host uses mod_proxy_balancer to round-robin the Zope clients.

To get "sticky" sessions, I set a cookie using mod_headers that holds the "route" of the client. The routs are defined as additional information passed to the BalancerMember, and manifest in an evnironmen variable called "BALANCER_WORKER_ROUTE".

This way once someone comes into the site, as long as the cookie persists, they will always be proxied to the same client. I opted to let the cookie expire when the user closes their browser since that’s how Plone is set up by default.

SSL

The SSL virtual host is a very straight forward SSL setup. Since I used a self-signed certificate, I didn’t need a "ca-bundle" file.

The interesting bit is when I use ProxyPass to route all traffic to the load balancer. Here I utilize the Virtual Host Monster (standard issue in Zope/Plone these days) to ensure urls are properly re-written to use HTTPS. And while I’m at it, I also "hide" the plone root (called /plone in my case) so that it looks like https://www.myhost.com/ is the root of the plone site, instead of https://www.myhost.com/plone. Nice.

There’s another ProxyPass directive that proxies any access to /admin to the load balancer, but also uses some VHM magic to give the user access to the root of Zope, and the core ZMI. This allows access since we’re proxying / to /plone As a bonus, since this directive only exists in the SSL virtual host, this also "forces" the user to use SSL to do root ZMI stuff, which helps protect the admin password (and gives the administrator warm fuzzies).

I also add a RedirectMatch directive to push the user over to the non-ssl virtual host when they log out. No sense in keeping them on SSL when there’s no need.

Non-SSL HTTP

Finally, the "public" virtual host. It consists of a ProxyPass directive similar to the one used in the SSL virtual host, except in this case, we’re rewriting to port 80, standard http. The VHM bits are the same.

The last few directives are rewrite rules to get the user to use SSL if they are logged in, and redirect the user to the SSL virtual host when they click the "log in" link. I’m sure there’s more to do to sew this up (like alter the template for the login portlet), but this covers most of the bases.

The Future

Production Plone Infrastructure (Future Plans)

Logging

I’d like to set up separate logging for each virtual host, possibly to syslogd (or some other collective logging solution).

Virtual Crash Cart Access

I may add a fourth Zope client. This fourth client wouldn’t be in the load balancer group and is solely responsible for handling ZMI requests. This was something Joel Burton mentioned at the Advanced Bootcamp, and I’ve always been keen on it. All the Zope clients can be busy and/or locked up, apache can be hosed, and I can still get to an unhosed client’s ZMI to try to shut things down or manually fix whatever caused the problem.

Caching

I haven’t included caching in this setup for two reasons:

1. I don’t like to preemptively optimize, and
2. Cache-fu does an extraordinarily good job just being there. :)

This infrastructure makes it easy to drop in a caching mechanism between the public virtual host and the load balancer virtual host if I need it.

I’m keen on mod_cache, since it can be loaded easily and keeps with my "do as much as we can in apache" mantra. However, my unscientific tests have shown little benefit from it’s use. Of course, that was before I came up with this scheme, and I was trying to keep everything in one apache host (which I’ve dubbed the "one host to unite them, one host to rule them" folly). Stay tuned for more on that.

Even if mod_cache doesn’t work out, there are still lots of other options, from Varnish to Squid to PigeonNet.

Summary

So here’s my attempt to do Plone Up Right^TM, and distill all of the many tutorials that are out there into what I think is one pretty good approach. Please, leave comments and let me know if I’m out of my mind, or setting myself up for some pain later. All criticism is welcome. If I’m doing it right, that might be good to hear too :)

I’ve been faced with a user requirement I’ve never had to deal with before, and I’m on the fence as to what would be the best solution. I’m looking for war stories, implementation strategies, other ideas, anything that might help me make this decision.

Here are my basic requirements:

• Multiple users will log into the following services with the same username/password:
  • Multiple Plone Sites
  • Samba
• Most will be affiliated with the university I work for, but some will be outside collaborators.
• We also need to store extra metadata about users:
  • Contact info
  • Application-specific info

Softer requirements:

• Group/Role Management
• Softer metadata requirements:
  • Store service info (which plone sites are they on, what Samba file shares do they have access to, etc)
  • Plone site-specific info

Our campus uses Active Directory. I’m able to authenticate against it in Plone and Samba. I have no direct access, however, to the AD or would ever be able to routinely modify it.

This situation is further complicated by the addition of users who are not affiliated with our university. For Plone this isn’t too big of a deal, but I have yet to figure out how to get Samba to authenticate against multiple sources (e.g. user not in AD? auth against local user).

So I’ve come up with some proposed scenarios, and have drawn up some diagrams to help convey the ideas.

Scenario 1

Here LDAP is used as the central storage for users, groups, roles, etc, and LDAP handles both authentication and authorization. The information is managed via an external management interface. There are many LDAP management tools out there, so this wouldn’t require any development to that end.

There’s an alternate way to handle authentication here using Kerberos directly in Plone instead of letting LDAP do it. I added this after talking to a colleague of mine who’s ultimately in the same boat, and had a lot of trouble getting OpenLDAP to auth over Kerberos.

Scinerio 2

Here the situation is almost the same, except all of the user and group management is handled in Plone. This centralizes the user management to one common web-fronted UI (including Samba). I’m a little apprehensive of this situation. I don’t know if I like filesystem access decisions being made by Plone, and I also am not sure how well Plone works with LDAP in a read-write relationship.

Scinerio 3

This option involves building a user management system, and storing all of our user info in a relational database. The user management system pushes out user and group info to the Plone sites via XML-RPC (or some sort of RESTful addon), and separately alters the local userbase and samba config files on the Samba server.

In this scenario, Plone does authentication via a PAS plugin that talks to the AD, and falls back to the RDBMS. Samba authenticates against PAM, which tries AD and falls back to the local system.

One caveat here, I haven’t researched PAM yet to be sure it can handle this.

So, I’m at an impasse. I don’t know what to do. Avoiding development effort would be nice, but I’m not afraid of building a solution if I need to. I’m apprehensive about cramming application-specific data into LDAP, but such things have been done before.

I don’t know what to do. Any input would be greatly appreciated. :)

I gave a talk on the “Great Saga of Content Type Creation in Plone” (read: AGX sucks but you should still do UML) at the Plone Symposium East 2009 at Penn State a month or two ago.

The video is available online. Give it a look.

I was scheduled against Nate Aune’s talk on Deliverance, a very hot topic, so I didn’t get as many people as I’d hoped, but the audience was great and the whole thing was a lot of fun. I turned some people onto other ways of making content, and got a ton of really positive and constructive feedback.

This was my first “major” presentation, so it was a big learning experience for me, as such, comments and suggestions are highly encouraged.

I’ll even put up with mild flaming ;)

If you’re skimming this, I’d like to direct your attention to the poll below

I’ve been using ZopeSkel and advocating it’s use for over a year now. I’ve participated in a major sprint on it, I’ve taught it to friends and colleagues, and I’ve come to a conclusion:

ZopeSkel SUCKS.

OK, so it doesn’t really suck. IMHO, it sucks a lot less than ArchGenXML. It’s still an essential part of my development tool kit, and I can’t see doing Plone development without it.

I’m giving a talk about ZopeSkel at the Plone Symposium East next month. I’m going to be advocating ZopeSkel over AGX for content type creation. It will surely be epic.

Back to the suckage: ZopeSkel is not without its flaws. It’s inherent architecture isn’t quite right. ZopeSkel took a somewhat specific tool (PasteScript) and abused it to do something outside the realm of the tools purpose. This is bad!

I hit upon some of the shortcomings of ZopeSkel in the Sprint Reportout I posted last year. Over the past few months I’ve been pondering those shortcomings and think it’s time for a major change.

So here are the major issues:

• ZopeSkel is out of PasteScript’s scope
• ZopeSkel is impossible to test accurately
• ZopeSkel interactive sessions are not easily reproducable
• ZopeSkel can’t respond to user input
• ZopeSkel’s documentation varies from overly adequate to non-existent
• ZopeSkel’s development is severely fragmented.
• People complain about ZopeSkel not having a graphical user interface

That said, ZopeSkel is still extremely useful and intuitive, it just needs some work by a few dedicated individuals. Ultimately, I’d like to see the product position itself as the defacto code generation tool in the Plone universe.

So here’s what I think needs to be done:

We need to re-write zopeskel from the ground up. I like the use of cheetah templates and context-aware commands. I’d like to see a robust framework for collecting information from the developer, saving it into an INI file (or something else, maybe YAML) and providing a framework where the user interface is decoupled from the data collection and template execution.

There’s even an opportunity here for an integration point with AGX.

It’s ambitious, yes, but I think it’s well within the realm of possibility.

Now we get to the reason why I’m posting this: I’m seriously contemplating sprinting on this at Penn State next month.

Due to funding problems I’m paying my own way to the symposium. I don’t want to plan a sprint and stay 2 extra nights if nobody is going to sprint with me. I’m also open to the idea of taking on a sponsor if someone wants to put up some money to see this work progress.

I need to know what you think. Please leave a comment, or use the poll below.

I had a template with a handful of tables that basically just displayed the same tabular data, just different data sets depending on the table. In case your curious, my data is segregated by workflow state.

I’ve had this problem before but in the past I had a skin layer registered too (thanks to ArchGenXML).

Defining macros is pretty easy, the Zope Book does a good job of explaining how to define them. It’s actually easier than it looks. If you don’t have a need for slots, it basically just comes down to slapping a metal:define-macro attribute into any arbitrary block of code.

Here’s a quick example:

<html>
<body>
    <p metal:define-macro="title">
        <span tal:replace="view/context/Title" />
    </p>
</body>
</html>

I’ve left out most of the usual ZPT window-dressings (like a i18n domain declaration), but I’ll assume you already know how to do that :).

This is also of course, a pretty stupid example. But it does illustrate an important concept in METAL: when your macro is being expanded, it’s as if you wrote the code right where the metal:use-macro attribute is used. So any variables that are available to the template that’s calling the macro are available here.

You have to be careful of this though, because it does limit how much your macro can be re-used.

In the past, because of the skin layer, I was able to drop a ZPT file into my skins directory, and call them like this (assume I put the last macro into a file called “macros.pt” in the skins directory along side this template):

<html>
<body>
    <p metal:use-macro="here/macros/macros/title" />
</body>
</html>

I saw a lot of examples where the name of the file itself was used (e.g. here/macros.pt/macros/title), but I don’t recall doing it that way.

As far as I can tell, this doesn’t work with templates on the filesystem that aren’t registered in a skin layer. So when you aren’t using a skins folder, because you’ve had the forethought to put that sort of stuff in a policy or theme product, this sort of macro expansion doesn’t seem to work.

I found a solution to this by accident. I don’t have my Zope 3 book (the essential Web Component Development with Zope 3) with me today, and was poking around Google aimlessly to solve this problem. I stumbled upon a page from a hereto unidentified Zope 3 course that showed me what I needed to do.

Zope 3 registers page templates, much like Zope 2/Plone does with the skins concept. But in Zope 3 you do it with ZCML as opposed to the CMF/GenericSetup glue that you need to register a Plone skin layer.

So in a handy configure.zcml (I put mine in the browser directory of my product), you first need to register the template containing your macros as a browser page:

<configure
    xmlns="http://namespaces.zope.org/zope"
    xmlns:browser="http://namespaces.zope.org/browser">
...
    <browser:page
      for="*"
      name="my.product.macros"
      template="macros.pt"
      permission="zope.Public"
    />

</configure>

Note that I since my macro file (macros.pt) is in the same directory as my configure.zcml, I didn’t have to specify any sort of relative path information in the template attribute.

Since the macro template is being registered like any other browser view, you can specify a particular interface that is only allowed to use it, using the for attribute. Likewise, you can set a specific permission that will restrict access to the macros using the permission attribute. I haven’t tested either attribute with a browser page that’s being used for macros, so I don’t know what the expected behavior is.

OK, so with the ZCML tag above added to configure.zcml and a quick restart of the server (I had trouble with plone.reload in this case, FYI), any macros in macros.pt are available, to any view on any content type that feels inclined to use them.

To access the macros within a browser view/page (or viewlet, etc), the syntax is similar to what it was in the “skins layer” scenario mentioned earlier, but it’s very Zope 3:

<html>
<body>
    <p metal:use-macro="view/context/@@my.product.macros/title" />
</body>
</html>

You access the macro by getting the browser page defined in the ZCML above applied to the current context. That’s it.

So to summarize: you can use a browser page like a sort of template/code snippet resource. And as a bonus, said resource can contain METAL macros. METAL macros are a nice way to clean up your code, enforce DRY, and reduce bugs.

Side Note:
I’ve also come to realize that the same @@viewname syntax works to apply a view to an object inside of a page template. This is a natural predecessor to the macro stuff I was just talking about, but I didn’t realize it was possible until now. Whoo-hoo!

I wrote the Adventures In Theming series about a month ago as I was working my way through developing my first theme product for Plone 3. Initally it was a very long single blog post, but I figured it would be more useful to people if I split it up and released it over the course of a couple of weeks.

WordPress didn’t seem to like that after the first few posts, at least in the atom feed, so I think I lost a lot of Planet Plone views. Sorry if anybody was following along. :) It looks like google is helping people find the posts, so it’s not soo bad, and anyway, I doubt anybody was waiting with bated breath for the each installment :P. It’s pretty hard to read if you just happen upon the individual articles, which sucks.

I’ve just put all of the posts into one page as promised. If you’ve done themeing, or want to do theming, please take a gander. As with anything I write, I’m always open to suggestions, corrections, etc. I got some editing help from Rob Lineberger before I broke my original post up into a series, which I think helped the series a lot (so buy him a beer and tell him to get a blog already if you see him… or ever meet him). I haven’t audited the code in a while, that’s on my todo list, but I can always use help with that. Getting to the point: be a good blogospherizan and give me some feedback :)

So anyway, enough epilouge-ing, there’s some new developments that need to be at least documented (does anyone actually read this far down into a blog post?) to help wayword googlers and stranded plonistas learn from my mistakes.

Recall my original implementation idea from my first post:

I thought that because viewlets were registered providers, I could easily get the stock plone viewlets I wanted to re-organize to show up in my new viewlet by simply using the provider prefix in TALES expressions in my page template.

It just didn’t work. I was met with weird errors, and spent a lot of time trying to work around them (I just couldn’t blame Plone, it had to me my fault)

I didn’t do much digging until after my theme was complete. I traced the problem down to the stock viewlets, but didn’t realize that the problem was actually with the stock viewlets.

I ranted a bit about the situation during a psuedo-pair programming session I was having with my coworker, Mark Biggers (who also doesn’t have a blog yet). I was putting the finishing touches on my theme product at the time, and showing Mark what I did.

I was a little peeved that something that should work, going by every book and piece of documentation I had, not to mention logic, just didn’t work. I also felt stupid because I didn’t take the time to really understand what was wrong. Instead I wasted days trying to work around it, and spent even more time implementing my own viewlet manager.

Mark agreed that the situation seemed really odd, and suggested I take a look for some known Plone bugs to see if anyone noticed the problem before. In my haste and fustration, I never thought to look into the plone trac to see if somebody had submitted a bug ticket with the same problem I had. Taking Mark’s advice, I gave it a shot, and within seconds, in the first search I did, I found a bug ticket that was precisely the problem I was having. And the person who posted the bug was dilligent (and estute!) enough to figure out how to fix the bug and submit a patch. Score!

But the celebration was short lived. Upon further inspection, I realized the ticket was marked fixed. But wait, the bug also says “milestone changed from 3.x to 4.0″. This was just confusing, so I did some digging.

I was able to verify that the bug was truely fixed (6 months prior), and that it truely wasn’t in the latest release of plone.app.layout. That made no sense to me, especially given that the change was 5 characters.

I figured this warrented a post to the plone core mailing list, just to see what the reasoning was. I didn’t really get the response I wanted.

Note to self: I need to follow up on this. If there is anything I can do to help out, I really need to put some effort into finding out how. I’m not sure how releases work in Plone, so I didn’t want to volunteer, but in retrospect I should have said “So how do I do that” instead of “can that someone be me” :P

In case it’s not obvious, this small change would have prevented the need for me to create my own viewlet manager. It’s probably best, in the long run, that I went ahead and built my own viewlet manager. I’ve learned a lot, I’ve got a more stable and flexible means of customizing Plone, and I was able to document it pretty thouroughly, which (hopefully) will help somebody (most likely myself) someday.

In spite of that most Excellent Adventure (surely not a Bogus Journey), bugs suck. When you find out that days and days of fustration could have been avoided by such a minor change, it really makes you want to do whatever you can to avoid (and fix!) bugs in general.

Future Plans

So since I wrote my initial post my theme has been deployed in a live site and it’s gone over well. It’s holding up to multi-browser scrutiny in a very hostile environment (research lab at a university :)).

One thing I failed to document before, however, is that this first deployment is also the first time I extended my theme. This worked surprisingly well. The only customization I’ve had to do so far is change the logo, but between skin-layer changes and zcml overrides I don’t think making major changes to the way it looks without changing the layout should be pretty easy, and I think more drastic re-arrangement wouldn’t be much harder.

So I plan to document that in a blog post in the near future.

I also want to see how well my theme plays with CSSManager, and see how I can integrate with it, or add my own tool that’s specific to my applicaitons and base theme.

One thing I didn’t do much of that I’ll need to expand on is writing my own viewlets. The theme I created will be used in conjunction with a suite of applications I’m co-developing. You may have noticed I left a large empty space in my theme on the upper left-hand side. I plan to put a context-aware viewlet there with application-specific information and links to help and common resources. It should be pretty cool, I’ll document it here on my blog when I get to writing it.

This is the seventh post in a short series about the ideas, trials and tribulations of my first theme product. I’ll be posting the whole story in one place when the series is complete.

I’m Not Done Yet

Now that the theme is finished, there are a few more steps before I can consider it truly complete.

1. Does it work in IE? Or FireFox, or Safari, or Konqurer or Your Cellphone or a screen reader…. It’s important to test your theme across a few browsers. Chances are if you follow the Plone Way^TM, you won’t have too much trouble, but it’s always a good idea. This goes beyond the usual web designer’s perogative. You’ve opted to use a CMS that prides itself in graceful fallbacks, cross-browser compatibility and accessability. Make sure you’re not stomping all over those ends just so you can have a "rad" theme.

2. Is anything missing? Much like how I missed the footer in my theme, there are other bits and pieces of Plone that your theme may not have taken into account. A good example of this is the calendar portlet. There are some very specific CSS rules that you may have to override to get your theme to take affect.

   It’s a good idea to add all the stock portlets, and install any third-party products you plan to use with your site into your theme product’s development buildout. Poke around the site, and make sure you didn’t miss anything. It may be the product or portlet developer’s fault, but it’s up to you as the theme creator to cover all the bases.

3. Can you change themes? There is a theme control panel configlet (http://localhost:8080/plone/@@skins-controlpanel) that has a dropdown list allowing you to select the current theme. You should be able to select the "Default Plone" theme without any lingering effects.

   You will however, notice some immediate effects :). Reload the page, restart the server and poke around again like you did earlier just to be sure.

   If you find reminates, they can probably be attributed to forgetting to associate a stylesheet or viewlet customization specifically with your theme.

4. Does it uninstall? If you do more than add theme-specific viewlets, make sure that nothing breaks when you uninstall your theme. This is especially important if you ever want to distribute your theme to other people, but even if it’s for internal use only, it will make you happier in the long run.

Walking through these steps myself, here’s what I discovered.

It looks funny in IE

A preliminary glance in IE 6 (installed on my Ubuntu Hardy system via IEs4Linux) showed some messy bits under my theme relating to portlets, but the same issues come up with the default Plone theme, so I’m counting this one passable.

I’m in the process of getting a new windows VM up and running so I can test "real" IE6 and IE7 (and probably IE8-beta just for kicks)

I need to dig a bit and see if IE6 compatibility is a priority for Plone, and see if there are any bugs relating to this issue (I’m sure there are, but I should verify)

Odd Portal Rows

I hadn’t noticed it before, but there are special odd/even classes for portlet content like there are for listing tables (e.g. folder contents). I added a bunch of portlets to my layout, and added some events and recent changes. I tried to cover as many bases as I could for the sake of throughness.

After adding all the content, I noticed that the portal footers and headers looked nice, but the content seemed a bit messy, as each item in the events and recent changes portlets would run together. I investigated a bit and found that there was class="portletItem even" and class="portletItem odd" attributes specified in the portlet content.

To account for this, I added the following to my CSS

src/my.theme/my/theme/browser/stylesheets/main.css

...
.portletItem.even {
    background-color: #ecf0f5;
}
...

Uninstallability

In testing the theme, I found out that the theme itself isn’t "deleted" when the product is uninstalled, inspite of the fact that all of it’s required resources and viewlets are no longer available.

And worse, the theme is still set as the default theme. This leaves the site in a more-or-less broken state to the end user, who may not realize they need to change the default theme to a theme that’s actually available still. And the theme stays in the dropdown list.

To fix this requires setting up an uninstall profile. This is documented on plone.org, but I’ll show you what I did in my case.

First, I created a new directory called uninstall in my profiles directory.

$ cd ~/my.theme/src/my/theme/profiles
$ mkdir uninstall

Then I added the XML files that made up the profile. It’s basically what I did in the default profile, but with remove="True" added to the bits I wanted to remove, and the default theme changed back to a sane default.

src/my.theme/my/theme/profiles/uninstall/skins.xml

<?xml version="1.0"?>
<object name="portal_skins"
   default_skin="Plone Default">

 <object name="my_theme_custom_images" remove="True"/>
 <object name="my_theme_custom_templates" remove="True"/>
 <object name="my_theme_styles" remove="True"/>

 <skin-path name="My Theme" based-on="Plone Default" remove="True">
  <layer name="my_theme_custom_images"
     insert-after="custom" remove="True"/>
  <layer name="my_theme_custom_templates"
     insert-after="my_theme_custom_images" remove="True"/>
  <layer name="my_theme_styles"
     insert-after="my_theme_custom_templates" remove="True"/>
 </skin-path>

</object>

You need just enough information to let GenericSetup find the configuration bits that need to be removed, so you may notice that I removed some of the extra attributes from this skins.xml.

Then, I had to wire up the profile. This is a multi-part process, that starts with registering the profile with ZCML

src/my.theme/my/theme/profiles.zcml

...
  <genericsetup:registerProfile
      name="uninstall"
      title="My Theme - Uninstall"
      directory="profiles/uninstall"
      description='"My Theme" Plone theme. - UNINSTALL'
      provides="Products.GenericSetup.interfaces.EXTENSION"
      />
...

This is esentially the same as the other <genericsetup:registerProfile> tag that ZopeSkel already put into profiles.zcml for you, modified slightly to reflect the location and name of the new profile.

Now I’ve got a brand new GenericSetup profile, but no way to invoke it using the add/remove products interface that my users will be acustomed to. This means I’ll have to apply the uninstall profile with python code.

This is accomplished by bringing back an old-school, but still effective, method of controlling what happens when Plone installs or uninstalls a product: Extensions/install.py

The Extensions directory is not created for us by ZopeSkel, so we need to create it ourselves.

$ cd ~/my.theme/src/my/theme/
$ mkdir Extensions

Then we need to create the install.py file. This file can contain 3 functions: install(), uninstall(), and upgrade(). Each function will only be called if it’s present, and can basically do whatever you want. This is a very straight-forward way to add in some "magic" activity upon install, do some migration upon upgrade, or as in our case, tidy up when the product is being removed. It’s definately the go-to proceedure when GenericSetup doesn’t do what you want, or time prevents you from implementing a GenericSetup-based solution effectively.

src/my.theme/my/theme/Extensions/install.py

from Products.CMFCore.utils import getToolByName

def uninstall(portal):
    setup_tool = getToolByName(portal, 'portal_setup')
    setup_tool.setImportContext('profile-my.theme:uninstall')
    setup_tool.runAllImportSteps()
    return "Ran all uninstall steps."

You can find out more about the setup_tool’s API by inspecting it with DocFinderTab or looking at the source code (in parts/plone/CMFQuickInstallerTool)

This is the sixth post in a short series about the ideas, trials and tribulations of my first theme product. I’ll be posting the whole story in one place when the series is complete.

I Forgot The Footer! (Overriding Stock Viewlets)

I’ve managed to get all of the styling nailed down. Here’s what it looks like now:

Looking Good.... But wait!

So here’s what’s different: The extra viewlets are gone. I’ve added in all of the background images. I’ve styled the portlets. It’s become obvious that I’m going to have to do some more specific customization of the calendar portlet, but that’s not a big deal right now.

You may have been wondering what I did with the search viewlet that’s normally up in the right hand corner. In this screenshot, you can see I opted to use the search portlet instead… and yeah, that was my plan all along. :)

In addition, I’ve added my own copy of the modern plone logo, with a transparent background. I accomplished this by putting a file called "logo.jpg" into /src/my.theme/my/theme/skins/my_theme_custom_images.

Here’s the CSS as it stands now:

src/my.theme/my/theme/browser/stylesheets/main.css

/* Stylesheet for the 'My Theme' Plone theme */

body {
    background-color: #cfd8e5;
}

#siice_header #main {
    height: 90px;
}

#my_header #left_side {
    float: left;
    clear: left;
    height: 90px;
    width: 50%;
    background-image: url("++resource++my.theme.images/left_background.png");
    background-repeat: no-repeat;
}

#my_header #right_side {
    float: right;
    clear: right;
    height: 90px;
    width: 50%;
    background-repeat: no-repeat;
    background-position: right;
    background-image: url("++resource++my.theme.images/right_background.png");
}

#my_header #personal_bar_sections {
    clear: both;
}

#my_header #personal_bar {
    float: right;
    clear: right;
}

#my_header #sections {
    float: left;
    clear: left;
}

#my_header #crumbtrail {
    clear: both;
}

#my_header #personal_bar_sections {
    background-image: url("++resource++my.theme.images/tabs_background.png");
    height: 20px;
    background-repeat: repeat-x;
    background-color: #b7c4c8;
    border-bottom: 1px solid #93a7ac;
    background-position: top;
}
/************* Override Plone Styles **********/

/* Logo */
#portal-logo {
    float: right;
    padding-right: 5px;
}

/* Site Actions */
#portal-siteactions {
    float: left;
}

#portal-siteactions li a, #portal-siteactions li a:hover, #portal-siteactions li a:active, #portal-siteactions li a:visited {
    border: 0px;
    color: #5f8dd3;
    background-color: transparent;
}

/* Top Sections */
#portal-globalnav {
    padding: 0px;
    font-size: 110%;
}

#portal-globalnav li.selected a {
    color: #000000;
    background-color: transparent;
    border: none;
}

#portal-globalnav li.plain a {
    color: #000000;
    background-color: transparent;
}

/* get rid of the left border for the first tab, the index page */
#portaltab-index_html {
    border: 0px !important;
}

#portal-globalnav li {
    border: 0px;
    border-left: 1px solid #000000;
}
#portal-globalnav li a {
    border: 0px;
    color: #000000;
    background-color: transparent;
    margin: 0px;
}

/* Personal Bar */
#portal-personaltools {
    padding: 0px;
    padding-right: 1em;
    background-color: transparent;
    border: 0px;
}

/* Bread Crumbs Nav */
#portal-breadcrumbs {
    background-color: #b7c4c8;
    border-bottom: none;
}

#portal-breadcrumbs a {
    color: #4b93ff;
}

/* "Green" action tabs */
.contentViews li a {
    background-color: #e3f4d7;
}

/* Portlets */
.portletHeader, .portletFooter, .portletItem, .portlet {
    border: 0px !important;
}

.portletHeader {
    text-transform: none;
    font-size: 110%;
}

.portletHeader, .portletFooter {
    background-color: #b5c3d7 !important;
}

.portletItem {
    background-color: #dfe5ee;
}

.portletHeader a, .portletFooter a, .portletItem a {
    border: 0px;
}

/* Page Body */
.documentContent {
    border: 1px #447821 solid;
}

Considering the amount of CSS that Plone uses, this isn’t that much to override. I’m sure if I was delving deepter into the content area it might be a little more cumbersome, but it looks like the core theme is pretty well built for utilizing CSS to it’s fulllest.

A nice bonus of this layout: it works with large fonts. I haven’t tested it in Internet Explorer or Safari yet, but I can hit ctrl-+ over and over with no ill effects. w00t!

You’ll notice, of course, that I’ve missed the portal footer, the bit just above the colophon at the bottom of the page. I styled it in my mockup but have neglected to give it any attention in my theme product. Now is the time.

I don’t like the idea of just slapping a background on the div around the footer via CSS. I like my styles to be flexible, and if the font size got beyond a certain point, the text would exceed the height of the background. That’s just messy. There’s also the chance that I may need to put some static content inside of the portal footer that I don’t want overriden by any user-configurable options. Examples include a special copywright notice or link to a help center.

Normally, this isn’t a big deal. In my application, however, I may have people with the Manager role running around on the sites that use this theme that aren’t under my jurisdiction.

What this all means is that I need a new viewlet to replace the portal footer viewlet. Luckily, executing such a task is easier than explaining why I need to do it :).

Anyway, overriding a viewlet can be accomplished entirely in ZCML if you want. The same goes for viewlet managers, in theory at least. But, if you need it, you have the option to produce a class that can give you more flexibility.

The Theme Reference has a good overview of what you need to do. Here’s what my <browser:viewlet> tag looks like:

src/my.theme/my/theme/browser/configure.zcml

...
  <browser:viewlet
        name="plone.footer"
        for="*"
        manager="plone.app.layout.viewlets.interfaces.IPortalFooter"
        template="footer.pt"
        permission="zope.Public"
        layer=".interfaces.IThemeSpecific"
        />
...

With this configuration, I just had to drop a footer.pt file into my browser directory and bickety-bam, my footer shows up!

Note that I made sure to specify a layer so my override will only be enabled when my theme is in effect.

I took a look at the footer template that comes with Plone to be sure nothing special was happening. No worries there, so in my footer.pt, I was free to code it how I wished. I did borrow the date/time stuff to display the copywright year, however. Clever!

For the sake of illustration, in my examples here I’ve pulled the full copywright from Plone’s storck footer template. Here’s what it looks like:

src/my.theme/my/theme/browser/footer.pt

<div id="my_footer">
    <div id="my_copywright">
        <p>
    <span i18n:translate="description_copyright" tal:omit-tag="">
    The
    <span i18n:name="plonecms" tal:omit-tag="">
        <a href="http://plone.org" i18n:translate="label_plone_cms">Plone<sup>&reg;</sup> CMS &mdash; Open Source Content Management System</a>
    </span>
    is
    <acronym title="Copyright" i18n:name="copyright" i18n:attributes="title title_copyright;">&copy;</acronym>
    2000-<span i18n:name="current_year"
               tal:omit-tag=""
               tal:define="now modules/DateTime/DateTime"
               tal:content="now/year" />
    by the
    <span i18n:name="plonefoundation" tal:omit-tag="">
        <a href="http://plone.org/foundation" i18n:translate="label_plone_foundation">Plone Foundation</a></span>
    et al.
    </span>
</p>

<p>
    <span i18n:translate="description_trademark" tal:omit-tag="">
    Plone<sup>&reg;</sup> and the Plone logo are registered trademarks of the
        <span i18n:name="plonefoundation" tal:omit-tag="">
            <a href="http://plone.org/foundation" i18n:translate="label_plone_foundation">Plone Foundation</a></span>.
    </span>

    <span i18n:translate="description_license" tal:omit-tag="">
    Distributed under the
        <span i18n:name="license" tal:omit-tag="">
            <a href="http://creativecommons.org/licenses/GPL/2.0/" i18n:translate="label_gnu_gpl_licence">GNU GPL license</a></span>.
    </span>
</p>
    </div>
    <div id="my_footer_shadow"></div>
</div>
<br />

Here’s what I added to my existing CSS file to make the footer look like it does in my original design:

src/my.theme/my/theme/browser/stylesheets/main.css

...
#my_footer {
    border-top: 2px solid #a9dd89;
        background-color: #d2dce2;
    background-image: url("++resource++my.theme.images/footer_background.png");
    height: 32px;
    background-repeat: repeat-x;
    background-position: top;
    clear: both;
}

#my_footer #my_copywright {
    text-align: center;
}

#my_footer #my_footer_shadow {
    background-image: url("++resource++my.theme.images/footer_shadow.png");
    border-top: 2px #879ba3 solid;
    height: 10px;
    background-repeat: repeat-x;
    background-position: top;
    clear: both;
}
...

A quick note about my background images

Each of my gradient backgrounds are much larger than they need to be height-wise, and the color of the last row of pixels is the same as the background color of the element they’re in the background of. This helps to ensure that the text can stretch to a severe degree, either in amount or font-size, and the gradient still looks good.

With the new CSS added, here’s what it looks like:

Ah, that's much better

So almost everything is done! All that’s left is some cleanup and testing.

This is the fifth post in a short series about the ideas, trials and tribulations of my first theme product. I’ll be posting the whole story in one place when the series is complete.

A Touch Of Style

I’m going to continue using the Zope 3 resource approach with my style sheet. I don’t need DTML, I’m not overriding any specific stock Plone CSS file, and I like keeping all of my browser-related stuff in the browser directory of my product.

Doing things this way requires an additional step. For Plone to incorporate the style sheet(s), they have to be registered with the portal_css tool. This is accomplished via GenericSetup.

First, it’s a good idea to turn on “Debug/development mode” under portal_css in the ZMI of your plone instance (http://localhost:8080/plone/portal_css/manage_cssForm). This will ensure that your css will be refreshed in the browser as it’s updated on disk.

Then you need to create a css file, for example main.css in browser/stylesheets

To get the CSS registered, you need to add a tag to profiles/default/cssregistry.xml

What do you know! A quick glance at the profiles/default directory and we see that ZopeSkel has already done this for us. :). For the sake of illustration, here’s what that file looks like, after ZopeSkel has generated the skeleton:

src/my.theme/my/theme/profiles/default/cssregistry.xml

<?xml version="1.0"?>
<object name="portal_css">
 <stylesheet title=""
    id="++resource++my.theme.stylesheets/main.css"
    media="screen" rel="stylesheet" rendering="import"
    cacheable="True" compression="safe" cookable="True"
    enabled="1" expression=""/>
</object>

You can add additional style sheets in the same manner.

You can verify that the css is active by checking the portal_css tool (it will be second-to-last in the list), and making sure http://localhost:8080/plone/++resource++my.theme.stylesheets/main.css doesn’t throw any errors.

Now that the style sheet is registered, it needs to be populated with styles.

src/my.theme/my/theme/browser/stylesheets/main.css

/* Stylesheet for the 'MY Theme' Plone theme */

#my_header #main {
    height: 90px;
}

#my_header #left_side {
    float: left;
    clear: left;
}

#my_header #right_side {
    float: right;
    clear: right;
}

#my_header #personal_bar_sections {
    clear: both;
}

#my_header #personal_bar {
    float: right;
    clear: right;
}

#my_header #sections {
    float: left;
    clear: left;
}

#my_header #crumbtrail {
    clear: both;
}

I had to add another id to my template, and re-arrange things a bit to get it all to work. Here’s the modified template:

src/my.theme/my/theme/browser/myheader.pt

<div id="my_header">
    <div id="main">
        <div id="right_side">
            <div tal:replace="structure view/plone.logo/render" />
        </div>
        <div id="left_side">
            <div tal:replace="structure view/plone.site_actions/render" />
        </div>
    </div>
    <br />
    <div id="personal_bar_sections">
        <div id="sections">
            <div tal:replace="structure view/plone.global_sections/render" />
        </div>
        <div id="personal_bar">
            <div tal:replace="structure view/plone.personal_bar/render" />
        </div>
    </div>
    <br />
    <div id="crumbtrail">
        <div tal:replace="structure view/plone.path_bar/render" />
    </div>
</div>

To explain a bit, what I’ve done is used <div>’s to separate everything out, and then used float and clear to create the sort of layout I wanted. float moves the block-level elements to one side, and clear controls where float-ing elements are allowed to go. Here’s what it looks like at this point:

We're Almost There!

As you can see from the screen shot, I’ve achieved my primary goal of re-arranging the stock plone viewlets into the orientation that I specified in my initial design.

So now, I’ve just got to add in styles to handle the more looky-feely elements (background images, colors, fonts) and override bits of Plone style that either don’t look good in my theme or are counter to it (like the padding around the personal_bar)

Ousting The Doppelgängers

Now’s a good time to get rid of the superflous viewlets that I’ve re-implemented in my viewlet manager. This is accomplished through the <hidden> tag, in our GenericSetup profile’s viewlets.xml file.

src/my.theme/my/theme/profiles/default/viewlets.xml

<?xml version="1.0"?>
<object>
    <hidden manager="plone.portaltop" skinname="My Theme">
     <viewlet name="plone.header" />
     <viewlet name="plone.personal_bar" />
     <viewlet name="plone.path_bar" />
   </hidden>
</object>

The <hidden> tag takes two attributes, manager and skinname. manager is the name of the viewlet manager that you want to hide viewlets in. In this case, it’s plone.portaltop. You can figure out where the viewlets you want to hide are by using GloWorm or @@manage-viewlets.

Inside of the <hidden> tag are <viewlet> tags, each specifying the name of a viewlet you’d like to hide.

You’ll need to re-install your theme product to see the change.