The Crushinator project aims to replace PasteScript as a de facto standard for developing boilerplate code generation systems. My interest in starting the project came from use of and work on ZopeSkel/templer.core. There are several issues with the way that PasteScript solicits values from the user that are restrictive to the developer and often frustrating for the user.

If you’re not familiar with how PasteScript works, you typically invoke it via the paster create sub-command. You then get prompted for a series of values that correspond to template values. It works in a linear fashion. The values may be manipulated prior, but are essentially passed verbatim to the underlying template

For more background, see the design document.

Terminology Clarification

‘user’ vs ‘developer’

When I talk about users here, I mean someone who invokes the paster (or, zopeskel/templer) command to run a code generation ‘template’. When I talk about developers I’m talking about folks that write the ‘template’ code.

‘template’ vs Template vs template

There’s a confusing bit of nomenclature in PasteScript regarding the word ‘template’. Typically when people talk about templates, they mean a predefined structure; most people think of Word templates or cut pieces of plastic or paper used to guide a pen or saw. In both cases, there’s an implication of structure and repeatability.

When programmers talk about templates, they typically mean text files with special syntax; these text files provide structure and repeatability as in the common sense, but templates for programmers are typically filled with place holders, and often simplified logic. Examples are vast, but include Cheetah, Jinja2, Smarty, Mason, etc.

In PasteScript, Template is a base class that handles the interaction between the user and (typically) a template (in the programmer sense). It also does some things with treating code layouts like templates, doing additive injections of code, etc.

Separating these concerns and quelling this confusion is a core goal of the Crushinator project. That said, the only real frame of reference anyone reading this will likely have is PasteScript, so when I use the term ‘template’, I’ll put it in quotes, and I’ll be referring to templates in the PasteScript sesne. If I don’t quote it, assuming it wasn’t just a typo, I mean it in the programmer sense.

Specific Issues This Document Addresses

This section covers the specific problems with PasteScript that are being addressed by the Probe/Interrogation implementation, and the new concepts and features that will also be implemented.

Skipping

We’ve often found it useful to skip some prompts. Lets say the user enters a response to a prompt, and that implies some sane defaults. It’s difficult for the developer to handle this situation with the core PasteScript library (ZopeSkel and templer.core have overcome this on their own).

A concrete example: there are about a half-dozen template variables in the python egg-related templates (package, namespace, nested-namespace) that deal with package metadata (license, author, keywords, etc). These values are strictly optional in the sense of building the code skeleton successfully and creating a working package. Aside from the ZopeSkel hacking around the issue, there’s no way to just ask the user “do you want to set metadata now?”, and if they say “no”, skip all of those questions, substituting sane defaults. (See ZopeSkel’s ‘easy mode’ to see this in practice)

This happens because the prompting happens independently of the processing and template transformation. You have to wait until the user is finished answering all of the questions before you can act.

Validation

In PasteScript, there is no ‘baked-in’ facility for checking a value from the user. If you do detect an error, you have to rerun the prompt loop, or intervene on your own.

This has caused some pain; aside from the obvious (e.g. the developer assumes a raw integer and the user enters ‘one’), often the values entered by the user will correspond directly to a python identifier, or a package name. A good developer will typically massage the value to make it conform, but it doesn’t always happen.

What’s worse, it can confuse the user. Say they entered “My Class”, understanding that the mangling will happen. They expected “MyClass”, but instead got “My_Class” due to an assumption on the developers part. I’d rather see an error here, or feedback about what happened (‘My Class’ is not a valid class name, is My_Class ok?’, or ‘You set the content type to “My Class”, so we’re going to call the class My_Class’, etc).

The way I look at it, the code generated by a tool like Crushinator should be ‘sound’ out of the box. The user should be able to understand what happened by looking at it, it shouldn’t do anything ‘magical’ for its own sake, and as much as possible, it should run without error.

Validation and allowing some sort of feedback to the user will alleviate these issues.

Branching

Building on the ‘skipping’ concept, this involves presenting the user with a whole new set of prompts given their specific response to a certain prompt. As far as I know, there is no current implementation of this concept in any PasteScript-based boilerplate generators.

An example: The user is building a new python package. It asks them, “would you like to set up Sphinx documentation”. If the user says “yes”, they are asked specific questions that are necessary for Sphinx to function. If they say “no”, they go on without it. A similar approach could be used for adding a zc.buildout structure.

Grouping

There’s utility in being able to arbitrarily group prompts, primarily in ‘alternative’ front ends, such as the web or desktop. This could be implemented as a tabbed interface, or a ‘wizard’, or just allow the developer to provide more guidance to the user by giving a description and/or help-related text that covers a subset of a larger suite of prompts.

To my knowledge, this is not implemented in PasteScript.

More information

In PasteScript, developers are limited to a single, simple string that gets passed to the user when solicited for a value. Often developers want to provide a more detailed explanation for what the value means, what its implications are, etc.

Crusinator will provide several mechanisms for the developer to communicate with the user (but most of these will be optional)

Markup in Prompts

Currently there’s no way to effectively provide emphasis, urls, or otherwise enrich the prompts as they’re given to the user. This limits both the amount of help the developer can give the user, and makes more robust user interfaces (web, desktop) more difficult to implement.

There will be a single markup standard used throughout Crushinator. This is independent from any template processors that the developer uses.

Markup would apply to any text that is presented to the user.

Internationalization

i18n is not currently a common component of PasteScript (I don’t think it’s impossible to implement it with gettext, but AFAIK nobody’s doing it). As PasteScript-based generators have often become the go-to ‘quick-start’ for new users of a framework, the lack of this feature is a hindrance to acceptance.

It will be optional but recommended (using gettext makes it pretty easy to open the door to other folks translating it for you). However it is implemented, it will need to be able to handle the markup mentioned above. All text provided to the user will be expected to be passed through that translation mechanism.

User Feedback

When a value fails validation, it’s useful to tell the user why. Crushinator will expand on this concept, allowing user feedback when validation passes.

An example: The user is building a python egg, and they opt to not fill out the metadata. The developer can point the user to the generated setup.py file, and a link to the setuptools page explaining what the values mean, so they’ll have a pointer to what they’re supposed to do.

Implementation

Terms

In the design document, I’ve settled on a separation of many of the concerns and picked nomenclature to identify the parts. In this document, I’m focusing on the Probe, Interrogation, and (to a lesser extent) the Runner classes. There will be mention of the Collector and UserInterface classes, and fleeting mention of Skeleton and Injector, but the specifics of UserInterface, Skeleton and Injector are still in flux. Although Collector has a base implementation, it’s only utilized by the UserInterface, so there’s not much need for detail here.

A basic overview of the terms/base classes:

All of these classes reside in crushinator.framework. As typical implementations are built, they will be part of the crushinator.toolkit package.

Basic Communications

Crushinator itself provides a setuptools entry-point facility to register (plug-in) User Interfaces, Collectors, and Runners. The User Interface provides a plug-in point for Runners. In this way the Runner expresses compatibility with a given set of User Interfaces. The User Interface will also register any setuptools ‘console scripts’ that the user will use to invoke it.

Help Documentation

A user will invoke the User Interface via the console script. The User Interface must provide a help() method (and corresponding UI element or command line switch), which is expected to return a list of all Runners that have expressed compatibility with that User Interface. The help() method will also take a Runner class or string containing a dotted-package notation (e.g. ‘my.package.MyRunnerClass’).

The Runner class must also provide a help() method. When a Runner specifier is passed to the help() method for a User Interface, after look-up, this method is to be invoked by the User Interface. The help() method for a Runner will also take a specifier. In this case the value can be: a Probe class/package-notation, an Interrogation class/package-notation, a two-tuple  (of Interrogation class/package-notation, Probe name) or just a Probe name.

Probes and Interrogations will also (optionally) have a help() method, returning general text about what they do. This might not typically differ from the description property also expected, but is provided in the event that more context can be given outside of a short clarification as to what a Probe/Interrogation’s purpose or expected value is.

In all cases, (as mentioned earlier) it will be possible that the text will contain a lightweight markup.

The goal here is to encourage developers to include useful help at each level of the system, thus providing a better end-user experience. The user should be able to pull up help on a specific Runner, or a specific Probe and/or Interrogation with relative ease, and the User Interface should be able to provide such help text in an intuitive way.

User Interaction

The User Interface will provide some mechanism for a user to select a Runner they wish to work with. Default values will be pulled via invoking one or more appropriate Collectors. Collectors return a dictionary, where the keys are Runner class specifications (dotted package notation), and the values are dictionaries of individual Probe names and their default values. The defaults for a specific Runner are passed to the Runner’s constructor.

The Runner will internally maintain a collection of Interrogations, which, in turn, maintain a collection of Probes. The User Interface invokes the Runner’s next() and set() methods as it interacts with the user.

The next() method returns a Probe object. The set() method takes a value and a Probe specifier. This ‘commits’ the value back to the Runner. The set() method returns any success output from the Probe, or raises an exception upon error. This gives us a few useful API interactions:

• Branching becomes possible. The Runner consults the Probe’s next() method to retrieve the next Probe (or Interrogation) that should be presented to the user. What the Probe returns is dependent on its current value.
• Validation happens along the way. Each Probe implements a validate() method, which is invoked by the Runner’s set() method. The validate() method can also be used on its own to check a value before saving it.
• The Interrogation also implements a validate() method, allowing for invariant checks and other sorts of group validation.
• The User Interface can build dynamic interfaces by pre-polling the Probes without calling set(). This is necessary when building UIs for the web or desktop.
• The user can get feedback about successes as well as errors.

Another feature in this communication is a high-level of autonomy. A Runner must provide a list of Probes to a User Interface outside of the next()/set() interaction, as well as a separate list of Interrogations. Each Probe or Interrogation can be invoked, validated, and then set() independently. This provides the sort of flexibility necessary for asynchronous UIs. Those lists will react to existing values just like the next() method does (the API here is still a bit cloudy, they may actually take values as a method argument). In this way the User Interface can present the user with a ‘wizard’ view or tabbed interface.

After all of the Probe values are collected from the user and set back on the Runner, the User Interface will then call the invoke() Runner method. This is where the actual code generation will happen. The invoke() method will raise an exception which contains error text from any failed Skeletons or Injectors. If it runs successfully, the return value will be a success string, much like the validate() method of a Probe.

The exact details here may change quite a bit over time as the Skeleton and Injector classes are fleshed out. There’s a good chance that the Runner will be passed a list of files that were changed, or other useful info beyond “code generated successfully”. However, the intent is the same: let the user know what’s going on.

More Details

I’m planning to use reStructuredText to handle markup in labels, help text, and other text passed to the user. It will probably happen automatically, but I may need to provide a way to prevent processing if it gets in the way.

Gettext will be used to set up translations of all core implementations. It may be rolled into the reStructuredText processor if that ends up being automatic. The idea is to make help text and translations a seamless part of the development process.

Interrogations and Runners will implement some sort of dictionary-like API to retrieve specific Probes. I see the use case here in a fancy web UI that validates via AJAX as the user enters values.

As I mentioned earlier, heavy use of setuptools entry points will be employed to register User Interfaces as console apps, and to provide a way for Runners to integrate with specific UIs. That said, there will be a set of stock implementations for user interfaces that should be compatible with any Runner.

Additional Products

In the crushinator.toolkit package, the aim is to provide useful implementations and helper libraries to make building code generators as easy as possible. To facilitate the implementations outlined above, a few specific utilities start to emerge:

• Factory methods for Probes and Interrogations. It should be easy to make a Probe or Interrogation. The factory pattern makes sense here.
  NOTE: This might actually be best suited for crushinator.framework
  Some convenience features would include:
• passing a callable to dynamically set the validate() method
• generating a template variable name (‘short name’) for the  probe automatically based on the label
• Implementation of validating Probes for many useful data types:
• Integer
• Float
• E-mail address
• Python identifier
• URI
• User Interface registration helper for Runners. There should be automatic registration for all of the stock UI implementations in crushinator.toolkit.
• Gettext/reStructuredText parsing for labels, help, etc. Preferably one function that handles both tasks at once. This may be rolled into the base classes and become transparent to the developer.
• Logging – I’ve already begun to implement logging in both the crushinator.framework and crushinator.toolkit packages. I want to make that as streamlined as possible for the developer. The idea is to swallow as many exceptions as possible, returning ‘friendly’ text to the user, but always log that something happened with as much detail as possible. I’m impressed with the way Python handles logging, and it’s ideal for systems like this.

Wrap Up

I’m preparing to implement things as I’ve outlined in this document. I’m interested in hearing what others think about this plan; is there anything I’ve omitted? Are there any assumptions I’m making that aren’t accurate? Have I overlooked any potential problems with user interface implementations you’re interested in seeing developed?

Comments, corrections, discussion, questions… really any feedback would be greatly appreciated.

Adding folks from the developer communities I’m involved with, and noticed a tweet from @optilude, mentioning this: http://www.screenr.com/b1Rs.

I want to take a closer look at the UI later today, but it looks pretty great so far. This tweeing thing might be more fruitful than I ever thought.

I’ve been working on a new project that’s based on the Pyramid framework. The framework creators claim 99% test coverage. The other developer and I thought that was pretty great, and decided that setting a similar goal for our project would be worth the trouble.

I don’t think I need to go into intimate details about why tests of the various forms (unit, functional, integration, systems, load, stress) are important (I will if somebody asks :)). I think it can be summed it up in a simple truth:

In the long run, writing tests saves you time, and makes your code stronger.

I’ve spent most of this week refactoring tests in an effort to get to that mythical 100% goal, and I’ve found the experience to be extremely fruitful.

Note: we’re using Nose to run the tests and generate the coverage report.

I’ve learned a lot.

I know much more about how Pyramid works (and it’s been refreshingly straight-forward). I’ve become a fan of WebTest, and intimately familiar with WebOb. All of this was a side-effect of the need to break down the tests into units that could be adequately run at the various levels, to touch all of the code. I had to dig into the APIs for each package to figure out how to test certain things, and again, at each level. Luckily, these particular packages are very well documented, and their code is relatively clean and easy to understand, so the process was much less arduous than it might have been given a different toolset.

I had to dig into protocol-level aspects of the application. Some of more opaque parts of Pyramid, WebOb, WSGI and HTTP required closer inspection. After this week, I know so much more about multi-part form encoding (MIME encoding technically, RFC 2046), file uploads (RFC 1867), and perhaps more importantly, how Pyramid and WebOb implement these concepts.

The separation between unit and functional testing became more obvious, and more than ever, I appreciate the need for them to be separate.

The code’s gotten better.

When faced with a line of code that wasn’t covered, I had to figure out why. The answer to ‘why’ forced me to re-evaluate the way the tests were written and think critically about how the application is structured.

This is, of course, in addition to the expected benefit of catching untested code. That untested code manifested in several ways:

• I found a handful of use cases that just weren’t being tested.
• I found some edge cases that were not being tested (accounted for in the code, but not being executed by the tests)
• I found API interfaces that were implemented but never executed.
• I found tests that were passing, but not testing what they were supposed to test.
• I found one specific edge case that was accounted for in the code, but it turned out to be effectively impossible.

What this amounts to is exposure of: false assumptions, bad tests, and some potential design flaws. In fixing and/or addressing them, the code has attained a new level of solidity. And even the tests themselves are a lot better than they were last week.

In summation…

I’d have to say working toward 100% coverage, even if you can’t get there, is more than worth the effort. Just exposing one or two bad tests or use cases that were neglected is worth it, and I gained a lot more than that from this endeavor.

Epilogue

I started writing this yesterday morning, when I had one or two stubborn lines of code that kept eluding coverage. As of about 6 o’clock last night, I’ve actually achieved my goal of 100% test coverage. I need to do a bit more digging to make sure this isn’t a misleading number, but given what’s gone into getting here, I feel pretty confident that even if it’s not as impressive an achievement as it sounds, that it is a reflection of a very good test suite, and a well put together application.

I’m interested in hearing others thoughts on the requirements, approach, design, etc.

It’s a work in progress; I feel pretty good about most of the decisions made but everything is up for discussion. I’m currently working on the implementation details.

I also presented on this project at the last TriZPUG meeting. For a more general, brief overview (‘elevator pitch’ of sorts), see my slides (PDF).

Using buildout.extensionscripts, I whipped up a quick proof of concept, that I think might be helpful. I should create a full-blown buildout extension for this someday (prodding is welcome).

The file and an example buildout using it are up on my google code SVN repo.

Here’s how you can try it out (make sure you have the usual pre-requisites for zc.buildout: python, gcc, etc):

$ svn checkout http://lionfacelemonface.googlecode.com/svn/trunk/cfg-dump
$ cd cfg-dump
$ python bootstrap.py
$ bin/buildout -c dump.cfg

Don’t worry, the buildout will exit before anything is actually installed. It will download templer.core, mr.developer, and the one or two recipes that are used in the buildout files. I wanted the dumping process to be as quick and painless as possible, but from what I can tell it’s unavoidable. Buildout will install requisite recies. mr.developer will download templer.core. I wanted to test how my code got along with another buildout extension, and I used mr.developer, and used templer.core since I had it in mind. It seems to play nice but because extensions are evaluated after the config is parsed, but before the installation, I can’t really stop other extensions from doing things if buildout invokes them first.

If you peek into the buildout itself, you’ll notice its kind of contrived and somewhat odd. It’s a mishmash of buildouts I’m currently putting together. It’s not intended to be run directly, but it will probably not fail if you did.

Be careful using the extension on a buildout that you care about; it’s not destructive, but it could use more checks for things like existing dump files and such. It’s also not extensively tested.

Once the buildout is complete (shouldn’t take more than a 20 seconds or so), you will see a file called buildout-dump.cfg. Ta-da!

The formatting of the dumped config leaves a little bit to be desired (buildout section is not at the top, no comments injected or preserved), but it should be runnable. It makes an assumption or two (sections/options prefixed with ‘_’ are skipped, computed values are skipped if they can be detected, etc), so YYMV (please let me know of specific recipes you have trouble with). I consider this a bonus feature at this point, but reliable round-tripping would be pretty cool.

Here are the options available (no configuration is necessary out of the box):

• dump-skip-empty – boolean. If true, and a value is empty (or false I think, need to verify that – it may not be what we want), omit it from the output instead of outputting param = . Defaults to false.
• dump-file – the path to a file to put the dump into (the contents will be overwritten!). If this option is set but empty, no file will be written. Defaults to ${buildout:directory}/buildout-dump.cfg
• dump-eject – boolean, if true, the buildout will terminate after dumping the cfg file. If false, it will proceed to complete the buildout.

The extension also looks at the verbosity level switch (-v). If present, the config file will be dumped to stdout as well as being written.

So please, try it out, let me know how it could be improved (I’ve only scratched the surface of the zc.buildout internals), if you find bugs, etc.

I’m in the planning stages for a new project, and although I’m still on the fence as to what we’ll use to develop it, I’m sweet on Pyramid. However, I’m also a WSGI n00b, and haven’t used the paste stack for anything beyond building plone buildouts and archetypes boilerplate. That means that PasteDeploy is not in my playbook. There are some great PasteScript templates provided by the Pyramid project to get started quickly, but I didn’t see any good explanation of why the files were created, and how they actually worked, even in the super-awesome Pyramid docs.

Disclaimer: I didn’t look that hard. :P

So, to ease my pain, I got back to basics. I started with a fresh egg, the PasteDeploy documentation and PEP 333. I came up with the absolute, bare-minimum code, setup, and config necessary to produce a functioning (albeit boring) WSGI application, and the necessary glue to run it via PasteDeploy.

If you’re the type that doesn’t need much explaination and just want to look at the code, go right to the source (checkout instructions)

Prerequisites

You’ll need, at minimum, a modern version of python (I used the 2.6 version that ships with Debian Squeeze), virtualenv, and some way to install virtualenv.

I’ll leave getting virtualenv installed as an exercise for the reader :)

I’m going to assume you’re in a Linux environment in the few times we need to interact with the operating system. As far as I know, this should work on any operating system Python runs on, but as always, your mileage may vary.

Environment

$ virtualenv fun-minimal-wsgi
$ cd fun-minimal-wsgi
$ source bin/activate
(fun-minimal-wsgi)$ easy_install PasteScript PasteDeploy

If for some reason you’re not familiar with virtualenv (and if you’re a python developer, you really should be, it’s great), these commands:

1. Sets up a sandbox python environment (but will use system packages if they exist and haven’t been overridden locally)
2. Puts you into that environment
3. Installs PasteDeploy

To get out of the environment, you can type deactivate:

(fun-minimal-wsgi)$ deactivate

Bare-Minimum Egg

We’ll store our WSGI code in a simple python egg, that is setuptools-enabled (pretty much the standard). This requires a couple of directories, and some special files (I based this off of output of the PasteScript ‘basic_namespace’ template, for details about what this means, see the setuptools docs).

Here’s what the layout looks like:

fun-minimal-wsgi\
--> minimal\
-----> minimal\
---------> __init__.py
-----> setup.py

Here’s a quick list of commands to create an empty version of that layout (assume that we’re still in the fun-minimal-wsgi directory):

(fun-minimal-wsgi)$ mkdir minimal
(fun-minimal-wsgi)$ touch minimal/setup.py
(fun-minimal-wsgi)$ mkdir minimal/minimal
(fun-minimal-wsgi)$ touch minimal/minimal/__init__.py

The basic idea here is we have an egg source directory minimal, containing a setup.py file, used to install and configure it, and a singular package, also called minimal (coincidentally… what it’s named is not a requirement of setuptools).

Now for the contents of setup.py:

from setuptools import setup, find_packages
import os

version = '1.0'

setup(name='minimal',
      version=version,
      description="",
      long_description="""
         Very, very minimal example of a WSGI application and middleware.
      """,
      # Get more strings from
      # http://pypi.python.org/pypi?:action=list_classifiers
      classifiers=[
        "Programming Language :: Python",
        ],
      keywords='',
      author='Josh Johnson',
      author_email='none of your beeswax@somehost.com',
      url='http://lionfacelemonface.wordpress.com',
      license='BSD',
      include_package_data=True,
      packages=['minimal',],
      zip_safe=False,
      install_requires=[
          'setuptools',
      ],
      entry_points="""
      """,
      )

This is, at its bare minimum, all you need to say you've got an egg. :)

You could run python setup.py develop and start using it in your virtual environment (but don't yet), not that it does anything yet :)

Bare-Minimum WSGI App

This code goes into minimal/minimal/__init__.py. This is right out of PEP 333, with a couple of changes to take out any logic, and show all of the possible ways to create a WSGI app, and WSGI middleware.

# simple_app and AppClass are right out of PEP 333
def simple_app(environ, start_response):
    """Simplest possible application object"""
    status = '200 OK'
    response_headers = [('Content-type', 'text/plain')]
    start_response(status, response_headers)
    return ['Hello world!\n']

class AppClass:
    """Produce the same output, but using a class

    (Note: 'AppClass' is the "application" here, so calling it
    returns an instance of 'AppClass', which is then the iterable
    return value of the "application callable" as required by
    the spec.

    If we wanted to use *instances* of 'AppClass' as application
    objects instead, we would have to implement a '__call__'
    method, which would be invoked to execute the application,
    and we would need to create an instance for use by the
    server or gateway.
    """

    def __init__(self, environ, start_response):
        self.environ = environ
        self.start_response = start_response

    def __iter__(self):
        status = '200 OK'
        response_headers = [('Content-type', 'text/plain')]
        self.start_response(status, response_headers)
        yield "Hello world!\n"

# not in PEP 333, but mentioned in the comments to AppClass
class AlternateAppClass:
    def __call__(self, environ, start_response):
        status = '200 OK'
        response_headers = [('Content-type', 'text/plain')]
        start_response(status, response_headers)
        return ['Hello world!\n']

class MinimalMiddleware:
    """
    Bare-minimum, doesn't do anything at all, middleware.
    """
    def __init__(self, application):
        self.application = application

    def __call__(self, environ, start_response):
        return self.application(environ, start_response)

class SimpleMiddleware:
    """
    Takes a prefix, and appends it to each line in the response.
    """
    def __init__(self, application, prefix):
        self.application = application
        self.prefix = prefix

    def __call__(self, environ, start_response):
        response = self.application(environ, start_response)
        return ['%s %s' % (self.prefix, s) for s in response]

Read PEP 333 for full explainations, but here's the gist:

WSGI apps are super simple. At minimum, they're a simple function (simple_app), and at their most complex, a simple that implements the iterator protocol (AppClass). The function, or the __call__ method (in the case of AppClass, __init__ is standing in for __call__) take a 'start_response' callable and a dictionary with environment information (think CGI variables). They then use the callable to set the necessary response code and headers (in our case '200 OK' and 'Content-Type: text/plain'), then return an iterable, where each member is a line in the output to send to the browser.

The AppClass example is probably a good way to implement a WSGI app class, but it's not the simplest. What's required is a callable, so any object that implements a __call__ method and returns an iterable. AppClass implements the iterator protocol and short-circuits the need for instantiation. For the sake of clarity and throughness, I added the AlternateAppClass class to illustrate this.

We'll see the functional difference later when it's being wired up for use by PasteDeploy.

WSGI Middleware works like a bucket-brigade. An application object is passed from middleware class to middleware class until all are called. The middleware is instantiated with an application object, and when called is expected to return a response. I've included a simplified minimal implementation (MinimalMiddleware), and an implementation that manipulates the request before returning it (SimpleMiddleware).

Note: In the event that you're trying to jump ahead, this code might not run out of the box. I'm allowing a configuration option to be passed to SimpleMiddleware, and since it's required, it would fail.

PasteDeploy Configuration - Application

Now for the server part. You'll need to call this file something useful and put it somewhere you can get to (I've put it in the egg, minimal/minimal.ini).

[app:main]
use = egg:minimal

[server:main]
use = egg:Paste#http
host = 0.0.0.0
port = 6543

This does two things. It tells PasteDeploy to look in the egg called 'minimal' for an 'app-factory' entry point called main (more on this in a bit). It then configures the server.

Glue - Application

Before we can run the server we have to do three things: create an application factory, add the entry point specified in minimal.ini, and then install the egg.

The factory is simply a callable that returns a WSGI application. We'll add this to minimal/minimal/__init__.py.

...
def main(global_config, **settings):
    # settings comes from paste deploy, whatever values were in the section of the
    # deployment config file
    if settings.get('use_class', False):
        return AppClass
    elif settings.get('use_alt_class', False):
        return AlternateAppClass()
    else:
        return simple_app
...

An application factory, for PasteDeploy, is a callable that takes a config object (the contents of the whole config file), and any number of keyword arguments, which correspond to the other options mentioned in the config file (in our case, had we put toggle = True in [app:main], that argument would be passed along to our factory)

I've added a few config options that the factory can respond to, to make it easy to try out the different ways of implementing the same WSGI application.

Now for the entry point, the real glue that ties PasteDeploy with our code.

We'll add the following to setup.py in the call to setup():

...
      entry_points="""
      [paste.app_factory]
      main = minimal:main
      """,
...

Note that minimal:main maps directly to the main() function we created as our application factory. Had we named it something else, or if we wanted to provide multiple app factories for whatever reason, we could specify them one after the other, the name of the entry point (what we'd reference in minimal.ini), and a path to the function. The name main is just the default, and used as a convention/convienence. Here are some other examples:

• Put use = egg:minimal#otherapp in minimal.ini, and otherapp = minimal:some_other_function_in_init in setup.py
• Put use = egg:minimal#main in minimal.ini, and main = minimal.some_package:other_function in setup.py
• Put use = egg:minimal#other in minimal.ini, and main = minimal.some_package.package_deeper:other_function in setup.py

You get the idea.

Install the Egg

We'll want to install this in develop mode, so changes to the code will be reflected as soon as they are made.

Note: entry points are only updated when the egg is installed, so you'll have to repeat this procedure every time you make a change to them in your egg.

Make sure you've activated your virtual environment before you do this!

(fun-minimal-wsgi)$ cd minimal
(fun-minimal-wsgi)$ python setup.py develop
(fun-minimal-wsgi)$ cd ..

Starting The Server

Now we should be able to start the server using the paster serve command.

(fun-minimal-wsgi)$ paster serve minimal/minimal.ini --reload

You can now navigate to http://127.0.0.1:6543 and you should see 'hello world'

Since we specified --reload, we should be able to make changes to the application code and minimal.ini and the server will reload for us (remember the note about entry points though).

PasteDeploy Configuration - Middleware

Things will change a little bit now:

[app:minimal_app]
use = egg:minimal

[filter:middleware]
use = egg:minimal
prefix = yikes:

[pipeline:main]
pipeline =
    middleware
    minimal_app
...

Here we changed the [app:main] heading to an easy-to-refer-to name. We then defined the middleware part (middleware is called 'filters' in PasteDeploy configs), and a pipeline, which links up the middleware and the application.

The middleware specified here is also called 'main'.

Note we've also specified a parameter to the middleware. If we didn't need to, we could have just used egg:minimal in the [pipeline:main] section, in replace of [middleware]

Note: This will break your running server instance, since we are referring to a filter factory that doesn't exist.

Glue - Middleware

Now to make the middleware work. We'll need to a new entry point to minimal/setup.py, so it now looks like this:

...
      entry_points="""
      [paste.app_factory]
      main = minimal:main

      [paste.filter_factory]
      main = minimal:middleware
      """,
...

Again, the name of the entry point is not significant, and main is used as the default so we don't have to specify it in the ini file.

Now for the filter factory in minimal/minimal/__init__.py:

...
# middleware factory for paste deploy
def middleware(global_config, **settings):
    prefix = settings.get('prefix', 'Booyeah:')

    def factory(app):
        return SimpleMiddleware(app, prefix)

    return factory

Here, we're using the SimpleMiddleware class, and finally utilizing/providing the prefix parameter.

Now we can re-install the egg (to get the new entry point) and restart the server.

(fun-minimal-wsgi)$ cd minimal
(fun-minimal-wsgi)$ python setup.py develop
(fun-minimal-wsgi)$ cd ..
(fun-minimal-wsgi)$ paster serve minimal/minimal.ini --reload

Now, visiting http://127.0.0.1:6543 will look a little different.

The same middleware can be added to the pipeline multiple times as well, try adding another copy of middleware to [pipeline:main] and see what happens. You could also add another section referencing the same filter factory, but call it something else, and use a different setting for the prefix.

Conclusion

So, this helped me understand how WSGI works, and how PasteDeploy is used to serve it. This is the foundation of what Pylons, TurboGears and Pyramid use, and so it's important to understand to help understand the mentality behind everything else these frameworks do.

Please feel free to comment! Any feedback is appreciated, this was primarily a learning exercise for me :)

/***
 * Overload default ajax methods and show a spinner graphic under the mouse
   pointer when things are loading
 ***/

$(function(){
    var spinner = $('');
    spinner.css({
        display: 'none',
        position: 'absolute',
    });
    $('body').append(spinner);
});

jQuery.ajaxSetup({
  beforeSend: show_spinner,
  complete: hide_spinner,
});

function show_spinner(){
    $('img#ajax_waiting_spinner').css('display', 'block');

    $('body').mousemove(spinner_follow_mouse);

    /// position it ititially
    $('body').mousemove();
}

function hide_spinner(){
    $('img#ajax_waiting_spinner').css('display', 'none');

    // this prevents us from killing other
    // mousemove handlers that might be registered.
    $('body').unbind('mousemove', spinner_follow_mouse);
}

function spinner_follow_mouse(event){
     $('img#ajax_waiting_spinner').css('top', event.clientY+10).css('left', event.clientX+10);
}

Notes:

You’ll want to register this in your default profile if you’re installing this as part of a product. Add

	    <javascript cacheable="True" compression="none" cookable="False"
	              enabled="True" id="ajax_spinner.js"
	              expression=""
	              inline="True"/>

to jsregistry.xml. I’ll leave adding it via the ZMI an an exercise for the reader :)

This works *almost* perfectly. If you don’t move the mouse after initialting an ajax call via a click, the mousemove event never fires and you don’t see the waiting graphic.

YYMV, could be optimized (especially the way I’m registering the js), I probably shouldn’t be using $() (no trouble yet *shrug*), etc, etc. Just a quick note so I don’t forget and so I can get some feedback on the approach.

You know that Add… menu in the green bar when you have edit privileges on a page (I think the actual permission is ‘Modify portal content’)?

Have you ever wanted to replace the default call to portal_factory, via the createObject script with something else? Maybe a custom browser view that will create an object or give the user something else to do? (in my case it was ‘create a contact’ or ‘associate some existing contacts’).

Trying to use the five tag in zcml doesn’t work (it does work, however, to add new items to the display menu).

This does (at least in plone 4.0+, should also apply to 3.5). Given a standard plone egg-based product called my.product:

In my/product, add the following:

• overrides.zcml
• a menu module, I used a package, you can just use menu.py

In overrides.zcml:

<configure
    xmlns="http://namespaces.zope.org/zope"
    xmlns:five="http://namespaces.zope.org/five"
    xmlns:i18n="http://namespaces.zope.org/i18n"
    xmlns:gs="http://namespaces.zope.org/genericsetup"
    xmlns:browser="http://namespaces.zope.org/browser"
    i18n_domain="blitzen.services">

   <browser:menu
    id="plone_contentmenu_factory"
    title="The 'add' menu - allows the user to add new content items in the context"
    class=".menu.MyFactoriesMenu"
    />

</configure>

In menu.py:

from plone.app.contentmenu.menu import FactoriesMenu, FactoriesSubMenuItem

class MyFactoriesMenu(FactoriesMenu):
    """
    Overload factories menu to do fancy stuff with adding new contacts
    """

    def getMenuItems(self, context, request):
        # menuitems is a list of tal-friendly dictionaries
        menuitems = super(MyFactoriesMenu, self).getMenuItems(context, request)

        # we only want to pull these contact switcheroo shenanagians
        # if we're on a workspace object
        if context.portal_type != 'Workspace':
            return menuitems

        new_items = []

        for item in menuitems:
            if item.get('id', False) == 'Contact':
                new_item = item.copy()
                new_item['action'] = "%s/@@new_contact" % (context.absolute_url())
                new_items.append(new_item)
            else:
                new_items.append(item)

        return new_items

There you go. Restart Zope and it should be using your menu items class instead of Plone’s. You can do whatever you want in there, change the order of the items, remove some, inject new ones, etc. See plone.app.contentmenu.menu‘s configure.zcml file for more explaination, and the FactoriesMenu class for more information about what the list of dictionaries returned by getMenuItems is expecting.

One note: menus are aggressively cached in plone, so you might have to restart the app server where you’d normally just use plone.reload. Enjoy!

One other note: there are adapters you can override to do full-blown customizations. This approach seems a bit weird to me but I think it’s sound for the use case (I just want to override the createObject link in the menu).

So I consolidated my code, optimized the crap out of it, checked out the trunk of ZCatalog (hastily following some new developments) and sat down to wrap python’s cProfiler around my browser view, in a vain attempt at trying to make some sense out of my issue (I’ll probably post about it in detail soon, I’m still tinkering)

Anyway, after some trial and error, I came across a feature of Zope that I didn’t know existed: it has a built-in profiler. :P

Here’s how to turn it on in a typical buildout (this is untested, so please, if you try it out let me know if you run into any weirdness):

[buildout]
parts =
    instance

extends = 

http://dist.plone.org/release/4.1-latest/versions.cfg

find-links =

http://dist.plone.org/release/4.1-latest

http://pypi.blitzen.unc.edu

http://dist.plone.org/thirdparty

[instance]
recipe = plone.recipe.zope2instance
user = admin:secret
http-address = 1040
debug-mode = on
verbose-security = on
blob-storage = var/blobstorage

eggs =
    Zope2
    Plone
    ${buildout:eggs}

environment-vars =
    PROFILE_PUBLISHER 1

zope-conf-additional =
    publisher-profile-file ${buildout:directory}/profile.dat

The important part would be the environment-vars and zope-conf-additional bits. I’m not sure if the environment var is necessary or not.

This gives you a new tab under Debug Settings in the ZMI Control Panel.

So you’d navigate to http://127.0.0.1:1040/Control_Panel/manage_main, and you’d see something like this:

Clicking on the ‘Debug Information’ link brings you to a page with a couple of tabs at the top. Clicking on the “Profiling” tab, brings up this interface: (assuming you’ve clicked around your plone site a bit)

So this is really great. I now have the full power of python’s built-in profiling tools right there in Zope. I can use this interface to try to identify issues, or use the pstats module to analyze the profiler.dat file.

The downside is that nothing jumped out at me when I looked at the results after running my slow, memory-eating code. But hey, what I said before about Plone not having profiling is now totally False! I think I just need to get more familair with the results and processing them. Maybe something will shake out eventually.

One caveat: I’m not 100% sure just yet, but I think it’s best to do profiling when you’re not running in foreground mode (bin/instance fg). It seems a lot of time and function calls are spent checking/loading file system resources, which is done more frequently in that mode. I’m thinking that it’s best to try to emulate your production environment as much as possible (get rid of things like plone.reload, PDBDebugMode, DocFinderTab, etc).

w00t!

Only about 3000 objects, bad performance, lots of CPU and RAM used, etc. And it’s Plone 4. It was also eating up disk space, I’d have to pack it multiple times during the course of the import or the Data.fs would climb to 60+ GB.

I have a sinking suspicion that the bulk of the bad performance is due to some very fancy and very hastily put together roll up screens, our virtualization environment (KVM on RHEL 5), or some other factor, but I’m not sure. Profiler tools don’t seem to work well with the zope that comes with Plone 4, and what did work (CallProfiler) didn’t work with five browser views, so when optimizing we were basically flying blind.

We’ve gotten rid of any brain.getObject() calls, prudent use of memoize, plone.app.caching… we’ve done whatever we can to make it faster, and it was better, but still no bueno.

Anyway on a whim I switched my development copy of the app to use RelStorage with MySQL on the back-end (only using mysql because it’s what we’ve already got deployed). I also added memcached into the mix. The non-scientific, contrived tests I did saw a massive increase in performance (only a slight difference between using memcached and not). Almost 4000 requests per second for the default home page, vs about 900 using a stock ZEO setup on basically the same hardware (I was inadvertently testing RelStorage on 2GB of RAM vs 4 in the initial tests). Logged in. Our expensive roll up screens showed an increase from 150 requests per second to 223, with twice as much data in play (there’s an especially slow report that pulls about 1600 objects from the catalog and provides pagination.. when I did my relstorage tests I used a script to generate 900 and then copy-and-pasted them until I got over 3k).

RAM no longer climbed to the point where the VM would swap itself to death, the site remained responsive under heavy load (importing from the MySQL db via wsapi4plone) with only one client running, the database files have barely climbed over 17GB and I forgot to tweak some innodb optimization factors before I created the RelStorage tables (granted the import isn’t done yet, about 50% of the data is left to import..).

So, I’m way happy about all this. But my question now is, why doesn’t it seem like more people are using this? It’s documented very well, it’s a little weird to set up if you’re not a RDBMS person, but it was very easy. It’s fast, it’s efficient, and doesn’t require anything from the developer. You can even build it without undo if you want to make it presumably faster (I opted to not do that, but it’s a nice option). You can scale out the DB server and memcached easily (probably not easier than ZEO + FileStorage, but still very easy). You can move up to postgres or oracle if you need more heft. You can theoretically go back to ZEO if you want (I had trouble with the conversion scripts and cron4plone, but they’re fast and would probably do the trick).

RelStorage is, like, my favourite thing ever. So what’s the downside?