FrontPage NamespacePassingRevisited / backlinks

Note - this document is still a work in progress. Please do not make inline comments in the proposal - use the NamespacePassingRevisitedDiscussion to make comments.

Contact

http://www.zope.org/Members/hathawsh

Background

Namespaces are the key to the magic of DTML. To resolve a <dtml-var> reference, the DTML machinery simply requests the reference from the current namespace.

Namespace objects are a stack of dictionaries. The most recently added dictionary is searched first. This algorithm is implemented in cDocumentTemplate.c, where it is called a "MultiMapping" or "MM" for short, and is exposed to Python as a "TemplateDict". The file pDocumentTemplate.py is the equivalent non-optimized version.

The name lookup gets more interesting when mixed with acquisition. By using an adapter called InstanceDict, it is possible to put any object in the namespace stack, including objects that are wrapped in acquisition wrappers.

ChrisW

Is there a diagram/list of what your avarage namespace (for example if you do <dtml-var object> or <dtml-var method>) looks like in terms of what is actually in it and what order it is searched? If not, then how about a NamespaceExplainer^x

The effect is that it's possible to search for a name among a potentially large set without spending a lot of time on namespace construction. In fact, most namespace searches occur entirely in C-optimized code.

When a DTML method calls another DTML method, the namespace is usually passed. The subtemplate, rather than create a new namespace, just pushes entries onto the stack of dictionaries. That way it can access all names that were available in the first method. Before the subtemplate is finished, it pops the added entries.

Problem

While working on the PythonMethod^x product, it became apparent that it would be useful to allow a method to access the DTML namespace. However, it also became apparent that the current implementation of namespaces has created a lot of inconsistency in calling conventions for different kinds of methods.

The document template invocation mechanism has also necessitated the infamous hack:

<dtml-var expr="someTemplate(_.None, _)">

This effectively duplicates what happens when a name is looked up in a namespace that refers to a DTML method. By default, namespace lookups try to execute the object found, and when the object has an isDocTemp attribute, it is invoked with two arguments: None and the namespace object.

The relevant code, in OFS/DTMLMethod^x.py and DocumentTemplate/DT_String.py, is very difficult to understand. Only after studying it in depth does it become clear that the REQUEST variable passed to DTMLMethod^x is not necessarily a REQUEST object; when called as a subtemplate it is a namespace object (a TemplateDict^x) and the "client" argument is None. When DTMLMethod^x objects are invoked through a URL, the mapply() method is tricked into sending the correct arguments by the apparently meaningless func_code object. All of this also depends on an obscure attribute called "isDocTemp", which is set in a base class. This level of obfuscation leads to poor maintainability and makes it very difficult to standardize on method invocation semantics.

Proposed Solution

Zope 2.2 now provides execution stacks for security purposes. The stacks are essentially bound to the thread. Until now, Zope had no dependency on thread-bound variables, but now that it does, why not extend the dependency by using a thread-bound namespace?

The namespace should be created when first invoking a method that uses a namespace. Subtemplates could then get the namespace from the thread rather than through an argument. Subtemplates would push new entries and remove them before finishing, the same as is done now.

DT_String.__call__ currently checks whether the "mapping" argument is an instance of TemplateDict^x, and if it is then new entries will be pushed.

In this proposal DT_String would try to get the namespace from a thread-bound variable. If not created yet, it be would created. The mapping argument would still be added to the namespace, but for backward compatibility and safety it should not be added if it is an instance of TemplateDict^x.

cDocumentTemplate and DT_Util would be modified since they don't need to check for the presence of an isDocTemp attribute anymore. Subtemplates would be invoked with no arguments by default.

Benefits

• Calling conventions simplified.

• There would no longer be a need to pass namespace objects. The infamous hack mentioned above would no longer be necessary.

• Methods other than DTML methods could safely access the namespace.

Risk Factors

• If Zope makes use of microthreads, the code that binds variables to threads will have to adapt. Actually this will be an issue that affects ZODB and the security machinery as well, so we will probably need a module that implements thread-bound variables through either OS threads or microthreads.

• This makes namespace passing more reliable. As a result, some DTML methods may pick up objects from the namespace where they didn't before.

• This changes the meaning of certain arguments for often used methods. This proposal has been carefully crafted to not break existing code, but experience says that somehow, somewhere, it will.

Scope

This project is focused on reducing the complexity of namespace passing so as to make namespaces accessible to method types other than DTML methods.

Deliverables

• Changes to DocumentTemplate^x to provide the specified behavior

• A new threadvars module for maintaining and eventually cleaning up thread-bound variables

• A note in the CHANGES file