Pylons with Tokyo Cabinet Beaker Sessions

Maintaining a session is something most web apps need to do, and all web frameworks implement some sort of session management system.

Pylons allows you to specify the type of session your web app should use: database, file, memcached, etc. The great thing about it is that you can easily implement your own session manager and plug it right in.

Here I am implementing a session manager that uses Tokyo Cabinet and JSON (instead of cPickle). The simplejson module now has speedups that use C extensions so load and dump are more or less on par with cPickle. With JSON serialization you'll be able to share the session data in apps written in other languages (such as Java, Perl, Ruby).

Tokyo Cabinet is a class of DBM that is a key-value store. It differs from RDBMS in that data are stored solely as key-value pairs, with no relations to each other. The advantage over RDBMS is that Tokyo Cabinet is lightning fast in comparison. If you're familiar with memcached, you can think of it like memcached except with persistent data. Since sessions need to be accessed frequently, and sessions are essentially key-value pairs -- a key/namespace mapping to a session object -- I thought it'd be a good candidate to use Tokyo Cabinet.

Assuming you already have a Pylons project setup, let's create a file under [project path]/lib/ called tcjson.py (Tokyo Cabinet JSON... get it?), and fill it with the following.

import logging

from beaker.container import NamespaceManager, Container
from beaker.synchronization import file_synchronizer
from beaker.util import verify_directory

try:
  from pytyrant import PyTyrant
except ImportError:
    raise InvalidCacheBackendError("PyTyrant cache backend requires the 'pytyrant' library")

try:
  import json
except ImportError:
  try: 
    import simplejson as json
  except ImportError:
    raise InvalidCacheBackendError("PyTyrant cache backend requires the 'simplejson' library")

log = logging.getLogger(__name__)

class TokyoCabinetManager(NamespaceManager):
  def __init__(self, namespace, url=None, data_dir=None, lock_dir=None, **params):
    NamespaceManager.__init__(self, namespace)
     
    if not url:
      raise MissingCacheParameter("url is required") 
    
    if lock_dir:
      self.lock_dir = lock_dir
    elif data_dir:
      self.lock_dir = data_dir + "/container_tcd_lock"
    if self.lock_dir:
      verify_directory(self.lock_dir)            
    
    host, port = url.split(':')
    self.tc = PyTyrant.open(host, int(port))

  def get_creation_lock(self, key):
    return file_synchronizer(
      identifier ="tccontainer/funclock/%s" % self.namespace,
      lock_dir = self.lock_dir)

  def _format_key(self, key):
    return self.namespace + '_'  

  def __getitem__(self, key):
    return json.loads(self.tc.get(self._format_key(key)))

  def __contains__(self, key):
    return self.tc.has_key(self._format_key(key))

  def has_key(self, key):
    return key in self

  def set_value(self, key, value):
    self.tc[self._format_key(key)] =  json.dumps(value)

  def __setitem__(self, key, value):
    self.set_value(key, value)
    
  def __delitem__(self, key):
    del self.tc[self._format_key(key)]

  def do_remove(self):
    self.tc.clear()
  
  def keys(self):
    raise self.tc.keys()


class TokyoCabinetContainer(Container):
  namespace_manager = TokyoCabinetManager

Now, put this in your [project path]/config/environment.py file.

import investor.lib.tcjson as tcjson 
import beaker
beaker.cache.clsmap['ext:tcjson'] = tcjson.TokyoCabinetManager

You can then specify ext:tcjson as your beaker session in your INI file (e.g. development.ini).

beaker.session.type = ext:tcjson
beaker.session.url = 127.0.0.1:1978

Fire up your app and it should be using Tokyo Cabinet for your sessions! :)