The sTate of Things - Work

Better E-mail validation

nospam@example.com (Joseph Tate) — Wed, 02 Dec 2009 15:05:30 -0600

Due to several shortcomings of the stock formencode email validator, I forked it and extended the test suite. This fixes the two most glaring issues I know of, namely the inability to handle unicode strings (international domains), and several problems with input checking (e.g., allowing commas) where invalid e-mail addresses make it through.

I did not write most of the code, I just refined it and added tests to exercise it. Let me know if it's useful to you, and if you find problems with it.

Web Browser Posers

nospam@example.com (Joseph Tate) — Sat, 09 May 2009 22:27:46 -0500

Ok, I'm not a novice when it comes to developing websites: I've been building web pages for close on 15 years. But within the last week, I've come across two browser behaviors (or perhaps they're browser addon behaviors) that make me scratch my head.

First, a request coming from something sending the User-Agent "Mozilla/4.0"-- yes, that's all, no clarifiers or parentheticals-- is lopping off the GET parameters when a popup is launched through a button click via an onclick handler. This site states that this is a Yahoo! search something, but the links are not something that a Bot would come across. On the other hand, there is no referrer sent, whick makes me think it could be some kind of link preloader or some other browser add on. Also, I saw a very similar error today coming from Firefox 3.0, though I'm not sure it's related.

Second, and this is really baffling: Sometimes I'm getting requests from a browser identifying itself as IE 6.x that has the entire URL made lowercase. I'm use nice REST-ful URLs for my application, so when a identifier comes across as lowercase, it throws off the lookup. Of course my own copy of IE 6 doesn't exhibit the behavior. For this particular case, I'm using JavaScript to build a URL, and then sticking it as the src attribute of an embedded iframe that is also being created by JavaScript. I'm seeing other errors in my logs though of IE6 and IE7 browsers going to different links (links that would typically be clicked or pasted from an e-mail) that are all lower case as well. Again, not sure if that's related, or if people are just typing them in (lazily) or if it's a browser bug. The only thing I can seem to find about this is this forum (news?) post from 2005 with no replies.

Of course my Google searching is revealing nothing to help me keep my hair, so I turn to the Lazy Web. Any ideas?

In search of good [flash] help

nospam@example.com (Joseph Tate) — Mon, 19 Jan 2009 11:47:24 -0600

I'm working on some freelance work to rebuild a website that has a whole bunch of flash v4 movies that need to be moved forward to flash v9 or higher. I received a reference of a guy who does good work on the flash programming side of things, but finding a flash animator who isn't afraid of a little action scripting has proven extremely challenging. Anyone know of someone who is free for a project immediately?

Services, libraries, and consumers "Oh my!"

nospam@example.com (Joseph Tate) — Mon, 19 May 2008 13:52:20 -0500

I've been working on a project for a month or so now where we were told to create a piece of software that did a few tasks, and were instructed to "...make it run as a service". We've been struggling with the idea of creating a remote service that could also be used as a library in some cases (like for our first use case), and we still haven't got it right. We argue about rich objects vs marshalable simple object data structures, and premature optimization vs. marshalling times. Finally I said today, "It's easier to produce a service for a library than it is to consume a service as a library". Write the library, rich objects and all, and while you're doing that, create a sample service and client layers that marshall and unmarshall those rich objects, hopefully with some off the shelf object marshalling utilities. With this you get the added benefit, that if the service/client layers you chose don't work for the consumer, you can create a different set.

I guess in languages where the service layer comes free, or cheap, or where serialization is innate, it's not important to make the distinction, but in Python where service layers are built from scratch, and serialization (cPickle) only works in highly controlled environments it's important to distinguish between libraries that work like import foo; foo.frobinate() and services that work like proxyObject = Proxy(serviceURL); proxyObject.frobinate().

Layers, it's not just good on cakes.

Slice Host and rBuilder Online Images

nospam@example.com (Joseph Tate) — Fri, 09 May 2008 01:28:51 -0500

I host this blog, as well as e-mail for dragonstrider.com and a host of other services on an old PC behind my cable modem at home. This has served me well for the most part, but it requires onsite maintenance when it goes down. This is bad when I'm at work, or vacation, as happened this week. So, I bit the bullet and researched some Virtual Priate Server (VPS) hosting providers.

I ended up choosing Slice Host as a no-frills, just the tech if you please, Linux/Xen-based VPS host. Their entry level plan (slice) gives you 256 MB RAM, 10GB storage, and 100GB of bandwidth for $20/month, and you can scale it with a reboot up to 4GB/160GB/1600GB for $280. /proc/cpuinfo shows that the host for my entry level slice is a two way "Dual-Core AMD Opteron(tm) Processor 2212" operating at 2.0 GHz. There's a separate swap partition (so swap doesn't count against the 10GB limit), as well as web based management tools for rudimentary Name Services, starting, stopping and rebuilding your slice, a web console (in case ssh isn't working for some reason), some statistics and reporting, and my favorite part, a rescue mode.

Rescue mode lets you boot your slice in a rescue environment, mount your root file system in an alternate location, and do what you want (or need) with it. This makes it pretty easy to run your appliance from rBuilder Online on a hosted slice. Here are the steps to get this working. Choose a Xen Appliance Image (32 or 64 bit, though 64 bit is preferred) that is a single file system image.

Create a slice (doesn't matter what kind, we're going to blow it away anyway).

Reboot your slice into rescue mode

SSH or console in using the password mailed to you (yes, rescue mode gets started with a randomized password)

wget -O - <link to the rBO image> | tar -xz # This downloads and extracts the filesystem image

dd if=<path to filesystem image file> of=/dev/sda1

e2fs.ext3 -f /dev/sda1 # This forces a file system check, without this check the next step will fail.

resize2fs /dev/sda1 # Resize the file system image to match the available size

mount /dev/sda1 /mnt

copy the following networking configuration files from your rescue image to your new slice image mounted in /mnt
- /etc/sysconfig/network /etc/sysconfig/network-scripts/ifcfg-eth0 /etc/resolv.conf

Edit /etc/sysconfig/network to fix the hostname to the desired value

chroot /mnt

passwd # changes the root password since the rBO images ship with root's password blanked

At this point you can do any additional configuration you wish, such as adding additional users, making sure that openssh-server is installed and configured to start on boot, etc.

When that's done, shutdown, exit rescue mode from the Slice Host panel, and log in to your new appliance.

There is quite a bit of noise when the slice boots up with an rPath Linux based appliance because the kernel in the image isn't used for booting, and modules.dep isn't located for the booting kernel, but that seems harmless.

Now to build an appliance to run on the thing... I used the rPL 2 beta 3 text devel image as my image while developing this HOWTO.

PyCon 2008

nospam@example.com (Joseph Tate) — Mon, 17 Mar 2008 09:57:40 -0500

I just returned from PyCon 2008 in Chicago, where I connected up again with a few people I met last year and met a few new people. Rather than write a travelogue, I'll just highlight a few of the main things that I learned/did.

It looks like some of the niceties of Zope2 are going to leave the silo under the Repose project. This means that the zope transaction manager (repoze.tm), retry engine (repoze.retry) and other zope features can be used by your WSGI application. Also it means that Zope can be hosted as a WSGI app.

I learned there are ways to extend _import_ that don't involve rewriting _import_. I don't know if I'll ever need to know this, but it might be useful in a plugin loader of some sort. There's a flow chart in Brett's SVN.

I should probably use more weak references in my Python code. Dr. Tim Couper gave a lucid overview of references weak and strong, and practical ways of dealing with circular references, including how to detect them using unittest. I hope he posts his slides soon.

Kevin Dangoor gave a slick demo of TG 2 and Dojo, and gave a quick introduction to Comet servers. It looks like dojo does pretty much everything that Mochikit does, but adds support for Javascript UI widgets. Comet looks really cool. More about that later.

Tahoe is a nifty looking distributed remote filesystem concept. It needs fuse drivers though before it's really a filesystem rather than a storage mechanism (like S3). It's all RESTy though, which makes some people happy.

There are metrics for measuring code complexity, and python has tools to do that. Basically every branch in a method increases complexity by one. Keep complexity down to 10 or less to make unit testing feasible. More than 10 and the number of unittests required to cover every branch starts to get unmanageable. Apparently there is a PyMetrics module for measuring code complexity. Here are the slides from a very interesting talk.

I held a BoF on i18n'izing web applications, and a few people helped me to brainstorm this problem. The general consensus was that you should do translation as close to the user as prudent; view certainly, controller is ok, but never in the model, if you have messages that are generated by some other process, or that gets cached to the database, the "data" should be stored separately from the operational message. Generally you should avoid sending this kind of data to the end user, and instead abstract it with messages of your own. Barry Warsaw came in to the BoF with a hard problem; usually when you're handling web templates, you filter out the non-element text and build a string table out of that. What about the cases however, where you have <em> or <b> tags? I think that if you had an XML parser that would extract the text elements, and if sub-elements are present lump them with the text. For example, if you had "This text <b>needs</b> to be translated", it would be collected as one string, but if you had "<div><span>Phrase 1</span><span>Phrase 2</span></div>" it would be two phrases.

There are a couple of cool tools that have stemmed from the PyPy project. The first offers a framework for distributed testing. Py.test does stack introspection so that simple "assert" statements can be used instead of the pseudo self.assertEquals().

I got together with some people to talk about Python Packaging. Here are some notes from the meeting, and there's been some additional discussion on the distutils-SIG mailing list. Jeff Rush did a tutorial on Thursday, and his slides and exercises are available. Hopefully something coherent comes out of this, but it looks like more of the same resistance to making application installation reasonable.

I talked briefly with Ivan Krstić and Noah Kantrowitz about python plugin frameworks. Hopefully we'll be able to collaborate some on a generic system for application extensions.

Probably the coolest thing I learned about was a couple of protocols, both implemented by orbited. Orbited's client libraries allow for a user to connect to a orbited server to provide a push mechanism for sending messages to a web browser. The Orbited team was shooting for a 0.4 release after the PyCon Sprints this week. Check out their IRC client demo.

PyCryptoPP was suggested to me as a decent Python based Crypto library, since it's simply python wrappers for Crypto++. PyOpenssl has a new maintainer, and should be getting some much needed attention.

I also enjoyed Raymond Hettinger's talk on "Core Python Containers". It was very helpful in understanding what list, dict, deque and set do behind the scenes, and how to use them most efficiently.

Overall, I had a great experience at PyCon 2008. The venue was big enough for all of us (there were more than 1000 registrants), and there was more than enough room for Open Space/BoF talks.

Chicago style pizza is definitely different from anything I have eaten before. The jury is still out as to whether it's worth the hype.

PyCon 2007

nospam@example.com (Joseph Tate) — Mon, 26 Feb 2007 15:08:39 -0600

Last night I returned home after 3 days at PyCon 2007 in Dallas Texas. I think my brain is full. It was great to meet people to whom I've talked on IRC or email, or whose software I've used. It was also fun to hang out with Mark Ramm, Robert Brewer, Ian Bicking, Ben Bangert, Rob Orsini and Chad Whitacre at Robert and Chad's suite on Friday night. I haven't laughed so hard in a long time.

As Robert mentioned in his blog post, there was some mind melding, er, meeting on the part of Zope, Pylons and CherryPy as well as others. I'll look at the way that sockets are handled and contribute some code. I'm looking forward to this collaboration. In addition to what Robert mentioned another thing we decided was to come up with several "stories" for deployment and help guide users down the right path for them when deploying a Python web application.

For the rest of the conference, most of the keynote and full length talks were good, the lightning plenary talks were great, having several rooms to hold Birdts of a Feather (BoF) or non-scheduled talks provided much enlightenment, owning an 802.11A wireless chipset was a difference maker in connectivity and bandwidth, and lots of Pythonistas have beards (some of which are out of control)

I noted several things that I thought merited additional attention:

trac -- I didn't know it could be used with mercurial instead of svn

pyjamas -- generate javascript from python code

Idea (some kind of testing tool, can't find a link)

IPython -- Interactive Python including a replacement for pdb

Jack Slocum's YahooUI extensions/datagrid

Fitnesse -- A framework for collecting functional and integration tests

Zope's TestRunner -- Has something that could be used for memory/object leak detection

Monit -- might be used for the same

Star Schema -- Data reporting in memory instead of in db

Packaging eggs instead of building new setuptools targets for linux packaging systems.

zdaemon, supervisor2 and other libraries that may make daemonizing a python program trivial.

Lots of cool testing tools: figleaf, twill, scotch

Hopefully I can make it again next year.

Nice and Easy Server Calls (or How to keep your hair when developing rAA plugins)

nospam@example.com (Joseph Tate) — Mon, 15 Jan 2007 10:24:49 -0600

Previous to last weekend, the process for setting up a rAA plugin to talk to the backend looked like the following:

Set up a schedule object (one time or immediate based on whether you want to block or not).
Set up a table to hold whatever data you need when doing the operation, or more recently use plugin properties.
Link the schedule Id from the schedule object in 1 with the data in 2. For immediate tasks, you needed to do this through a callback.
Implement XMLRPC interfaces on the webside to retrieve the data in 2, and to save the results.
Add a logical branch in your backend doTask method to do the new functionality, making xmlrpc calls as appropriate to retrieve the data for processing, and store the result.
Write a callback method to retrieve the result saved above.

Basically that sucked. Now, the process is as follows.

On the web side, make a call to self.callBackend passing in the method name on the backend to call as well as any arguments to pass.
On the server side, implement that function and return a result.

Notice that no XMLRPC interfaces need designing, no additional tables need creating, and no callbacks are needed to retrieve the data.

This currently only works for synchronous tasks where the page request will block until the results are available, but in the future, there will be another method (called callBackendAsync) that will make longer running tasks easy (in conjunction with the CallbackStatusWidget). I hope that I can get to that soon.

Announcing Port 25 version 0.11

nospam@example.com (Joseph Tate) — Wed, 01 Nov 2006 07:59:35 -0600

I'm pleased to announce a Xen enabled virtual appliance download for Port 25. Get it NOW!.

This release contains a new version of SpamAssassin as well as security fixes from rPL.

KDE 0.90 beta Released

nospam@example.com (Joseph Tate) — Fri, 15 Sep 2006 20:54:17 -0500

I am pleased to announce the first beta release of the KDE Distribution. This is based on rPath Linux 1 with updated kde, hal, avahi, cups and other packages, as well as added amarok, kopete, konversation, and k3b. DVD and CD images are available from the release page.
Continue reading "KDE 0.90 beta Released"

Turbogears help wanted

nospam@example.com (Joseph Tate) — Fri, 15 Sep 2006 20:22:02 -0500

My employer, rPath, is hiring web developers to work on the rPath Appliance Agent team. rAA of course is built on TurboGears. See this page for more details.

rAA in the News

nospam@example.com (Joseph Tate) — Tue, 29 Aug 2006 10:41:04 -0500

So my last entry, oh so many moons ago, mentioned that I was working on a new project. Now that that project has been released, I can talk about it, but better yet, how about an article about rPath Linux, what we're doing, and the rPath Appliance Agent (or rAA) which I helped design and build.

A little bit about rAA itself; it provides a pluggable framework for system configuration tasks, like setting up networking, setting the root password and checking for and applying software updates. There are some screenshots of rAA in action linked from the eWeek article above. rAA is written in python and built around TurboGears, a web application framework for RAD and MVC web development using Python.

rAA is Open Source Software, released under the Reciprocal Public License. A conary package group that can be added to a rBuilder Project to insert rAA into your own custom software appliance is available from the rAA Project.

Some basic documentation for rAA is available here. It's built with rPath Linux as an assumed installation platform, but several appliances have rAA built-in, and have VMware images available so you can try it out without installing a new Linux distribution.

Catching up

nospam@example.com (Joseph Tate) — Thu, 30 Mar 2006 05:58:33 -0600

First of all it sucks to be sick. I don't know what I caught, nor where I caught it from, but now I'll be coughing for at least a week and my lungs are already sore from the last two days of hacking. I just hope that my wife and son don't catch it. Alex already has ear infections and a nasty cough coinciding with the arrival of four new teeth. At least his black eye is better.

A lot has happened since December, and I won't even begin to enumerate it all, just to tell you that I was working feverishly to get VMware and QEMU/RAW Hard Drive images working in rBuilder Online. With that task completed I am no longer working on rBuilder, instead I'm working with Ken VanDine and Xiaowen on something to be announced soon.

Yesterday I modified the PHPAppPackageRecipe class to make it more friendly to sofware upgrades. Previously the tarball containing the PHP application was extracted directly to /srv/www using whatever root folder was in the tarball. Then the recipe made a link connecting /srv/www/ to whatever folder was extracted. The problem with this arrangement is that if that directory contained the version, as many php apps do, the next version would be installed to a different location causing a manual step to move settings and data to the new folder.

Now the recipe extracts the application as most other packages do to the conary build folder. Then I copy the contents to /srv/www/ and eliminate the link. Hopefully this will ease upgrade pain. I rebuilt Gallery bumping its version to the newly released 2.1 and Serendipity in the LAMP rBuilder project so if you're running either of those be aware that there will be some one-time transitional pain. Feel free to contact me for help in upgrading to these new packages.

PHP Application Packaging

nospam@example.com (Joseph Tate) — Mon, 19 Dec 2005 09:10:37 -0600

I spent some time this weekend packaging Gallery2 and Serendipity for rPath based Linux Distributions. As part of that effort, I built a conary recipe superclass to make packaging other PHP applications easier.

This class currently resides in the LAMP repository in the phpapppackage:source trove. The superclass handles creation of the apache configuration file to drop in to /etc/httpd/conf.d, provides stubs for creating empty files and directories for use by the PHP application, and sets up the requirement on PHP.

So, a new recipe for a simple application like gallery looks like the one here.

Notice the calls to MakeWriteableDirs and CreateWriteable. Those create the empty config files and directories needed by the application. These are created with ownership "apache", as which user the stock httpd server runs. Also notice the r.macros.dirconf macro. This data gets inserted into the apache configuration file between <Directory foo> directives so that you can set application specific php configuration values, or even set overrides.

If you use phpapppackage.recipe as a root class to package some other php application, shoot me an e-mail to let me know. I'd love to hear about it.

AJAX and rBuilder

nospam@example.com (Joseph Tate) — Thu, 10 Nov 2005 09:22:54 -0600

For the past week or so, I have worked on creating a new AJAX-y group trove builder for rBuilder Online. This widget will allow anyone who can shop online to create new purpose built Linux images. The images can then be downloaded, burned to a CD-R, and installed, all in a day.

The best part of the work I've been doing is that it elegantly falls back to POSW (Plain old static web) when javascript is disabled. I do this by rewriting the urls on the page to make the AJAX calls to our XMLRPC interface to rBuilder Online. If the urls don't get rewritten, they will make page load calls to perform the same action.
Continue reading "AJAX and rBuilder"