The sTate of Things - Technology

As Promised to TriZPUG: EPDB

nospam@example.com (Joseph Tate) — Fri, 29 Jan 2010 00:36:37 -0600

So I did some digging around after giving my off-the-cuff lightning talk at TriZPUG tonight and it looks like some other ex-rpathers (Thanks Dugan and Gafton!) have forked epdb. There's also the the rPath tree synchronized from here but this tree is missing some of the latest changes. The dugan tree is "python setup.py installable" now, instead of using make, and some shortcut documentation has been created, so I don't have to make this post as long as I thought I was going to have to.

For those who didn't see my little demo, epdb is like pdb (the standard Python debugger), but it adds multi-line text input, history and tab completion, nested debugging from the debug prompt, shortcuts to introspecting code, and a very nice post mortem debugger. Last, but not least, it also contains a server and client for remote debugging. The docs are still pretty sparse, but hopefully more attention can help fix that. I'd also be happy to answer questions about it.

An Exercise in Python planet link recursion

nospam@example.com (Joseph Tate) — Thu, 31 Dec 2009 13:16:43 -0600

Thanks to Chris Calloway, TriZPUG has a planet now. I don't know why it's taken me so long to connect with this group of people (I've been working with Python for 5 years now), but it's a pretty cool group from what I've seen so far. Thanks for making me (a TurboGears guy) feel welcome among all you Zope/Plone/Django developers.

Better E-mail validation

nospam@example.com (Joseph Tate) — Wed, 02 Dec 2009 15:05:30 -0600

Due to several shortcomings of the stock formencode email validator, I forked it and extended the test suite. This fixes the two most glaring issues I know of, namely the inability to handle unicode strings (international domains), and several problems with input checking (e.g., allowing commas) where invalid e-mail addresses make it through.

I did not write most of the code, I just refined it and added tests to exercise it. Let me know if it's useful to you, and if you find problems with it.

New Biofuel System

nospam@example.com (Joseph Tate) — Wed, 11 Nov 2009 08:01:49 -0600

This technology is probably the only biofuel technology I'm really excited about. Unlike E85, it doesn't use food crops, and although algae based programs don't compete directly with food crops, they still require fermentation of cellulose, or refining of algae produced oil to create fuel. More links here and here. Joule Biotechnologies website here. I've been thinking for a while that we should be able to extract CO2 from the atmosphere and produce fuel. Now Joule has gone and built something that might be able to do that.

Usability and "Linuxification"

nospam@example.com (Joseph Tate) — Fri, 18 Sep 2009 00:23:00 -0500

This week, Neil McAllister at InfoWorld wrote about User Interface (UI) design in applications (whether for in house or general use). He argues that the UI should be left to professionals, that the professional UI designers should be given final say in UI design, and that software suffers because developers are building the UI or the usability expert's concerns are dismissed or overruled by developer interests. I certainly have seen the "damage" that software developers can do when left in charge of user interaction; terse messages, techno-babble, pointless configuration options, arcane defaults, etcetera. I'm guilty of such damage myself, but I make no claims to expert status, though I'm a bit more motivated to acquire that status to improve my consulting business.

Continue reading "Usability and "Linuxification""

Working around KDE bug 162485

nospam@example.com (Joseph Tate) — Mon, 06 Jul 2009 09:39:25 -0500

If you want to add support for third party certificates in your KDE 4 desktop, you'll have to work around this languishing bug. KDE for some arrogant reason includes its own certificate authority bundle located in /usr/share/kde*/apps/kssl/ca-bundle.crt, but doesn't provide the tools needed to modify the collection as a normal user. Therefore, as root, move this file out of the way, and link to your distribution's certificate bundle (typically in /etc/ssl/certs). This will let you use your distribution's SSL tools for managing SSL, rather than waiting for KDE to implement these important features. Changes to the distro's CA bundle will require restarting the applications using SSL/TLS before they can see the new root certificate authorities, but that's better than having to click through nag screens for certificates that should be trusted. We still have the security problem of not being able to verify certificates in any app but Konqueror, but the above fix removes the need to do that if you have a Root CA.

Web Browser Posers

nospam@example.com (Joseph Tate) — Sat, 09 May 2009 22:27:46 -0500

Ok, I'm not a novice when it comes to developing websites: I've been building web pages for close on 15 years. But within the last week, I've come across two browser behaviors (or perhaps they're browser addon behaviors) that make me scratch my head.

First, a request coming from something sending the User-Agent "Mozilla/4.0"-- yes, that's all, no clarifiers or parentheticals-- is lopping off the GET parameters when a popup is launched through a button click via an onclick handler. This site states that this is a Yahoo! search something, but the links are not something that a Bot would come across. On the other hand, there is no referrer sent, whick makes me think it could be some kind of link preloader or some other browser add on. Also, I saw a very similar error today coming from Firefox 3.0, though I'm not sure it's related.

Second, and this is really baffling: Sometimes I'm getting requests from a browser identifying itself as IE 6.x that has the entire URL made lowercase. I'm use nice REST-ful URLs for my application, so when a identifier comes across as lowercase, it throws off the lookup. Of course my own copy of IE 6 doesn't exhibit the behavior. For this particular case, I'm using JavaScript to build a URL, and then sticking it as the src attribute of an embedded iframe that is also being created by JavaScript. I'm seeing other errors in my logs though of IE6 and IE7 browsers going to different links (links that would typically be clicked or pasted from an e-mail) that are all lower case as well. Again, not sure if that's related, or if people are just typing them in (lazily) or if it's a browser bug. The only thing I can seem to find about this is this forum (news?) post from 2005 with no replies.

Of course my Google searching is revealing nothing to help me keep my hair, so I turn to the Lazy Web. Any ideas?

In search of good [flash] help

nospam@example.com (Joseph Tate) — Mon, 19 Jan 2009 11:47:24 -0600

I'm working on some freelance work to rebuild a website that has a whole bunch of flash v4 movies that need to be moved forward to flash v9 or higher. I received a reference of a guy who does good work on the flash programming side of things, but finding a flash animator who isn't afraid of a little action scripting has proven extremely challenging. Anyone know of someone who is free for a project immediately?

Sort Tabular Data

nospam@example.com (Joseph Tate) — Sat, 10 Jan 2009 15:25:42 -0600

I hate the way that tabular sorting is typically done in web apps (make links on every column with sort_order="columnname" or similar). It is tedious to code, and requires a lot of bandwidth and round trips from the server, not to mention additional load on the database.

Well, today I Googled a bit, and found SortTable.js. Add the script, and add a "sortable" class to the table you wish to sort, and you're done. It automatically detects string, numeric and date columns and sorts them using a very quick (though non-stable) sort.

I only had a very small problem (some of the CSS styles in FF3 stopped working) with the mechanism used to set the table up to be sortable (window.onload replacement), so I switched it to use jquery(document).ready, which happens later in the page loading process. Works nicely.

Check out the documentation for additional features.

KDE in Foresight

nospam@example.com (Joseph Tate) — Mon, 04 Aug 2008 20:48:28 -0500

I get asked a couple of times a week what's going on with the Foresight KDE edition, what is the status of KDE 4.x, and when is the KDE edition going to be released. I thought I'd blog so that something exists in Internet-firma that can be referred to and even "Googled".

First the status of KDE 4.1: I have no plans to build KDE 4.1 into Foresight. While a giant leap forward from what KDE 4.0 was, KDE 4.1 is still disappointing. Part of what happens when you rewrite rather than make incremental improvements is that functionality gets left behind, and some core functionality that I feel is essential to making a decent desktop experience for the end user is missing or buggy. Also, two of KDE 4.x's killer apps are less usable in 4.1 than they are in KDE 3.5.9: Amarok 2.0 lacks basic wholesale tag editing, dynamic playlists, and is a very buggy Alpha; and Kmail 4.2, part of Kontact is significantly slower than it's 3.5.x counterpart. Kontact itself doesn't yet feel cohesive when it comes to user interface interactions. There are a few KDE 4 apps that I would prefer over kde 3 though: kopete, konversation, potatoguy and many others show marked improvement.

Even though we won't be shipping KDE 4.1 in Foresight's KDE edition, it will be available to install side by side with 3.5.x in kde.rpath.org@fl:2-kde4.1-devel so long as someone is willing to maintain it.

For the KDE edition, there remain a few cosmetic issues with the theme, but mostly what we're missing is new ISOs on the release label and testing. If you'd like to help, jump in #foresight-kde on Freenode. I'm hoping that I'll be able to free up some time in the next few weeks to get us over the final hump.

Icon View Style Grid Layout?

nospam@example.com (Joseph Tate) — Sat, 26 Jul 2008 13:28:16 -0500

For a couple of projects now, I've wanted a grid layout engine that is similar to how desktops display lists of icons: nearly fixed width items, but varying slightly in height, displayed on a variable width page, so your layout could end up with 1 column or 8 depending on the width of the browser window. Tables are no good because they're always a fixed number of columns. Div elements using float works, so long as you make all the elements a fixed width, but they also have to be the same height, or you'll end up with gaps. I'm thinking it's going to have to be javascript driven including redrawing when the page size changes, and to manually size all items to the tallest item in the row, but I can't seem to find an example on the web anywhere (or my Google foo is weak).

Dear lazy web, can you point me in the right direction?

WRT54GL, now more useful

nospam@example.com (Joseph Tate) — Mon, 23 Jun 2008 08:48:35 -0500

I've had an OpenWRT compatible wireless router for almost 5 years now, starting with a Belkin FD7230-4, and then getting a Linksys WRT54GL about a year ago. When I first got the 54GL, I tried to load a few third party firmwares on the Belkin, and after much consternation, went back to the original. I was not interested in compiling my own firmware, but it seemed that I would have had to. Over the past month or two, I've tried three times to get openwrt or xwrt to work on my 54GL. I was attempting to use Kamikaze 7.09 based images, and after flashing several different times, was never able to get the Wide Area Network connection to load (Actully, when I tried DD-WRT I did get it to load, but I couldn't figure out how to do some of the things I needed it to do). Command line attempts to bring up the WAN connection yielded "invalid substitution" errors on the OpenWRT based images. Google didn't show anyone else with this problem, and I didn't have time to dig into it (minutes that my router are down are minutes that my website is down). So I always reloaded the linksys firmware, (via TFTP) and reloaded my settings from the backup I'd made.

Enter Tomato Firmware: An "it just works" firmware for WRT54GL and similar. No building your own from the ground up, no busted start scripts, and it includes all of the features that I need from my router right now. It runs DNSmasq, so it is a full featured DHCP, DNS and TFTP server (which takes three jobs away from my main server). I installed it, the web interface worked, it actually kept most of the settings from my linksys setup, so I didn't have to reload wireless settings. I just had to add my static IP clients, re-add my port forwarding rules, and set up the DNS records. I'll have to see if I can get it to run OpenVPN server and NTP, but if not, that's not a deal breaker (I may even do without NTP for my small network). Then I can really start migrating off my old Athlon XP hardware.

Services, libraries, and consumers "Oh my!"

nospam@example.com (Joseph Tate) — Mon, 19 May 2008 13:52:20 -0500

I've been working on a project for a month or so now where we were told to create a piece of software that did a few tasks, and were instructed to "...make it run as a service". We've been struggling with the idea of creating a remote service that could also be used as a library in some cases (like for our first use case), and we still haven't got it right. We argue about rich objects vs marshalable simple object data structures, and premature optimization vs. marshalling times. Finally I said today, "It's easier to produce a service for a library than it is to consume a service as a library". Write the library, rich objects and all, and while you're doing that, create a sample service and client layers that marshall and unmarshall those rich objects, hopefully with some off the shelf object marshalling utilities. With this you get the added benefit, that if the service/client layers you chose don't work for the consumer, you can create a different set.

I guess in languages where the service layer comes free, or cheap, or where serialization is innate, it's not important to make the distinction, but in Python where service layers are built from scratch, and serialization (cPickle) only works in highly controlled environments it's important to distinguish between libraries that work like import foo; foo.frobinate() and services that work like proxyObject = Proxy(serviceURL); proxyObject.frobinate().

Layers, it's not just good on cakes.

Slice Host and rBuilder Online Images

nospam@example.com (Joseph Tate) — Fri, 09 May 2008 01:28:51 -0500

I host this blog, as well as e-mail for dragonstrider.com and a host of other services on an old PC behind my cable modem at home. This has served me well for the most part, but it requires onsite maintenance when it goes down. This is bad when I'm at work, or vacation, as happened this week. So, I bit the bullet and researched some Virtual Priate Server (VPS) hosting providers.

I ended up choosing Slice Host as a no-frills, just the tech if you please, Linux/Xen-based VPS host. Their entry level plan (slice) gives you 256 MB RAM, 10GB storage, and 100GB of bandwidth for $20/month, and you can scale it with a reboot up to 4GB/160GB/1600GB for $280. /proc/cpuinfo shows that the host for my entry level slice is a two way "Dual-Core AMD Opteron(tm) Processor 2212" operating at 2.0 GHz. There's a separate swap partition (so swap doesn't count against the 10GB limit), as well as web based management tools for rudimentary Name Services, starting, stopping and rebuilding your slice, a web console (in case ssh isn't working for some reason), some statistics and reporting, and my favorite part, a rescue mode.

Rescue mode lets you boot your slice in a rescue environment, mount your root file system in an alternate location, and do what you want (or need) with it. This makes it pretty easy to run your appliance from rBuilder Online on a hosted slice. Here are the steps to get this working. Choose a Xen Appliance Image (32 or 64 bit, though 64 bit is preferred) that is a single file system image.

Create a slice (doesn't matter what kind, we're going to blow it away anyway).

Reboot your slice into rescue mode

SSH or console in using the password mailed to you (yes, rescue mode gets started with a randomized password)

wget -O - <link to the rBO image> | tar -xz # This downloads and extracts the filesystem image

dd if=<path to filesystem image file> of=/dev/sda1

e2fs.ext3 -f /dev/sda1 # This forces a file system check, without this check the next step will fail.

resize2fs /dev/sda1 # Resize the file system image to match the available size

mount /dev/sda1 /mnt

copy the following networking configuration files from your rescue image to your new slice image mounted in /mnt
- /etc/sysconfig/network /etc/sysconfig/network-scripts/ifcfg-eth0 /etc/resolv.conf

Edit /etc/sysconfig/network to fix the hostname to the desired value

chroot /mnt

passwd # changes the root password since the rBO images ship with root's password blanked

At this point you can do any additional configuration you wish, such as adding additional users, making sure that openssh-server is installed and configured to start on boot, etc.

When that's done, shutdown, exit rescue mode from the Slice Host panel, and log in to your new appliance.

There is quite a bit of noise when the slice boots up with an rPath Linux based appliance because the kernel in the image isn't used for booting, and modules.dep isn't located for the booting kernel, but that seems harmless.

Now to build an appliance to run on the thing... I used the rPL 2 beta 3 text devel image as my image while developing this HOWTO.

PyCon 2008

nospam@example.com (Joseph Tate) — Mon, 17 Mar 2008 09:57:40 -0500

I just returned from PyCon 2008 in Chicago, where I connected up again with a few people I met last year and met a few new people. Rather than write a travelogue, I'll just highlight a few of the main things that I learned/did.

It looks like some of the niceties of Zope2 are going to leave the silo under the Repose project. This means that the zope transaction manager (repoze.tm), retry engine (repoze.retry) and other zope features can be used by your WSGI application. Also it means that Zope can be hosted as a WSGI app.

I learned there are ways to extend _import_ that don't involve rewriting _import_. I don't know if I'll ever need to know this, but it might be useful in a plugin loader of some sort. There's a flow chart in Brett's SVN.

I should probably use more weak references in my Python code. Dr. Tim Couper gave a lucid overview of references weak and strong, and practical ways of dealing with circular references, including how to detect them using unittest. I hope he posts his slides soon.

Kevin Dangoor gave a slick demo of TG 2 and Dojo, and gave a quick introduction to Comet servers. It looks like dojo does pretty much everything that Mochikit does, but adds support for Javascript UI widgets. Comet looks really cool. More about that later.

Tahoe is a nifty looking distributed remote filesystem concept. It needs fuse drivers though before it's really a filesystem rather than a storage mechanism (like S3). It's all RESTy though, which makes some people happy.

There are metrics for measuring code complexity, and python has tools to do that. Basically every branch in a method increases complexity by one. Keep complexity down to 10 or less to make unit testing feasible. More than 10 and the number of unittests required to cover every branch starts to get unmanageable. Apparently there is a PyMetrics module for measuring code complexity. Here are the slides from a very interesting talk.

I held a BoF on i18n'izing web applications, and a few people helped me to brainstorm this problem. The general consensus was that you should do translation as close to the user as prudent; view certainly, controller is ok, but never in the model, if you have messages that are generated by some other process, or that gets cached to the database, the "data" should be stored separately from the operational message. Generally you should avoid sending this kind of data to the end user, and instead abstract it with messages of your own. Barry Warsaw came in to the BoF with a hard problem; usually when you're handling web templates, you filter out the non-element text and build a string table out of that. What about the cases however, where you have <em> or <b> tags? I think that if you had an XML parser that would extract the text elements, and if sub-elements are present lump them with the text. For example, if you had "This text <b>needs</b> to be translated", it would be collected as one string, but if you had "<div><span>Phrase 1</span><span>Phrase 2</span></div>" it would be two phrases.

There are a couple of cool tools that have stemmed from the PyPy project. The first offers a framework for distributed testing. Py.test does stack introspection so that simple "assert" statements can be used instead of the pseudo self.assertEquals().

I got together with some people to talk about Python Packaging. Here are some notes from the meeting, and there's been some additional discussion on the distutils-SIG mailing list. Jeff Rush did a tutorial on Thursday, and his slides and exercises are available. Hopefully something coherent comes out of this, but it looks like more of the same resistance to making application installation reasonable.

I talked briefly with Ivan Krstić and Noah Kantrowitz about python plugin frameworks. Hopefully we'll be able to collaborate some on a generic system for application extensions.

Probably the coolest thing I learned about was a couple of protocols, both implemented by orbited. Orbited's client libraries allow for a user to connect to a orbited server to provide a push mechanism for sending messages to a web browser. The Orbited team was shooting for a 0.4 release after the PyCon Sprints this week. Check out their IRC client demo.

PyCryptoPP was suggested to me as a decent Python based Crypto library, since it's simply python wrappers for Crypto++. PyOpenssl has a new maintainer, and should be getting some much needed attention.

I also enjoyed Raymond Hettinger's talk on "Core Python Containers". It was very helpful in understanding what list, dict, deque and set do behind the scenes, and how to use them most efficiently.

Overall, I had a great experience at PyCon 2008. The venue was big enough for all of us (there were more than 1000 registrants), and there was more than enough room for Open Space/BoF talks.

Chicago style pizza is definitely different from anything I have eaten before. The jury is still out as to whether it's worth the hype.