Qwicap 1.4a34

Qwicap 1.4a34 was released this weekend. With four busy months elapsed since the previous release, 1.4a12, it was time to declare something "done" again, because a lot fixes and improvements have accumulated over that time. The development of this version has coincided with the development of our second and third large, Qwicap-based web applications which will be supplying the end-user and administrator interfaces to UT Austin's new identity management system. The same four developer team is responsible for both applications. They're up against the challenge of schedule-constraints, the rapidly evolving, still-under-development code of the identity management system itself, and the fact that none of them had used Qwicap before (well, one had used its XML engine a few years back). So, I've been watching their progress with interest, wondering which aspects of Qwicap would work well for them, and which wouldn't.

So far, with both applications in an advanced state of development, and user testing of the end-user application already complete, the results are encouraging. The team seems uniformly pleased with the experience of working with Qwicap, and they're already planning to use it as the basis of their future projects. Though I visit their office daily to see how things are going, and to take a look at any problems they want a hand with, there have been remarkably few problems. They've found some minor bugs (minor in that there were small, simple fixes), a few more troublesome bugs, and they've needed a few new features. The more troublesome bugs were interesting, because they were the result of the team choosing a flow-control and markup handling strategy that had never occurred to me, and consequently it was a bit surprising to Qwicap, too. What stunned me was that it worked, and worked well in all but a few obscure cases. I had to sit down with the team-lead and walk through the relevant code several times before I even understood why it worked at all.

Qwicap 1.4a4 and 1.3.3 Released

Qwicap 1.4a4 and 1.3.3 have been released. The change history has all the details fit to print, but the main motivation behind the 1.4a4 release was to make available a fix for a race condition during session tear-down that's been in the Qwicap code base more-or-less forever. Originally, its effects were of little importance—mostly some exceptions showing-up in the logs—and it wasn't at all clear how the problems came about. With 1.4's support for dynamically generated downloadables, those effects started to have an actual impact by sometimes preventing access to the downloadables in the final page of the session. Fortunately, that made the nature of the problem more obvious and led to this bug fix.

Another old bug (or inadequately developed feature) was in the Qwicap.goodbye method which is the method that sends an application's final page to the user. The bug was that the method didn't inform Qwicap that the application was shutting-down, so Qwicap did nothing to help the application shutdown. Which was fine, for applications that invoked goodbye and gracefully returned their way back to, and out of, their main method, but those that didn't could inadvertently persist in the delusion that they were still operating normally by calling a variety of Qwicap methods that should have no role in, and could delay, the shutdown of an application (retrieving user input, sending pages to the user, etc.). Now that goodbye informs Qwicap that the application is shutting down, those methods that shouldn't be used during shutdown will throw a QwicapSessionDeathException when invoked, in order to help the application along in its shutdown activities. (By the way, QwicapSessionDeathException now includes a message that states why the session is dead, or dying.)

In addition, some enhancements have been made since 1.4a0, including: the addition of the Qwicap.getFloat method; moving the Downloadable class into the public API; changing all of the XML attribute manipulation methods to accept arbitrary objects as attribute values; and various improvements to the Javadocs. In addition, the Hunt The Wumpus example application has had its game logic further debugged, and it has been extended to support larger maps, upon request. The extra "config" page that allows larger maps to be specified, also provides a long-overdue example of Qwicap features like the high-level form manipulation API, while further illustrating how multi-page applications function.

Qwicap 1.3.3

Qwicap 1.3.3 was created by back-porting the bug fixes and minor enhancements that have accumulated in the 1.4 branch as of version 1.4a4. So, that race condition is gone, Qwicap.goodbye is beefed-up, Qwicap.getFloat exists, XML attribute values can be specified as arbitrary objects, QwicapSessionDeathExceptions are more informative, Qwicap.convertSubmitButtonsToInputs has had its documentation substantially expanded, various Context methods that didn't need to be public have been removed from the public API, the XML tag hierarchy validator includes document names in any exceptions it throws, all subclasses of Markup now include getMutable and getImmutable methods, the Context.getFile and Context.getExistingFile methods have been documented at long last, some useless code has been removed, and the Javadocs have suffered various improvements. Also, its version of Hunt The Wumpus received a bug fix, and therefore sucks less than it traditionally has done.

Which Reminds Me

While trying to implement the large map generator for the Qwicap 1.4a4 version of Hunt The Wumpus, I was frustrated for quite a while by my inability to come up with a simple implementation for the generator. The generator had a conceptually simple job to do, so it seemed the implementation should be simple, too. We all know such things frequently don't work out that way, but it was bugging me all the same. At one point, I considered using a cellular automaton to generate a hexagon-based map, and that sent me off to Wolfram's A New Kind of Science (NKS) pages, which was more convenient than winching down my copy of the book from its shelf and finding a spot for it on my desk. What I stumbled upon in doing so was WolframTones, a site that generates music using Wolfram's one-dimensional cellular automata. I found it very impressive, and a welcome demonstration of Wolfram's NKS ideas in an easily appreciated, real-world application - which is the sort of thing I regretted the absence of in the NKS book.

My only complaint with WolframTones is that it limits its compositions to thirty seconds. Ideally, it'd be possible to generate arbitrarily long compositions. What I wonder about long compositions is this: Will they seem to hold-together as well as the thirty-second ones usually do? Or does one's brain eventually reject a longer composition due to the extended lack of the sort of patterns designed into music by human intelligences? This would probably be more of an issue with some types of music than others; in particular, some flavors of jazz might not be a problem. And if it's not a problem for some types of music, what does that tell us about music, and will this eventually result in, say, a music generator box for use in stores that will continuously compose and play new music to save the stores the cost of playing background music that someone owns? As background music, shortcomings in the compositions would be masked by the lack of attention being paid to them. Which is fine for customers passing through, but what about the staff that has to hear the stuff all day? Will they eventually hunt down and destroy the box to save themselves? Or would it work well enough that it wouldn't drive people nuts, and, in fact, would periodically produce something really catchy? With every composition produced by an automaton chosen randomly from a set of around 4 billion possible automata, and with no direct access to the box to ask it to save its state, one would end-up in the frustrating situation of hearing a great song that no one had ever heard before, and knowing that you'd never, ever hear it again. That might promote some useful contemplation of the transitory nature of life, but I think, for the most part, it would just be a new way of pissing people off. Technology is great for that.

Qwicap Feature Summary

I've never succinctly described the features of the Qwicap web application API in one place. I'd assumed most of them would be obvious from reading the introductory documents, but that was probably a mistake on several levels. For one thing, without having been given a good reason to care, who's going to bother with the introductory documents? For another, my colleagues at UT Austin have been exposed to Qwicap for years, and I doubt that they're aware of a lot of these features. So here goes:

• Pure Java
• Open Source
• Designed for Developers, Unapologetically
• Discards the CGI Model
• Very Rich State, Totally Automatic
• Developer-Friendly APIs
• Security: Automatically Prevents Replay Attacks
• Security: Automatically Prevents Cross-Site Scripting Attacks
• Markup Integrity: Completely Standards Compliant
• Markup Integrity: Automatic HTML Encoding & Decoding
• Markup Integrity: Automatic Integrity Checking
• Automatic Input Verification, Error Messages and User Correction Requests
• Transparent File Uploads
• Transparent File Downloads
• High-Level XHTML Form Manipulation API
• Custom XML Engine - Fast and convenient, with thread-safe, fine-grained caching.
• Zero JavaScript
• No Metadata, No Configuration Files
• Simplified Web Application Deployment
• Not a Framework - It fits into your code, rather than vice versa.

For more details, start with the version of that list that includes explanatory text. If that doesn't frighten you off, there's plenty more reading material on the Qwicap site. If you have any questions, drop me a line.

And Now For Something Completely Different....

Those familiar with Christopher Alexander's work—most famously his book "A Pattern Language", which was frequently referenced by Richard P. Gabriel in his book "Patterns of Software"—may find interesting Alexander's response to criticism of his more recent work, "The Nature of Order, Book One". (Thanks to Carfree Times for the pointer.)

Oh, and a bit of computation and UT Austin history.... While going through the Austin History Center's archives a few weeks back, I came across the 1958 memo announcing the appointment of the first director of UT's Computation Center. (My first job with The University was in the Comp. Center, and I've never left.) One thing I found interesting about the memo is the reference to our big iron of the day, the IBM 650. (Thankfully, unlike most computer companies, IBM has not forgotten its old models.) Among other things, that beast could do 138,000 logical operations per minute. And you have to love that magnetic drum - echoes of Atanasoff & Berry. Also interesting is the fact that, 47 years later, Dr. Young is still at The University, in our Computer Science department.

Qwicap 1.4a0 Released

A holiday week's worth of coding has advanced Qwicap 1.4 from a scrap of dusty paper with a feature list scrawled across it, to an alpha-test release. As befits an alpha release, it is not feature complete, but it is stable and useful, and lays the groundwork for the rest of what I want to accomplish with 1.4.

The noteworthy new features in Qwicap 1.4a0 are:

• Support for dynamically generated downloads - Generate your downloadable material any way you like, convert it to a byte array, and invoke the new Qwicap.addDownloadable method with a name, and an optional MIME type. It returns a unique URI for the downloadable, which you embed in your page markup wherever you wish. Downloadables automatically become eligible for garbage collection when the page they were first used in ceases to be "in play". (In other words, when the method that manages that page ceases to be in the call chain.)
• Input is now accepted from links ("a href" and "area href"), so you don't have to use forms exclusively, anymore. Combine this feature with the support for dynamically generated downloads, and things like dynamic graphics with corresponding client-side image maps become simple to implement. (The revised version of the Hunt The Wumpus example application demonstrates just such a map.)
• Simplified deployment - Most applications can now use a generic "web.xml" file, as seen in the revised example apps. Qwicap will automatically discover the entry-point of your application code at runtime, and will also handle serving-out any web pages, style sheets, etc. that would otherwise be inaccessible due to the default servlet url path being "/". This means one less thing to learn/remember, and also keeps the URLs for your web applications pleasantly short.

Major features remaining to be implemented in version 1.4 include internationalization, and asynchronous prompts. The former represents unfamiliar ground for me, but it seems clear that there should be some way for developers to alter the default input-validation error messages that Qwicap automatically adds to web pages. Given Qwicap's facility with XML files, I'm inclined to have it retrieve those messages from a language-specific XML file. What I haven't wrapped my head around yet is how to cope with the various permutations of those messages, like pluralization. Perhaps a language-specific class is in order - a method could be invoked on it that is specific to particular problem (bad integer input, for instance), and the code could supply whatever markup it wished in response (it could be loaded from a language-specific XML file, or produced by any other means convenient to the international developer).

I have a much better handle on the idea of asynchronous prompts (my favorite would-be feature in 1.4), but the implementation details to be considered are numerous, and, in some cases, rather non-trivial. The idea, however, is simple enough: instead of blocking until user input arrives, as the current Qwicap.prompt methods do, an asynchronous prompt would be non-blocking, and would therefore allow the web application to continue processing after sending a page to the user. The page would presumably be set to refresh periodically, and would thereby receive status updates until the server had concluded its processing, at wich point the client and server become synchronized once again. Equivalent (or fancier) functionality can be created in many web application schemes already, but Qwicap's asynchronous prompts should be vastly simpler for developers to comprehend, use, and support, while also being more predictable/dependable in as much as the code remains strictly server-side.

Barring another week of holiday materializing out of thin air, I don't know when I'll have a chance to make headway on these outstanding features. (Things are busy at the office, and our current and pending web applications don't have a compelling need for either of those features.) Nonetheless, I would value insights on internationalization in the mean time. And anyone who can put a sexy interface on the new Hunt The Wumpus example would be appreciated. (At present its interface is pleasantly minimal and clean, but doesn't exactly manage to catch one's eye.)

The "Programmers Hate Programmers" Hypothesis

I'd like to offer my fellow computer programmers the following hypothesis, which I've been kicking around for a few years: Programmers hate programmers. To explain: This hypothesis contends that the best explanation for the torments that programmers all-too-often lavish on one another with the software they create (especially libraries and APIs) is that programmers hate other programmers and therefore they design their creations to screw with each other. This hypothesis is obviously wrong, because, among other things, we could certainly screw with each other far more effectively than we do, if we were really trying. Nonetheless, at first glance, it seems to explain a lot of our experiences, and that's what I think is useful about it.

So, by way of example, this hypothesis seems to explain why...

• ...someone's code presents you with a "file not found" error, but doesn't tell you what file wasn't found, or where it was looking for the file.
• ...someone's code carefully checks a host of conditions to determine whether it can proceed with your request, and, finding some condition not met, fails, but won't let you in on the secret of why it failed. (In extreme cases, it won't even tell you that it failed.) It knew why, of course, because it went to the trouble of meticulously checking; it just didn't believe that you had need-to-know.
• ...someone's code does its error checking/reporting in such a way that the errors it reports have nothing to do with the actual problem and therefore—far from assisting you with debugging—they actually send you down the wrong path entirely.
• ...someone's API is so abstract that even after you learn it, you still don't know what it does, or, alternately, having learned what it does, you can't seem to make it do it.
• ...someone's system saves you from having to waste ten or twenty minutes writing code by forcing you to spend a week figuring-out how to write their config files, metadata, etc. (Hmm... what's the one thing my fellow programmers do best?.... Well, we can't have them doing that.)
• ...someone's API makes routine, simple operations lengthy and tedious, or makes implementing common, seemingly trivial, features, into major coding efforts, even though they had to know that everyone who used their API would be doing those sorts of things regularly. (Perhaps programmers who grow up with abusive APIs are destined to abuse their own APIs.)
• ...someone attempts to solve problems stemming from complexity by adding new layers of complexity.
• ...some APIs look as if they were designed exclusively to satisfy a feature check-list, with no concern for how the elements of the API would work together in practice.

There are probably a lot of other good examples that would resonate with much of the programming community, but I seem to have blocked from my memory the others that I've encountered. I leave extending the list as an exercise for the reader. The important thing about this hypothesis is that when you, the developer, find yourself in a situation where it seems to explain what you are up against, or it might describe another developer's experience with your code, something is wrong. Think of it as a diagnostic tool.

A few of those problems originate with judgement calls with which I've come to disagree. I think most, intentionally or otherwise, are examples of externalizing development costs. In other words, to save themselves a day or week of hard work, developers will cut corners in, or under-build, their code, and, having done so, they'll expose hundreds or thousands of other developers to hours of extra work. So, the original developers save a day or a week, but they cost the community months or years... the larger the community, the greater the damage.

Writing Non-Conformant Code

In this context, the best thing that can be said of an API, library, or other body of code, is that it does not conform to the Programmers Hate Programmers hypothesis, and I have some ideas about how to achieve such non-conformance.

I spent about 15 years writing Macintosh software, and I remember that Apple's engineers used to stress that the most important part of a Mac application was its user interface (GUI), since it's the GUI that defines an application to its user. In order to produce the best possible GUIs, they advised developers to design and, if possible, user-test and refine, their ideal GUI before writing the first real line of code, and then engineer the code around realizing that interface design, because a great many implementation details would be dictated by the requirements of the interface. In other words, the core code must be designed around the demands of the GUI, rather than the GUI being designed around it. Doing otherwise tended to create GUIs that interacted poorly with the user, and were clumsy and/or difficult to use.

While it might have been more honored in the breach than the observance, I believe they had the right idea. Over the years, I have also come to believe that the interfaces provided by GUIs, vital as they are, are not nearly as important to the world as the interfaces provided by APIs, because the effects of bad APIs propagate up into all of the code built on them, including GUIs. In extreme cases, bad APIs will even prevent code from being developed at all by making it appear effectively impossible, or so costly that the coding projects end-up abandoned. In other cases, potentially valuable projects won't be started at all due to past experiences poisoning the developer's cost/benefit guesstimate. (And then there's the matter of programmer burn-out....)

Once we conclude that the quality of APIs is profoundly important, the same advice that made sense for GUIs comes to make sense for APIs (and all aspects of the "developer experience" provided by a body of code): Design something as close as possible to an ideal API first, and then engineer the underlying code to make that API possible. If possible, observe how other programmers interact with the API, and refine it to be as intuitive, comprehensible, convenient, and simple as possible.

Bear in mind how your code is likely to be perceived. For instance, I suspect that many programmers (myself included) tend to conflate the number of lines of code they have to write in order to perform a particular task with the efficiency of the resulting implementation. If it takes a lot of lines, it's likely to be perceived as being less efficient than something that performs the same task in one or two lines. We all know that such a perception has nothing to do with runtime performance, but it can have a lot to do with how we perceive the effectiveness of an API, and our efficiency as programmers while using it. So, convenience is good, and is often worth a hit at runtime, if it's convenient enough. If reversion to a lower-level, but more efficient, API is possible in the optimization phase of development on an as-needed, piecemeal basis, so much the better.

Some APIs may not be able to represent every last feature that their underlying code could supply (or, as mentioned above, they may have to trade absolute performance for developer effectiveness). I believe that it's appropriate to offer both high- and low-level APIs. The high-level API should make, say, the most commonly-used 80% of the functionality as convenient to use, and as easy to learn, as possible. The low-level API should make all of the functionality available as best it can. Ideally, it should be possible to freely mix usage of the two APIs, so that the client code can enjoy the benefits of the high-level API, without foreclosing the option of later embracing, on a piecemeal basis, the more esoteric, or efficient, features exclusive to the low-level API. Even the low-level APIs should be designed to provide the best possible developer experience, but the high-level API should be designed first; its needs are likely to impose some requirements on the design of the low-level API.

(High-level APIs built over pre-existing low-level APIs can still be valuable, but they are in danger of having to adopt severely compromised internal or external designs, because they can't completely work-around the fixed low-level structure. Put another way, the kind of building you can build is severely restricted by the foundation it has to sit on. In order to have flexibility in the building's design, the foundation must be designed around the building's needs. Otherwise, you may be forced to build a shed where you needed a home.)

Criticisms and suggestions are easily produced, of course. If they aren't backed-up by code, they're too easy. For a couple of years, beneficially interleaved with other projects, I've worked on a Java web application development API designed on the aforementioned principles. It's called Qwicap. It's open-source. You can begin using it after reading just one or two (longish) pages of introductory documentation. Most of its API is confined to just two classes. It's a clean-slate design intended to eliminate most of what I perceive as the common tedium of web app. development. It works to fit into the natural flow of your code, rather than forcing you to fit your code into it. We use it in production at The University of Texas at Austin. And, yes, I'm trying to draw attention to it. With more than a hundred thousand projects on Sourceforge, alone, and countless web application development schemes available, it's hard to get noticed. It's even hard to know whether anyone has already done something like it. Much as I detest self-promotion, it seems I'm going to have to try my hand at it. Wish me luck.

That Remindes Me...

If you haven't already seen it, Richard P. Gabriel's book Patterns of Software (free PDF, 1.2 MB) is well worth a read. The problems of abstraction, and the idea of code habitability, are two good topics from the book that spring to mind. I first became aware of him through an article on Sun's site, which is a quick read and also worthwhile. Finally, if you happen to be an old Mac programmer, the book "Revolution in the Valley", by Andy Hertzfeld, is a treat. The web site of Mac folklore from which it was derived includes more material, but the book is a thing of beauty.