Qwicap Turns Five

Qwicap—the Java web development "framework" that advances the (apparently) unpopular idea that the structure of a web application should be no more dictated, or even affected, by sending a web page to a user and processing the resulting input than a conventional application is affected by reading from, or writing to, a file*—turned five years old today.

Work on it began June 4th, 2003 while I was a member of U.T. Austin's Networking group, and the first plausibly complete version (1.0) was ready four months later, on September 30, 2003. It was open-sourced a little less than three years later, in August of 2005, with the release of version 1.3. The current version, as of last Friday, is 1.4b25. (Work on 1.4 has been ongoing since the beginning of 2006, which makes it a fine example of setting too many goals for one release of a product. I anticipate declaring it finished as soon as there's time to give its multi-lingual features a good workout.)

From the start, Qwicap's development was driven by the goals of simplicity, security and standards-compliance. I believe it has proved that those are not opposing concerns, as it delivers first-rate intrinsic security, by far the simplest development experience of any Java web framework I've seen, and compliance with all relevant IETF and W3C standards. The design decisions that followed from those goals, especially the simplicity, mean that Qwicap will never be suitable for developing, say, the web interface for Bank of America, but they make it a very good choice for developing the hundreds of much smaller web applications that most large organizations use internally, or to serve other relatively small user communities.

---

* Blocking is Good

It'd be an enormous pain in the ass if conventional applications had to exit after every read or write operation they performed, and then start over from scratch when they were later notified that the read or write had completed. (That description may sound somewhat similar to asynchronous I/O, but even that doesn't require the application to exit between I/O operations.) For some reason, however, it is commonly accepted that web applications should have to operate in exactly that manner, if you understand the writes to be equivalent to sending web pages to users, and reads to be equivalent to acquiring the input produced by those web pages.

Why have web application developers been willing to have such an absurd burden dumped in their laps? The fact that many of us started by writing CGIs may have something to do with it, but that was a long time ago – there's been ample time to reconsider the CGI structure since then. Scalability is probably one reason, but what percentage of the web applications developed in the world really need to scale to thousands of concurrent users? My belief is that it's a small percentage, and that the tools suited to high scalability aren't ideal for smaller apps. HTTP's statelessness may be another reason for believing that it's appropriate to process each hit in isolation, but how many web applications are, or can be, truly stateless? Reality is inherently state-rich and I don't believe that most web applications can really get away from that, nor do I believe that it's generally worth trying to do so.

Qwicap was, and is, my response to that absurd burden. Its most basic principle is that sending a web page to a user and obtaining the resulting input should be a blocking operation – the application should simply halt after sending a web page, and it should resume exactly where it left off when the user's input from that page arrives. Not so much as a local variable should be disturbed in the process. It shouldn't be any different from the blocking that automatically occurs when a program reads from, or writes to, a file or stream.

Of course, for some purposes, blocking I/O isn't the most efficient approach to I/O, but it is the hands down winner for most applications. I imagine that most developers have never done anything other than blocking I/O. By the same token, I believe that most web application developers should never have to do anything other than blocking user interaction. Qwicap makes that possible. Therefore, a Qwicap web application is not only a set of plain-old-Java-objects (POJO), but is also a plain-old-application that begins execution at its "main" method, interacts with the user when it needs to, and exits when it has accomplished its purpose. It'd be hard to make things any simpler than that.

Nonetheless, gentle reader, you'll have noticed that Qwicap has not taken the industry by storm. I don't think the industry is even aware of it. Well, I've never been any damn good at self-promotion (I really hate the idea, in fact, and I don't even like writing this post), so Qwicap's obscurity is, perhaps, unsurprising. But, if any of the above struck a chord with you, take a look at Qwicap, and let me know how it goes. (What works? What doesn't? What do you find yourself doing repeatedly or clumsily that Qwicap might simplify or even handle for you?) Over the years, I've incorporated a lot of feedback from local developers into Qwicap's design and/or API, and I remain interested in incorporating more developer feedback, where possible.

Qwicap 1.4b12 Released

Qwicap 1.4b12 has been released. Thanks go out to Sunil for noticing the problem with extracting numeric values from XML when using the Qwicap XML engine stand-alone. That bug has been fixed. Also, the Qwicap.getStatusReport method works again. Full details in the change history. Source code, binaries and examples are available on SourceForge, as usual.

Qwicap 1.4b11 Released

Version 1.4b11 of the Qwicap Java web application framework (or anti-framework, IMO) has been released. This beta kind of got away from me, frankly. It really should have been broken-up into a succession of betas, and possibly version 1.5 betas, at that. But I was running low on things to do before declaring version 1.4 done, and some untouched items on the original 1.4 "to do" list were staring at me, and I started to think that it'd just be a downright shame to go to release without a few more of those features. One thing led to another, of course, and here we are.

My project management skills aside, good things have happened to Qwicap. Notably, this version embraces Java 1.5, provides Qwicap's first support for internationalization/localization/multi-lingualism, and allows Qwicap applications to run both directly from their WAR files, and within the default configuration of the Security Manager in Tomcat. It also wraps-up the effort to make Qwicap fully character-set aware. (It's so character-set aware at this point, that you can make web applications that operate entirely in the EBCDIC code page of your choice, or in any mix of the many other character sets known to Java. The limiting factor is the range of character sets supported by the various browsers, and the bugs therein.) See the change history for full details.

By the way, in connection with Qwicap's new internationalization features, I'm looking for people with good non-English language skills who'd be willing to translate Qwicap's internal error messages (there are something like 22 of them), and its "number guess" example application (one web page, and a few other strings). I expect that Qwicap's internationalization features will evolve with use, so translations will help not only with the actual translation problem, but will probably aid in the refinement and growth of Qwicap's internationalization API. If you're interested, there's contact information on the Qwicap main page, or you can leave me a comment here.

Qwicap 1.4b9 Released

Qwicap version 1.4b9 was released on May 31. It includes the minor enhancements of the unannounced beta 7 and 8 releases (logging in Qwicap.reportException, and more informative exception messages from the MutableMarkup class, respectively). The work on beta 9 is of a whole 'nother order, hence this announcement.

Last Friday, after lunch, I was looking forward to a quiet afternoon, and was reading my usual collection of computing-related web sites when I arrived at Elliot Rusty Harold's blog, Cafe au Lait. As usual there was a lot of material there, and, also as usual, I wasn't even going to try to deal with it all; I was just there for the quote of the day, which turned out to be about character set support in various web application development systems. And it included a harmless looking little link attached to the word "Unicode" (just like that one, in fact). I clicked the innocent word and that was the end of my peaceful Friday afternoon. The link was to Joel Spolsky's 2003 article The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!). If you haven't read it, consider it recommended. If you're in a hurry, just read the bit at the end about the problems faced by web browsers when they need to interpret the bytes of a web page. That gives you the flavor of many of the problems posed by character set handling. And they're fairly obvious problems, if you happen to think about them. Unfortunately, I hadn't.

I wasn't exactly unaware of character set issues (I sorted out EBCDIC to ASCII translation issues back in the days of the IBM 3033 and the original IBM PC - issues that had IBM's support engineers embarassed and stumped), and I've been a big fan of Unicode, but while reading that I article I quickly realized that I'd failed to grasp just how significant character set issues are, or can be, to everyday software development. Specifically, in the case of Qwicap, I realized that I'd ignored character set issues when I wrote Qwicap's XML engine. I knew that, having emerged from Reader objects, the characters in memory were Unicode (sort of) and I'd just left my concern at that, without thinking about the encoding of the source materials that went into those Reader objects. I'd also neglected an issue associated with transmitting web pages to clients: They were always sent as UTF-8, regardless of what the document markup called-for. (In a way, that's OK because a character set specification in the HTTP "Content-Type" header, which Qwicap was in the habit of supplying, takes precedence over any specification in the markup, but it's both rude and confusing to ignore the document author's wishes as expressed by their markup.)

So, seeing all of those basic oversights at once was quite a kick in the teeth (self-inflicted, no less). And, of course, one of the great things about open source development is that you have the opportunity to make big, embarassing mistakes in public. On the bright side, if I knew how difficult it was going to be to develop Qwicap when I started, I might have come to my senses and found something less painful to occupy my time, like hitting myself in the head with blunt objects. So, since I still think Qwicap is a good idea, and I yet cling to the hope that it'll find a niche for itself beyond the bounds of my worthy employer, The University of Texas at Austin, (and for all I know, perhaps it already has) maybe it's just as well that I didn't fully appreciate what I was getting myself into when I first had this idea.

Anyway, with the release of version 1.4b9, Qwicap is now character-set aware, and I can take some comfort from the facts that (1) nobody but me ever noticed this problem in Qwicap, and (2) there are vastly more popular web application development schemes that still neglect character set issues. In the latter case, though they were absurdly late in coming, the fixes in 1.4b9 do become one more feather in Qwicap's hat. Of course, I still feel plenty stupid.

Qwicap 1.4b6 Released

Version 1.4b6 of Qwicap is a trivial release, correcting a serious bug in a method, in the non-public API, which Qwicap doesn't even use. (Qwicap does use other methods in the same utility class, however, which is the only reason that this issue is relevant.) I've also tweaked the markup of the pages in the example applications such that Eclipse won't choke when trying to load them anymore. So, this release is a big yawn, but that's good news late in the beta process.

Qwicap 1.4b5 Released

Qwicap version 1.4b5 eliminates a case in QwicapServlet in which HttpServletResponse.sendError could throw an IllegalArgumentException, because an Auth2Scheme implementation had refused access to the web application, and had already caused the response to be committed (for instance, by redirecting the user to an authentication page). QwicapServlet now invokes sendError only when the response has not been committed. And that's as exciting as this release gets, which is good news, because it means the beta cycle is winding-down toward a true release.

For those unfamiliar with Qwicap, allow me to throw in the standard blurb:

Qwicap is a Java web application development API that discards the traditional hit-run-exit CGI model, automatically provides rich state, and supplies a powerful, extremely easy-to-use XML "templating" engine to avoid mixing code and content. It builds on Servlet, XHTML and CSS technologies.

Qwicap 1.4b4 Released

Oh, stupid, stupid me. I screwed-up the SecureRandom service provider implementation in Qwicap 1.4b3 by doing nothing more than setting the visibility modifier on my subclass of SecureRandomSpi to "package", rather than public. (As always, I was trying to keep as much out of the public API as possible.) Having just written and used nearly identical code in another project, and because I couldn't readily create a JUnit test for anything that depended on the environment provided by an application server, I put the code in without testing it, thereby compounding my mistake with a "what could go wrong?" attitude. Of course, when we actually came to use the one project that utilized that service provider this morning, exceptions were thrown, and time wasted.

That mistake has been corrected in the just-released version 1.4b4. Also corrected is a mistake in the Javadocs that explain how to get a SecureRandom instance based on Qwicap's provider. Also, I found a way to create a JUnit test for the visibility-modifier induced bug, so I won't re-create it in the future.

Since the SecureRandom service provider was a feature new to 1.4b3, I'm the only one who has had time to create any code that depends on it, so this bug won't affect any existing Qwicap users. Still, it's embarassing.

Qwicap 1.4b3 Released

This is just a quick note to announce the release of Qwicap version 1.4b3. Download links, and the usual documentation and example applications, can be found on the Qwicap home page. This version of Qwicap fixes a bug introduced late in the 1.4b2 development cycle that can cause a NullPointerException to be thrown when rejecting inputs supplied by certain form controls. It also makes Qwicap's internal Rule 30 pseudo-random number generator (PRNG) implementation (see also page 974) available through the "Provider" mechanism of the java.security.Security class. Thus, it is now possible to create SecureRandom instances which utilize Qwicap's PRNG implementation, and which are seeded from Qwicap's persistent, internal instance of that PRNG.

By the way, I'd hoped, by now, to be offering a write-up of the Qwicap 1.4 development process, discussing the feedback loop that was established with the developers who were using it at the time, and the additional adjustments that were made to it following a review of their completed code, but I write slowly, and it's still nowhere near complete.

Qwicap 1.4b1 Released

Qwicap version 1.4b1 includes a more flexible, rearchitected authentication & authorization system, includes cleaner core code wherever the code became involved with Servlet life-cycles, futher simplifies web application deployment descriptors, adds three convenience methods to the Results class, and includes numerous improvements to the Javadocs.

The new authentication & authorization system is not backward compatible with the previous one, but neither is it fundamentally different, so adapting an existing Auth2Scheme implementation to it is a minor matter. Primarily, the architecture has benefitted from the adoption of the factory pattern, while new features permit access to the static content in a web application (web pages, style sheets, etc.) to be regulated, and regulated independently of access to the web application, itself.

As usual, the documentation, download links, etc. can be found on the Qwicap main page.

Change Log for Version 1.4b1

• Filled-in some old gaps in the Javadoc, and made numerous improvements to the Javadoc markup throughout the classes of the public API.
• Added to the authentication & authorization (Auth²) system the ability to regulate access to the static content in a web application, independently of access to the application itself.
• Rearchitected the authentication & authorization (Auth²) code. The new implementation uses an Auth2SchemeFactory associated with each servlet to produce an Auth2Scheme instance (optionally associated with each session) which is used to authenticate & authorize each hit in a session. It returns to Qwicap an Auth2SchemeReply instance describing its conclusions, upon which Qwicap acts, restricting access to the web application and/or its static content accordingly. The client application may retrieve the current reply object by using the new Qwicap.getAuth2SchemeReply method. (The old Qwicap.getAuth2Scheme method has been discontinued.)
• Modified a number of core classes to reflect a better understanding of servlet life cycles. This cleaned-up the code in a number of places, and outright eliminated the need for the internal QwicapContext class.
• Simplified the web application deployment descriptor (the "web.xml" file) by eliminating the need to specify edu.utexas.its.eis.tools.qwicap.servlet.QwicapContext as a "listener". Revised the Deployment Descriptor documentation accordingly. (Web applications using previous versions of Qwicap 1.4 will need to remove that "listener" specification from their "web.xml" files.)
• Added several new methods to the Results class: getAttribute, everyNth, and IF_NOT_EMPTY.

Qwicap 1.4 Goes Beta with Version 1.4b0

Qwicap 1.4b0 corrects a bug that may have interfered with Tomcat shutdown, and a bug that caused newline characters within HTML "textarea" tags to be encoded. In addition, it simplifies web application setup by eliminating the requirement to include the "qwicap.css" style sheet in web applications, and, for convenience, it provides access to Qwicap's internal pseudo-random number generator.

Links to all of the documentation, source, etc. can be found on the Qwicap main page. I have also, at long last, placed the source code into a Subversion repository on SourceForge. I have doubts about whether I've structured the repository in a way that I'll be satisfied with in the long run, but there it is. (Even in the short run, I didn't intend to include the compiled classes in the repository, but there they are, despite my best efforts to delete them.)

Not documented anywhere (except here, obviously) is the fact that version 1.4b0 contains the germ of a new feature that adds the first elements of AJAX technology to Qwicap. (In the interest of reliability, I have no intention of ever making Qwicap depend on AJAX, or scripting in general, but I am open to using those technologies to provide non-critical, optional enhancements.) The immediate goal of this work is a feature that will cause pages in client-side caches to visually reflect their changing status as valid, or invalid, targets for user interaction when the user goes back to them. It is implemented using Javascripts that are automatically embedded in each web page. At present, those scripts are fully functional only in Firefox version 2. My best efforts to make them work in Safari 2.0.4 have failed utterly, and I currently hold no hope of being able to improve on that. However, they come close to working in the current WebKit builds; the remaining problem seems to be that the changes my scripts make to a document's DOM are not rendered. Plainly, that's not good enough for a "real" feature, so it is disabled by default. However, the brave may enable it using the Qwicap.setScriptsEnabled method. You'll want to enable it before your application sends its first web page to the client in order to get consistent behavior for all pages.

By the way, those wishing to add their own application-specific AJAX functionality into their Qwicap applications can do so by creating a FormDataSetFilter to intercept and reply to the asynchronous inputs from their scripts. Just remember to commit your response by closing the output stream before returning from the filter. Such filters are executed within the full context of your Qwicap application, so they have access to all of your accumulated state (with the obvious exception of the local variables in your blocked methods, which can't be in your filter's scope). May you have better luck with your scripts than I've had with mine thus far.

Change Log for Version 1.4b0

• Corrected the handling of newline ('\n') characters within the content of HTML "textarea" tags. Previous versions of 1.4 had replaced those newlines with "br" tags, which was incorrect. Newlines in "textarea" tags are now left as they are; they are not encoded. In this respect, "pre" and "textarea" tags are now handled identically.
• Added a few more JUnit tests, and improved a few others.
• Added the Qwicap.getRandom method to make available to client applications Qwicap's internal pseduo-random number generator instance.
• Updated the "guess" and "wumpus" example applications to use the Qwicap.getRandom method. Also, removed the "wumpus" application's dependence on the "servlet.jar" library file, and duly removed that file from the distribution.
• Corrected a bug that may have caused Qwicap to interfere with normal Tomcat shutdown.
• It is no longer necessary to include the "qwicap.css" style sheet in Qwicap applications. Qwicap will now supply that document automatically. However, it remains necessary for an application's style sheet(s) to import the "qwicap.css" style sheet, unless it overrides all of Qwicap's styles. Applications may still override the default Qwicap styles in their own style sheets, of course, and may outright replace the default "qwicap.css" style sheet, if they prefer, by providing a document of the same name in the application's top-level directory.
• Various Javadoc improvements, chiefly in classes that are not part of Qwicap's public API.
• Corrected an error in a Javadoc block and in a separate comment in the QwicapServlet class. The word "send" in both cases had been transformed into "sendFile" due to some overzealous refactoring in the distant past. These corrections have no functional significance, but do make a little bit of the documentation more comprehensible.

Qwicap 1.4a42 Released

Qwicap 1.4a42 has been released. (Download it from SourceForge.) This version has been in the works for almost six months, off and on. It was bedeviled by two problems: (1) a thread synchronization bug that could manifest when a user submitted new form data before the processing of their previous hit was complete, and (2) a misunderstanding between myself, a colleague who administers some of our servers, some obscure error reports from the JVM, and some Google searches, that led us to conclude (incorrectly) that Qwicap absolutely had to have a thread pool in order to operate reliably over long periods of time.

With regard to the thread synchronization bug: Ouch. I could not reproduce it when running the server and the web browser on the same box (my normal method of working), even when the box was a multi-processor machine. If a colleague, Kevin Wood, hadn't noticed this bug and then invested serious effort in exploring it, I don't know when it would have been found. Just to keep it interesting, the fixes to Qwicap 1.4a37 that he provided to me didn't work in his test setup when I supplied him with builds from the official codebase that incorporated all of his changes. (Those builds worked fine on my machine, naturally.) The builds he produced from the older codebase worked fine in his test setup, and mine. Even after we both sat down and satisfied ourselves that I hadn't mangled, or omitted, any of his modifications, the problem persisted in my builds, when he ran the tests on his box. In the end, he set up a box for me that was configured essentially the same as his machine, and I did my own testing. Eventually, I identified one last race condition that even he'd missed. (I throw no stones here; if I hadn't missed it in the first place, he couldn't have missed it in the second.)

With regard to the perceived need for a thread pool, I put a lot of work into building, instrumenting and testing a thread pool based on the mistaken belief that our production servers were encountering a memory-related resource exhaustion problem related to the total number of threads created over the lifetime of any given Qwicap application. In the end, it turned-out that the problem was confined to the amount of memory consumed by the stacks of the threads running at any given time, when running under Java 1.5, which dramatically increased the default stack size relative to Java 1.4 (it seems to have gone from 256K to a megabyte or more). Initially, configuring our servers to run Tomcat in a 64-bit address space solved the problem, and later configuring them to use only 256K as the default stack size provided an alternate solution.

So, Qwicap doesn't turn-out, strictly speaking, to need a thread pool to operate reliably, but I implemented one before that became clear to us, and therefore it now has a thread pool. There's no point in ripping it out, because (a) some people will actually want it, (b) a lot of nice status data gathering machinery became tied to it, (c) it can be configured such that it doesn't hang onto any threads, if you wish, and therefore won't get in the way of people who don't want pooling, and (d) the configuration parameters for the pool allow the size of the Qwicap thread stacks to be controlled, which goes to the heart of the real problem.

Other noteworthy changes in Qwicap 1.4a42 include:

• The concept of "blocking listeners" has been added. Such listeners, registered using the new Qwicap.addBlockingListener method, are invoked just before and after Qwicap blocks to wait for user activity. This applies to the prompt methods, and the redirect method, and will apply to any future blocking methods. Thus a "blocking listener" can perform actions like releasing and reacquiring limited resources like database connections, or log application activity, transparently from the perspective of a web application's code.
• There is now a "service data recorder" (SDR), for lack of a better term, built into Qwicap. The SDR accumulates data in hourly chunks (up to 72 of them). Each hour's data includes information on the thread pool (total size, number of active and inactive threads, number of threads created and allowed to die, number of threads unavailable due to the max. pool size limit), user sessions (number completed and expired, and the mininum, average and maximum duration), hits (total number, hits directed to invalid pages, and the minimum, average and maximum response time), and RAM usage (amounts free, used, and total). For the time being, the SDR reports are available only in a human-readable format, because I am hesitant to commit to what data will be gathered, how it will be represented, etc.

At this point, my main goal for Qwicap is to finalize the 1.4 release. I'm prepared to declare it feature-complete now. (Therefore, I should have released it as a "beta", rather than an "alpha".) While it doesn't include all of the features I'd originally intended it to have, it's the most significant revision to Qwicap thus far, and at some point you just have to draw a line across such an effort, declare it complete, and add the missing features to the list for the next major revision. Furthermore, the current release version, 1.3.3, is almost a year old at this point, and lacks a number of bug fixes and important improvements that are present in 1.4. (Backporting the fixes and improvements to common code became an unsupportable burden some time ago, with the never-quite-ready-for-release version 1.3.4.) The thought of people putting-up with 1.3-isms when they should be benefitting from 1.4-isms is really getting on my nerves. So, expect the 1.4 release in the not-too-distant future, bug reports permitting.

Qwicap Turns Three

Qwicap, my pet project for simplifying the development of Java web applications by allowing them to be structured much like conventional applications (for "conventional" think of interactive command-line apps), turned three years old today.

Actually, it's more like 3.4 years, total, but three years since version 1.0 was released. The releases were internal to U.T. Austin until my boss, his boss, and her boss, all agreed to open-sourcing the product. Then there was the matter of figuring-out whether it could be done at all, given that nobody in my department (the IT dept., no less) had ever done it, or knew of policies that would permit it. Ultimately, I found the lawyer at UT System who dealt with these matters, and discovered that, contrary to all expectations, we actually did have a policy that allowed—and even encouraged—open-sourcing. After reading the policy document countless times, and having many conversations with the ever-helpful lawyer, I even managed to figure-out how to apply the policy to my situation. With that done, there was a lot of paper-work to be filled out, months of waiting for a critical signature while I frantically polished the code and wrote documentation (while also doing all of the things they normally pay me to do), the filing of the paper-work, and, then, figuring-out how to use SourceForge to publish the code.

Finally, Qwicap was posted to SourceForge, and I began trying to figure-out how to make people aware of it. After a few anxious days I was overjoyed to see the news item announcing its release on the SourceForge main page. Wow. All those years of work would not live, die and vanish within the confines of The University. It was actually out there.

That news item lasted three days, and the response to it was deafening, in the same sense that silence can be deafening. Ouch. So, there followed efforts to get Qwicap listed on more topic-specific sites. At the recommendation of colleagues, I started-up this blog, much as it goes against my grain (twenty years ago, it would have been a different matter, but that was then). At the recommendation of my boss, I submitted a presentation proposal to the regional Educause conference. And, of course, I kept working on Qwicap.

The work on Qwicap has gone well. There were a number of 1.3 revisions, then work began on 1.4, the biggest update yet. Several test versions have been released, several of my colleagues have contributed improvements to the project (Kevin, Jay, thanks), and work on it is nearing completion. And we've deployed three more major campus web applications that are built on Qwicap, with very positive feedback from the development team that built them. Most of the team members have baselined Qwicap for their upcoming projects. Since Qwicap's goal was to make life better for developers, that has all been music to my ears.

In stark contrast, the work on publicizing Qwicap has failed miserably. The topic-specific sites ignored it. The blog, which lead-out with my "Programmers Hate Programmers" hypothesis, generated some attention, but if anyone actually did any work with Qwicap as a result, I've never heard about it. The regional Educause deadline came and went without a word, but eventually I was informed that they might be inclined to use the presentation as a backup, in case one of the presentations they actually wanted fell-through. Shortly before the deadline, I swallowed hard and agreed to be a backup, but that never happened. (On the plus side, I never had to write that presentation.) When the national conference came along later, I never quite managed to submit the presentation proposal again. 'Should have, of course, meant to, but still felt the bruises. (On the plus side, I still haven't had to write that presentation, let alone present it.)

I keep an eye on the Qwicap web site, and find that it continues to generate some attention, but most of it relates to the example source code for Java number-guess applications (though not the Qwicap version, naturally). I presume that some people find the examples useful as the interactive equivalent of a "Hello, World" program, but I'm also inclined to suspect that it's the answer to some professor's CS 101 assignment. It's not quite the contribution I'd hoped to make with Qwicap.

Well, Hell.

Last time I looked, there were about 100,000 open-source projects on SourceForge, alone. It seems pretty clear that most of them have utterly failed to make an impression in their niches of the industry, so, statistically speaking, it's no big surprise to be in the same situation. On the other hand, more than a few of those projects seem to have skipped the unpleasant "writing the code" phase of project development, so I kind of thought Qwicap had an edge in that respect. Shows you what I know.

Ironically, when I first proposed open-sourcing Qwicap, I'd been concerned that the product might generate enough interest that I'd have a hard time keeping up with it, so I'd warned my bosses that it might take time away from my usual duties, but they were supportive, nonetheless. Thank you, bosses. [Now there's something you don't hear every day.]

That concern wasn't just wishful thinking. In the late '80s and early '90s I built and maintained the "Gatekeeper" anti-virus system for Macintosh. It was freeware, and developed on my own time. It broke new ground in its field, and achieved a significant level of success; postcards from users arrived from all over the world, those being the days before most people had even heard of email, let alone the Internet. The people who could use email had nice things to say, too, but satisfied users are the quiet ones. The ones you hear from the most are the folks who aren't altogether satisfied; they want features added, or bugs fixed. I encouraged the feedback, and especially the bug reports, of course. Can't fix 'em if you don't know about 'em, and every bug is embarrassing. But, if you haven't had this experience, imagine looking at your email every day for years and finding what comes to seem like nothing but problem reports. Many come with kind words about the value of the product, but the praise is ephemeral, while the worries and the bug hunting and the other work grinds on and on, in every spare moment you can find.

And it wasn't even the case that the Gatekeeper system had a lot of actual bugs. But it seemed like there were, because Gatekeeper had to be implemented as a set of complex, runtime patches to the Mac operating system, and my patches had to coexist with a startling variety of other people's patches, and with all of the undocumented idiosyncrasies of an operating system that was "quirky" on a good day. In practice, the problem-space was limitless and the problems often unreproducible, and uncorrectable. Of course, each bug would be reported many times, and with every version of Gatekeeper (and every OS patch, and every OS version) ever released out there running on somebody's computer, somewhere, they never really went away, even if they'd been fixed years before. And I wanted to represent my handiwork well, so, naturally, I tried to respond to everybody.

After enough years of dealing with all that, day in, day out, I couldn't bear to read the email anymore, or open the letters, or read the postcards. It seemed like there was never any good news, but an endless supply of bad news, all of it directed to me personally, and awaiting my reply. It was relentless, and endlessly depressing. All those years of work, and that was what I had to show for it. The many successes—the product did what I set out to do, and more—couldn't compete in my awareness with the cumulative pounding of the bad news. I had to walk away. Some people were annoyed when their mail went unanswered. I wasn't happy about it, either. Some told me they felt betrayed, which hurt, but walking away had become a matter of survival. (And the product was free, after all. What did they want? Blood?)

The burnout was intense. There came a day when I was contacted by a development house that wanted to turn Gatekeeper into a commercial product. On the one hand, I didn't want to deal with it ever again. On the other hand, it felt like it would be a mistake to have gone through so much and not at least look into this possibility, so I met with a local, commercially successful Mac developer in order to talk about what sort of money I could reasonably expect out of commercializing a product like Gatekeeper. He told me that over something on the order of three years, a product of its kind should be worth about a million dollars. It was an impressive number (still is), but even so I couldn't bring myself to go back to that work.

It felt like about ten years before I could find the will to take on another major programming project, another big idea. So, naturally, I worried about whether I'd go through the same thing again trying to support Qwicap. Turns out that it hasn't been much of a problem. What luck.

One of the nastier traps one can step in, I suspect, is to discover the burn of a really great idea. The kind of idea that you can't not work on, no matter what anybody else thinks of it. At that point, the idea has you right where it wants you, and you have to go where it takes you, while paying mightily for the ride. Great things may come of it, they may make it all worthwhile in the end, but there's only room for so many great outcomes in the world, so the odds can't possibly be in your favor. But you go on anyway.

Maybe Qwicap will make a difference outside The University one day. So far, not so good. Oh, well. I'm not done with Qwicap, and it's not done with me.

'More than I'd intended to say, but I assume I'm talking to myself at this point. If not, good luck with your own great ideas. And have a look at Qwicap.

Qwicap 1.4a34

Qwicap 1.4a34 was released this weekend. With four busy months elapsed since the previous release, 1.4a12, it was time to declare something "done" again, because a lot fixes and improvements have accumulated over that time. The development of this version has coincided with the development of our second and third large, Qwicap-based web applications which will be supplying the end-user and administrator interfaces to UT Austin's new identity management system. The same four developer team is responsible for both applications. They're up against the challenge of schedule-constraints, the rapidly evolving, still-under-development code of the identity management system itself, and the fact that none of them had used Qwicap before (well, one had used its XML engine a few years back). So, I've been watching their progress with interest, wondering which aspects of Qwicap would work well for them, and which wouldn't.

So far, with both applications in an advanced state of development, and user testing of the end-user application already complete, the results are encouraging. The team seems uniformly pleased with the experience of working with Qwicap, and they're already planning to use it as the basis of their future projects. Though I visit their office daily to see how things are going, and to take a look at any problems they want a hand with, there have been remarkably few problems. They've found some minor bugs (minor in that there were small, simple fixes), a few more troublesome bugs, and they've needed a few new features. The more troublesome bugs were interesting, because they were the result of the team choosing a flow-control and markup handling strategy that had never occurred to me, and consequently it was a bit surprising to Qwicap, too. What stunned me was that it worked, and worked well in all but a few obscure cases. I had to sit down with the team-lead and walk through the relevant code several times before I even understood why it worked at all.

Qwicap 1.4a4 and 1.3.3 Released

Qwicap 1.4a4 and 1.3.3 have been released. The change history has all the details fit to print, but the main motivation behind the 1.4a4 release was to make available a fix for a race condition during session tear-down that's been in the Qwicap code base more-or-less forever. Originally, its effects were of little importance—mostly some exceptions showing-up in the logs—and it wasn't at all clear how the problems came about. With 1.4's support for dynamically generated downloadables, those effects started to have an actual impact by sometimes preventing access to the downloadables in the final page of the session. Fortunately, that made the nature of the problem more obvious and led to this bug fix.

Another old bug (or inadequately developed feature) was in the Qwicap.goodbye method which is the method that sends an application's final page to the user. The bug was that the method didn't inform Qwicap that the application was shutting-down, so Qwicap did nothing to help the application shutdown. Which was fine, for applications that invoked goodbye and gracefully returned their way back to, and out of, their main method, but those that didn't could inadvertently persist in the delusion that they were still operating normally by calling a variety of Qwicap methods that should have no role in, and could delay, the shutdown of an application (retrieving user input, sending pages to the user, etc.). Now that goodbye informs Qwicap that the application is shutting down, those methods that shouldn't be used during shutdown will throw a QwicapSessionDeathException when invoked, in order to help the application along in its shutdown activities. (By the way, QwicapSessionDeathException now includes a message that states why the session is dead, or dying.)

In addition, some enhancements have been made since 1.4a0, including: the addition of the Qwicap.getFloat method; moving the Downloadable class into the public API; changing all of the XML attribute manipulation methods to accept arbitrary objects as attribute values; and various improvements to the Javadocs. In addition, the Hunt The Wumpus example application has had its game logic further debugged, and it has been extended to support larger maps, upon request. The extra "config" page that allows larger maps to be specified, also provides a long-overdue example of Qwicap features like the high-level form manipulation API, while further illustrating how multi-page applications function.

Qwicap 1.3.3

Qwicap 1.3.3 was created by back-porting the bug fixes and minor enhancements that have accumulated in the 1.4 branch as of version 1.4a4. So, that race condition is gone, Qwicap.goodbye is beefed-up, Qwicap.getFloat exists, XML attribute values can be specified as arbitrary objects, QwicapSessionDeathExceptions are more informative, Qwicap.convertSubmitButtonsToInputs has had its documentation substantially expanded, various Context methods that didn't need to be public have been removed from the public API, the XML tag hierarchy validator includes document names in any exceptions it throws, all subclasses of Markup now include getMutable and getImmutable methods, the Context.getFile and Context.getExistingFile methods have been documented at long last, some useless code has been removed, and the Javadocs have suffered various improvements. Also, its version of Hunt The Wumpus received a bug fix, and therefore sucks less than it traditionally has done.

Which Reminds Me

While trying to implement the large map generator for the Qwicap 1.4a4 version of Hunt The Wumpus, I was frustrated for quite a while by my inability to come up with a simple implementation for the generator. The generator had a conceptually simple job to do, so it seemed the implementation should be simple, too. We all know such things frequently don't work out that way, but it was bugging me all the same. At one point, I considered using a cellular automaton to generate a hexagon-based map, and that sent me off to Wolfram's A New Kind of Science (NKS) pages, which was more convenient than winching down my copy of the book from its shelf and finding a spot for it on my desk. What I stumbled upon in doing so was WolframTones, a site that generates music using Wolfram's one-dimensional cellular automata. I found it very impressive, and a welcome demonstration of Wolfram's NKS ideas in an easily appreciated, real-world application - which is the sort of thing I regretted the absence of in the NKS book.

My only complaint with WolframTones is that it limits its compositions to thirty seconds. Ideally, it'd be possible to generate arbitrarily long compositions. What I wonder about long compositions is this: Will they seem to hold-together as well as the thirty-second ones usually do? Or does one's brain eventually reject a longer composition due to the extended lack of the sort of patterns designed into music by human intelligences? This would probably be more of an issue with some types of music than others; in particular, some flavors of jazz might not be a problem. And if it's not a problem for some types of music, what does that tell us about music, and will this eventually result in, say, a music generator box for use in stores that will continuously compose and play new music to save the stores the cost of playing background music that someone owns? As background music, shortcomings in the compositions would be masked by the lack of attention being paid to them. Which is fine for customers passing through, but what about the staff that has to hear the stuff all day? Will they eventually hunt down and destroy the box to save themselves? Or would it work well enough that it wouldn't drive people nuts, and, in fact, would periodically produce something really catchy? With every composition produced by an automaton chosen randomly from a set of around 4 billion possible automata, and with no direct access to the box to ask it to save its state, one would end-up in the frustrating situation of hearing a great song that no one had ever heard before, and knowing that you'd never, ever hear it again. That might promote some useful contemplation of the transitory nature of life, but I think, for the most part, it would just be a new way of pissing people off. Technology is great for that.

Qwicap Feature Summary

I've never succinctly described the features of the Qwicap web application API in one place. I'd assumed most of them would be obvious from reading the introductory documents, but that was probably a mistake on several levels. For one thing, without having been given a good reason to care, who's going to bother with the introductory documents? For another, my colleagues at UT Austin have been exposed to Qwicap for years, and I doubt that they're aware of a lot of these features. So here goes:

• Pure Java
• Open Source
• Designed for Developers, Unapologetically
• Discards the CGI Model
• Very Rich State, Totally Automatic
• Developer-Friendly APIs
• Security: Automatically Prevents Replay Attacks
• Security: Automatically Prevents Cross-Site Scripting Attacks
• Markup Integrity: Completely Standards Compliant
• Markup Integrity: Automatic HTML Encoding & Decoding
• Markup Integrity: Automatic Integrity Checking
• Automatic Input Verification, Error Messages and User Correction Requests
• Transparent File Uploads
• Transparent File Downloads
• High-Level XHTML Form Manipulation API
• Custom XML Engine - Fast and convenient, with thread-safe, fine-grained caching.
• Zero JavaScript
• No Metadata, No Configuration Files
• Simplified Web Application Deployment
• Not a Framework - It fits into your code, rather than vice versa.

For more details, start with the version of that list that includes explanatory text. If that doesn't frighten you off, there's plenty more reading material on the Qwicap site. If you have any questions, drop me a line.

And Now For Something Completely Different....

Those familiar with Christopher Alexander's work—most famously his book "A Pattern Language", which was frequently referenced by Richard P. Gabriel in his book "Patterns of Software"—may find interesting Alexander's response to criticism of his more recent work, "The Nature of Order, Book One". (Thanks to Carfree Times for the pointer.)

Oh, and a bit of computation and UT Austin history.... While going through the Austin History Center's archives a few weeks back, I came across the 1958 memo announcing the appointment of the first director of UT's Computation Center. (My first job with The University was in the Comp. Center, and I've never left.) One thing I found interesting about the memo is the reference to our big iron of the day, the IBM 650. (Thankfully, unlike most computer companies, IBM has not forgotten its old models.) Among other things, that beast could do 138,000 logical operations per minute. And you have to love that magnetic drum - echoes of Atanasoff & Berry. Also interesting is the fact that, 47 years later, Dr. Young is still at The University, in our Computer Science department.

Qwicap 1.4a0 Released

A holiday week's worth of coding has advanced Qwicap 1.4 from a scrap of dusty paper with a feature list scrawled across it, to an alpha-test release. As befits an alpha release, it is not feature complete, but it is stable and useful, and lays the groundwork for the rest of what I want to accomplish with 1.4.

The noteworthy new features in Qwicap 1.4a0 are:

• Support for dynamically generated downloads - Generate your downloadable material any way you like, convert it to a byte array, and invoke the new Qwicap.addDownloadable method with a name, and an optional MIME type. It returns a unique URI for the downloadable, which you embed in your page markup wherever you wish. Downloadables automatically become eligible for garbage collection when the page they were first used in ceases to be "in play". (In other words, when the method that manages that page ceases to be in the call chain.)
• Input is now accepted from links ("a href" and "area href"), so you don't have to use forms exclusively, anymore. Combine this feature with the support for dynamically generated downloads, and things like dynamic graphics with corresponding client-side image maps become simple to implement. (The revised version of the Hunt The Wumpus example application demonstrates just such a map.)
• Simplified deployment - Most applications can now use a generic "web.xml" file, as seen in the revised example apps. Qwicap will automatically discover the entry-point of your application code at runtime, and will also handle serving-out any web pages, style sheets, etc. that would otherwise be inaccessible due to the default servlet url path being "/". This means one less thing to learn/remember, and also keeps the URLs for your web applications pleasantly short.

Major features remaining to be implemented in version 1.4 include internationalization, and asynchronous prompts. The former represents unfamiliar ground for me, but it seems clear that there should be some way for developers to alter the default input-validation error messages that Qwicap automatically adds to web pages. Given Qwicap's facility with XML files, I'm inclined to have it retrieve those messages from a language-specific XML file. What I haven't wrapped my head around yet is how to cope with the various permutations of those messages, like pluralization. Perhaps a language-specific class is in order - a method could be invoked on it that is specific to particular problem (bad integer input, for instance), and the code could supply whatever markup it wished in response (it could be loaded from a language-specific XML file, or produced by any other means convenient to the international developer).

I have a much better handle on the idea of asynchronous prompts (my favorite would-be feature in 1.4), but the implementation details to be considered are numerous, and, in some cases, rather non-trivial. The idea, however, is simple enough: instead of blocking until user input arrives, as the current Qwicap.prompt methods do, an asynchronous prompt would be non-blocking, and would therefore allow the web application to continue processing after sending a page to the user. The page would presumably be set to refresh periodically, and would thereby receive status updates until the server had concluded its processing, at wich point the client and server become synchronized once again. Equivalent (or fancier) functionality can be created in many web application schemes already, but Qwicap's asynchronous prompts should be vastly simpler for developers to comprehend, use, and support, while also being more predictable/dependable in as much as the code remains strictly server-side.

Barring another week of holiday materializing out of thin air, I don't know when I'll have a chance to make headway on these outstanding features. (Things are busy at the office, and our current and pending web applications don't have a compelling need for either of those features.) Nonetheless, I would value insights on internationalization in the mean time. And anyone who can put a sexy interface on the new Hunt The Wumpus example would be appreciated. (At present its interface is pleasantly minimal and clean, but doesn't exactly manage to catch one's eye.)

The "Programmers Hate Programmers" Hypothesis

I'd like to offer my fellow computer programmers the following hypothesis, which I've been kicking around for a few years: Programmers hate programmers. To explain: This hypothesis contends that the best explanation for the torments that programmers all-too-often lavish on one another with the software they create (especially libraries and APIs) is that programmers hate other programmers and therefore they design their creations to screw with each other. This hypothesis is obviously wrong, because, among other things, we could certainly screw with each other far more effectively than we do, if we were really trying. Nonetheless, at first glance, it seems to explain a lot of our experiences, and that's what I think is useful about it.

So, by way of example, this hypothesis seems to explain why...

• ...someone's code presents you with a "file not found" error, but doesn't tell you what file wasn't found, or where it was looking for the file.
• ...someone's code carefully checks a host of conditions to determine whether it can proceed with your request, and, finding some condition not met, fails, but won't let you in on the secret of why it failed. (In extreme cases, it won't even tell you that it failed.) It knew why, of course, because it went to the trouble of meticulously checking; it just didn't believe that you had need-to-know.
• ...someone's code does its error checking/reporting in such a way that the errors it reports have nothing to do with the actual problem and therefore—far from assisting you with debugging—they actually send you down the wrong path entirely.
• ...someone's API is so abstract that even after you learn it, you still don't know what it does, or, alternately, having learned what it does, you can't seem to make it do it.
• ...someone's system saves you from having to waste ten or twenty minutes writing code by forcing you to spend a week figuring-out how to write their config files, metadata, etc. (Hmm... what's the one thing my fellow programmers do best?.... Well, we can't have them doing that.)
• ...someone's API makes routine, simple operations lengthy and tedious, or makes implementing common, seemingly trivial, features, into major coding efforts, even though they had to know that everyone who used their API would be doing those sorts of things regularly. (Perhaps programmers who grow up with abusive APIs are destined to abuse their own APIs.)
• ...someone attempts to solve problems stemming from complexity by adding new layers of complexity.
• ...some APIs look as if they were designed exclusively to satisfy a feature check-list, with no concern for how the elements of the API would work together in practice.

There are probably a lot of other good examples that would resonate with much of the programming community, but I seem to have blocked from my memory the others that I've encountered. I leave extending the list as an exercise for the reader. The important thing about this hypothesis is that when you, the developer, find yourself in a situation where it seems to explain what you are up against, or it might describe another developer's experience with your code, something is wrong. Think of it as a diagnostic tool.

A few of those problems originate with judgement calls with which I've come to disagree. I think most, intentionally or otherwise, are examples of externalizing development costs. In other words, to save themselves a day or week of hard work, developers will cut corners in, or under-build, their code, and, having done so, they'll expose hundreds or thousands of other developers to hours of extra work. So, the original developers save a day or a week, but they cost the community months or years... the larger the community, the greater the damage.

Writing Non-Conformant Code

In this context, the best thing that can be said of an API, library, or other body of code, is that it does not conform to the Programmers Hate Programmers hypothesis, and I have some ideas about how to achieve such non-conformance.

I spent about 15 years writing Macintosh software, and I remember that Apple's engineers used to stress that the most important part of a Mac application was its user interface (GUI), since it's the GUI that defines an application to its user. In order to produce the best possible GUIs, they advised developers to design and, if possible, user-test and refine, their ideal GUI before writing the first real line of code, and then engineer the code around realizing that interface design, because a great many implementation details would be dictated by the requirements of the interface. In other words, the core code must be designed around the demands of the GUI, rather than the GUI being designed around it. Doing otherwise tended to create GUIs that interacted poorly with the user, and were clumsy and/or difficult to use.

While it might have been more honored in the breach than the observance, I believe they had the right idea. Over the years, I have also come to believe that the interfaces provided by GUIs, vital as they are, are not nearly as important to the world as the interfaces provided by APIs, because the effects of bad APIs propagate up into all of the code built on them, including GUIs. In extreme cases, bad APIs will even prevent code from being developed at all by making it appear effectively impossible, or so costly that the coding projects end-up abandoned. In other cases, potentially valuable projects won't be started at all due to past experiences poisoning the developer's cost/benefit guesstimate. (And then there's the matter of programmer burn-out....)

Once we conclude that the quality of APIs is profoundly important, the same advice that made sense for GUIs comes to make sense for APIs (and all aspects of the "developer experience" provided by a body of code): Design something as close as possible to an ideal API first, and then engineer the underlying code to make that API possible. If possible, observe how other programmers interact with the API, and refine it to be as intuitive, comprehensible, convenient, and simple as possible.

Bear in mind how your code is likely to be perceived. For instance, I suspect that many programmers (myself included) tend to conflate the number of lines of code they have to write in order to perform a particular task with the efficiency of the resulting implementation. If it takes a lot of lines, it's likely to be perceived as being less efficient than something that performs the same task in one or two lines. We all know that such a perception has nothing to do with runtime performance, but it can have a lot to do with how we perceive the effectiveness of an API, and our efficiency as programmers while using it. So, convenience is good, and is often worth a hit at runtime, if it's convenient enough. If reversion to a lower-level, but more efficient, API is possible in the optimization phase of development on an as-needed, piecemeal basis, so much the better.

Some APIs may not be able to represent every last feature that their underlying code could supply (or, as mentioned above, they may have to trade absolute performance for developer effectiveness). I believe that it's appropriate to offer both high- and low-level APIs. The high-level API should make, say, the most commonly-used 80% of the functionality as convenient to use, and as easy to learn, as possible. The low-level API should make all of the functionality available as best it can. Ideally, it should be possible to freely mix usage of the two APIs, so that the client code can enjoy the benefits of the high-level API, without foreclosing the option of later embracing, on a piecemeal basis, the more esoteric, or efficient, features exclusive to the low-level API. Even the low-level APIs should be designed to provide the best possible developer experience, but the high-level API should be designed first; its needs are likely to impose some requirements on the design of the low-level API.

(High-level APIs built over pre-existing low-level APIs can still be valuable, but they are in danger of having to adopt severely compromised internal or external designs, because they can't completely work-around the fixed low-level structure. Put another way, the kind of building you can build is severely restricted by the foundation it has to sit on. In order to have flexibility in the building's design, the foundation must be designed around the building's needs. Otherwise, you may be forced to build a shed where you needed a home.)

Criticisms and suggestions are easily produced, of course. If they aren't backed-up by code, they're too easy. For a couple of years, beneficially interleaved with other projects, I've worked on a Java web application development API designed on the aforementioned principles. It's called Qwicap. It's open-source. You can begin using it after reading just one or two (longish) pages of introductory documentation. Most of its API is confined to just two classes. It's a clean-slate design intended to eliminate most of what I perceive as the common tedium of web app. development. It works to fit into the natural flow of your code, rather than forcing you to fit your code into it. We use it in production at The University of Texas at Austin. And, yes, I'm trying to draw attention to it. With more than a hundred thousand projects on Sourceforge, alone, and countless web application development schemes available, it's hard to get noticed. It's even hard to know whether anyone has already done something like it. Much as I detest self-promotion, it seems I'm going to have to try my hand at it. Wish me luck.

That Remindes Me...

If you haven't already seen it, Richard P. Gabriel's book Patterns of Software (free PDF, 1.2 MB) is well worth a read. The problems of abstraction, and the idea of code habitability, are two good topics from the book that spring to mind. I first became aware of him through an article on Sun's site, which is a quick read and also worthwhile. Finally, if you happen to be an old Mac programmer, the book "Revolution in the Valley", by Andy Hertzfeld, is a treat. The web site of Mac folklore from which it was derived includes more material, but the book is a thing of beauty.