Twenty 42s

My recent work on Qwicap has concentrated on character set issues, first concerning the loading of XML/XHTML markup, and now concerning receiving input from browsers. I've learned a lot. For instance, unlike output sent to a client, input sent from the client in HTTP includes no provision for identifying the character set. You can, of course, add an "accept-charset" attribute to the "form" elements in your web application's pages, but there's no guarantee that the client will support the character set you specify, unless you confine yourself to ISO-8859-1. You could try inferring the client's supported character sets from the "accept-charset" headers of its HTTP requests, but then you would almost always be lied to, because most browsers throw in a wildcard that means that they accept all official character sets. Since there are currently 254 of those, what do you suppose the chances are that a browser sending that wildcard really supports them all? Zilch, in my tests.

The icing on this particular cake is that a Java application server is required to provide your web application with the client's input in the form of Java String objects, via the ServletRequest class. How can the application server reliably translate the input bytes it receives into the Unicode characters that will populate those String objects? Not knowing the character set used to encode the characters represented by those bytes, it can't, but the API requires that it blunder ahead with String creation regardless.

Twenty 42s

All of these character set torments led me to take a close look at the first 65,536 characters of Unicode, you know, for fun. Did you know that among those characters there are twenty ways to represent the number, just to pick one arbitrarily, forty-two? There are. And that's assuming that you don't mix-and-match amongst languages. If you do, there are 400 representations. Specifically, in those first 65,536 Unicode characters, there are twenty discrete groups of characters for representing decimal digits. Your browser probably doesn't even have a font containing glyphs for all of them, but here are those twenty forty-twos, along with their character codes expressed in hexadecimal:

４２	0xFF14 and 0xFF12
᠔᠒	0x1814 and 0x1812
៤២	0x17E4 and 0x17E2
፬፪	0x136C and 0x136A
၄၂	0x1044 and 0x1042
༤༢	0x0F24 and 0x0F22
໔໒	0x0ED4 and 0x0ED2
๔๒	0x0E54 and 0x0E52
൪൨	0x0D6A and 0x0D68
೪೨	0x0CEA and 0x0CE8
౪౨	0x0C6A and 0x0C68
௪௨	0x0BEA and 0x0BE8
୪୨	0x0B6A and 0x0B68
૪૨	0x0AEA and 0x0AE8
੪੨	0x0A6A and 0x0A68
৪২	0x09EA and 0x09E8
४२	0x096A and 0x0968
۴۲	0x06F4 and 0x06F2
٤٢	0x0664 and 0x0662
42	0x0034 and 0x0032

By the way, the Java methods for converting from strings to binary numeric values, like Integer.parseInt, accept all of those character sequences as valid inputs, as well as all of the cross-language mixes. Worth thinking about if you have a web application that accepts numeric inputs. And the fun doesn't stop with decimal digits; for instance, you can find a replication of most of ASCII between 0xFF00 and 0xFF60.

What's the point? Nothing in particular, except to reinforce the general warning that one is ignorant of character encoding issues at one's peril. (Before you ask, no, neither my software's handling, nor my understanding, of these issues is perfect; that's why I know a little something about the "peril" part.)

"How to Design a Good API and Why it Matters"

A kindred spirit at Google, Joshua Bloch, delivers a Google Tech Talk: How to Design a Good API and Why it Matters. If any part of my Programmers Hate Programmers anti-hypothesis resonated with you, you'll appreciate his presentation with its much greater detail.

The Lost Tank of Brownsville

I visited the Bamberger Ranch again on Saturday. Generous folks that they are, the Bambergers allowed me the use of the guest quarters once again, so I could drive down in the middle of the night when the roads are free of traffic, get some sleep, then go out and take photos whenever I regain consciousness. If I happen to be present around lunch time, they usually throw in a sandwich and related hospitality. It's a helluva deal.

With the summer heat settling itself in nicely, I did notice one odd thing about this arrangement, however: the guest quarters aren't air conditioned. OK, I already knew that, and I've been through Austin summers without air conditioning in years past (long past, thankfully), so the idea of life without chilled air isn't utterly absurd to me. What made the absence of air conditioning stand-out this time was the fact that I kept hearing the air conditioner in their office kicking-in just outside the open window I was trying to sleep under. Like it was taunting me. It's not hard to understand that an office would need air conditioning. What had me stumped as I lay there in the too-still night air was why one would spend the money to air-condition 80% of a building, but not the other 20% of it. The guest quarters, you see, are part of the same building as the office. Maybe the architecture of the structure contains the explanation, or maybe it's a test. Knowing David Bamberger, it could be a test. If so, I passed this time, but summer is just getting started. If they keep extending their hospitality to me, I may have to donate a fan to the ranch. It'd be a bargain at twice the price, of course.

I was lucky enough to get a free lunch with my visit (and they say there's no such thing), along with pleasant conversation with David, Margaret, her daughter Margie, and various friends. Afterward, Margaret and co. went off for a swim, which I foolishly declined in the name of photographic necessity. I talked with David a bit about some of the areas of the ranch that I'd already visited, and asked him about other noteworthy places. He picked-out two on a map of the ranch; one was an earthen-dammed tank in the "Brownsville" region of the ranch, the other an overlook in the "airstrip" section (so-named because a small aircraft once landed there to the detriment of all concerned). I headed out for the tank, figuring I'd have plenty of time for airstrip afterward.

As far as I know, that tank in the Brownsville section, like several other tanks on the ranch, has no name. I hereby rectify that situation by dubbing it the "lost tank of Brownsville." I take David at his word that he built it out there, somewhere, but I spent hours carefully going back and forth on hot, dusty gravel roads—roads that I'd been warned had eaten trucks which, from the sound of 'em, could have comfortably carried mine in their glove compartments—and carefully picking my way on foot through fields of rocks to descend, with still greater care, into the valleys below to look for that tank, and I never did find it. It's not even a big area of the ranch, but it's very slow going on foot, and bears only an eccentric similarity to the maps, at least if you don't already know where you're going. On the plus side, I did stumble upon a few spots of photographic interest in the valleys I explored, and shot a few panoramas, clouds permitting. I also found where they keep all of their chiggers and was able to give them a good feeding. (Note to the afflicted: doctors can prescribe anti-itch lotions far superior to the over-the-counter rubbish. Believe me.)

By the time I'd given up my search for the lost tank, there was no hope of getting to Airstrip, at the far other end of the ranch, before sunset, so that was that. That'll give me something to do on a future visit; not that I felt I was in any danger of running out of material out there.

Anyway, I'm months behind in assembling panoramas, so I don't have any panos, new or old, to show, but this trip did produce some readily displayed conventional photographs, which I hereby include below for your amusement and possible edification. (And if anyone can identify that dragonfly or the unidentified plants, I hope they'll edify me.)

Monarda citriadora (Purple Horsemint) near Madrone Lake.

Danaus gilippus (Queen) butterfly on Eupatorium greggii (Greg's Blue Mistflower) near Madrone Lake.

Odocoileus virginianus (White-tailed deer) among grass and wildflowers. The wildflowers are primarily Zexmenia hispida (Zexmenia).

Unknown dragonfly on unknown plant at the edge of the thistle field near the windmill.

Papilio glaucus (Tiger swallowtail) feeding from the flower of an unknown variety of thistle in the thistle field near the windmill.

An icon of rural America, an Aermotor 702 windmill, the only windmill left on the ranch, stands just to one side of the highest point on the ranch. Remarkably, the 702 was in continuous production from 1933 until at least 1981. (It may still be in production; the company's history page is not entirely clear on that point.) I've been saying that I needed to shoot a panorama with one of these in it for so long that a friend had suggested that I carry an inflatable model of one, so that I could add rural color to any scene.

My first entry in the Jay Lake Artifacta Americana photo contest.

Rana berlanderi (Rio Grande leopard frog), probably female, at the windmill pool.

Rana berlanderi (Rio Grande leopard frog), probably male, at the windmill pool.

For reasons best known to themselves, these frogs not only come out at the height of the afternoon heat, but regard a photographer standing twelve feet away as a threat from which they have to retreat, while accepting as harmless that same photographer crawling on his elbows and belly at a mere five feet. I suspect that this species likes to humiliate its prey before moving in for the kill. I was probably lucky to escape with my life.

Qwicap 1.4b9 Released

Qwicap version 1.4b9 was released on May 31. It includes the minor enhancements of the unannounced beta 7 and 8 releases (logging in Qwicap.reportException, and more informative exception messages from the MutableMarkup class, respectively). The work on beta 9 is of a whole 'nother order, hence this announcement.

Last Friday, after lunch, I was looking forward to a quiet afternoon, and was reading my usual collection of computing-related web sites when I arrived at Elliot Rusty Harold's blog, Cafe au Lait. As usual there was a lot of material there, and, also as usual, I wasn't even going to try to deal with it all; I was just there for the quote of the day, which turned out to be about character set support in various web application development systems. And it included a harmless looking little link attached to the word "Unicode" (just like that one, in fact). I clicked the innocent word and that was the end of my peaceful Friday afternoon. The link was to Joel Spolsky's 2003 article The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!). If you haven't read it, consider it recommended. If you're in a hurry, just read the bit at the end about the problems faced by web browsers when they need to interpret the bytes of a web page. That gives you the flavor of many of the problems posed by character set handling. And they're fairly obvious problems, if you happen to think about them. Unfortunately, I hadn't.

I wasn't exactly unaware of character set issues (I sorted out EBCDIC to ASCII translation issues back in the days of the IBM 3033 and the original IBM PC - issues that had IBM's support engineers embarassed and stumped), and I've been a big fan of Unicode, but while reading that I article I quickly realized that I'd failed to grasp just how significant character set issues are, or can be, to everyday software development. Specifically, in the case of Qwicap, I realized that I'd ignored character set issues when I wrote Qwicap's XML engine. I knew that, having emerged from Reader objects, the characters in memory were Unicode (sort of) and I'd just left my concern at that, without thinking about the encoding of the source materials that went into those Reader objects. I'd also neglected an issue associated with transmitting web pages to clients: They were always sent as UTF-8, regardless of what the document markup called-for. (In a way, that's OK because a character set specification in the HTTP "Content-Type" header, which Qwicap was in the habit of supplying, takes precedence over any specification in the markup, but it's both rude and confusing to ignore the document author's wishes as expressed by their markup.)

So, seeing all of those basic oversights at once was quite a kick in the teeth (self-inflicted, no less). And, of course, one of the great things about open source development is that you have the opportunity to make big, embarassing mistakes in public. On the bright side, if I knew how difficult it was going to be to develop Qwicap when I started, I might have come to my senses and found something less painful to occupy my time, like hitting myself in the head with blunt objects. So, since I still think Qwicap is a good idea, and I yet cling to the hope that it'll find a niche for itself beyond the bounds of my worthy employer, The University of Texas at Austin, (and for all I know, perhaps it already has) maybe it's just as well that I didn't fully appreciate what I was getting myself into when I first had this idea.

Anyway, with the release of version 1.4b9, Qwicap is now character-set aware, and I can take some comfort from the facts that (1) nobody but me ever noticed this problem in Qwicap, and (2) there are vastly more popular web application development schemes that still neglect character set issues. In the latter case, though they were absurdly late in coming, the fixes in 1.4b9 do become one more feather in Qwicap's hat. Of course, I still feel plenty stupid.

Qwicap 1.4b6 Released

Version 1.4b6 of Qwicap is a trivial release, correcting a serious bug in a method, in the non-public API, which Qwicap doesn't even use. (Qwicap does use other methods in the same utility class, however, which is the only reason that this issue is relevant.) I've also tweaked the markup of the pages in the example applications such that Eclipse won't choke when trying to load them anymore. So, this release is a big yawn, but that's good news late in the beta process.

"For the Love of Light"

Jim Austin's article on high-dynamic range (HDR) photography, "HDR For the Love of Light", features my work "Hamilton Pool Panorama no. 2". Thanks for the kind words, Jim, and the excellent article. Anyone who wants an introduction to high dynamic range photography, or impressive examples of its use, will find that article worth a look.

Ironically, I recently discovered a problem in my panoramic workflow that has been muting the colors in my work for some time. Most of my panoramas, including that one, need to be recomputed to bring out the true quality of their color. I'm in the process of doing that in between work on my ongoing Spring panorama project at the Bamberger Ranch Preserve. Unfortunately, recomputing the panoramas, and then processing them as HDR, is very time consuming, so I don't expect to have everything fixed any time soon.

To elaborate on the problem, Apple's Aperture application, which I use for managing my photos (when it works, it's great; when it doesn't, it reminds me of Photoshop), has a preset export option labelled something like "Original Size, 16-bit TIFF". Because that was the only preset export option that suggests that it preserves all of the information in RAW originals, and because Aperture is designed for professional photographers, I naively assumed that it would also preserve the color space of the images. Not so. In fact, it converts the image to the sRGB color space, a big step down from the AdobeRGB color space of my originals. This isn't a bug, but I do think the name of that preset export option should be changed to something that makes its behavior explicit, like "Original Size, 16-bit TIFF, sRGB".

Once I noticed that problem and fixed it, I was bit by another. I needed a utility that would let me assign a color space to the non-color-managed 16-bit TIFF images produced by the software I use for assembling panoramas (PTMac). I tried doing it with Photoshop CS, but it launches so slowly that it created a strong incentive to find a lighter-weight tool for this job. (Aperture, frustratingly, provides no mechanism for assigning a color space to an image.) After a bit of Googling, I found an article in Apple's "Pro" area titled "Image Editing in ColorSync" which told me that Apple's "ColorSync Utility" was precisely what I was looking for. I started using it, and, for a while, I thought it was a great tool. Then I realized that when it saved my 16-bit TIFFs with their newly assigned color spaces, it was saving them as 8-bit TIFFs. Put simply, this "Pro" tool was automatically throwing away half of the data in my images without a single word of warning. I'm still dumbstruck by this behavior. And there's no way around it; while its "Save As" dialog offers an enticing pop-up menu that allows the image bit-depth to be specified, the only option it offers is 8-bit.

So, I'm back to Photoshop for my color-space assignment needs. (Sigh.) If anyone can suggest an alternative color-space assignment tool for Mac OS X, please do.

Qwicap 1.4b5 Released

Qwicap version 1.4b5 eliminates a case in QwicapServlet in which HttpServletResponse.sendError could throw an IllegalArgumentException, because an Auth2Scheme implementation had refused access to the web application, and had already caused the response to be committed (for instance, by redirecting the user to an authentication page). QwicapServlet now invokes sendError only when the response has not been committed. And that's as exciting as this release gets, which is good news, because it means the beta cycle is winding-down toward a true release.

For those unfamiliar with Qwicap, allow me to throw in the standard blurb:

Qwicap is a Java web application development API that discards the traditional hit-run-exit CGI model, automatically provides rich state, and supplies a powerful, extremely easy-to-use XML "templating" engine to avoid mixing code and content. It builds on Servlet, XHTML and CSS technologies.