Geminids and Mysteries

I watched the peak of the Geminids meteor shower last-night/this-morning from the Bamberger Ranch, with the kind permission of the Bambergers. (They even threw in a slice of Margaret's birthday cake. Happy birthday, Margaret.) I setup a pair of the original Canon Digital Rebels, one with a Sigma 20mm, f1.8 lens, and the other with a Sigma 10-20mm, f4-5.6 lens set for 10mm and f4. Both were shooting RAW, manual-focused on infinity, set for ISO 400, and using a shutter speed of 30 seconds. They were aimed such that the field of view of the 20mm lens was entirely subsumed in the field of the view of the 10-20mm lens, set at 10mm. I shot about 270 frames with each camera, concurrently, but with the shutters staggered to try to avoid missing meteors while one or the other camera had its shutter closed.

From past experience, I knew the 20mm, f1.8 lens could capture some meteors. I had doubts that the 10-20mm would be of any use at f4. My first look at the results credits the 20mm, f1.8 lens with 8 meteors, and the 10-20mm at f4 with 4 meteors, even though it covered four times as much sky as the 20mm lens. So, the 10-20mm, f4-5.6 lens isn't completely useless for meteor showers, but it's a very poor choice for such work. Not much of a surprise there, but now I know.

The good news is that all of that driving, and those four hours of shivering and photography produced my best meteor photograph so far (taken with the 20mm, f1.8 lens, of course). That meteor wasn't the most spectacular of the 198 I counted, but it was a good one. The best of them all was really two meteors that entered at the zenith at the same moment, but going in precisely opposite directions. One burned-out quickly, but the other had a good, long run, and burned so bright against the black of the sky that it reminded me of seeing a thin ribbon of magnesium burn in a high school chemistry class.

One peculiar happening while I lay out there on the cold ground controlling the cameras and counting occasional meteors, was a sound off in the dark distance that I can only say made me imagine an angry hawk, expressing its anger in high-pitched wheezes. This sound came several times, then the usual quiet returned. It's quiet out there in a way that city dwellers, like me, may have a hard time grasping. There are no man-made sounds to be heard. Not even a distant hum of traffic on some highway. There's just the wind blowing through trees, and a few crickets. It takes a while to notice it, but it stands out in a big way once you do. So, the wheezing came from nowhere and then vanished into that amazing quiet. A minute or two later, the wheezing returns from the same direction, but closer. Soon, hooves are heard running on the hard ground, coming from that direction towards me. They seem to account for at least four animals, but what sort of animal, I can't figure. My untrained ears claim that the hoof-falls are too heavy to be deer, but too light to be cattle. The sound of the hooves seems to be ahead of the wheezing, which chases after them. The sounds come closer and closer and I begin to wonder if some panicked livestock are about to show-up and trample my cameras. Their eyesight and hearing are almost certainly better than mine, so they should know I'm ahead, but they keep coming anyway. Just ahead of that wheezing. I let out a quick whistle to try to ensure that they're factoring me into their plans. It doesn't seem to have any effect on their pace or course, but soon the hoof-falls stop, seemingly for their own reasons. Whatever was making them is nearby; I can hear them breathing hard, but can't see a thing. Their breathing dies away, and the quiet is beginning to settle in again when the wheezing returns, and then hooves are running again, but now parallel to me, and then away and out into the invisible grass, and the remarkable quiet.

Qwicap 1.4a42 Released

Qwicap 1.4a42 has been released. (Download it from SourceForge.) This version has been in the works for almost six months, off and on. It was bedeviled by two problems: (1) a thread synchronization bug that could manifest when a user submitted new form data before the processing of their previous hit was complete, and (2) a misunderstanding between myself, a colleague who administers some of our servers, some obscure error reports from the JVM, and some Google searches, that led us to conclude (incorrectly) that Qwicap absolutely had to have a thread pool in order to operate reliably over long periods of time.

With regard to the thread synchronization bug: Ouch. I could not reproduce it when running the server and the web browser on the same box (my normal method of working), even when the box was a multi-processor machine. If a colleague, Kevin Wood, hadn't noticed this bug and then invested serious effort in exploring it, I don't know when it would have been found. Just to keep it interesting, the fixes to Qwicap 1.4a37 that he provided to me didn't work in his test setup when I supplied him with builds from the official codebase that incorporated all of his changes. (Those builds worked fine on my machine, naturally.) The builds he produced from the older codebase worked fine in his test setup, and mine. Even after we both sat down and satisfied ourselves that I hadn't mangled, or omitted, any of his modifications, the problem persisted in my builds, when he ran the tests on his box. In the end, he set up a box for me that was configured essentially the same as his machine, and I did my own testing. Eventually, I identified one last race condition that even he'd missed. (I throw no stones here; if I hadn't missed it in the first place, he couldn't have missed it in the second.)

With regard to the perceived need for a thread pool, I put a lot of work into building, instrumenting and testing a thread pool based on the mistaken belief that our production servers were encountering a memory-related resource exhaustion problem related to the total number of threads created over the lifetime of any given Qwicap application. In the end, it turned-out that the problem was confined to the amount of memory consumed by the stacks of the threads running at any given time, when running under Java 1.5, which dramatically increased the default stack size relative to Java 1.4 (it seems to have gone from 256K to a megabyte or more). Initially, configuring our servers to run Tomcat in a 64-bit address space solved the problem, and later configuring them to use only 256K as the default stack size provided an alternate solution.

So, Qwicap doesn't turn-out, strictly speaking, to need a thread pool to operate reliably, but I implemented one before that became clear to us, and therefore it now has a thread pool. There's no point in ripping it out, because (a) some people will actually want it, (b) a lot of nice status data gathering machinery became tied to it, (c) it can be configured such that it doesn't hang onto any threads, if you wish, and therefore won't get in the way of people who don't want pooling, and (d) the configuration parameters for the pool allow the size of the Qwicap thread stacks to be controlled, which goes to the heart of the real problem.

Other noteworthy changes in Qwicap 1.4a42 include:

• The concept of "blocking listeners" has been added. Such listeners, registered using the new Qwicap.addBlockingListener method, are invoked just before and after Qwicap blocks to wait for user activity. This applies to the prompt methods, and the redirect method, and will apply to any future blocking methods. Thus a "blocking listener" can perform actions like releasing and reacquiring limited resources like database connections, or log application activity, transparently from the perspective of a web application's code.
• There is now a "service data recorder" (SDR), for lack of a better term, built into Qwicap. The SDR accumulates data in hourly chunks (up to 72 of them). Each hour's data includes information on the thread pool (total size, number of active and inactive threads, number of threads created and allowed to die, number of threads unavailable due to the max. pool size limit), user sessions (number completed and expired, and the mininum, average and maximum duration), hits (total number, hits directed to invalid pages, and the minimum, average and maximum response time), and RAM usage (amounts free, used, and total). For the time being, the SDR reports are available only in a human-readable format, because I am hesitant to commit to what data will be gathered, how it will be represented, etc.

At this point, my main goal for Qwicap is to finalize the 1.4 release. I'm prepared to declare it feature-complete now. (Therefore, I should have released it as a "beta", rather than an "alpha".) While it doesn't include all of the features I'd originally intended it to have, it's the most significant revision to Qwicap thus far, and at some point you just have to draw a line across such an effort, declare it complete, and add the missing features to the list for the next major revision. Furthermore, the current release version, 1.3.3, is almost a year old at this point, and lacks a number of bug fixes and important improvements that are present in 1.4. (Backporting the fixes and improvements to common code became an unsupportable burden some time ago, with the never-quite-ready-for-release version 1.3.4.) The thought of people putting-up with 1.3-isms when they should be benefitting from 1.4-isms is really getting on my nerves. So, expect the 1.4 release in the not-too-distant future, bug reports permitting.

The Real Thing(s)

From my most recent visit to the Bamberger Ranch: Above, real, wild turkeys. Below, J. David Bamberger and an old friend.

Apple's Backup 3 - Hopeless Junk

What good is backup software that can't perform a restore? That's what I've been wondering in the week since the main hard drive in my PowerMac G5 died. Having lost another drive earlier in the year for which I had no backup, I'd finally setup a proper backup system, using external SATA drives as the destinations, and Apple's Backup version 3.1 (a part of the ".Mac" package) to perform nightly incremental backups. So, my latest drive failure seemed like one failure too many for the year, but one for which I was prepared. Or so I thought until I tried to restore my files. Then I found that Backup 3.1 would only restore a certain number of the files early in the backup set before it would crash. Any attempt to restore other files from the set resulted only in a crash. Specifically, Backup would proceed to very slowly consume about 2.1 GB of virtual memory, and then it would crash, writing the following to "backup.log":

Backup(1104,0x1ace800) malloc: *** vm_allocate(size=1069056) failed (error code=3)
Backup(1104,0x1ace800) malloc: *** error: can't allocate region
Backup(1104,0x1ace800) malloc: *** set a breakpoint in szone_error to debug

As best I can tell, this means that between the 2.1 GB of virtual memory it used-up (as observed in Activity Monitor), it also consumed a further 1.9 GB of combined real, private and shared memory, and then died because its 32-bit address space was exhausted. Brilliant. Thank you so much, Apple, for backup software that can backup, but not restore. That's innovative, alright, but not in a good way.

Needless to say, I submitted many crash reports, and a proper bug report, complete with a plea for help, but none has been offered in the following week.

I have, however, salvaged a fair number of files by manually mounting the ".sparseimage" files hidden inside the backup "files", but with seventy incrementals since the last full backup, sifting through everything by hand is a major problem. And files with resource forks, along with executables, can't be salvaged that way at all. Oh, thank you, Apple.

So, based on this experience, let me warn anyone else out there who's using Apple's Backup program: Don't. If your backup sets are small, it may well work for you, but once they grow large enough, you'll reach the same point I did, where it'll crash rather than restore your files. And you won't know that you've reached that point until it's too late.

S.M.A.R.T.?

As an aside, the drive that died (a 250 GB Maxtor that Apple sold me with the G5) had a controller that supports SMART (Self-Monitoring, Analysis and Reporting Technology), and I had installed the DiskWarrior extension that routinely checks the SMART status of the mounted drives and is supposed to display an alert when a drive reports that it is beginning to fail. No such warning appeared. This is the second SMART-equipped mechanism that I've lost (but the first since installing the DiskWarrior extension), and I haven't seen SMART do a thing. Has anyone out there actually seen SMART do its stuff?

Aperture Vaults

The one thing that did work well in this mess was the backup scheme ("vaults") that Apple's Aperture team integrated into that application. There was no problem restoring from the vault I'd created on one of my backup drives, and because support for vaults is well integrated into Aperture, I'd (1) actually setup a vault (the splash screen gently nags you to do so), and (2) kept it current. If any members of the Aperture team happen to read this, let me just say: Thank you. And please go beat the metaphorical crap out of whoever's responsible for the Backup application; they make everybody else at Apple look bad.

Failure in a Creek

Last Friday afternoon I drove out to the Bamberger Ranch to lend Margaret a hand with her Macintosh, and to shoot some panoramas while the fall color was burning bright, and the valleys were still safely free of gunfire. Unfortunately, I didn't finish with the computer until the sun was well on its way to the horizon, so there wasn't a lot of time to find a good location and get my panoramic rig setup. After a quick consultation with David, I set off for the Louis Bromfield trail, which I was assured was especially worth seeing, and where he figured we'd cross paths later on, as he had some work to do down there.

Unfortunately, the first stretch of the trail was hard to distinguish from any of a dozen deer paths, and I headed off into the hills, squandering the rapidly diminishing light as I wandered in what eventually became exactly the wrong direction. When, at last, my mistake became completely obvious, I turned back and eventually intersected the Bromfield trail. It was a shame to lose that time, because the portion of the Bromfield trail that I did see was gorgeous: a creek running down a course cut through limestone, bordered by thick native grasses, and shaded by bald cypress trees in full fall color. Not that the shade was evident, however, because the sun had already settled behind a hill.

Cursing my timing, and daylight savings, I stopped at the first dense group of cypress trees, found my way down to the creek bottom, and hurriedly assembled my panoramic rig astride the creek. I thought I might still be able to shoot something useful, due to the reasonably bright sky, and the flexibility afforded by the High Dynamic Range (HDR) photographic technique I began using in August. It turns out I was right, but, as I learned tonight, in my rushed setup I ruined all of the panoramas I was about to shoot by botching the horizontal alignment of the camera; I read my setup notes correctly, but aligned the wrong bits to the diagrammed positions. Damn.

Lest the occasion be a total loss, I present here the best fragments of the ruined panoramas. The final one was actually shot after nightfall. I had already watched the sun set while I was still looking for a good vantage point. By the time I'd found one, parked my truck so it wouldn't appear in the would-be panorama, sprinted up a valley wall, and setup the camera, night had already fallen. The scene was rescued, if it was rescued, by using HDR to combine three exposures, the longest of which lasted 25 seconds.

I met up with the Bambergers afterward, and David pointed-out that I must have missed the Bromfield trail, because he'd ended-up working there, as expected, and hadn't come across me. I told my little tale of woe, while silently kicking myself for missing the opportunity to walk one of the most beautifully restored sections of that now famous ranch with the man responsible for it all. It's a lot like having missed a chance to walk with Aldo Leopold through his little piece of Wisconsin's sand counties, or with Bromfield through his pleasant valley. Damn, again.

Jay Lake's "Rocket Science"

Last time I checked, the list of people who had it in for me included Nazis, Commies, the Kansas City Mob, the United States Army, the Augusta Police and the Butler County Sheriff's Department. Not to mention Mr. Bellamy's gang, Doc Milliken and Lois. I was sure I'd left someone off the list, but I figured they'd let me know in due time.

—Rocket Science by Jay Lake.

I seldom read non-fiction, on the grounds that the Universe has a ≅14 billion year head start on me, and that leaves me with a lot to learn and not a lot of time to do it. But I've always had a soft spot for science fiction, and, months back, when an old friend, Jay Lake, sent me a link to a press release announcing that he'd just signed a contract for his second and third novels, my immediate reaction was "when did you have a first novel?" Let your friends move out of town, and little details like that start getting lost, apparently. He filled in the blanks for me, and I ordered a copy of Rocket Science immediately. I must admit, however, that I don't always enjoy Jay's writing. The skill and intelligence he brings to his craft are always evident, but the style and subject don't always appeal to me. Just a personal thing.

Jay and I met in 1986, working jobs that eventually payed about a buck over minimum wage. We were both in on the ground floor of the desktop publishing revolution, not that that meant a lot at the time. I was developing Mac software in my spare time and periodically interviewing for jobs in the Microcomputer Support Group in UT Austin's Computation Center. Jay was writing sporadically, and I can't remember what else. One thing about the kind of guy Jay was: with both of us basically living paycheck to paycheck, he was the sort of fellow who'd find our boss working herself up to firing me over one or another of my pointed critiques about the way our operation was run, and he'd tell her that I had a point, and if I was fired, he'd quit. Probably saved my job a couple of times that way. And I always had to hear about it from somebody else. You've got to respect a guy like that. And he was amazingly smart, had already lived a very odd life which gave him a fascinating perspective on things, and was a thoroughly nice guy. It's easy to end-up friends with a fellow like that.

So, an old friend like that gets his first novel published, and I think you pretty much have to go buy a copy and read it. And I'm glad I did. It was a hoot - the quickest read I've had in a long time. It's sci-fi set in post-World War II Kansas and centers around a mild-mannered fellow who quite unexpectedly crosses paths with a super-secret aircraft stolen from the Nazis right under the noses of Army Intelligence. Mayhem ensues. It's fun. Recommended.

---

Other recent-ish page-turners: The Soviet Space Race with Apollo, A World Lit Only By Fire, and Tales of a Shaman's Apprentice.

Conflating Atomicity with Thread-Safety

I've been adding a thread pool to Qwicap (in the unreleased version 1.4a40), and, with it, a lot of machinery for gathering information about the pool, the threads, the rate at which they're processing hits, etc. The thread pool implementation, of course, depends on careful use of synchronization for its correctness. Because synchronization can be costly, I tried to minimize its use in the information gathering machinery. I thought I had a lot of scope for such minimization due to the fact that operations on Java primitives that occupy 32 bits, or less, are atomic. So I did things like declaring various counters as "int" types and using the increment operator to change them, e.g. "ThreadCount++;", without bothering to execute them inside an appropriately synchronized block.

Very quick, but, during testing I soon realized that it was also very wrong.

A run of javap made the problem obvious. The "ThreadCount++;" statement, where "ThreadCount" is an instance variable, compiles into the following Java instructions:

   2:   getfield        #2; //Field ThreadCount:I
   5:   iconst_1
   6:   iadd
   7:   putfield        #2; //Field ThreadCount:I

Each of those instructions is operating on an integer primitive, and is therefore operating atomically, but, of course, there are four instructions, and if other threads are executing any of the same instructions at the same time, you can lose the effect of one or more of the increment operations. So, operations on 32-bit, and smaller, primitives are atomic in the JVM, but this fact does nothing to help you avoid synchronization.

My dim recollections of 68000 assembly language programming seem to have steered me in the wrong direction in this case, by leaving me convinced that an operation as simple and common as "ThreadCount++;" would probably be represented by a single instruction. Wrong. And returning to my 68000 documentation, I see that I must have been thinking of facilities like the "address register indirect post-increment" addressing mode, which was nifty, but also wouldn't have allowed that increment operation to be represented as a single 68000 instruction. Wrong again. That'll teach me.

By the way, am I the only old C programmer who wishes from time to time that the "asm" statement had been carried over into Java? I mean, once you have a platform independent assembly language, why not? I admit that the practical value would be low, and code readability would suffer, [OK, that's two good "why nots" right there] but I'd have found it very educational. Having actually used the JVM instruction set (which I wouldn't have been able to resist), rather than having merely read about it, I certainly would have known better than to believe that my increment operation would execute atomically.

Squirrels Return, So Does the Owl Cam'

I silently discontinued the owl cam' on Saturday, when I noticed that the fox squirrel family that had been living in the nest box had moved out. I was surprised they stayed as long as they did. However, cool weather this evening (Sunday) seems to have motivated them to return to the nest box, so I've fired-up the nest box cam' again. As before, my preferred view is the five-up, but you can choose from several options on the screech owl cam' main page.

Qwicap Turns Three

Qwicap, my pet project for simplifying the development of Java web applications by allowing them to be structured much like conventional applications (for "conventional" think of interactive command-line apps), turned three years old today.

Actually, it's more like 3.4 years, total, but three years since version 1.0 was released. The releases were internal to U.T. Austin until my boss, his boss, and her boss, all agreed to open-sourcing the product. Then there was the matter of figuring-out whether it could be done at all, given that nobody in my department (the IT dept., no less) had ever done it, or knew of policies that would permit it. Ultimately, I found the lawyer at UT System who dealt with these matters, and discovered that, contrary to all expectations, we actually did have a policy that allowed—and even encouraged—open-sourcing. After reading the policy document countless times, and having many conversations with the ever-helpful lawyer, I even managed to figure-out how to apply the policy to my situation. With that done, there was a lot of paper-work to be filled out, months of waiting for a critical signature while I frantically polished the code and wrote documentation (while also doing all of the things they normally pay me to do), the filing of the paper-work, and, then, figuring-out how to use SourceForge to publish the code.

Finally, Qwicap was posted to SourceForge, and I began trying to figure-out how to make people aware of it. After a few anxious days I was overjoyed to see the news item announcing its release on the SourceForge main page. Wow. All those years of work would not live, die and vanish within the confines of The University. It was actually out there.

That news item lasted three days, and the response to it was deafening, in the same sense that silence can be deafening. Ouch. So, there followed efforts to get Qwicap listed on more topic-specific sites. At the recommendation of colleagues, I started-up this blog, much as it goes against my grain (twenty years ago, it would have been a different matter, but that was then). At the recommendation of my boss, I submitted a presentation proposal to the regional Educause conference. And, of course, I kept working on Qwicap.

The work on Qwicap has gone well. There were a number of 1.3 revisions, then work began on 1.4, the biggest update yet. Several test versions have been released, several of my colleagues have contributed improvements to the project (Kevin, Jay, thanks), and work on it is nearing completion. And we've deployed three more major campus web applications that are built on Qwicap, with very positive feedback from the development team that built them. Most of the team members have baselined Qwicap for their upcoming projects. Since Qwicap's goal was to make life better for developers, that has all been music to my ears.

In stark contrast, the work on publicizing Qwicap has failed miserably. The topic-specific sites ignored it. The blog, which lead-out with my "Programmers Hate Programmers" hypothesis, generated some attention, but if anyone actually did any work with Qwicap as a result, I've never heard about it. The regional Educause deadline came and went without a word, but eventually I was informed that they might be inclined to use the presentation as a backup, in case one of the presentations they actually wanted fell-through. Shortly before the deadline, I swallowed hard and agreed to be a backup, but that never happened. (On the plus side, I never had to write that presentation.) When the national conference came along later, I never quite managed to submit the presentation proposal again. 'Should have, of course, meant to, but still felt the bruises. (On the plus side, I still haven't had to write that presentation, let alone present it.)

I keep an eye on the Qwicap web site, and find that it continues to generate some attention, but most of it relates to the example source code for Java number-guess applications (though not the Qwicap version, naturally). I presume that some people find the examples useful as the interactive equivalent of a "Hello, World" program, but I'm also inclined to suspect that it's the answer to some professor's CS 101 assignment. It's not quite the contribution I'd hoped to make with Qwicap.

Well, Hell.

Last time I looked, there were about 100,000 open-source projects on SourceForge, alone. It seems pretty clear that most of them have utterly failed to make an impression in their niches of the industry, so, statistically speaking, it's no big surprise to be in the same situation. On the other hand, more than a few of those projects seem to have skipped the unpleasant "writing the code" phase of project development, so I kind of thought Qwicap had an edge in that respect. Shows you what I know.

Ironically, when I first proposed open-sourcing Qwicap, I'd been concerned that the product might generate enough interest that I'd have a hard time keeping up with it, so I'd warned my bosses that it might take time away from my usual duties, but they were supportive, nonetheless. Thank you, bosses. [Now there's something you don't hear every day.]

That concern wasn't just wishful thinking. In the late '80s and early '90s I built and maintained the "Gatekeeper" anti-virus system for Macintosh. It was freeware, and developed on my own time. It broke new ground in its field, and achieved a significant level of success; postcards from users arrived from all over the world, those being the days before most people had even heard of email, let alone the Internet. The people who could use email had nice things to say, too, but satisfied users are the quiet ones. The ones you hear from the most are the folks who aren't altogether satisfied; they want features added, or bugs fixed. I encouraged the feedback, and especially the bug reports, of course. Can't fix 'em if you don't know about 'em, and every bug is embarrassing. But, if you haven't had this experience, imagine looking at your email every day for years and finding what comes to seem like nothing but problem reports. Many come with kind words about the value of the product, but the praise is ephemeral, while the worries and the bug hunting and the other work grinds on and on, in every spare moment you can find.

And it wasn't even the case that the Gatekeeper system had a lot of actual bugs. But it seemed like there were, because Gatekeeper had to be implemented as a set of complex, runtime patches to the Mac operating system, and my patches had to coexist with a startling variety of other people's patches, and with all of the undocumented idiosyncrasies of an operating system that was "quirky" on a good day. In practice, the problem-space was limitless and the problems often unreproducible, and uncorrectable. Of course, each bug would be reported many times, and with every version of Gatekeeper (and every OS patch, and every OS version) ever released out there running on somebody's computer, somewhere, they never really went away, even if they'd been fixed years before. And I wanted to represent my handiwork well, so, naturally, I tried to respond to everybody.

After enough years of dealing with all that, day in, day out, I couldn't bear to read the email anymore, or open the letters, or read the postcards. It seemed like there was never any good news, but an endless supply of bad news, all of it directed to me personally, and awaiting my reply. It was relentless, and endlessly depressing. All those years of work, and that was what I had to show for it. The many successes—the product did what I set out to do, and more—couldn't compete in my awareness with the cumulative pounding of the bad news. I had to walk away. Some people were annoyed when their mail went unanswered. I wasn't happy about it, either. Some told me they felt betrayed, which hurt, but walking away had become a matter of survival. (And the product was free, after all. What did they want? Blood?)

The burnout was intense. There came a day when I was contacted by a development house that wanted to turn Gatekeeper into a commercial product. On the one hand, I didn't want to deal with it ever again. On the other hand, it felt like it would be a mistake to have gone through so much and not at least look into this possibility, so I met with a local, commercially successful Mac developer in order to talk about what sort of money I could reasonably expect out of commercializing a product like Gatekeeper. He told me that over something on the order of three years, a product of its kind should be worth about a million dollars. It was an impressive number (still is), but even so I couldn't bring myself to go back to that work.

It felt like about ten years before I could find the will to take on another major programming project, another big idea. So, naturally, I worried about whether I'd go through the same thing again trying to support Qwicap. Turns out that it hasn't been much of a problem. What luck.

One of the nastier traps one can step in, I suspect, is to discover the burn of a really great idea. The kind of idea that you can't not work on, no matter what anybody else thinks of it. At that point, the idea has you right where it wants you, and you have to go where it takes you, while paying mightily for the ride. Great things may come of it, they may make it all worthwhile in the end, but there's only room for so many great outcomes in the world, so the odds can't possibly be in your favor. But you go on anyway.

Maybe Qwicap will make a difference outside The University one day. So far, not so good. Oh, well. I'm not done with Qwicap, and it's not done with me.

'More than I'd intended to say, but I assume I'm talking to myself at this point. If not, good luck with your own great ideas. And have a look at Qwicap.

The Screech Owl Nest Box Cam' Temporarily Returns

Temporarily featured on my screech owl nest box cam': A family of fox squirrels. There's mom, and three pups. The pups stay in the nest all of the time at the moment, while mom is out most of the day inhaling the contents of my bird feeder to make enough milk to keep the pups satisfied. The pups appear to suckle in their sleep, so it's surprising she can meet the demand.

The pups were not born in the nest box; Mme. Squirrel moved them to the nest box a few days ago, presumably in response to rain and/or our increasingly cool weather. Past experience suggests that she will rotate the pups among several nest sites, but for the moment they're in my nest box (stop that gnawing!) and therefore observable.

I don't plan on providing daily commentary, or resuming any of my other owl cam'-related work, but you can watch the critters live. My preferred way of watching them is this five-up view. Other viewing options can be found on the nest box cam' main page.

Qwicap 1.4a34

Qwicap 1.4a34 was released this weekend. With four busy months elapsed since the previous release, 1.4a12, it was time to declare something "done" again, because a lot fixes and improvements have accumulated over that time. The development of this version has coincided with the development of our second and third large, Qwicap-based web applications which will be supplying the end-user and administrator interfaces to UT Austin's new identity management system. The same four developer team is responsible for both applications. They're up against the challenge of schedule-constraints, the rapidly evolving, still-under-development code of the identity management system itself, and the fact that none of them had used Qwicap before (well, one had used its XML engine a few years back). So, I've been watching their progress with interest, wondering which aspects of Qwicap would work well for them, and which wouldn't.

So far, with both applications in an advanced state of development, and user testing of the end-user application already complete, the results are encouraging. The team seems uniformly pleased with the experience of working with Qwicap, and they're already planning to use it as the basis of their future projects. Though I visit their office daily to see how things are going, and to take a look at any problems they want a hand with, there have been remarkably few problems. They've found some minor bugs (minor in that there were small, simple fixes), a few more troublesome bugs, and they've needed a few new features. The more troublesome bugs were interesting, because they were the result of the team choosing a flow-control and markup handling strategy that had never occurred to me, and consequently it was a bit surprising to Qwicap, too. What stunned me was that it worked, and worked well in all but a few obscure cases. I had to sit down with the team-lead and walk through the relevant code several times before I even understood why it worked at all.

Qwicap 1.4a4 and 1.3.3 Released

Qwicap 1.4a4 and 1.3.3 have been released. The change history has all the details fit to print, but the main motivation behind the 1.4a4 release was to make available a fix for a race condition during session tear-down that's been in the Qwicap code base more-or-less forever. Originally, its effects were of little importance—mostly some exceptions showing-up in the logs—and it wasn't at all clear how the problems came about. With 1.4's support for dynamically generated downloadables, those effects started to have an actual impact by sometimes preventing access to the downloadables in the final page of the session. Fortunately, that made the nature of the problem more obvious and led to this bug fix.

Another old bug (or inadequately developed feature) was in the Qwicap.goodbye method which is the method that sends an application's final page to the user. The bug was that the method didn't inform Qwicap that the application was shutting-down, so Qwicap did nothing to help the application shutdown. Which was fine, for applications that invoked goodbye and gracefully returned their way back to, and out of, their main method, but those that didn't could inadvertently persist in the delusion that they were still operating normally by calling a variety of Qwicap methods that should have no role in, and could delay, the shutdown of an application (retrieving user input, sending pages to the user, etc.). Now that goodbye informs Qwicap that the application is shutting down, those methods that shouldn't be used during shutdown will throw a QwicapSessionDeathException when invoked, in order to help the application along in its shutdown activities. (By the way, QwicapSessionDeathException now includes a message that states why the session is dead, or dying.)

In addition, some enhancements have been made since 1.4a0, including: the addition of the Qwicap.getFloat method; moving the Downloadable class into the public API; changing all of the XML attribute manipulation methods to accept arbitrary objects as attribute values; and various improvements to the Javadocs. In addition, the Hunt The Wumpus example application has had its game logic further debugged, and it has been extended to support larger maps, upon request. The extra "config" page that allows larger maps to be specified, also provides a long-overdue example of Qwicap features like the high-level form manipulation API, while further illustrating how multi-page applications function.

Qwicap 1.3.3

Qwicap 1.3.3 was created by back-porting the bug fixes and minor enhancements that have accumulated in the 1.4 branch as of version 1.4a4. So, that race condition is gone, Qwicap.goodbye is beefed-up, Qwicap.getFloat exists, XML attribute values can be specified as arbitrary objects, QwicapSessionDeathExceptions are more informative, Qwicap.convertSubmitButtonsToInputs has had its documentation substantially expanded, various Context methods that didn't need to be public have been removed from the public API, the XML tag hierarchy validator includes document names in any exceptions it throws, all subclasses of Markup now include getMutable and getImmutable methods, the Context.getFile and Context.getExistingFile methods have been documented at long last, some useless code has been removed, and the Javadocs have suffered various improvements. Also, its version of Hunt The Wumpus received a bug fix, and therefore sucks less than it traditionally has done.

Which Reminds Me

While trying to implement the large map generator for the Qwicap 1.4a4 version of Hunt The Wumpus, I was frustrated for quite a while by my inability to come up with a simple implementation for the generator. The generator had a conceptually simple job to do, so it seemed the implementation should be simple, too. We all know such things frequently don't work out that way, but it was bugging me all the same. At one point, I considered using a cellular automaton to generate a hexagon-based map, and that sent me off to Wolfram's A New Kind of Science (NKS) pages, which was more convenient than winching down my copy of the book from its shelf and finding a spot for it on my desk. What I stumbled upon in doing so was WolframTones, a site that generates music using Wolfram's one-dimensional cellular automata. I found it very impressive, and a welcome demonstration of Wolfram's NKS ideas in an easily appreciated, real-world application - which is the sort of thing I regretted the absence of in the NKS book.

My only complaint with WolframTones is that it limits its compositions to thirty seconds. Ideally, it'd be possible to generate arbitrarily long compositions. What I wonder about long compositions is this: Will they seem to hold-together as well as the thirty-second ones usually do? Or does one's brain eventually reject a longer composition due to the extended lack of the sort of patterns designed into music by human intelligences? This would probably be more of an issue with some types of music than others; in particular, some flavors of jazz might not be a problem. And if it's not a problem for some types of music, what does that tell us about music, and will this eventually result in, say, a music generator box for use in stores that will continuously compose and play new music to save the stores the cost of playing background music that someone owns? As background music, shortcomings in the compositions would be masked by the lack of attention being paid to them. Which is fine for customers passing through, but what about the staff that has to hear the stuff all day? Will they eventually hunt down and destroy the box to save themselves? Or would it work well enough that it wouldn't drive people nuts, and, in fact, would periodically produce something really catchy? With every composition produced by an automaton chosen randomly from a set of around 4 billion possible automata, and with no direct access to the box to ask it to save its state, one would end-up in the frustrating situation of hearing a great song that no one had ever heard before, and knowing that you'd never, ever hear it again. That might promote some useful contemplation of the transitory nature of life, but I think, for the most part, it would just be a new way of pissing people off. Technology is great for that.

Qwicap Feature Summary

I've never succinctly described the features of the Qwicap web application API in one place. I'd assumed most of them would be obvious from reading the introductory documents, but that was probably a mistake on several levels. For one thing, without having been given a good reason to care, who's going to bother with the introductory documents? For another, my colleagues at UT Austin have been exposed to Qwicap for years, and I doubt that they're aware of a lot of these features. So here goes:

• Pure Java
• Open Source
• Designed for Developers, Unapologetically
• Discards the CGI Model
• Very Rich State, Totally Automatic
• Developer-Friendly APIs
• Security: Automatically Prevents Replay Attacks
• Security: Automatically Prevents Cross-Site Scripting Attacks
• Markup Integrity: Completely Standards Compliant
• Markup Integrity: Automatic HTML Encoding & Decoding
• Markup Integrity: Automatic Integrity Checking
• Automatic Input Verification, Error Messages and User Correction Requests
• Transparent File Uploads
• Transparent File Downloads
• High-Level XHTML Form Manipulation API
• Custom XML Engine - Fast and convenient, with thread-safe, fine-grained caching.
• Zero JavaScript
• No Metadata, No Configuration Files
• Simplified Web Application Deployment
• Not a Framework - It fits into your code, rather than vice versa.

For more details, start with the version of that list that includes explanatory text. If that doesn't frighten you off, there's plenty more reading material on the Qwicap site. If you have any questions, drop me a line.

And Now For Something Completely Different....

Those familiar with Christopher Alexander's work—most famously his book "A Pattern Language", which was frequently referenced by Richard P. Gabriel in his book "Patterns of Software"—may find interesting Alexander's response to criticism of his more recent work, "The Nature of Order, Book One". (Thanks to Carfree Times for the pointer.)

Oh, and a bit of computation and UT Austin history.... While going through the Austin History Center's archives a few weeks back, I came across the 1958 memo announcing the appointment of the first director of UT's Computation Center. (My first job with The University was in the Comp. Center, and I've never left.) One thing I found interesting about the memo is the reference to our big iron of the day, the IBM 650. (Thankfully, unlike most computer companies, IBM has not forgotten its old models.) Among other things, that beast could do 138,000 logical operations per minute. And you have to love that magnetic drum - echoes of Atanasoff & Berry. Also interesting is the fact that, 47 years later, Dr. Young is still at The University, in our Computer Science department.

Qwicap 1.4a0 Released

A holiday week's worth of coding has advanced Qwicap 1.4 from a scrap of dusty paper with a feature list scrawled across it, to an alpha-test release. As befits an alpha release, it is not feature complete, but it is stable and useful, and lays the groundwork for the rest of what I want to accomplish with 1.4.

The noteworthy new features in Qwicap 1.4a0 are:

• Support for dynamically generated downloads - Generate your downloadable material any way you like, convert it to a byte array, and invoke the new Qwicap.addDownloadable method with a name, and an optional MIME type. It returns a unique URI for the downloadable, which you embed in your page markup wherever you wish. Downloadables automatically become eligible for garbage collection when the page they were first used in ceases to be "in play". (In other words, when the method that manages that page ceases to be in the call chain.)
• Input is now accepted from links ("a href" and "area href"), so you don't have to use forms exclusively, anymore. Combine this feature with the support for dynamically generated downloads, and things like dynamic graphics with corresponding client-side image maps become simple to implement. (The revised version of the Hunt The Wumpus example application demonstrates just such a map.)
• Simplified deployment - Most applications can now use a generic "web.xml" file, as seen in the revised example apps. Qwicap will automatically discover the entry-point of your application code at runtime, and will also handle serving-out any web pages, style sheets, etc. that would otherwise be inaccessible due to the default servlet url path being "/". This means one less thing to learn/remember, and also keeps the URLs for your web applications pleasantly short.

Major features remaining to be implemented in version 1.4 include internationalization, and asynchronous prompts. The former represents unfamiliar ground for me, but it seems clear that there should be some way for developers to alter the default input-validation error messages that Qwicap automatically adds to web pages. Given Qwicap's facility with XML files, I'm inclined to have it retrieve those messages from a language-specific XML file. What I haven't wrapped my head around yet is how to cope with the various permutations of those messages, like pluralization. Perhaps a language-specific class is in order - a method could be invoked on it that is specific to particular problem (bad integer input, for instance), and the code could supply whatever markup it wished in response (it could be loaded from a language-specific XML file, or produced by any other means convenient to the international developer).

I have a much better handle on the idea of asynchronous prompts (my favorite would-be feature in 1.4), but the implementation details to be considered are numerous, and, in some cases, rather non-trivial. The idea, however, is simple enough: instead of blocking until user input arrives, as the current Qwicap.prompt methods do, an asynchronous prompt would be non-blocking, and would therefore allow the web application to continue processing after sending a page to the user. The page would presumably be set to refresh periodically, and would thereby receive status updates until the server had concluded its processing, at wich point the client and server become synchronized once again. Equivalent (or fancier) functionality can be created in many web application schemes already, but Qwicap's asynchronous prompts should be vastly simpler for developers to comprehend, use, and support, while also being more predictable/dependable in as much as the code remains strictly server-side.

Barring another week of holiday materializing out of thin air, I don't know when I'll have a chance to make headway on these outstanding features. (Things are busy at the office, and our current and pending web applications don't have a compelling need for either of those features.) Nonetheless, I would value insights on internationalization in the mean time. And anyone who can put a sexy interface on the new Hunt The Wumpus example would be appreciated. (At present its interface is pleasantly minimal and clean, but doesn't exactly manage to catch one's eye.)