Twenty 42s

My recent work on Qwicap has concentrated on character set issues, first concerning the loading of XML/XHTML markup, and now concerning receiving input from browsers. I've learned a lot. For instance, unlike output sent to a client, input sent from the client in HTTP includes no provision for identifying the character set. You can, of course, add an "accept-charset" attribute to the "form" elements in your web application's pages, but there's no guarantee that the client will support the character set you specify, unless you confine yourself to ISO-8859-1. You could try inferring the client's supported character sets from the "accept-charset" headers of its HTTP requests, but then you would almost always be lied to, because most browsers throw in a wildcard that means that they accept all official character sets. Since there are currently 254 of those, what do you suppose the chances are that a browser sending that wildcard really supports them all? Zilch, in my tests.

The icing on this particular cake is that a Java application server is required to provide your web application with the client's input in the form of Java String objects, via the ServletRequest class. How can the application server reliably translate the input bytes it receives into the Unicode characters that will populate those String objects? Not knowing the character set used to encode the characters represented by those bytes, it can't, but the API requires that it blunder ahead with String creation regardless.

Twenty 42s

All of these character set torments led me to take a close look at the first 65,536 characters of Unicode, you know, for fun. Did you know that among those characters there are twenty ways to represent the number, just to pick one arbitrarily, forty-two? There are. And that's assuming that you don't mix-and-match amongst languages. If you do, there are 400 representations. Specifically, in those first 65,536 Unicode characters, there are twenty discrete groups of characters for representing decimal digits. Your browser probably doesn't even have a font containing glyphs for all of them, but here are those twenty forty-twos, along with their character codes expressed in hexadecimal:

４２	0xFF14 and 0xFF12
᠔᠒	0x1814 and 0x1812
៤២	0x17E4 and 0x17E2
፬፪	0x136C and 0x136A
၄၂	0x1044 and 0x1042
༤༢	0x0F24 and 0x0F22
໔໒	0x0ED4 and 0x0ED2
๔๒	0x0E54 and 0x0E52
൪൨	0x0D6A and 0x0D68
೪೨	0x0CEA and 0x0CE8
౪౨	0x0C6A and 0x0C68
௪௨	0x0BEA and 0x0BE8
୪୨	0x0B6A and 0x0B68
૪૨	0x0AEA and 0x0AE8
੪੨	0x0A6A and 0x0A68
৪২	0x09EA and 0x09E8
४२	0x096A and 0x0968
۴۲	0x06F4 and 0x06F2
٤٢	0x0664 and 0x0662
42	0x0034 and 0x0032

By the way, the Java methods for converting from strings to binary numeric values, like Integer.parseInt, accept all of those character sequences as valid inputs, as well as all of the cross-language mixes. Worth thinking about if you have a web application that accepts numeric inputs. And the fun doesn't stop with decimal digits; for instance, you can find a replication of most of ASCII between 0xFF00 and 0xFF60.

What's the point? Nothing in particular, except to reinforce the general warning that one is ignorant of character encoding issues at one's peril. (Before you ask, no, neither my software's handling, nor my understanding, of these issues is perfect; that's why I know a little something about the "peril" part.)