Here is an example:

var table = new dojox.layout.TableContainer({
    cols: 2,
    spacing: "5px"
});

table.addChild(
    new dijit.form.TextBox({
        id: "textbox",
        name: "textbox",
        label: "Text Box:",
        title: "Enter your text here!",
        colspan: 2
    })
);

table.startup();

In most browsers this works just fine, but I’ve had several problems related to Internet Explorer 7.  One of these is that column spanning doesn’t work the way it should in tables.  This is happening because IE7 uses the attribute “colSpan” to set the column span on a TD element.  However, all the other browsers I test with use “colspan”.  (Notice the capitalization!)  The Dijit code is building the HTML by modifying the dom nodes directly instead of using the innerHtml attribute.  This means that it is calling code that looks like this:

element.colspan = x;

When it needs to be calling code like this:

element.colSpan = x;

To get around this I used the dojo.query method to get a list of all elements with a colspan attribute and then set the colSpan attribute to match here.

Here is my fix:

dojo.addOnLoad(function() {
    if(dojo.isIE > 0) {
        console.info("Fixing IE colSpan issue.");
        // Fix colSpan attributes
        dojo.forEach(dojo.query("[colspan]"), function(tag) {
            tag["colSpan"] = tag.colspan;
        });
    }
});

This fix isn’t needed for IE 8 or IE 9, but it doesn’t hurt to leave the code in for those as well. You could change the if condition to check for dojo.isIE > 0 && dojo.isIE < 8 if you don’t want to run the code on IE 8 or newer.

For example, I can create an instance of CurrencyTextBox (a subclass of NumberTextBox) and call set(“value”,  2589632).  This will display the value as follows (assuming that my locale is set to en_US):

If I click in the box to edit the value, it changes back to just numbers and looks like this:

Exiting the field will reformat the displayed value to match my locale’s number formatting guidelines.  However, when I call the get(“value”) method on the widget, I will get back the number 2589632 instead of the formatted string.  This makes it easily to implement locale sensitive number formatting for web applications.

The problem I’ve been having is that percents aren’t handled quite right. Setting the “type” constraint to “percent” will properly display percent values so that setting the value of the widget to “0.02″ will display the string “2.00%”.

However, When the user clicks on the box to edit the value, the “2.00%” is replaced by it’s real value: “0.02″.

This causes two problems:

1. The user is expecting to enter a percentage, not a decimal value.  Users are often confused when the value they were expecting to see suddenly changes.  Also, a user might enter “2″ in the field, expecting it to show “2.00%”.  Instead it will show “200.00%”.
2. Truncation can occur when the number being edited has too many decimal places. Setting the “places” constraint on the widget to “2″ will display the value “0.0256″ as “2.56%”. However, when you edit the value it truncates back to 2 decimal places, showing you only “0.02″. Those extra two decimal places are now lost forever.

There is an open defect for this issue, but I couldn’t wait for them to fix it in the main dojo code.  To get around these issues I started digging into the NumberTextBox code and found that I can replace the parse() method on the NumberTextBox object to get around the problem. This solution also appends a percent symbol to the end of the string if one was left off, making it easier for a user to simply enter “10″ and have that converted to “10.00%” when they move to the next field.

Here is my solution:

function createPercentTextBox(name, title, extraParams) {
    if(!dojo.isObject(extraParams)) {
        extraParams = {};
    }

    var parseFunction = function(expression, options) {
        if(dojo.isString(expression)) {
            expression = dojo.trim(expression);
            var i = expression.lastIndexOf("%");
            if(i == -1) {
                expression = expression + "%";
            }
        }
        var value = dojo.number.parse(expression, { pattern: '0%' });
        return value;
    };

    var params = {
        id: name,
        name: name,
        label: title ? title + ":" : "",
        title: title ? title : "",
        constraints: {
            type: 'percent',
            places: 2
        },
        editOptions: {
            pattern: "##0.00%"
        },
        parse: parseFunction
    };

    dojo.mixin(params, extraParams);

    return new dijit.form.NumberTextBox(params);
}

Note: I’m using Dojo 1.6 for this example. Future versions of Dojo may resolve this issue more elegantly.

Klavaro de Esperanto

Today I wrote a simple Esperanto keyboard for the iPhone. It’s really just a little HTML page with some JavaScript and CSS to make it look like an iPhone app when you view it from an iPhone. It looks pretty bad if you view it outside of an iPhone though, because the buttons don’t line up right. Maybe I’ll make it more fully featured on non-iPhone browsers, but there are better tools out there for that already.

Since starting at my current company, I’ve been trying to learn the current version of RPG, which is RPG IV (aka RPGLE or ILE/RPG). Most of the running code that I see in RPG is actually written using RPG III syntax despite the fact that RPG IV has been out since 1994. This is mostly due to the fact that much of it was either generated programmatically or was written before 1994. My goal in learning RPG isn’t to become proficient enough to program RPG for a living, but instead to become proficient enough to help our organization transition their existing systems to more modern technologies as needed. However, my “outsider” view of RPG (coming from a Java/Perl/Ruby/etc background) has helped me do some things with it that long time RPG programmers might not think of trying to do. This is an example of that.

Like most large enterprise development shops these days, we use a Service Oriented Architecture (SoA) for most of our newer Java and .NET services. The problem is integrating our legacy RPG systems into these newer SoA systems. The cleanest approach is to use data queues (aka JMS queues). RPG can use these queues natively without having to make use of outside libraries as it would need to do in order to make socket calls. The problem is how to structure the data that is placed on the queue. The best thing to do would be to place a SOAP message – a type of XML document – onto the queue. This way the SoA systems could easily pick the message up off the queue and process it just like any other SOAP message they receive. However, this turns out to be problematic because it is difficult to produce properly formatted XML from RPG without the use of external libraries.

The good news is that there are products available that can help solve this problem. (For example, The RPG XML Suite and Scott Klement’s presentation on using free software to produce XML from RPG.) However, for various reasons we are unable to use these solutions, and since we have a large pool of RPG programmers who don’t yet know Java or .NET, there will still be a need to develop applications in RPG and have them interact via XML with our SoA applications.

To help solve this problem, and to help me learn RPG, I wrote a program that lets the programmer generate a sort of “mini DOM” for an XML document that can then produce a properly formatted and escaped XML string. Below is the code for the program. However, please note that I haven’t tested it throughly and since I am a novice RPG programmer it is bound to have bugs in it. I also used lots of “newer” RPG programming concepts, such as pointers, dynamic memory allocation, variable length null terminated strings, and subprocedures (which are like subroutines except that they have their own local variables and call stack that allow them to be run recursively.)

      // This program implements a simple XML DOM Builder.
      // Written by Andrew Young 

     D MAX_NODES       C                   65535

     D NODE_DEF        DS
     D   nodeId                       5U 0 INZ(1)
     D   nodeParent                   5U 0 INZ(0)
     D   nodeType                     1A   INZ('E')
     D   nodeName@                     *   INZ(*NULL)
     D   nodeNameLen                  5U 0 INZ(0)
     D   nodeValue@                    *   INZ(*NULL)
     D   nodeValLen                   5U 0 INZ(0)

     D NODE_LIST       DS
     D nodes                           *   DIM(MAX_NODES)
     D nodeEnd                        5U 0 INZ(1)

     D buildXML        PR         65535A   VARYING
     D                                 *   VALUE

     D createNode      PR              *
     D                                5U 0 VALUE

     D setNodeName     PR             3U 0
     D                                 *   VALUE
     D                            65535A   VALUE VARYING

     D getNodeName     PR         65535A   VARYING
     D                                 *   VALUE

     D setNodeValue    PR             3U 0
     D                                 *   VALUE
     D                            65535A   VALUE VARYING

     D getNodeValue    PR         65535A   VARYING
     D                                 *   VALUE

     D getNode         PR              *
     D                                5U 0 VALUE

     D getParentNode   PR              *
     D                                 *   VALUE

     D nodeTest        PR             3U 0

     D printNode       PR             3U 0
     D                                 *   VALUE

     D displayString   PR             3U 0
     D                            65535A   VARYING VALUE

     D escapeXML       PR         65535A   VARYING
     D                            65535A   VARYING VALUE

     D cleanup         PR

      /FREE
       nodeTest();
       cleanup();
       EXSR doPause;
       *INLR = *ON;
       RETURN;
      /END-FREE

      * This method pauses the output
     C     doPause       BEGSR
     C                   DSPLY                   PAUSE             1
     C                   ENDSR

     P cleanup         B
     D i               S             10I 0 INZ(1)
     D ret             S              3U 0
     D node@           S               *
     D node            DS                  LIKEDS(NODE_DEF) BASED(node@)
      /FREE
       DOW i < nodeEnd;
         node@ = getNode(i);
         IF node@ <> *NULL;
           IF node.nodeName@ <> *NULL;
             DEALLOC node.nodeName@;
             node.nodeName@ = *NULL;
           ENDIF;
           IF node.nodeValue@ <> *NULL;
             DEALLOC node.nodeValue@;
             node.nodeValue@ = *NULL;
           ENDIF;
           DEALLOC node@;
           node@ = *NULL;
         ENDIF;
         i = i + 1;
       ENDDO;
      /END-FREE
     P cleanup         E

     P nodeTest        B
     D                 PI             3U 0

     D node@           S               *
     D node            DS                  LIKEDS(NODE_DEF) BASED(node@)
     D nodeIndex       S              5U 0
     D ret             S              3U 0

      /FREE
       node@ = createNode(0);
       ret = setNodeName(node@:'documents');

       node@ = createNode(node.nodeId);
       ret = setNodeName(node@:'document');

       node@ = createNode(node.nodeId);
       ret = setNodeName(node@:'name');
       node.nodeType = 'A';
       ret = setNodeValue(node@:'My First Document');

       node@ = getParentNode(node@);

       node@ = createNode(node.nodeId);
       node.nodeType = 'T';
       ret = setNodeValue(node@:'Some text.');

       node@ = getParentNode(node@);
       node@ = getParentNode(node@);

       node@ = createNode(node.nodeId);
       ret = setNodeName(node@:'document');

       node@ = createNode(node.nodeId);
       node.nodeType = 'A';
       ret = setNodeName(node@:'name');
       ret = setNodeValue(node@:'A Second Document <>&"&"><');

       node@ = getParentNode(node@);

       node@ = createNode(node.nodeId);
       ret = setNodeName(node@:'field');

       node@ = createNode(node.nodeId);
       node.nodeType = 'A';
       ret = setNodeName(node@:'name');
       ret = setNodeValue(node@:'url');

       node@ = getParentNode(node@);

       node@ = createNode(node.nodeId);
       node.nodeType = 'T';
       ret = setNodeValue(node@:'http://www.google.com/');

       node@ = getParentNode(node@);
       node@ = getParentNode(node@);

       node@ = createNode(node.nodeId);
       ret = setNodeName(node@:'description');

       node@ = createNode(node.nodeId);
       node.nodeType = 'T';
       ret = setNodeValue(node@:'Here is a link: ');

       node@ = getParentNode(node@);

       node@ = createNode(node.nodeId);
       setNodeName(node@:'a');

       node@ = createNode(node.nodeId);
       node.nodeType = 'A';
       setNodeName(node@:'href');
       setNodeValue(node@:'http://www.google.com/');

       node@ = getParentNode(node@);

       node@ = createNode(node.nodeId);
       node.nodeType = 'T';
       setNodeValue(node@:'Google');

       node@ = getParentNode(node@);

       node@ = getParentNode(node@);

       node@ = createNode(node.nodeId);
       node.nodeType = 'T';
       setNodeValue(node@:', and here is some text that needs to be escaped: '+
           '"<>"&><&');

       nodeIndex = 1;
       DOW nodeIndex < nodeEnd;
         node@ = getNode(nodeIndex);
         ret = printNode(node@);
         nodeIndex = nodeIndex + 1;
       ENDDO;

       ret = displayString(buildXML(getNode(1)));

       RETURN 0;
      /END-FREE
     P nodeTest        E

       // This routine prints out the value of the current node
     P printNode       B
     D                 PI             3U 0
     D node@                           *   VALUE

     D node            DS                  LIKEDS(NODE_DEF) BASED(node@)
     D i               S              3U 0
      /FREE
       i = displayString('Node: ' + %char(node.nodeId));
       i = displayString('  Parent: ' + %char(node.nodeParent));
       i = displayString('    Type: ' + node.nodeType);
       i = displayString('    Name: ' + getNodeName(node@));
       i = displayString('   Value: ' + getNodeValue(node@));
       RETURN 0;
      /END-FREE
     P printNode       E

       // This routine write the output to standard out using DSPLY.
     P displayString   B
     D                 PI             3U 0
     D output                     65535A   VARYING VALUE

     D message         S             52A   VARYING
     D i               S             10I 0 INZ(0)
     D j               S             10I 0 INZ(0)
      /FREE
       i = 1;
       j = %len(%trim(output));
       DOW i < j;
         message = %subst(output:i);
         DSPLY message;
         i = i + 52;
       ENDDO;
       RETURN 0;
      /END-FREE
     P displayString   E

     P getNode         B
     D                 PI              *
     D nodeIndex                      5U 0 VALUE
      /FREE
       RETURN nodes(nodeIndex);
      /END-FREE
     P getNode         E

     P getParentNode   B
     D                 PI              *
     D node@                           *   VALUE

     D node            DS                  LIKEDS(NODE_DEF) BASED(node@)
     D parentNode@     S               *
      /FREE
       IF node.nodeParent > 0;
         parentNode@ = getNode(node.nodeParent);
       ENDIF;
       RETURN parentNode@;
      /END-FREE
     P getParentNode   E

     P createNode      B
     D                 PI              *
     D parentIndex                    5U 0 VALUE

     D nodeIndex       S              5U 0
     D node@           S               *
     D node            DS                  LIKEDS(NODE_DEF) BASED(node@)
     D i               S              3U 0
      /FREE
       node@ = %alloc(%size(NODE_DEF));
       node.nodeId = nodeEnd;
       node.nodeType = 'E';
       node.nodeParent = parentIndex;
       nodes(nodeEnd) = node@;
       nodeEnd = nodeEnd + 1;
       RETURN node@;
      /END-FREE
     P createNode      E

     P setNodeName     B
     D                 PI             3U 0
     D node@                           *   VALUE
     D value                      65535A   VALUE VARYING

     D node            DS                  LIKEDS(NODE_DEF) BASED(node@)
     D i               S              3U 0
      /FREE
       IF node.nodeName@ <> *NULL;
         // Deallocate currently used space
         DEALLOC node.nodeName@;
         node.nodeName@ = *NULL;
       ENDIF;
       // Allocate new space
       node.nodeNameLen = %len(value);
       node.nodeName@ = %alloc(node.nodeNameLen+1);
       %str(node.nodeName@:node.nodeNameLen+1) =
           %subst(value:1:node.nodeNameLen);
       RETURN 0;
      /END-FREE
     P setNodeName     E

     P getNodeName     B
     D                 PI         65535A   VARYING
     D node@                           *   VALUE

     D node            DS                  LIKEDS(NODE_DEF) BASED(node@)
     D value           S          65535A   VARYING
     D i               S              3U 0
      /FREE
       IF node.nodeName@ <> *NULL;
         value = %str(node.nodeName@:node.nodeNameLen+1);
       ENDIF;
       RETURN value;
      /END-FREE
     P getNodeName     E

     P setNodeValue    B
     D                 PI             3U 0
     D node@                           *   VALUE
     D value                      65535A   VALUE VARYING

     D node            DS                  LIKEDS(NODE_DEF) BASED(node@)
     D i               S              3U 0
      /FREE
       IF node.nodeValue@ <> *NULL;
         // Deallocate currently used space
         DEALLOC node.nodeValue@;
         node.nodeValue@ = *NULL;
       ENDIF;
       // Allocate new space
       node.nodeValLen = %len(value);
       node.nodeValue@ = %alloc(node.nodeValLen+1);
       %str(node.nodeValue@:node.nodeValLen+1) =
           %subst(value:1:node.nodeValLen);
       RETURN 0;
      /END-FREE
     P setNodeValue    E

     P getNodeValue    B
     D                 PI         65535A   VARYING
     D node@                           *   VALUE

     D node            DS                  LIKEDS(NODE_DEF) BASED(node@)
     D value           S          65535A   VARYING
      /FREE
       IF node.nodeValue@ <> *NULL;
         value = %str(node.nodeValue@:node.nodeValLen+1);
       ENDIF;
       RETURN value;
      /END-FREE
     P getNodeValue    E

     P buildXML        B
     D                 PI         65535A   VARYING
     D node@                           *   VALUE

     D childNode@      S               *
     D outp            S          65535A   VARYING
     D i               S             10I 0
     D node            DS                  LIKEDS(NODE_DEF) BASED(node@)
     D childNode       DS                  LIKEDS(NODE_DEF) BASED(childNode@)

      /FREE
       // This function writes out the XML structure for the current node and
       //     its children.
       IF node.nodeType = 'E';
         // Write start of element
         outp = outp + '<' + %trim(getNodeName(node@));
         // Look for child attributes
         i = 1;
         DOW i < nodeEnd;
           childNode@ = getNode(i);
           IF childNode.nodeType = 'A' AND childNode.nodeParent = node.nodeId;
             outp = outp + ' ' + %trim(getNodeName(childNode@)) + '="' +
                 escapeXML(getNodeValue(childNode@)) + '"';
           ENDIF;
           i = i + 1;
         ENDDO;
         outp = outp + '>';
         // Look for child elements and text nodes
         i = 1;
         DOW i < nodeEnd;
           childNode@ = getNode(i);
           IF childNode.nodeParent = node.nodeId;
             IF childNode.nodeType = 'T';
               outp = outp + escapeXML(getNodeValue(childNode@));
             ELSEIF childNode.nodeType = 'E';
               outp = outp + buildXML(childNode@);
             ENDIF;
           ENDIF;
           i = i + 1;
         ENDDO;
         outp = outp + '';
       ENDIF;  // node.nodeType = 'E'

       RETURN outp;

      /END-FREE

     P buildXML        E

     P escapeXML       B
     D                 PI         65535A   VARYING
     D input                      65535A   VARYING VALUE

     D inputSize       S              5U 0 INZ(0)
     D i               S              5U 0 INZ(1)
     D currentChar     S              1A   INZ(*BLANKS)
     D outputChar      S              6A   VARYING
     D output          S          65535A   VARYING
      /FREE
       inputSize = %len(input);
       DOW i <= inputSize;
         currentChar = %subst(input:i:1);
         outputChar = currentChar;
         IF currentChar = '&';
           outputChar = '&amp;';
         ELSEIF currentChar = '<';
           outputChar = '&lt;';
         ELSEIF currentChar = '>';
           outputChar = '&gt;';
         ELSEIF currentChar = '"';
           outputChar = '&quot;';
         ENDIF;
         output = output + outputChar;
         i = i + 1;
       ENDDO;
       RETURN output;
      /END-FREE
     P escapeXML       E



Huffman coding is also a variable length encoding. It uses the distribution of bytes in a given document to assign shorter bit sequences to more common bytes and longer bit sequences to less common bytes. This means that the most common bytes only take up a few bits (as few as 1 or 2), whereas very uncommon bytes can take up more than 8 bits (and thus take more space to store than they did in the original document.) However, in the worst case the short bit sequences and long bit sequences will even out and the final product will be a file of the same size as the original. In practice the final product is usually a file that is much smaller than the original.

Because the Huffman coding algorithm doesn’t know about multibyte sequences in UTF-8 documents, it doesn’t take into account the fact that certain bytes (those with the high bit set) only occur together with other bytes and therefore their distribution is not independent. This means that the entropy of the file is actually smaller than the algorithm thinks it is and therefore the algorithm needs fewer bits to represent the file than it realizes. In order to make use of multibyte sequences – not only in UTF-8, but in other multibyte encodings as well – I modified the algorithm to think in terms of Unicode characters instead of bytes. By converting the file from its native encoding into a stream of characters before performing the Huffman coding calculations, the entropy of the file is reduced and the file can be stored more compactly.

Since a file written in an Asian language will consist mostly of Unicode characters that translate to two or more bytes in UTF-8, it stands to reason that the standard Huffman coding algorithm won’t produce as compact a file as a Huffman coding algorithm that works in terms of characters instead of in terms of bytes. To test this I compressed a Chinese text file with both the standard and improved algorithms and compared the results. Although my encoding method didn’t save as much space as other common compression utilities such as GZip, I still saw an improvement when encoding characters instead of raw bytes.

Compressing the raw UTF-8 bytes of a 69KB Chinese file created a 48KB file. Compressing the underlying characters created a 32KB file, which is a significant improvement.  Likewise, starting with a 46KB GBK encoded version of the same file created a 34KB file when compressing the raw bytes and a 32KB file when compressing characters instead.  (As a matter of fact, compressing characters creates the same file in both cases because the input characters are converted to Unicode internally.)

Here is the raw data from the experiment:

Note: The “Huffman Bytes” files were created using the same algorithm as the “Huffman Characters” files, except the input encoding was set to ISO-8859-1 so that the algorithm would ignore multibyte sequences in the input files and work purely with 8-bit bytes. This produces the same result that the original Huffman coding would produce.

To accomplish this I rewrote jpdv in Java.  It took me about 18 hours of solid coding, minus time for food of course.

The software can now generate both context-based and dependency-based vector spaces for Japanese text that has been pre-parsed with CaboCha.  It can also generate a similarity matrix for a given vector space using the cosine similarity measurement.  I still need to add a path selection function to throw out paths that are too long and a basis element selection function that determines which N basis elements to keep out of all those discovered, but I will add those to the next release.  I’m thinking of writing the path selection and basis element selection functions as Groovy scripts so that they can be supplied at run time.  This would allow for better customization of the system at run time for a given task.

More information can be found here and on the GitHub page.

Here is an example similarity matrix generated by the current version of jpdv:

However, jpdv still has a way to go before it works as promised.  So far the tool can parse CaboCha formatted XML and produce both a word co-occurrence based vector space and a slightly modified XML representation that better demonstrates the dependency relationships of the words in the text.  The next step is to use the dependency information to produce the vector space that I need.  Unfortunately, I only have until the end of next week to finish it, because I’m working on this as the final project in my NLP class this semester.  I also plan to use the vector spaces created by the tool to do word sense disambiguation for the SEMEVAL-2 shared task on Japanese WSD.

(The image included here was generated by jpdv as a LaTeX file from one of the sentences I’m using for testing.)

I’m working on a project for my NLP class that involves generating a semantic vector space for Japanese text, and I decided that this might be a good time to learn one of the LISP dialects.  I’ve been looking at Clojure for a while now, but I hadn’t taken the time to learn it before.  I must say, I’m quite impressed so far.  The fact that reading a Japanese XML document into a data structure “just works” without any tweaking is pretty nice.  I’m still coming to grips with functional programming, but I’m liking it so far.  And the best thing as far as I’m concerned is that Clojure code can easily be made massively parallel thanks to all of its data types being immutable (ala Erlang).  That will help me out a lot when I need to run my code on multi-core / multi-processor machines (which I often have to do because of the sheer amount of data being crunched.)