Huffman coding is also a variable length encoding. It uses the distribution of bytes in a given document to assign shorter bit sequences to more common bytes and longer bit sequences to less common bytes. This means that the most common bytes only take up a few bits (as few as 1 or 2), whereas very uncommon bytes can take up more than 8 bits (and thus take more space to store than they did in the original document.) However, in the worst case the short bit sequences and long bit sequences will even out and the final product will be a file of the same size as the original. In practice the final product is usually a file that is much smaller than the original.

Because the Huffman coding algorithm doesn’t know about multibyte sequences in UTF-8 documents, it doesn’t take into account the fact that certain bytes (those with the high bit set) only occur together with other bytes and therefore their distribution is not independent. This means that the entropy of the file is actually smaller than the algorithm thinks it is and therefore the algorithm needs fewer bits to represent the file than it realizes. In order to make use of multibyte sequences – not only in UTF-8, but in other multibyte encodings as well – I modified the algorithm to think in terms of Unicode characters instead of bytes. By converting the file from its native encoding into a stream of characters before performing the Huffman coding calculations, the entropy of the file is reduced and the file can be stored more compactly.

Since a file written in an Asian language will consist mostly of Unicode characters that translate to two or more bytes in UTF-8, it stands to reason that the standard Huffman coding algorithm won’t produce as compact a file as a Huffman coding algorithm that works in terms of characters instead of in terms of bytes. To test this I compressed a Chinese text file with both the standard and improved algorithms and compared the results. Although my encoding method didn’t save as much space as other common compression utilities such as GZip, I still saw an improvement when encoding characters instead of raw bytes.

Compressing the raw UTF-8 bytes of a 69KB Chinese file created a 48KB file. Compressing the underlying characters created a 32KB file, which is a significant improvement.  Likewise, starting with a 46KB GBK encoded version of the same file created a 34KB file when compressing the raw bytes and a 32KB file when compressing characters instead.  (As a matter of fact, compressing characters creates the same file in both cases because the input characters are converted to Unicode internally.)

Here is the raw data from the experiment:

Note: The “Huffman Bytes” files were created using the same algorithm as the “Huffman Characters” files, except the input encoding was set to ISO-8859-1 so that the algorithm would ignore multibyte sequences in the input files and work purely with 8-bit bytes. This produces the same result that the original Huffman coding would produce.

To accomplish this I rewrote jpdv in Java.  It took me about 18 hours of solid coding, minus time for food of course.

The software can now generate both context-based and dependency-based vector spaces for Japanese text that has been pre-parsed with CaboCha.  It can also generate a similarity matrix for a given vector space using the cosine similarity measurement.  I still need to add a path selection function to throw out paths that are too long and a basis element selection function that determines which N basis elements to keep out of all those discovered, but I will add those to the next release.  I’m thinking of writing the path selection and basis element selection functions as Groovy scripts so that they can be supplied at run time.  This would allow for better customization of the system at run time for a given task.

More information can be found here and on the GitHub page.

Here is an example similarity matrix generated by the current version of jpdv:

However, jpdv still has a way to go before it works as promised.  So far the tool can parse CaboCha formatted XML and produce both a word co-occurrence based vector space and a slightly modified XML representation that better demonstrates the dependency relationships of the words in the text.  The next step is to use the dependency information to produce the vector space that I need.  Unfortunately, I only have until the end of next week to finish it, because I’m working on this as the final project in my NLP class this semester.  I also plan to use the vector spaces created by the tool to do word sense disambiguation for the SEMEVAL-2 shared task on Japanese WSD.

(The image included here was generated by jpdv as a LaTeX file from one of the sentences I’m using for testing.)

I’m working on a project for my NLP class that involves generating a semantic vector space for Japanese text, and I decided that this might be a good time to learn one of the LISP dialects.  I’ve been looking at Clojure for a while now, but I hadn’t taken the time to learn it before.  I must say, I’m quite impressed so far.  The fact that reading a Japanese XML document into a data structure “just works” without any tweaking is pretty nice.  I’m still coming to grips with functional programming, but I’m liking it so far.  And the best thing as far as I’m concerned is that Clojure code can easily be made massively parallel thanks to all of its data types being immutable (ala Erlang).  That will help me out a lot when I need to run my code on multi-core / multi-processor machines (which I often have to do because of the sheer amount of data being crunched.)

Installation Steps:

1. Install XCode and the Developer tools off of your Mac OSX install CD if you haven’t already.
2. Install MacPorts if you haven’t already.
3. Download this tar file: cabocha-macports.tar.gz
4. Extract the tar file.
5. Run install.sh from inside the directory that the tar file created.

The install script first installs the following two programs via MacPorts:

• ChaSen (a Japanese part of speech and morphological analyzer)
• MeCab (another Japanese part of speech and morphological analyzer that uses )

Then the install script will download and install the following programs, patching them as necessary:

• TinySVM (A Support Vector Machines library)
• YamCha (A text chunker)
• CaboCha (The Japanese dependency parser we’re interested in)

All off the applications will be installed into the MacPorts directory structure in /opt/local and they should therefore work out of the box without a problem. When you run CaboCha you can choose whether to use ChaSen or MeCab for part of speech and morphological analysis. The default is normally ChaSen, but it seems that ChaSen is no longer being actively maintained. Also, MeCab uses Conditional Random Fields (CRF) which generally perform better than the Hidden Markov Models (HMM) used by ChaSen. Because of this, my install script configures CaboCha to use MeCab as its default part of speech and morphological analyzer. If you want to use ChaSen instead, you can pass CaboCha the “-a chasen” flag or change the install.sh script to use ChaSen instead of MeCab.

Note: This script works for me in Snow Leopard, but I give no guarantees for how well it will work for you. Also, be aware that all of these utilities work with EUC-JP encoded text, not UTF-8 encoded text, so you will need to configure your terminal to use EUC-JP. If you’re using the standard Mac OSX Terminal app, the easiest way to do this is:

1. Open the Terminal app.
2. Click on the ‘Terminal’ menu and then click on ‘Preferences’.
3. Click on ‘Settings’.
4. Highlight your favorite terminal profile (I use ‘Pro’ myself).
5. Click on the button with the gear on it and select ‘Duplicate Settings’.
6. Name the new profile something useful, like ‘EUC-JP’.
7. Click on the new profile and then click on the ‘Advanced’ tab.
8. Change the ‘Character encoding’ drop down to ‘Japanese (EUC)’.
9. Close the preferences window and quit the terminal app to make sure it picks up the changes.

To use your new profile, relaunch the terminal app, click on the ‘Shell’ menu, select ‘New Window’, and then select ‘EUC-JP’ (or whatever you called your new profile). This will open a new terminal window that is using EUC-JP for its character encoding. You might also want to mess with the fonts for your new profile to make the Japanese characters easier to read.

The following websites were especially helpful in showing me what files to modify in order to get TinySVM installed properly:

• Research/Cabocha+Yamcha+TinySVM+MeCabの導入
• Mac OS X Pantherに言語処理ツール

When I went back to school in the summer of 2004, I declared myself to be double majoring in computer science and Japanese. However, I discovered two years later – after finishing four semesters of Japanese and a handful of the undergraduate CS courses – that I wasn’t devoted enough to either of those majors to be able to complete a degree while working full time and raising a family. Luckily though, I took an introductory linguistics class in 2006 and discovered that I really enjoyed linguistics, so I switched my major at first to Japanese and linguistics, and then to English and linguistics, and then finally to just linguistics. But that didn’t mean that I wasn’t still interested in CS and Japanese. In fact, I found that my linguistic interests gravitated towards computational linguistics and Japanese linguistics.

I’ve had a long time interest in studying abroad in Japan, which recently led me to start looking for linguistics programs in Japan. Unfortunately, I don’t speak nearly enough Japanese to attend a Japanese university where all the courses are taught in Japanese. However, a handful of universities in Japan offer “international” degree programs now where the language of study is English. The Japanese government is providing grants and scholarships to promote these programs – especially at the graduate level – because they want to bring more international students into Japanese research universities.

While digging through lists of international graduate programs, I came across the program offered by Keio University’s Graduate School of Science and Technology. The admissions information mentioned the need to seek out an advisor before applying to the university and it provided a convenient link to look through the list of professors. While looking through this list, something interesting caught my eye:

Ohara, Kyoko Hirose Associate Professor

Research Area:

Cognitive Linguistics / Corpus Linguistics / Lexical Semantics / Contrastive analysis of Japanese and English / Natural Language Processing

While I had seen plenty of professors with “Natural Language Processing” listed before, they were usually researching machine translation, speech recognition, or some sort of human-machine interface, usually involving robotics. However, Ohara-sensei’s research is exactly the sort of thing that I am currently working on. In fact, she is in the process of producing a Japanese version of FrameNet, which is the annotated corpus I am currently using for my honors thesis research. Also, I discovered that Keio is Japan’s top private university and was the first university in Japan, opening a few years before the Meiji revolution occurred (it is named for the era that came before the Meiji era, which was called the Keio era).

I emailed Ohara-sensei and spoke with my current advisor about the possibility of studying at Keio after I finish my masters degree. So far it looks very promising. My main deficiency is my lack of a solid understanding of Japanese. Although the program is in English, and my dissertation would be in English, I would still be studying the linguistics of Japanese, and therefore I need a much better grasp of the Japanese language. However, it will be at least 3 years before I finish my masters degree, which will give me plenty of time to improve my Japanese to a sufficient level. I also plan to do my master’s thesis on a topic involving Japanese linguistics that will allow me to make use of the Japanese FrameNet project, which should hopefully help my chances of being admitted to Keio.

After confirming that this would indeed something that I could actually do, I spoke with my wife and daughter about it and they are both onboard with the idea, as long as it is actually feasible. I’ve discovered that the Japanese government has a scholarship available to international graduate students (the same one I mentioned above) that will pay for 3 years of tuition (which is the normal length of time that a PhD takes at Keio) and provide the student with about a $1,000 a month stipend. The scholarship is competitive, but I will try to get it and hope for the best. My wife was also won over by the fact that the school mascot is a unicorn!

The Keio Mascot

The campus I would be attending is in Yokohama, so housing should be less expensive than in Tokyo proper. Also, we hope to teach my daughter as much Japanese as possible before then and then put her in the public school system so that she can experience the culture more closely and hopefully learn to speak fluent Japanese.

Of course, this is all very preliminary at the moment, and nothing is certain. However, I’m quite serious about it and I really think it is going to happen. I’m really looking forward to spending a few years in Japan. It would be a once-in-a-lifetime experience!