To accomplish this I rewrote jpdv in Java.  It took me about 18 hours of solid coding, minus time for food of course.

The software can now generate both context-based and dependency-based vector spaces for Japanese text that has been pre-parsed with CaboCha.  It can also generate a similarity matrix for a given vector space using the cosine similarity measurement.  I still need to add a path selection function to throw out paths that are too long and a basis element selection function that determines which N basis elements to keep out of all those discovered, but I will add those to the next release.  I’m thinking of writing the path selection and basis element selection functions as Groovy scripts so that they can be supplied at run time.  This would allow for better customization of the system at run time for a given task.

More information can be found here and on the GitHub page.

Here is an example similarity matrix generated by the current version of jpdv:

However, jpdv still has a way to go before it works as promised.  So far the tool can parse CaboCha formatted XML and produce both a word co-occurrence based vector space and a slightly modified XML representation that better demonstrates the dependency relationships of the words in the text.  The next step is to use the dependency information to produce the vector space that I need.  Unfortunately, I only have until the end of next week to finish it, because I’m working on this as the final project in my NLP class this semester.  I also plan to use the vector spaces created by the tool to do word sense disambiguation for the SEMEVAL-2 shared task on Japanese WSD.

(The image included here was generated by jpdv as a LaTeX file from one of the sentences I’m using for testing.)

I’m working on a project for my NLP class that involves generating a semantic vector space for Japanese text, and I decided that this might be a good time to learn one of the LISP dialects.  I’ve been looking at Clojure for a while now, but I hadn’t taken the time to learn it before.  I must say, I’m quite impressed so far.  The fact that reading a Japanese XML document into a data structure “just works” without any tweaking is pretty nice.  I’m still coming to grips with functional programming, but I’m liking it so far.  And the best thing as far as I’m concerned is that Clojure code can easily be made massively parallel thanks to all of its data types being immutable (ala Erlang).  That will help me out a lot when I need to run my code on multi-core / multi-processor machines (which I often have to do because of the sheer amount of data being crunched.)