I have written some macros to do the conversion, but they were scary looking and fragile. Fortunately, I now found a tool that does the same job better and with more flexibility. DocConverter by Danny Brewer and Dan Horwood allows to convert a whole directory of files at a time from any to any OpenOffice-understood format.

I have just converted more than a thousand documents from doc to txt without any problems.  Actually, I had a small problem, but it was my fault. I had some corrupted files that OO would not open and that was breaking DocConverter and throwing some ugly looking Basic runtime error. I had to delete the problem files, kill the Open Office (stop macro did not) and rerun the tool. Otherwise, it just run.

In 19th century, there lived a famous rabbinic scholar Ben Ish Chai, who among other writings had two collections of letters. Ben Ish Chai claimed that only one collection was his and that the other one was somebody else’s, found by him. Modern scholars thought both collections were his, but could not prove it conclusively as the style of writing was different.

Machine Learning to the rescue! In 2004, Moshe Koppel and Jonathan Schler have discovered that it may help to look not at the writing style differences (as the style may have been faked), but rather at how deep those differences were. For example, an author could fake a stylistic mismatch by consciously avoiding favorite words, but would still write in long overrun sentences, use more of passive verb forms or display many other measurable behaviours.

So, if the most obvious differences were removed one by one, the speed at which the rest of the features would look identical could be a good indicator. They called this technique unmasking and the mistery of Ben Ish Chai was solved for good.

I think what impressed me here was not the clever math. The whole field of determining authorship is based on clever math. It is rather the fact that the math was looking at hints within the hints of the language - the invisible aspects that become noticeable only after the eye learns to see beyond what the most obvious reality offers. I cannot explain it better, but to me it has a special elegance that just counting the words and sentence lengths does not offer.

Dance descriptions, while not really in English do have a regular structure and can be thought of as a sub-language with full set of syntactic, semantic and pragmatic levels.

There are basic words of the language (move names), correct ways of putting them in a sentence (a routine) and all the way up to good flowing text (classes that do not hurt the participants).

I was thinking about relationship between dance instructions and computational linguistics in context of Scottish Country Dancing for at least a year. My imagined benefits were that codified dance instructions would allow for automatic dance animations, superior teacher aids and other applications that currently require a lot of sweat and toil. Dance evening programmes that are currently put together manually for each event, could be assisted with automated evaluation pointing out awkward sequences of dances.

Unfortunately, my attempts at explaining the connection made no sense to the people around me. So, I was ecstatic to discover that such a link was already discovered by others before me.

Adam Bull, more than 10 years ago, has tried to apply principles of computational linguistics to Aerobics for his MPhil degree in the paper entitled The formal description of aerobic dance exercise - a corpus-based computational linguistics approach. While, the report is not complete, it puts down many of the same arguments I have tried myself.

Unfortunately, the electronic copy of the document was not available. After some effort, I got in touch with Adam and he send me the copy of the report with the permission to distribute. I have put a copy of it on my own server.

I hope his research will get rediscovered and improved upon. That way when I get some time to apply my own PhD skills to Scottish Country Dancing, there will be more than one person on whose shoulders I would be able to stand.

The problem is the user configuration file gate.xml that is stored in the shared location, usually home directory. On Windows, that is C:\Documents and Settings\[ProfileName]\.

One of those settings was pointing to where the plugins were loaded from and was still referring to GATE 3.1’s locations. That caused NullPointerExceptions in the GATE and everything was breaking from that point on.

I found this by using FileMon, but later realised that it might have been done easier by changing runtime.spawn property to false in GATE’s build.xml file that is used to start the program. Using ant to start a program is a new one for me, but I guess it makes sense in some cases. Setting the property to false shows the startup messages and the exception that the wrong directories cause.

I have deleted the old gate.xml and gate.session files in my home directory and everything started to work. Back to actually trying to use the software.

So, I was really happy to discover an audio course Story of Human Language from The Teaching Company taught by John McWhorter. It is quite long a covers a lot of material, but - apart from some overly long parts on universal language - it is really interesting and Professor McWhorter is a great presenter.

I actually had a chance to listen to both an audio version of the course and to see some of it on DVD. Personally, I prefer just audio for several reason.

Firstly, I can listen to the course on my MP3 player when I am walking or doing chores. Video version requires allocating dedicated time, which for such a long course would be difficult.

Secondly, I actually found visual part of the presentation quite boring - for the most part professor is just standing behind the lectern and talks from his notes. In fact, I found the visual part distracted me from the really great and expressive rhetorics.

There was a number of great section in the course, but I found the one explaining language structure of Arabic and Chinese particularly interesting. He talked about Arabic first and I was all keen to learn that language. Then, he switched over to Chinese and I found it even more fascinating. And then, there were comparisons of languages and his cat. This has to be heard to be believed.

The course is obviously available for purchase, but it is also found in quite a few libraries. If you do borrow it from the library, try requesting all volumes at once. I only requested one volume and it was quite annoying to then have to wait a long time for the rest of the course arrive. This is another way I knew for myself that the course was enjoyable, as I had plenty of other audio material to listen to otherwise.

Bureaucracy often has its place and reason. Laws protect exploitable minorities; procedures serve to avoid known problems; cross-referencing forms are filled in triplicate to allow for audit and protection against falsification. The problem is not the bureaucracy as such but rather the fact that it eventually outgrows any individual person’s ability to comprehend it. At that point, only dedicated specialists can understand the process and the rest of us have to offer sacrifices to those acolytes in hopes of beneficial results.

Enter computers. It turns out that computers can bring the complexity of information down, back within the reach of the non-specialist. The more bureaucratic a processes, the better a computer can figure it out. What is a mind-numbing in-triplicate form to a human is a structured source of information with cross-checking redundancy to the computer.

This area of research is called “Natural Language Processing” - NLP. It is not an obscure field - any Google user has benefited from this type of research. Other applications of NLP include speech recognition and machine translation.

NLP is not a new branch of science. Back in the 1950s, software was being developed in the USA to translate from German into English. The translation quality of grammar-based systems was very poor. Nevertheless, even the possibility of machine translation was so impressive that about US$20 million were spent on the research before the enchantment fizzled out and fund allocations virtually stopped. NLP did not die at that point, but it certainly slowed down.

Statistical approaches to NLP have been around nearly as long as grammar-based ones. However, as they require large quantities of data, these did not become feasible until the mid-1990s. Once they did reach popularity, however, the research advanced rapidly, taking advantage of ever increasing computer speed and available storage. Statistical approaches do not rely on language comprehension. Instead, with sufficient amounts of text, common patterns can be established without understanding the rules of their formation.

A good example is Google’s new translation engine from Arabic to English. The engine won the NIST 2005 machine translation competition, even though its software developers did not know Arabic. Instead, they used existing parallel documents of United Nations translated by professionals - some 200 billion words of content in total. It is perhaps symbolic that, even in such a deeply technical area, the Universal Declaration of Human Rights helps to ensure humans all over the world will be able to communicate with each other.

Standalone, however, a statistical approach is not a panacea either. Since there is no real understanding involved, a statistical NLP system has no way to recover from invalid conclusions.

There is more to the puzzle. Most of the real world texts are about somebody or something. The entity could be a person, a company, or a committee. Sometimes, the name of that entity is very long. Documents of the United Nations are known for names that even a human would struggle with. “The Ad Hoc Committee on the Scope of Legal Protection under the Convention on the Safety of United Nations and Associated Personnel” would be one of those. Other large organisations have similar problems.

Currently, neither of the above approaches is sufficient on its own. Grammar-based systems break on complex names; statistical ones mark ‘The Committee’ as a completely separate entity, rather than a reference to the full name.

The ideal system that we’re working on would be able to identify the complex names using a combination of techniques. It would also be capable of using multiple appearances in different contexts to confirm the identification, including linking different forms of the same name. Once these goals are achieved, documents in legal and medical domains can get the full benefits from other, already available, research.

Soon, the day will come when computers understand what humans write or say. Hopefully, without needing the triplicates.

The fix is extremely simple:

1. In maxent-2.4.0/src/java/opennlp/maxent/io/ObjectGISModelReader.java, replace
   • new FileInputStream(f) with
   • new BufferedInputStream(new FileInputStream(f), 1000000)
2. Recompile maxent library
3. Deploy new version of maxent-2.4.0.jar into OpenNLP’s lib directory

The comparison is striking (the numbers are File access system calls):

• build.bin.gz - 29830 ->  40
• chunk.bin.gz  -11853 -> 16
• tag.bin.gz - 11091 -> 14

Finally, I pinged my research colleague who did get the toolkit working (thanks Scott). Turns out to be there is a whole set of model files missing from the tool’s download. They are linked to from a separate page on the original website (not even in the download).

I am downloading the models now and hopefully will be on my way. But I can certainly see why this particular toolkit is mentioned much less frequently than Stanford’s or Bikel’s.

After the fact, I have also found a mini tutorial by Daniel McLaren explaining OpenNLP components and showing some sample code and output. Looks better than what’s bundled with OpenNLP itself. Maybe Daniel and Thomas Morton (author of OpenNLP) should talk.

The following wrapper seems to do the trick on windows (with your own values for|parserdir| :

String settingsFile = "|parserdir|\\settings\\collins.properties";
Settings.load(settingsFile);
Parser parser = new Parser("|parserdir|\\bikel\\wsj-02-21.obj.gz");
Sexp result = parser.parse(Sexp.read("(This is a funny world)").list());

There is a complaint when running the above code:

Settings different during training than now
------------------------------
parser.settingsFile
was |parsedir|\settings\collins.properties
is null

This however does not impact anything and correct values seem to be picked up.

Also, all the scripts are designed for *nix with a lot of flexibility and variables built in. To get it running on Windows, I hardcoded everything but the input file and this is the result:

set PDIR=|parserdir|
java -Xmx500m -cp "%PDIR%\dbparser.jar;%CLASSPATH%" -Dparser.settingsFile=%PDIR%\settings\collins.properties danbikel.parser.Parser -is %PDIR%\wsj-02-21.obj.gz -sa %1


Now, I need to wrap-up the parser in my own code to add input/output batching and I discover that this option is not accepted when constructing parser from the code. Despite javadoc saying that LexicalizedParser.setOptionFlags() takes the same parameters as the command line, the option sets are actually very different.

In the end, after much poking around, I built the code sequence that seems to produce identical effect:

LexicalizedParser lp = new LexicalizedParser("..../englishPCFG.ser.gz");
// lp.setOptionFlags(new String[]{"-tagSeparator", "/"});
WhitespaceTokenizer tokenizer = new WhitespaceTokenizer(new StringReader(text));
List<Word> words = tokenizer.tokenize();
WordToTaggedWordProcessor wttwp = new WordToTaggedWordProcessor('/');
words = wttwp.process(words);
Tree tree = (Tree) lp.apply(words);


Here, text variable is a string that is effectively pretokenized with white-space separator and ‘/‘ character is the word/tag separator token.

Update (3rd of August):

An email exchange with Christopher Manning and another look through the code proved that  flags in setOptionFlags() are a strict subset of flags accepted by main() method. However, 90% of flags in the setOptionFlags() are not documented in that method’s javadoc, so the only ones I cared about were the ones I saw in main() method.

Yet further digging found some documentation in classes Options, Test and Train, all within edu.stanford.nlp.parser.lexparser package. So, some additional documentation does exist, but one has to navigate the maze of code to find it. I guess that’s the normal curse of the open source software.