The most important issue to understand is that external part is accessed only when user is authenticated with full username/password. In any other context, users and groups are those that are copied/imported into the internal Crowd side of the directory. This produces a couple of cognitive problems:

1. One cannot lookup users from that directory just after the directory is created. The search runs against the internal database and does not even generate LDAP lookup. This is obvious once you realise that the directory has effectively both remote and local repositories in one interface and the search only goes against local (still empty) one.
2. Directory permissions are also about the local directory. In the past, I disabled all modify permissions when configuring LDAP directory, as I did not want to accidentally change external user. Doing the same thing with Delegated directory will causes very odd database integrity violation stack traces. (now CWD-911)
3. Wild card handling in user lookup screen is different between Crowd internal directories and LDAP directories. Internal directories use substring search, while LDAP requires explicit star (*) character. Searching against Delegated Directory is searching against Crowd directory, so putting star wildcard will actually cause no matches. (now CWD-912)
4. Local directory part seems to store a lot more information about user than just username and group association. It actually stores email, full name, etc. This means that if any information gets changed in the original external LDAP, it may not be reflected in Crowd’s directory (and therefore to the applications). As there does not seem to be a way for the administrator to easily check for mismatches, such problem will likely to be extremely hard to troubleshoot. (now CWD-913)
5. Finally, there is no easy way to copy small sets of users into local part of the Crowd’s directory from the remote counter-part. They have to be added (with full information) one by one or copied wholesale from another directory. I have opened a request to improve this.

Crowd’s Delegated directory option was eagerly awaited for a long time by great many people, but it is obviously still in a need of improvement or two. I am looking forward to having those issues addressed soon.

Looking back at it, I realise that I was right in the thick of Internet development:

• When I just started working with Java, we had to throw out all the printed Javadocs, because jdk1.0b2 was released and a lot of Java API (e.g. FTP and MAIL) from jdk1.0a3 has been hidden under sun’s internal packages
• I did a first (alpha) implementation of standard servlet API for W3C’s Jigsaw server, by porting it from Sun’s Jeeves
• I dabbled in hot 2.5D Apple technology (HotSause), by generating web server’s directory content in MCF format. The format has died, but apparently it turned into RDF. I was developing Semantic Web applications well before the term got popular.
• I contributed to an Open Source project, well before SourceForge’s first appearance
• I was a late-comer to /. and my ID is still below 36000

I am not bragging! I am just musing out loud at how much personal web history can be retrieved with few well placed searches.

The flip side of a coin of course, is that this history will not go away, even if I wanted it to. Which is why I do not link to my Slashdot account (and this is not an invitation for exercise in forensics). One just hopes that the future recruiter will look at timestamps of my various web appearances and makes appropriate adjustments to skills and effort.

My ideal digital picture frame for them would be one sitting in a living room or a bedroom with new photos to delight my parents every so often.

Such a device would have to be:

• Wi-Fi capable - My parents have a wireless router and there is no point for a picture frame to sit next to the computer
• Able to pull content from private online photo account, such as Flickr or PicasaWeb, to which our extended family could push photos
• No ongoing monthly costs - subscription would make it a gift that keeps taking, rather than giving
• Controllable over the internet
• Ideally with speakers and/or some way to show video to be more future proof

I have been on a lookout for such a device for more than a year and had no luck. Obviously, digital picture frames are still a personal purchase rather than a gift one. Or maybe less technical parents is a smaller niche than I imagine.

But I have hope. Yesterday, I have received a small package that contained a Chumby! Chumby is not a digital picture frame. It is quite small (I think the website’s image is real-size). But it has features that make up for its size.

It has Wi-Fi access, including password-protected; it has no monthly costs; it is configured over the internet and comes with speakers. It also has touch sensitive screen, microphone and accelerometer (like in Wii controller).

Notice I did not say anything about pictures or videos. That’s because Chumby is a more generic device. It allows to choose what widgets run on it and a widget is a program written in Flash, the same environment that allows us to watch Flickr slide-shows and youTube videos, listen to internet radio and play casual games. It can also double as alarm clock and iPod music player.

More importantly, because anybody can develop and share a widget, I am not married to any particular way of presenting photos. Flickr widget exists already, but other photo and video service widgets are on the way.

And, if I am still unhappy, I can write my own widgets. Chumby runs Linux under the covers and Flash Lite 3 interface. And, differently from Apple’s position with iPhone, Chumby Industries encourage people to modify their software, hardware and even basic device shape. Already, there are compilation packages for python, perl and even Java (actually JamVM).

Chumby is not yet for public sale, but that should happen any day now. I was on a mailing list, so got a pre-release invite. That is good, as it means I have some time to really play with my Chumby.

And if all goes well, my Chumby will soon have a new friend or two hiding under the Christmas tree overseas.

The problem is the user configuration file gate.xml that is stored in the shared location, usually home directory. On Windows, that is C:\Documents and Settings\[ProfileName]\.

One of those settings was pointing to where the plugins were loaded from and was still referring to GATE 3.1’s locations. That caused NullPointerExceptions in the GATE and everything was breaking from that point on.

I found this by using FileMon, but later realised that it might have been done easier by changing runtime.spawn property to false in GATE’s build.xml file that is used to start the program. Using ant to start a program is a new one for me, but I guess it makes sense in some cases. Setting the property to false shows the startup messages and the exception that the wrong directories cause.

I have deleted the old gate.xml and gate.session files in my home directory and everything started to work. Back to actually trying to use the software.

To start, RWE is much smaller. It is about 400 people as compared to 15 thousands at JavaOne. This obviously makes scheduling logistics and eating arrangements simpler, but there is also a very different feel in the air. It feels that it is much harder to walk around without bumping into speakers and/or other moderately famous web people. At JavaOne, it is all about learning, here it is more like sharing.

Another interesting thing I noticed is that a lot more people than I expected were coming from Java server side background. In fact, we had a round of introductions at Web design Birds-Of-Feather session and more than half of the  people in the room had some (often strong) background in Java. To me, this is a great sign as it shows that the path I am taking (adding HTML/CSS/JavaScript to my Java skills)  has already been done by multiple people before without too many problems.

I have gone to the following sessions:

1.  Secure application development with Ajax (by Dean H. Saxe)  - The presentation itself was great and covered interesting topic in details. I did not understand all of the advanced concepts and consequences, but the core message was very clear and the slides give enough hints and terms to do further research on my own. I would have liked a more detailed example (e.g. ‘This is why SOP is not applicable’ ), but overall it was great.
2. Merging Ajax and Accessibility (by Mark Meeker) - Another great presentation. I heard before that designing for accessibility actually has beneficial side-effects of increased general usability and better design practices, but it was good to see it confirmed with large commercial sites. Mark also had great examples and talked about Hijax a bit as a way of building accessibility into the process, rather than trying to bolt it on at the end.
3. Web Design for Server-Side Developers (by Greg Murray) - This one I have found somewhat disappointing. I knew that covering good HTML, CSS, Javascript,  modular design and supporting tools in one presentation might have been too ambitious.  Still, I was looking forward to some sort of high-level view consistent story tying together the bits together with some best practices thrown in. Unfortunately, Greg was not able to deliver that. He spent too much time jumping between the topics. He also talked about jMaki’s  implementation a lot. That might have been useful, but given that some very important issues (Internationalisation, classes vs. IDs, etc) were still not implemented correctly (by Greg’s own admission), I felt jMaki was not yet ready to be shown as an example of best practices.
4. Web design/architecture Birds-Of-Feather session with Aaron Gustafson, David Verba and couple of others. It was actually interesting, because I sat with them at the dinner table without realising who they were. But you could see they were really smart and interesting, even in their unstaged moments. True geeks, in the good sense of the word. The session itself was a very interesting discussing and somehow I even managed to hog the floor for a while with my questions. Hopefully, it did not annoy too many people.

I am looking forward to the second day.

The fix is extremely simple:

1. In maxent-2.4.0/src/java/opennlp/maxent/io/ObjectGISModelReader.java, replace
   • new FileInputStream(f) with
   • new BufferedInputStream(new FileInputStream(f), 1000000)
2. Recompile maxent library
3. Deploy new version of maxent-2.4.0.jar into OpenNLP’s lib directory

The comparison is striking (the numbers are File access system calls):

• build.bin.gz - 29830 ->  40
• chunk.bin.gz  -11853 -> 16
• tag.bin.gz - 11091 -> 14

Finally, I pinged my research colleague who did get the toolkit working (thanks Scott). Turns out to be there is a whole set of model files missing from the tool’s download. They are linked to from a separate page on the original website (not even in the download).

I am downloading the models now and hopefully will be on my way. But I can certainly see why this particular toolkit is mentioned much less frequently than Stanford’s or Bikel’s.

After the fact, I have also found a mini tutorial by Daniel McLaren explaining OpenNLP components and showing some sample code and output. Looks better than what’s bundled with OpenNLP itself. Maybe Daniel and Thomas Morton (author of OpenNLP) should talk.

The following wrapper seems to do the trick on windows (with your own values for|parserdir| :

String settingsFile = "|parserdir|\\settings\\collins.properties";
Settings.load(settingsFile);
Parser parser = new Parser("|parserdir|\\bikel\\wsj-02-21.obj.gz");
Sexp result = parser.parse(Sexp.read("(This is a funny world)").list());

There is a complaint when running the above code:

Settings different during training than now
------------------------------
parser.settingsFile
was |parsedir|\settings\collins.properties
is null

This however does not impact anything and correct values seem to be picked up.

Also, all the scripts are designed for *nix with a lot of flexibility and variables built in. To get it running on Windows, I hardcoded everything but the input file and this is the result:

set PDIR=|parserdir|
java -Xmx500m -cp "%PDIR%\dbparser.jar;%CLASSPATH%" -Dparser.settingsFile=%PDIR%\settings\collins.properties danbikel.parser.Parser -is %PDIR%\wsj-02-21.obj.gz -sa %1


Now, I need to wrap-up the parser in my own code to add input/output batching and I discover that this option is not accepted when constructing parser from the code. Despite javadoc saying that LexicalizedParser.setOptionFlags() takes the same parameters as the command line, the option sets are actually very different.

In the end, after much poking around, I built the code sequence that seems to produce identical effect:

LexicalizedParser lp = new LexicalizedParser("..../englishPCFG.ser.gz");
// lp.setOptionFlags(new String[]{"-tagSeparator", "/"});
WhitespaceTokenizer tokenizer = new WhitespaceTokenizer(new StringReader(text));
List<Word> words = tokenizer.tokenize();
WordToTaggedWordProcessor wttwp = new WordToTaggedWordProcessor('/');
words = wttwp.process(words);
Tree tree = (Tree) lp.apply(words);


Here, text variable is a string that is effectively pretokenized with white-space separator and ‘/‘ character is the word/tag separator token.

Update (3rd of August):

An email exchange with Christopher Manning and another look through the code proved that  flags in setOptionFlags() are a strict subset of flags accepted by main() method. However, 90% of flags in the setOptionFlags() are not documented in that method’s javadoc, so the only ones I cared about were the ones I saw in main() method.

Yet further digging found some documentation in classes Options, Test and Train, all within edu.stanford.nlp.parser.lexparser package. So, some additional documentation does exist, but one has to navigate the maze of code to find it. I guess that’s the normal curse of the open source software.

My trees are large. A sentence of 40 words is an average case, rather than an edge one. So, all of the display packages I have tried cut off large chunks of the tree. It might be possible to tinker with their LaTeX code to produce output that is not cut-off at letter, a4 or even a3 size, but I am not that good with LaTeX yet. And I need to produce this large trees quickly, as I am not even sure whether this parser would be suitable for my needs in the long run.

So, instead, I wrote my own bridging code in Java between penn treebank output of the parser and Graphviz, graph layout software that I use for many layout tasks. The whole implementation was in one file less than 100 lines total and that included the logic to highlight maximum spanning subtrees of a particular element (NounPhrase in this example). Click on the small image to see the full example. Graphviz input file is also available for the curious.

At the moment, it is sufficient to convert to image files. If I ever do convince the parser to understand my 80-word sentences, the resulting trees will probably be large enough to need ZGRViewer.

The Java bridging code is not available yet, as it is very ugly. The secret was in the PennTreeReader’s main() method that showed how to read the parser’s output back in and into Tree form suitable for recursive descent. After that, it was just the code to navigate the tree levels and spit out incredibly easy Graphviz format. I will probably clean the code up a bit over the next couple of weeks and then release it.

If somebody does like the output and wants to see the code sooner, send me an email at alex@thisdomain.