I have started (and probably finished) a small GitHub project that demonstrates how to . . . → Read More: Bulk processing Lotus Notes database]]> This article is for a niche audience even smaller than my usual readers. There are not that many Lotus Notes developers; even smaller is a number of Lotus Notes coders who have bulk integration/migration needs. But some use cases do exist.

I have started (and probably finished) a small GitHub project that demonstrates how to export Lotus Notes into an XML format and then – as an example app – how to extract external links from it for link checking or other purposes.

There are two specific pieces of advice in there, that was learned the hard way:

1. If you have access to a Lotus Notes database, you can export its content: text, embedded multimedia, history, permissions and all. And it does not take long, I had a two gigabyte database exporting in 20 minutes or so.
2. Big XML files are hard to process. Usually streaming-oriented processing is a way to deal with it, but Lotus Notes XML is too ugly to build a streaming state machine around. It is easy to use XPath, but streaming normally does not support that and a full in-memory DOM is too large. Fortunately, XOM is a Java library that gives you the perfect combination. You create a custom Node factory class and it gets called for each element. Two important points: you can through the element away, effectively having a streaming mode and when you get the element, you can run XPath queries against it. I found Lotus Notes’s document element to be perfect to run my processing on. Each document is self-contained, so I process it and through it away.

I have used this approach for a number of small projects. I have done analytics (such as dead document cross-references), management reports (based on update history) and format conversion. I have also migrated Lotus Notes contact database into external mailing lists.

I did not enjoy Lotus Notes programming, but I did enjoy this particular toolkit and approach. It felt more than a hammer looking for a nail; it felt like a whole chainsaw willing – not just able – to rip into the meaty innards of Lotus Notes database and carve it into useful pieces.

Everything was going reasonably well, except my regular expressions were not picking section name/numbers sequences in all of the cases, which was causing a problem with the 6-language alignment algorithm.

Normally, . . . → Read More: Arabic numerals’ non-WYSIWYG]]>

For my other project, I needed to process some Arabic text that was in HTML file derived from MSWord document.

Everything was going reasonably well, except my regular expressions were not picking section name/numbers sequences in all of the cases, which was causing a problem with the 6-language alignment algorithm.

Normally, I just examine the text visually, determine a new regular expression pattern and that particular problem is solved. This time it was not to be.

When I looked at the text what I saw was the phrase “Section 1٣” with the word Section written in Arabic (right-to-left of course). The problem here is 1٣ which means 13, but with first digit 1 coming from Arabic Numerals set (which is what we use in English language) and the second digit ٣ (3) coming from Arabic-Indic Numerals set (which is what at least some Arab countries use). Confusing, I know. We use their numbers and

they already use somebody else’s. What do they know that we haven’t yet figured out?

Of course this juxtaposition makes no sense. Why would somebody mix the two alphabets, especially in an official document. I contacted the authoring departments and – unbelievably to me – they looked at the document and it was looking correct to them.

I had nothing to go on with, so I left that puzzle unsolved for a couple of weeks. That is until it hit me – they were looking at it in the MSWord, while I was looking at it on the codepoint character level. They had WYSIWYG on and I did not. So that was the difference.

I went looking around the MSWord interface with Arabic enabled and sure enough there was a whole collection of options for Arabic fonts, numbers and other options. And one of them was to display all numbers as Arabic-Indic. So, when that mode is enabled, MSWord will display any digits as Arabic-Indic ones. That answered half of the puzzle of why the original authors could not see the difference. But how did that happen in first place?

My guess is that the original section was copied from somewhere else in the document. The person who worked on that original had the keyboard (not MSWord display) configured to use Arabic numbers and was actually entering all too familiar 1,2,3 but displaying them as ١,٢,٣. Then, the person who copied the section title had a keyboard configured to use Arabic-Indic characters and he/she replaced or added to the section number using her keyboard. It still displayed cohesively, but now had numbers from different numeric systems.

Of course since the documents were designed for printing nobody noticed and really had no reason to care. This issue only becomes important when those documents are used as input for bitext alignment or some other computational processing. Then, and only then, it bites the person trying to make sense out of it.

The lesson here is. WYSIWYG might be good if all you are doing is looking or printing. But if your documents serve as input to other processes as well, WYSIWYG can cause some very non-obvious issues.

But, every time I put this particular tool aside, I know that picking it back up will be like . . . → Read More: Making up with ANTLR]]> I like ANTLR! It is a specialized tool that can really be applied to many difficult tasks when regular expressions get all Dust Puppy like. And I have used it in the past with great success.

But, every time I put this particular tool aside, I know that picking it back up will be like making up after a bad break up. Things feel familiar, but you are still so uncomfortable you cannot get anything working. Only knowing how great the tool is underneath, makes me go through the effort of re-familiarization.

I just downloaded ANTLR 3.1.2 bundled with its own GUI ANTLRWorks that offers visual diagrams, debugger and templates. You would think that would make for an easy out-of-box experience. You would be wrong.

You start the GUI and end up facing a blank screen. Lots of options and tabs for sure, but the only easy start one seems to be ‘Insert rule from template’.

Ok, so here is a couple of rules from templates trying to parse “Hello World!” string:

ID    :    LETTER (LETTER | DIGIT)*
;
LETTER
:    ‘a’..’z’ | ‘A’..’Z’
;

DIGIT    :    ’0′..’9′
;

WS    :    (‘ ‘ | ‘\t’ | ‘\n’ | ‘\r’) { $setType(Token.SKIP); }
;

Not good. We are missing a start state apparently. Ok, let’s add one:

hello    :    ID ID ‘!’
;

Still no good. Start looking at examples, trying to see what bits are compulsory. Ok, the word grammar is missing at the top of the file. Of course, I have both grammar and lexer elements now in one file (ANTLR 3 feature, I believe), but let’s not worry about deep meaning here.

grammar test;

Now, suddenly, syntax diagram starts showing up. Let’s try saving (as test.g) and compiling. No good:

The following token definitions can never be matched because prior tokens match the same input: LETTER

So much for following a template. More digging in examples. Memory really starts to bring back the Dragon Book‘s lessons. What’s the problem with LETTER and who is the prior token here. Ah, we don’t want the lexer to return LETTER (or DIGIT), only ID. So, LETTER and DIGIT are both token fragments, not tokens. Add fragment in front of both definitions. All good?

Nope! Now we have a problem with:

attribute is not a token, parameter, or return value: setType

But I did not write setType, the template provided it! Back to the examples! Apparently, somewhere along the way Skip tokens have gone away and we now have hidden channels instead. Swap that bit with one from an example and try again.

SUCCESS. Switch to interpreter, enter “Hello World!” in input box and run hello rule. Beauty, we have a parse diagram.

The final running grammar example is here:

grammar test;

hello    :    ID ID ‘!’
;

ID    :    LETTER (LETTER | DIGIT)*
;
fragment LETTER
:    ‘a’..’z’ | ‘A’..’Z’
;

fragment DIGIT    :    ’0′..’9′
;

WS    :    (‘ ‘ | ‘\t’ | ‘\n’ | ‘\r’) {  $channel = HIDDEN;  }
;

Hello World! Now, on to the real grammar and (if things really, really work) GATE integration…..

One of such tools is Exhibit, one of graduates from SIMILE project. It allows to do a very interactive webpage driven by just HTML+Javascript, with no server-side component required. I really like SIMILE’s . . . → Read More: CiteULike Exhibit visualization]]> Homegrown visualization is not the only way to quickly navigate CiteULike references. There are other tools that display bibliographies in interesting ways.

One of such tools is Exhibit, one of graduates from SIMILE project. It allows to do a very interactive webpage driven by just HTML+Javascript, with no server-side component required. I really like SIMILE’s tools, even though it feels like development slowed somewhat recently.

There is an example of how to import and display bibtext within Exhibit. It is not difficult, just a couple of steps. It must have been a popular section, as there is now a dedicated new tool for it.

Citeline Exhibit Builder allows to load in bibtext and presents editing interface to customize Exhibit’s presentation of the publications. It looks great and seem to work well. A nice aspect is that it allows to chose which bibtext fields to expose as filter facets. With original tutorial that would require html editing and understanding Exhibit mindset. Citeline nicely hides user from it.

There was a couple of small problems. Apparently, there is a way to login and ‘claim’ your presentation. I couldn’t test that as OpenID authentication failed (something about a nonce). Also, there is jsMath library but, once the generated Exhibit is downloaded, it fails with cross-server issues. Finally, as with most end-to-end solutions, it does not do data preprocessing/normalization to allow me, for example, to combine author/editor fields for sorting purposes.

Citeline is a very promising tool and I am certainly going to keep it in mind for publishing my bibliographies.

Nice as CiteULike is, it is fairly difficult to get an overall picture of . . . → Read More: Visualizing CiteULike collections]]> I am collecting my reading and reference material in CiteULike. I like the service because it can capture details from multiple sources. It also allows to discover what was collected by other interesting people through tags, people and bookmarks graph navigation.

Nice as CiteULike is, it is fairly difficult to get an overall picture of one’s own collection. It is especially difficult to see quickly if there are people who serve as hubs by collaborating with multiple different groups. The information is there, but it requires a lot of clicks to find it out.

My usual solution is to export information out, massage it into Graphviz format and use graph segmentation and layout algorithms to get a better overview. I have talked about Graphviz a number of times on this blog before. This is yet another time it proved useful.

I started by exporting CiteULike’s content of my library. I found Endnote export format to be more structured and therefore easier to parse. I then run it through a custom Python program that basically spat out graph with titles pointing at authors. That produced a very large graph and was not particularly useful.

The next step was to discover disjointed clusters of titles/authors. I used ccomps with -v and -x flags (e.g. ccomps.exe -v -x -o comp.dot output.dot).

ccomps gave me partitioned graphs as well as statistics on number of nodes/edges in each graph. I could then choose a graph with large number of nodes/edges (eventually, all of them) and run it through neato with overlap=scale and splines=true (e.g. neato.exe -Tgif -o neato_1.gif -Goverlap=scale -Gsplines=true comp_1.dot).

The resulting graph was still not perfect, but it was a good start. I also tried fdp instead of neato, but that seemed to produce giraffe versions of the graph with graph edges being overly long.

You can see an example of neato output for one of my clusters. Warning: if it causes problems due to its size, try it with IrfanView; that program can display even improbably large graphs (e.g. unpartitioned ones).

I have run into some problems as well that would either cause partitions combine together or produce duplicate nodes and edges.

The first problem was that sometimes a person was an author and sometimes an editor. I was interested in both, so collapsed those fields together. That caused some non-people to then show up on the graph and connect clusters in unexpected ways. For my library the specific value was ‘European’, so I filtered it out in the code.

The second problem was to do with CiteULike’s parsing. Sometimes, it would split a first+last name into separate names, probably due to incorrect manual entry at some point. I had to fix those at the source by editing corresponding CiteULike entry. Probably a good thing to do anyway.

The other problem is right out of the co-reference resolution domain. Sometimes names would include full first names, sometimes only a first name initial. I have worked around that by normalizing all first names to the initials. Obviously, this could collapse entries belonging to multiple real people into one.

Further on name problems, in cases of non English names (e.g. Spanish names with multiple surnames), CiteULike would get confused which part is which and not display or export it correctly. Additionally, sometimes characters such as ñ would be entered as plain n. Those also needed to be corrected manually.

The project only took a couple of hours including writing code and cleanup. It is already useful to me, as I found a new person who was in unexpectedly large number of papers and also found a chain of connections that might be interesting to follow more closely.

There is of course a lot more that could be done. Automatic co-reference of misspelt names, layout hints based on number of times authors appeared together, color coding of tags – these are just some of the easy ideas.

There might even be a small project/paper in doing co-reference resolution and cleaning up CiteULike data? After all, similar projects were done for Wikipedia. I don’t think CiteULike currently makes a full export available, but they do have some so might be amendable to exporting a special set for research purposes.

I could not figure out where to even start troubleshooting this issue, . . . → Read More: Vista repeatedly dropping wireless connection – solution]]> I am visiting my parents and connect to their network via wireless router. My laptop, which is (still!) running Vista kept dropping wireless connection every couple of minutes and reconnecting again. Interestingly, the other computers connected to the same router had no problems.

I could not figure out where to even start troubleshooting this issue, until I noticed that the problem only happens while I am running on battery and not when I am connected to the mains. Once I notice that, the solution was simple – power management module must have been too eager and turning off wireless after 30 seconds of inactivity. Given that I was trying to read emails or webpages, that would occur fairly regularly.

The fix is to go to the power-management control panel and adjust on-battery behaviour to match the full-power one. I am putting this out because an hour of searching for this problem online did not bring any result. I hope the next person to be flummoxed by this repeated connection loss will find my blog entry fast.