Bulk processing Lotus Notes database

Problems and Solutions

This article is for a niche audience even smaller than my usual readers. 🙂 There are not that many Lotus Notes developers; even smaller is a number of Lotus Notes coders who have bulk integration/migration needs. But some use cases do exist.

I have started (and probably finished) a small GitHub project that demonstrates how to export Lotus Notes into an XML format and then - as an example app - how to extract external links from it for link checking or other purposes.

There are two specific pieces of advice in there, that was learned the hard way:

  1. If you have access to a Lotus Notes database, you can export its content: text, embedded multimedia, history, permissions and all. And it does not take long, I had a two gigabyte database exporting in 20 minutes or so.
  2. Big XML files are hard to process. Usually streaming-oriented processing is a way to deal with it, but Lotus Notes XML is too ugly to build a streaming state machine around. It is easy to use XPath, but streaming normally does not support that and a full in-memory DOM is too large. Fortunately, XOM is a Java library that gives you the perfect combination. You create a custom Node factory class and it gets called for each element. Two important points: you can through the element away, effectively having a streaming mode and when you get the element, you can run XPath queries against it. I found Lotus Notes’s document element to be perfect to run my processing on. Each document is self-contained, so I process it and through it away.

I have used this approach for a number of small projects. I have done analytics (such as dead document cross-references), management reports (based on update history) and format conversion. I have also migrated Lotus Notes contact database into external mailing lists.

I did not enjoy Lotus Notes programming, but I did enjoy this particular toolkit and approach. It felt more than a hammer looking for a nail; it felt like a whole chainsaw willing - not just able - to rip into the meaty innards of Lotus Notes database and carve it into useful pieces.