Bulk processing Lotus Notes database

This article is for a niche audience even smaller than my usual readers. 🙂 There are not that many Lotus Notes developers; even smaller is a number of Lotus Notes coders who have bulk integration/migration needs. But some use cases do exist.

I have started (and probably finished) a small GitHub project that demonstrates how to export Lotus Notes into an XML format and then – as an example app – how to extract external links from it for link checking or other purposes.

There are two specific pieces of advice in there, that was learned the hard way:

  1. If you have access to a Lotus Notes database, you can export its content: text, embedded multimedia, history, permissions and all. And it does not take long, I had a two gigabyte database exporting in 20 minutes or so.
  2. Big XML files are hard to process. Usually streaming-oriented processing is a way to deal with it, but Lotus Notes XML is too ugly to build a streaming state machine around. It is easy to use XPath, but streaming normally does not support that and a full in-memory DOM is too large. Fortunately, XOM is a Java library that gives you the perfect combination. You create a custom Node factory class and it gets called for each element. Two important points: you can through the element away, effectively having a streaming mode and when you get the element, you can run XPath queries against it. I found Lotus Notes’s document element to be perfect to run my processing on. Each document is self-contained, so I process it and through it away.

I have used this approach for a number of small projects. I have done analytics (such as dead document cross-references), management reports (based on update history) and format conversion. I have also migrated Lotus Notes contact database into external mailing lists.

I did not enjoy Lotus Notes programming, but I did enjoy this particular toolkit and approach. It felt more than a hammer looking for a nail; it felt like a whole chainsaw willing – not just able – to rip into the meaty innards of Lotus Notes database and carve it into useful pieces.

4 thoughts on “Bulk processing Lotus Notes database”

  1. Hi Alex,

    i like your article

    i have one case and i need your help
    i want to integrate the solr to lotus so that the better search can be implemented.
    can you please help me on this.

  2. Hi Pankaj,

    Try the GitHub project mentioned and see how far you get. Until you spent a bit of time staring at Lotus Notes XML dump and figuring out what information you need to extract for it, there is not much other help anybody would be able to offer you.

  3. Hi Alex,

    Thanks for reply,

    i just want to implement the better search for the docs, which are in the lotus(Multiple database) and we don’t have database access. we can only access doc through lotus
    we cant pull the data from lotus due to security restriction

    can you please help on this

  4. Pankaj,

    There is no “just” for SOLR+LotusNotes yet. You do have to build the connection because Lotus Notes document format could be whatever you design.

    If you don’t know where to start, try running the export agent in the project and then looking at XML output. Compare that against field definitions in Lotus Designer to get a feel at the parity between what you see on screen and what is under the covers. Then, you will know what the next step should be.

Comments are closed.