Reducing disk thrashing of OpenNLP/MaxEnt parser – with one line code change

Computational Linguistics, Java

When OpenNLP toolkit uses MaxEnt parser, it has to read in about 25 MBytes of model files. The model reader uses basic unbuffered FileReader. The result is the excessive number of system calls (and disk access calls) during the parser startup.

The fix is extremely simple:

  1. In maxent-2.4.0/src/java/opennlp/maxent/io/ObjectGISModelReader.java, replace
    • new FileInputStream(f) with
    • new BufferedInputStream(new FileInputStream(f), 1000000)
  2. Recompile maxent library
  3. Deploy new version of maxent-2.4.0.jar into OpenNLP’s lib directory

The comparison is striking (the numbers are File access system calls):

  • build.bin.gz _- 29830 -> _ 40
  • _chunk.bin.gz _ -11853 -> 16
  • tag.bin.gz - 11091 -> 14