But, every time I put this particular tool aside, I know that picking it back up will be like . . . → Read More: Making up with ANTLR]]> I like ANTLR! It is a specialized tool that can really be applied to many difficult tasks when regular expressions get all Dust Puppy like. And I have used it in the past with great success.

But, every time I put this particular tool aside, I know that picking it back up will be like making up after a bad break up. Things feel familiar, but you are still so uncomfortable you cannot get anything working. Only knowing how great the tool is underneath, makes me go through the effort of re-familiarization.

I just downloaded ANTLR 3.1.2 bundled with its own GUI ANTLRWorks that offers visual diagrams, debugger and templates. You would think that would make for an easy out-of-box experience. You would be wrong.

You start the GUI and end up facing a blank screen. Lots of options and tabs for sure, but the only easy start one seems to be ‘Insert rule from template’.

Ok, so here is a couple of rules from templates trying to parse “Hello World!” string:

ID    :    LETTER (LETTER | DIGIT)*
;
LETTER
:    ‘a’..’z’ | ‘A’..’Z’
;

DIGIT    :    ’0′..’9′
;

WS    :    (‘ ‘ | ‘\t’ | ‘\n’ | ‘\r’) { $setType(Token.SKIP); }
;

Not good. We are missing a start state apparently. Ok, let’s add one:

hello    :    ID ID ‘!’
;

Still no good. Start looking at examples, trying to see what bits are compulsory. Ok, the word grammar is missing at the top of the file. Of course, I have both grammar and lexer elements now in one file (ANTLR 3 feature, I believe), but let’s not worry about deep meaning here.

grammar test;

Now, suddenly, syntax diagram starts showing up. Let’s try saving (as test.g) and compiling. No good:

The following token definitions can never be matched because prior tokens match the same input: LETTER

So much for following a template. More digging in examples. Memory really starts to bring back the Dragon Book‘s lessons. What’s the problem with LETTER and who is the prior token here. Ah, we don’t want the lexer to return LETTER (or DIGIT), only ID. So, LETTER and DIGIT are both token fragments, not tokens. Add fragment in front of both definitions. All good?

Nope! Now we have a problem with:

attribute is not a token, parameter, or return value: setType

But I did not write setType, the template provided it! Back to the examples! Apparently, somewhere along the way Skip tokens have gone away and we now have hidden channels instead. Swap that bit with one from an example and try again.

SUCCESS. Switch to interpreter, enter “Hello World!” in input box and run hello rule. Beauty, we have a parse diagram.

The final running grammar example is here:

grammar test;

hello    :    ID ID ‘!’
;

ID    :    LETTER (LETTER | DIGIT)*
;
fragment LETTER
:    ‘a’..’z’ | ‘A’..’Z’
;

fragment DIGIT    :    ’0′..’9′
;

WS    :    (‘ ‘ | ‘\t’ | ‘\n’ | ‘\r’) {  $channel = HIDDEN;  }
;

Hello World! Now, on to the real grammar and (if things really, really work) GATE integration…..

In the . . . → Read More: Conjunctions in named entities]]> A recent article on lingpipe discussed conjuncted named entities such as Johnson and Johnson and Wallace and Gromit. They suggest that maybe a way of treating this is as a frozen expression. I assume that means relying on statistical measures to see this Multi-Word-Expression repeating enough times to be treated as a unit.

In the United Nations corpus, things can get even more interesting. Let’s look at a relatively easy example: draft resolution A/56/L.28 and Add.1.

Is this a one document (one draft resolution) or two? And if two, then which two? The first one is obviously A/56/L.28. But Add.1 is not a valid document symbol, it is actually an (additive?) coreference to the first one and resolves to A/56/L.28/Add.1?

The answer (as good as I can make it so far) could lie in FRBR distinction between Expression and Manifestation. A resolution is an expression of Member States’s proposals and negotiations. To some degree, it evolves over several meetings. However between the discussions, the latest version or changes need to be reported to make sure they are formally registered and also to ensure the next round of discussions could have latest documents to work from.

In our case, the first time the draft resolution had to be presented it was published under A/56/L.28 (which incidentally means a limited distribution document 28 of the General Assembly’s 56th regular session). So, the initial Manifestation of the draft resolution became this physical document with a distinct symbol assigned.

But apart from its text, draft resolution has a list of sponsoring Member States. That list can change as draft resolution gains sponsors. These additional sponsors were in the Addendum A/56/L.28/Add.1. But the addendum does not make sense without the original document, so actually both physical documents represent one logical draft resolution, which is reflected in the grammar of the text (draft resolution, not resolutions).

What this means for named entity annotations and for recognition algorithms is hard to say and is something I am looking at with my PhD research.

Nice as CiteULike is, it is fairly difficult to get an overall picture of . . . → Read More: Visualizing CiteULike collections]]> I am collecting my reading and reference material in CiteULike. I like the service because it can capture details from multiple sources. It also allows to discover what was collected by other interesting people through tags, people and bookmarks graph navigation.

Nice as CiteULike is, it is fairly difficult to get an overall picture of one’s own collection. It is especially difficult to see quickly if there are people who serve as hubs by collaborating with multiple different groups. The information is there, but it requires a lot of clicks to find it out.

My usual solution is to export information out, massage it into Graphviz format and use graph segmentation and layout algorithms to get a better overview. I have talked about Graphviz a number of times on this blog before. This is yet another time it proved useful.

I started by exporting CiteULike’s content of my library. I found Endnote export format to be more structured and therefore easier to parse. I then run it through a custom Python program that basically spat out graph with titles pointing at authors. That produced a very large graph and was not particularly useful.

The next step was to discover disjointed clusters of titles/authors. I used ccomps with -v and -x flags (e.g. ccomps.exe -v -x -o comp.dot output.dot).

ccomps gave me partitioned graphs as well as statistics on number of nodes/edges in each graph. I could then choose a graph with large number of nodes/edges (eventually, all of them) and run it through neato with overlap=scale and splines=true (e.g. neato.exe -Tgif -o neato_1.gif -Goverlap=scale -Gsplines=true comp_1.dot).

The resulting graph was still not perfect, but it was a good start. I also tried fdp instead of neato, but that seemed to produce giraffe versions of the graph with graph edges being overly long.

You can see an example of neato output for one of my clusters. Warning: if it causes problems due to its size, try it with IrfanView; that program can display even improbably large graphs (e.g. unpartitioned ones).

I have run into some problems as well that would either cause partitions combine together or produce duplicate nodes and edges.

The first problem was that sometimes a person was an author and sometimes an editor. I was interested in both, so collapsed those fields together. That caused some non-people to then show up on the graph and connect clusters in unexpected ways. For my library the specific value was ‘European’, so I filtered it out in the code.

The second problem was to do with CiteULike’s parsing. Sometimes, it would split a first+last name into separate names, probably due to incorrect manual entry at some point. I had to fix those at the source by editing corresponding CiteULike entry. Probably a good thing to do anyway.

The other problem is right out of the co-reference resolution domain. Sometimes names would include full first names, sometimes only a first name initial. I have worked around that by normalizing all first names to the initials. Obviously, this could collapse entries belonging to multiple real people into one.

Further on name problems, in cases of non English names (e.g. Spanish names with multiple surnames), CiteULike would get confused which part is which and not display or export it correctly. Additionally, sometimes characters such as ñ would be entered as plain n. Those also needed to be corrected manually.

The project only took a couple of hours including writing code and cleanup. It is already useful to me, as I found a new person who was in unexpectedly large number of papers and also found a chain of connections that might be interesting to follow more closely.

There is of course a lot more that could be done. Automatic co-reference of misspelt names, layout hints based on number of times authors appeared together, color coding of tags – these are just some of the easy ideas.

There might even be a small project/paper in doing co-reference resolution and cleaning up CiteULike data? After all, similar projects were done for Wikipedia. I don’t think CiteULike currently makes a full export available, but they do have some so might be amendable to exporting a special set for research purposes.

I have written some macros to do the conversion, but they were scary looking and fragile. Fortunately, I now found a . . . → Read More: Bulk converting doc files into txt (or html)]]> I have written about converting Microsoft Word files into text or html using OpenOffice before. However, the wizards I described in that article were crashing when the number of files crossed into several hundreds.

I have written some macros to do the conversion, but they were scary looking and fragile. Fortunately, I now found a tool that does the same job better and with more flexibility. DocConverter by Danny Brewer and Dan Horwood allows to convert a whole directory of files at a time from any to any OpenOffice-understood format.

I have just converted more than a thousand documents from doc to txt without any problems.  Actually, I had a small problem, but it was my fault. I had some corrupted files that OO would not open and that was breaking DocConverter and throwing some ugly looking Basic runtime error. I had to delete the problem files, kill the Open Office (stop macro did not) and rerun the tool. Otherwise, it just run.

That’s what happened to us with the last (5:15pm) slot . . . → Read More: Artificial Intelligence discussion at BarCampNYC3]]> They say at BarCamp that if you don’t like the session you are in, feel free to go to a better one. No hard feelings. But what do you do, if you show up for the announced moderated discussion session yet the moderator does not.

That’s what happened to us with the last (5:15pm) slot of the second day of BarCampNYC3. So, after waiting for 10 minutes past the start time, I decided to step in and moderate.

We talked a bit about everything: a definition of Artificial Intelligence (no agreement) and statistical algorithms that try to find the tanks, tune adverts and prevent SPAM. We discussed the state of art in computer vision and why once well-known consumer company in that space (Riya) still failed miserably. Near the end, we also talked about artificial intelligence as an emotional one and whether Pleo is intelligent.

All together, it was a very spirited discussion and most of the people contributed their opinion and their knowledge. We may not have discussed what the original moderator had in mind, but we certainly discussed interesting topics.

Bureaucracy often has its . . . → Read More: Unravelling the black magic of bureaucracy]]> Arthur C. Clarke once famously wrote “Any sufficiently advanced technology is indistinguishable from magic”. In the same vein, many people feel that any sufficiently established bureaucracy is like a black magic, sorcery even. Certainly, it often takes skills out of this world to follow the logic of modern tax return instructions.

Bureaucracy often has its place and reason. Laws protect exploitable minorities; procedures serve to avoid known problems; cross-referencing forms are filled in triplicate to allow for audit and protection against falsification. The problem is not the bureaucracy as such but rather the fact that it eventually outgrows any individual person’s ability to comprehend it. At that point, only dedicated specialists can understand the process and the rest of us have to offer sacrifices to those acolytes in hopes of beneficial results.

Enter computers. It turns out that computers can bring the complexity of information down, back within the reach of the non-specialist. The more bureaucratic a processes, the better a computer can figure it out. What is a mind-numbing in-triplicate form to a human is a structured source of information with cross-checking redundancy to the computer.

This area of research is called “Natural Language Processing” – NLP. It is not an obscure field – any Google user has benefited from this type of research. Other applications of NLP include speech recognition and machine translation.

NLP is not a new branch of science. Back in the 1950s, software was being developed in the USA to translate from German into English. The translation quality of grammar-based systems was very poor. Nevertheless, even the possibility of machine translation was so impressive that about US$20 million were spent on the research before the enchantment fizzled out and fund allocations virtually stopped. NLP did not die at that point, but it certainly slowed down.

Statistical approaches to NLP have been around nearly as long as grammar-based ones. However, as they require large quantities of data, these did not become feasible until the mid-1990s. Once they did reach popularity, however, the research advanced rapidly, taking advantage of ever increasing computer speed and available storage. Statistical approaches do not rely on language comprehension. Instead, with sufficient amounts of text, common patterns can be established without understanding the rules of their formation.

A good example is Google’s new translation engine from Arabic to English. The engine won the NIST 2005 machine translation competition, even though its software developers did not know Arabic. Instead, they used existing parallel documents of United Nations translated by professionals – some 200 billion words of content in total. It is perhaps symbolic that, even in such a deeply technical area, the Universal Declaration of Human Rights helps to ensure humans all over the world will be able to communicate with each other.

Standalone, however, a statistical approach is not a panacea either. Since there is no real understanding involved, a statistical NLP system has no way to recover from invalid conclusions.

There is more to the puzzle. Most of the real world texts are about somebody or something. The entity could be a person, a company, or a committee. Sometimes, the name of that entity is very long. Documents of the United Nations are known for names that even a human would struggle with. “The Ad Hoc Committee on the Scope of Legal Protection under the Convention on the Safety of United Nations and Associated Personnel” would be one of those. Other large organisations have similar problems.

Currently, neither of the above approaches is sufficient on its own. Grammar-based systems break on complex names; statistical ones mark ‘The Committee’ as a completely separate entity, rather than a reference to the full name.

The ideal system that we’re working on would be able to identify the complex names using a combination of techniques. It would also be capable of using multiple appearances in different contexts to confirm the identification, including linking different forms of the same name. Once these goals are achieved, documents in legal and medical domains can get the full benefits from other, already available, research.

Soon, the day will come when computers understand what humans write or say. Hopefully, without needing the triplicates.