Everything was going reasonably well, except my regular expressions were not picking section name/numbers sequences in all of the cases, which was causing a problem with the 6-language alignment algorithm.

Normally, I just examine the text . . . → Read More: Arabic numerals’ non-WYSIWYG]]>

For my other project, I needed to process some Arabic text that was in HTML file derived from MSWord document.

Everything was going reasonably well, except my regular expressions were not picking section name/numbers sequences in all of the cases, which was causing a problem with the 6-language alignment algorithm.

Normally, I just examine the text visually, determine a new regular expression pattern and that particular problem is solved. This time it was not to be.

When I looked at the text what I saw was the phrase “Section 1٣” with the word Section written in Arabic (right-to-left of course). The problem here is 1٣ which means 13, but with first digit 1 coming from Arabic Numerals set (which is what we use in English language) and the second digit ٣ (3) coming from Arabic-Indic Numerals set (which is what at least some Arab countries use). Confusing, I know. We use their numbers and

they already use somebody else’s. What do they know that we haven’t yet figured out?

Of course this juxtaposition makes no sense. Why would somebody mix the two alphabets, especially in an official document. I contacted the authoring departments and – unbelievably to me – they looked at the document and it was looking correct to them.

I had nothing to go on with, so I left that puzzle unsolved for a couple of weeks. That is until it hit me – they were looking at it in the MSWord, while I was looking at it on the codepoint character level. They had WYSIWYG on and I did not. So that was the difference.

I went looking around the MSWord interface with Arabic enabled and sure enough there was a whole collection of options for Arabic fonts, numbers and other options. And one of them was to display all numbers as Arabic-Indic. So, when that mode is enabled, MSWord will display any digits as Arabic-Indic ones. That answered half of the puzzle of why the original authors could not see the difference. But how did that happen in first place?

My guess is that the original section was copied from somewhere else in the document. The person who worked on that original had the keyboard (not MSWord display) configured to use Arabic numbers and was actually entering all too familiar 1,2,3 but displaying them as ١,٢,٣. Then, the person who copied the section title had a keyboard configured to use Arabic-Indic characters and he/she replaced or added to the section number using her keyboard. It still displayed cohesively, but now had numbers from different numeric systems.

Of course since the documents were designed for printing nobody noticed and really had no reason to care. This issue only becomes important when those documents are used as input for bitext alignment or some other computational processing. Then, and only then, it bites the person trying to make sense out of it.

The lesson here is. WYSIWYG might be good if all you are doing is looking or printing. But if your documents serve as input to other processes as well, WYSIWYG can cause some very non-obvious issues.

This random start position part of the problem was making me think, so I used it as an exercise to try some newish jQuery functions, such as . . . → Read More: jQuery: Cycling between multiple classes with random start]]>

I saw an interesting question on StackOverflow on how to cycle between 3 states for list items , but with initial state for each item being potentially different.

This random start position part of the problem was making me think, so I used it as an exercise to try some newish jQuery functions, such as delegate and advanced class selectors.

My solution was basically to build a reduced case of state transition diagram with a cycle.  The advantage of it is that any number of states can be iterated through.  It  could even be a plugin, if need be.

My full test example can be found in the original SO question, probably toward the bottom. I am also including it here as a Pastie:

Here is one of the South African Warthog, displayed using Microsof’s Zoom.it technology. Try zooming . . . → Read More: A new camera]]> I got myself a new digital camera recently, a Canon T2i. It feels really nice and makes it quite hard to go back to point-and-shoots afterward. And it takes really good 18 megapixel shots.

Here is one of the South African Warthog, displayed using Microsof’s Zoom.it technology. Try zooming on it:

As I know nearly nothing of at least . . . → Read More: jQuery for multilingual web development]]> I have (nearly) finished developing a mini-website in 6 languages (Arabic, Chinese, English, French, Russian, Spanish). The layout was the same, so ideally it would have been driven by a content management system. Not in this case unfortunately, as I was not given enough time to setup the infrastructure.

As I know nearly nothing of at least two of the languages above (Arabic and Chinese), I had to keep rechecking the content provided to ensure the right text ends up in the right place on a page. Google Translate helped with that by back-translating from another language back into English and making sure I got right sentence boundaries, etc.

However, even with content in the right place, I still needed to visually verify that things are correct. Also, some of the late arriving changes needed to be implemented for all 6 sets of files. For example, some of the URLs changed, some classes for javascript enhancements were added or removed, and so on.

Initially, I tried to check things in the editor by using regular expressions. This worked for basic things, but as the project progressed and markup (and javascript enhancements) became more complex, the regular expressions became not sufficient. I needed something that understood HTML structure and could easy to run interactively.

I already was using jQuery for progressive enhancement and my Firefox always has Firebug setup. And I have been poking at random web-pages with jQuerify bookmarklet for ages. But with this project, jQuery+Firebug combination of tools has now graduated to a 1st class development and troubleshooting toolkit specifically for multi-lingual content.

Here is a couple of basic queries I run in Firebug console window:

• I had most of the links going to a new window and needed to check I did not miss a target attribute: $(“a[target != '_blank']“)
• When comparing languages side-by-side, I needed to see whether URL links were the same. The easiest way to do that was by looking at where those links were actually pointing out. I could of course select an element with Firebug to see all of its content, but it was easier to print a particular attribute automatically, when I hovered over it with a mouse: $(“a”).mouseenter(function(){console.log( $(this).attr(‘href’));})
• If I quickly needed to check which elements were affected by a particular class, I would just highlight them: $(“.NYOnly”).css(“background-color”, “red”)

None of these are hidden secrets, however it may not always be obvious what can be done and how far a couple of lines of jQuery code can go. Here is an example that gets pasted right into Firebug window. It uses jQuery-translate extension to hook into Google Translate API and prints out translated content of a table cell that is clicked on:

$.getScript('http://jquery-translate.googlecode.com/files/jquery.translate-1.3.9.min.js');

$(“td”).click(function(){
$(this).translate(‘ar’, ‘en’, {
replace: false,
each: function(i){console.log( this.translation[i] ) }
})
});

But, every time I put this particular tool aside, I know that picking it back up will be like making up . . . → Read More: Making up with ANTLR]]> I like ANTLR! It is a specialized tool that can really be applied to many difficult tasks when regular expressions get all Dust Puppy like. And I have used it in the past with great success.

But, every time I put this particular tool aside, I know that picking it back up will be like making up after a bad break up. Things feel familiar, but you are still so uncomfortable you cannot get anything working. Only knowing how great the tool is underneath, makes me go through the effort of re-familiarization.

I just downloaded ANTLR 3.1.2 bundled with its own GUI ANTLRWorks that offers visual diagrams, debugger and templates. You would think that would make for an easy out-of-box experience. You would be wrong.

You start the GUI and end up facing a blank screen. Lots of options and tabs for sure, but the only easy start one seems to be ‘Insert rule from template’.

Ok, so here is a couple of rules from templates trying to parse “Hello World!” string:

ID    :    LETTER (LETTER | DIGIT)*
;
LETTER
:    ‘a’..’z’ | ‘A’..’Z’
;

DIGIT    :    ’0′..’9′
;

WS    :    (‘ ‘ | ‘\t’ | ‘\n’ | ‘\r’) { $setType(Token.SKIP); }
;

Not good. We are missing a start state apparently. Ok, let’s add one:

hello    :    ID ID ‘!’
;

Still no good. Start looking at examples, trying to see what bits are compulsory. Ok, the word grammar is missing at the top of the file. Of course, I have both grammar and lexer elements now in one file (ANTLR 3 feature, I believe), but let’s not worry about deep meaning here.

grammar test;

Now, suddenly, syntax diagram starts showing up. Let’s try saving (as test.g) and compiling. No good:

The following token definitions can never be matched because prior tokens match the same input: LETTER

So much for following a template. More digging in examples. Memory really starts to bring back the Dragon Book‘s lessons. What’s the problem with LETTER and who is the prior token here. Ah, we don’t want the lexer to return LETTER (or DIGIT), only ID. So, LETTER and DIGIT are both token fragments, not tokens. Add fragment in front of both definitions. All good?

Nope! Now we have a problem with:

attribute is not a token, parameter, or return value: setType

But I did not write setType, the template provided it! Back to the examples! Apparently, somewhere along the way Skip tokens have gone away and we now have hidden channels instead. Swap that bit with one from an example and try again.

SUCCESS. Switch to interpreter, enter “Hello World!” in input box and run hello rule. Beauty, we have a parse diagram.

The final running grammar example is here:

grammar test;

hello    :    ID ID ‘!’
;

ID    :    LETTER (LETTER | DIGIT)*
;
fragment LETTER
:    ‘a’..’z’ | ‘A’..’Z’
;

fragment DIGIT    :    ’0′..’9′
;

WS    :    (‘ ‘ | ‘\t’ | ‘\n’ | ‘\r’) {  $channel = HIDDEN;  }
;

Hello World! Now, on to the real grammar and (if things really, really work) GATE integration…..

In the United Nations . . . → Read More: Conjunctions in named entities]]> A recent article on lingpipe discussed conjuncted named entities such as Johnson and Johnson and Wallace and Gromit. They suggest that maybe a way of treating this is as a frozen expression. I assume that means relying on statistical measures to see this Multi-Word-Expression repeating enough times to be treated as a unit.

In the United Nations corpus, things can get even more interesting. Let’s look at a relatively easy example: draft resolution A/56/L.28 and Add.1.

Is this a one document (one draft resolution) or two? And if two, then which two? The first one is obviously A/56/L.28. But Add.1 is not a valid document symbol, it is actually an (additive?) coreference to the first one and resolves to A/56/L.28/Add.1?

The answer (as good as I can make it so far) could lie in FRBR distinction between Expression and Manifestation. A resolution is an expression of Member States’s proposals and negotiations. To some degree, it evolves over several meetings. However between the discussions, the latest version or changes need to be reported to make sure they are formally registered and also to ensure the next round of discussions could have latest documents to work from.

In our case, the first time the draft resolution had to be presented it was published under A/56/L.28 (which incidentally means a limited distribution document 28 of the General Assembly’s 56th regular session). So, the initial Manifestation of the draft resolution became this physical document with a distinct symbol assigned.

But apart from its text, draft resolution has a list of sponsoring Member States. That list can change as draft resolution gains sponsors. These additional sponsors were in the Addendum A/56/L.28/Add.1. But the addendum does not make sense without the original document, so actually both physical documents represent one logical draft resolution, which is reflected in the grammar of the text (draft resolution, not resolutions).

What this means for named entity annotations and for recognition algorithms is hard to say and is something I am looking at with my PhD research.

One of such tools is Exhibit, one of graduates from SIMILE project. It allows to do a very interactive webpage driven by just HTML+Javascript, with no server-side component required. I really like SIMILE’s tools, even . . . → Read More: CiteULike Exhibit visualization]]> Homegrown visualization is not the only way to quickly navigate CiteULike references. There are other tools that display bibliographies in interesting ways.

One of such tools is Exhibit, one of graduates from SIMILE project. It allows to do a very interactive webpage driven by just HTML+Javascript, with no server-side component required. I really like SIMILE’s tools, even though it feels like development slowed somewhat recently.

There is an example of how to import and display bibtext within Exhibit. It is not difficult, just a couple of steps. It must have been a popular section, as there is now a dedicated new tool for it.

Citeline Exhibit Builder allows to load in bibtext and presents editing interface to customize Exhibit’s presentation of the publications. It looks great and seem to work well. A nice aspect is that it allows to chose which bibtext fields to expose as filter facets. With original tutorial that would require html editing and understanding Exhibit mindset. Citeline nicely hides user from it.

There was a couple of small problems. Apparently, there is a way to login and ‘claim’ your presentation. I couldn’t test that as OpenID authentication failed (something about a nonce). Also, there is jsMath library but, once the generated Exhibit is downloaded, it fails with cross-server issues. Finally, as with most end-to-end solutions, it does not do data preprocessing/normalization to allow me, for example, to combine author/editor fields for sorting purposes.

Citeline is a very promising tool and I am certainly going to keep it in mind for publishing my bibliographies.

Nice as CiteULike is, it is fairly difficult to get an overall picture of one’s own . . . → Read More: Visualizing CiteULike collections]]> I am collecting my reading and reference material in CiteULike. I like the service because it can capture details from multiple sources. It also allows to discover what was collected by other interesting people through tags, people and bookmarks graph navigation.

Nice as CiteULike is, it is fairly difficult to get an overall picture of one’s own collection. It is especially difficult to see quickly if there are people who serve as hubs by collaborating with multiple different groups. The information is there, but it requires a lot of clicks to find it out.

My usual solution is to export information out, massage it into Graphviz format and use graph segmentation and layout algorithms to get a better overview. I have talked about Graphviz a number of times on this blog before. This is yet another time it proved useful.

I started by exporting CiteULike’s content of my library. I found Endnote export format to be more structured and therefore easier to parse. I then run it through a custom Python program that basically spat out graph with titles pointing at authors. That produced a very large graph and was not particularly useful.

The next step was to discover disjointed clusters of titles/authors. I used ccomps with -v and -x flags (e.g. ccomps.exe -v -x -o comp.dot output.dot).

ccomps gave me partitioned graphs as well as statistics on number of nodes/edges in each graph. I could then choose a graph with large number of nodes/edges (eventually, all of them) and run it through neato with overlap=scale and splines=true (e.g. neato.exe -Tgif -o neato_1.gif -Goverlap=scale -Gsplines=true comp_1.dot).

The resulting graph was still not perfect, but it was a good start. I also tried fdp instead of neato, but that seemed to produce giraffe versions of the graph with graph edges being overly long.

You can see an example of neato output for one of my clusters. Warning: if it causes problems due to its size, try it with IrfanView; that program can display even improbably large graphs (e.g. unpartitioned ones).

I have run into some problems as well that would either cause partitions combine together or produce duplicate nodes and edges.

The first problem was that sometimes a person was an author and sometimes an editor. I was interested in both, so collapsed those fields together. That caused some non-people to then show up on the graph and connect clusters in unexpected ways. For my library the specific value was ‘European’, so I filtered it out in the code.

The second problem was to do with CiteULike’s parsing. Sometimes, it would split a first+last name into separate names, probably due to incorrect manual entry at some point. I had to fix those at the source by editing corresponding CiteULike entry. Probably a good thing to do anyway.

The other problem is right out of the co-reference resolution domain. Sometimes names would include full first names, sometimes only a first name initial. I have worked around that by normalizing all first names to the initials. Obviously, this could collapse entries belonging to multiple real people into one.

Further on name problems, in cases of non English names (e.g. Spanish names with multiple surnames), CiteULike would get confused which part is which and not display or export it correctly. Additionally, sometimes characters such as ñ would be entered as plain n. Those also needed to be corrected manually.

The project only took a couple of hours including writing code and cleanup. It is already useful to me, as I found a new person who was in unexpectedly large number of papers and also found a chain of connections that might be interesting to follow more closely.

There is of course a lot more that could be done. Automatic co-reference of misspelt names, layout hints based on number of times authors appeared together, color coding of tags – these are just some of the easy ideas.

There might even be a small project/paper in doing co-reference resolution and cleaning up CiteULike data? After all, similar projects were done for Wikipedia. I don’t think CiteULike currently makes a full export available, but they do have some so might be amendable to exporting a special set for research purposes.

I am interested in both topics. I did 3 years as senior technical support at BEA and could see how applying NLP techniques on written notes . . . → Read More: New mailing list to discuss junction of NLP and Software Engineering]]> Dr. René Witte has just created a new mailing list (SENLP) to discuss applying NLP techniques to Software Engineering and also to discuss general Software Engineering issues in developing NLP systems.

I am interested in both topics. I did 3 years as senior technical support at BEA and could see how applying NLP techniques on written notes in support cases could have improved quality of technical support. I did not get to do any of that, but some interest remains.

The second topic is even more interesting and important to me. It can build on current discussions currently held on blogs (see ‘The USES Issue‘ at Niels Ott’s blog) and in journals (see: ‘Empiricism Is Not a Matter of Faith‘ by Ted Pedersen). While some of the issues are discussed on mailing lists for individual pieces of software, a place to discuss cross-cutting concerns is very welcome.

I have joined the list and hope to see at least some of my readers there as well.