Nice as CiteULike is, it is fairly difficult to get an overall picture of . . . → Read More: Visualizing CiteULike collections]]> I am collecting my reading and reference material in CiteULike. I like the service because it can capture details from multiple sources. It also allows to discover what was collected by other interesting people through tags, people and bookmarks graph navigation.

Nice as CiteULike is, it is fairly difficult to get an overall picture of one’s own collection. It is especially difficult to see quickly if there are people who serve as hubs by collaborating with multiple different groups. The information is there, but it requires a lot of clicks to find it out.

My usual solution is to export information out, massage it into Graphviz format and use graph segmentation and layout algorithms to get a better overview. I have talked about Graphviz a number of times on this blog before. This is yet another time it proved useful.

I started by exporting CiteULike’s content of my library. I found Endnote export format to be more structured and therefore easier to parse. I then run it through a custom Python program that basically spat out graph with titles pointing at authors. That produced a very large graph and was not particularly useful.

The next step was to discover disjointed clusters of titles/authors. I used ccomps with -v and -x flags (e.g. ccomps.exe -v -x -o comp.dot output.dot).

ccomps gave me partitioned graphs as well as statistics on number of nodes/edges in each graph. I could then choose a graph with large number of nodes/edges (eventually, all of them) and run it through neato with overlap=scale and splines=true (e.g. neato.exe -Tgif -o neato_1.gif -Goverlap=scale -Gsplines=true comp_1.dot).

The resulting graph was still not perfect, but it was a good start. I also tried fdp instead of neato, but that seemed to produce giraffe versions of the graph with graph edges being overly long.

You can see an example of neato output for one of my clusters. Warning: if it causes problems due to its size, try it with IrfanView; that program can display even improbably large graphs (e.g. unpartitioned ones).

I have run into some problems as well that would either cause partitions combine together or produce duplicate nodes and edges.

The first problem was that sometimes a person was an author and sometimes an editor. I was interested in both, so collapsed those fields together. That caused some non-people to then show up on the graph and connect clusters in unexpected ways. For my library the specific value was ‘European’, so I filtered it out in the code.

The second problem was to do with CiteULike’s parsing. Sometimes, it would split a first+last name into separate names, probably due to incorrect manual entry at some point. I had to fix those at the source by editing corresponding CiteULike entry. Probably a good thing to do anyway.

The other problem is right out of the co-reference resolution domain. Sometimes names would include full first names, sometimes only a first name initial. I have worked around that by normalizing all first names to the initials. Obviously, this could collapse entries belonging to multiple real people into one.

Further on name problems, in cases of non English names (e.g. Spanish names with multiple surnames), CiteULike would get confused which part is which and not display or export it correctly. Additionally, sometimes characters such as ñ would be entered as plain n. Those also needed to be corrected manually.

The project only took a couple of hours including writing code and cleanup. It is already useful to me, as I found a new person who was in unexpectedly large number of papers and also found a chain of connections that might be interesting to follow more closely.

There is of course a lot more that could be done. Automatic co-reference of misspelt names, layout hints based on number of times authors appeared together, color coding of tags – these are just some of the easy ideas.

There might even be a small project/paper in doing co-reference resolution and cleaning up CiteULike data? After all, similar projects were done for Wikipedia. I don’t think CiteULike currently makes a full export available, but they do have some so might be amendable to exporting a special set for research purposes.

I do not actually agree that his first discovered rule is actually “a skill” (I think it is more of a test procedure), but the question asked does require a thought.

I was . . . → Read More: My take on “problem solving skills”]]> cynicalman asks what the “problem solving skills” in a job description actually mean and then tries to discover and blog the answers.

I do not actually agree that his first discovered rule is actually “a skill” (I think it is more of a test procedure), but the question asked does require a thought.

I was frequently told that I have good problem solving skills. So, here is my quick take on what those skills are:

Being able to quickly deduce a structure and meaning from the given examples or behaviour

For example, my co-worker was having a problem with a Fortran program. I haven’t seen Fortran before, but was able to figure out the control structure rules on the fly and see where the bug was. Another example would be to take a program with an unfamiliar menu structure (or a familiar Microsoft one) and quickly find where a particular option could be.

Knowing how to ask question – of yourself and others – that would partition the problem space

So, you say the server has crashed? is not a good question. How about When you say the server crashed, did the program exit completely? If not, was it rather just very slow and/or returned to normal a while later? It is really a couple of questions here, but the idea is to split the possible problem space into segments and decide which segment to keep working with. Similarly a good question would be: If this was caused by a network problem, how/where/when would it show up? Which log or trace file do I need to check to confirm or reject this?

Knowing how to convert data into information

Getting a 3 page long java thread dump is a (bad?) luck. Knowing how to read that thread dump to identify what the program was doing at the time and what it was _not_ doing is a problem solving skill.

Being able to manipulate large datasets and extract required individual entries

To me, this covers not being scared by 2GBytes access logs when I search for a problem behaviour pattern. It also covers knowing the tools to slide/dice/reconstruct the data sets.

Having a toolkit and knowing it well

At the moment, my toolkit is Vim with couple of plugins for editing and RegExp work, Ethereal for network, FileMon/RegMon/TcpMon from SysInternals for Windows troubleshooting, XMLStarlet for XML files, and, of course, Cygwin for its full toolchain. I have also written custom programs, mostly for complex data extraction. Finally, I use GraphViz for visualization of my extraction results. One of the things that frustrates me most is a person reading a log file in Notepad. What can they do with it except looking at it one line at a time? No highlighting, no place marks, no way to interactively clean it up by regexp deleting some unrelated sections.

And, above all, not giving up

Most of the situations can be troubleshooted successfully. Especially if you can reproduce it, insert trace statements or do other developer level modifications. If you think you hit a writer’s block of troubleshooting, use the same techniques writers do (try anything, switch attention, define next action, etc.)

Comments, thoughts, references?

BlogicBlogger Over and Out

He is also asking the community to contribute their lists. Well, I have three tools that I like enough to mention (again).

While GraphViz by itself is not Java, it . . . → Read More: The power of GraphViz]]> Ever felt the need to extract some relations from the configuration or data and present it in a visual form nicely layed out. Ever given that up as too hard due to the hard problem of laying out the elements? If you did, then check out GraphViz.

While GraphViz by itself is not Java, it is cross-platform and is fairly easy to setup and invoke from Java. It is also open-source, if that is important to you.

So, what does GraphViz do? Well, it does fancy graph layouts. Sounds boring until you realize that graphs – or data that could become graphs – are actually all over the Java world.

Let me give you a few examples:

• Ant task dependancies are graphs. If your ant file is rather large, check out ant2dot, Vizant or AntGraph (WayBackMachine archive).
• Spring dependencies are graphs. If your Spring project is getting complex, get an overview of it with SpringViz.
• If your code is already deployed, but you experience hangs or slowdowns, Graphviz will help to visualize the thread dumps.
• If you are not happy with GIF/SVG/PNG/PS output that Graphviz is capable of generating, you could always try a more interactive approach with ZGRViewer (a bit raw, but very promising project)
• Many other uses exist, some of which are described at the GraphViz resource page
• Finally, if you are able to generate text file that says a->c; b->c; c->d, you can generate GraphViz input (dot file) and let the program itself do the magic of layout. And once you master a simple dot file, you can start adding such bells and whistles as colors, clusters and structures.

If you have an itch to scratch, let GraphViz help you with it.

BlogicBlogger Over and Out