Table of results

Below is the table of Precision and recall results commputed on the output provided by the participants with the help of the GroupEval class of the Alignment API.

java -cp lib/procalign.jar fr.inrialpes.exmo.align.util.GroupEval -c -l "karlsruhe2,umontreal,fujitsu,stanford"

I further provide some comments and about these results, in next section. I also comment on a number of modifications that have been made on the provided files.

Analysis of the results

Here are some consideration of the results obtained by the various participants. These are not statistically backed up and only corresponds to a rough analysis. More explainations are found in the papers presented by the participants.

There were two groups of competitors...

In this test, there are clear winners it seems that the results provided by Stanford and Fujitsu/Tokyo outperform those provided by Karlsruhe and Montréal/INRIA.

In fact, it can be considered that these constitute two groups of programs. The Stanford+Fujitsu programs are very different but strongly based on the labels attached to entities. For that reason they performed especially well when labels were preserved (i.e., most of the time). The Karlsruhe+INRIA systems tend to rely on many different features and thus to balance the influence of individual features, so they tend to reduce the fact that labels were preserved.

This intuition should be further considered in the light of more systematic tests which were planned but never made.

...and indeed three groups of tests

Without going through a throughout statistical analysis of the results, it seems that the separation between sets of test that we presented (indicated by the first digit of their numbers) is significant for the participants as well.

• The first four tests were relatively easily handled by all participants. All programs showed there a better recall than precision (but not very significant).
• The systematic tests show more difficulty for the programs in general and for those of the second group in particular. This can be seen as a good way of assessing precisely system capabilities.
• The real life test were even more difficicult for both groups of participants.

Additional remarks

It is very difficult indeed to compare these results. This is true because there were different tests and not all go in the same direction. So aggregating the results can be done in many ways (e.g., averaging, global P/R, counting dominance). Moreover, these results are based on two measures, precision and recall, which are very easily understood but dual in the sense that increasing one often decreases the other. This means that one algorithms can have sometimes the same results as another but they are found non comparable in the table.

As an indication, the average values are given below:

Modifications made to the files

Establishing these results has been as automated as possible. However, we add to modify the provided files in a number of ways that are listed below.

The complete results can be retrieved from this archive

One first remark is that we finally had some problems that we did not solved with accentuated characters (test #206).

Karlsruhe

Karlsruhe did not provided results for all the tests and we did not complete their tests for them.

The Karlsruhe results for 204-206 were inverted (i.e., ontology 2 was that of the 101 test). We used the ParserPrinter utility of the alignment API to revert the correct order. Otherwise, the results provided were 0.

Most of Karlsruhe results contained references to the first test series (notably Techreport and others in the # 230 which have been discarded).

It also contained references to special ontologies (onto-noinst) not provided with the tests and references to alignments with FOAF entities (perfectly valid but discarded for technical reasons). It is believed that these discarding tend to improve the results of the tests anyway.

The first set of result files from Karlsruhe were based on the initially released tests, had no threshold applied and some KAON objects were in the results. After asking to redo the tests these problems were solved.

Fujitsu and Kyoto university

The ontology namespaces and filenames provided in these tests were based on a local file system (moreover, filenames onto.rdf were changed to refonto.rdf). They thus have been automatically converted in those of the contest.

INRIA and University of Montréal

The results provided by INRIA and University of Montréal used the Alignment API. The only change that was made was in the namespace of the alignment API itself since that of the implementation did not correspond to that of the specification.

Stanford

The results of Stanford were provided in pure RDF files with heavy use of references and the use of XML attributes for datatyped attributes. Since our alignment parser expect RDF/XML files and works only on the XML (i.e., it is unable to resolve rdf references), we developped an XSLT stylesheet solving these problems.

Results #302 and #221 were however slightly different and deserved a separate treatment.

Results #230, 201, 202 were not referring to the last file so they have been corrected for the best (by discarding cells causing trouble).

Most URI were local (referring to my own filesystem), so we had to adjust this too.

Stanford provided two additional tests that where not included in these results. They are, however, perfectly legitimate tests that should be included in further campaign.

Lockheed Martin

Lockheed Martin provided a set of results in notation 3 (n3). Due to the size of the result files and the need to create another XSLT stylesheet on XML/RDF files returned by CWM, it was not possible to included this before the meeting.

Appendice 1: shell-script for adjusting most of the problems

#!/bin/sh

CWD=`pwd`

for i in `ls -d ???`
do
if [ -a $i/refalign.rdf ]
then
ed $i/refalign.rdf << EOF
1,$ s:http\://exmo.inrialpes.fr/align/1.0:http\://knowledgeweb.semanticweb.org/heterogeneity/alignment:
w
EOF
else
	echo no $i/refalign.rdf
fi
if [ -a $i/umontreal.rdf ]
then
ed $i/umontreal.rdf << EOF
1,$ s:http\://exmo.inrialpes.fr/align/1.0:http\://knowledgeweb.semanticweb.org/heterogeneity/alignment:
w
EOF
else
	echo no $i/umontreal.rdf
fi
if [ -a $i/karlsruhe.rdf ]
then
	echo "Normaly we should have to suppress Kaon objects" 
	echo "Normaly we should have to suppress non existing objects" 
ed $i/karlsruhe.rdf << EOF
1,$ s;101/onto.rdf#Techreport;101/onto.rdf#TechReport;
w
EOF
else
	echo no $i/karlsruhe.rdf
fi
if [ -a $i/fujitsu.rdf ]
then
ed $i/fujitsu.rdf << EOF
1,$ s;file:///D:/align/align/oacontest/;http://co4.inrialpes.fr/align/Contest/;
1,$ s;file:/D:/align/align/oacontest/;http://co4.inrialpes.fr/align/Contest/;
1,$ s;refonto.rdf;onto.rdf;
1,$ s:../../../dtd:../dtd:
w
EOF
else
	echo no $i/fujitsu.rdf
fi
if [ -a $i/stanford.rdf ]
then
	xsltproc stan.xsl $i/stanford.rdf > $i/stan.rdf
	mv $i/stan.rdf $i/stanford.rdf
else
	echo no $i/stanfordrdf
fi
done

Appendice 2: XSLT stylesheet for transforming RDF into parsable XML/RDF

This can be found here.

Appendice 3: Complementary results

This first table displays, together with the results of the participants of the contests, those obtained by the demonstration aligners provided with the Alignment API and the first results obtained by Karlsruhe.

java -cp /Volumes/Phata/JAVA/ontoalign/lib/procalign.jar fr.inrialpes.exmo.align.util.GroupEval -l "std,nea,ssda5,edna5,sdna5,karlsruhe,karlsruhe2,umontreal,fujitsu,stanford" -f "pr" -c

The following table presents the F-measure and overall results for the same set of algorithms.

java -cp /Volumes/Phata/JAVA/ontoalign/lib/procalign.jar fr.inrialpes.exmo.align.util.GroupEval -l "std,nea,ssda5,edna5,sdna5,karlsruhe,karlsruhe2,umontreal,fujitsu,stanford" -f "mo" -c