Ontology Alignment Evaluation Initiative - OAEI-2008 Campaign

Ontology Alignment Evaluation Initiative

Benchmark preliminary results

Like last year, 13 systems participated in the benchmark track of this year's campaign.

The evaluation has been performed on the files provided by the participants (available here) which have been processed by the following script). The full set of normalised alignments is also available.

Summarised results

Table 1 provides the consolidated results by groups of tests. We display the results of participants as well as those given by some simple edit distance algorithm on labels (edna). The computed values are real precision and recall and not an average of precision and recall.

These results show already that the three systems which last year were ahead of the pack, are still at the top with are relatively ahead (ASMOV, Lily and RiMOM) with three close followers (AROMA, DSSim, and Anchor-Flood replacing Falcon, Prior+ and OLA2 last year). No system had strictly lower performance than edna. Each algorithm has its best score with the 1xx test series. There is no particular order between the two other series.

Comparison to 2007 tests

After remarks of last year we made two changes on the tests this year:

• tests #249 and 253 still had instances in the ontologies, these have been suppressed this year. Hence the test is more difficult than previous years;
• test which scrambled all labels within the ontology (#201-202, 248-254 and 257-262), have been complemented by tests which respectively only scramble 20%, 40%, 60% and 80% of the labels. Globally, this make the tests easier to solve.

algo	refalign		edna		aflood		aroma		ASMOV		CIDER		DSSim		GeRoMe		Lily		MapPSO		RiMOM		SAMBO		SAMBOdtf		SPIDER		TaxoMap
test	Prec.	Rec.	Prec.	Rec.	Prec.	Rec.	Prec.	Rec.	Prec.	Rec.	Prec.	Rec.	Prec.	Rec.	Prec.	Rec.	Prec.	Rec.	Prec.	Rec.	Prec.	Rec.	Prec.	Rec.	Prec.	Rec.	Prec.	Rec.	Prec.	Rec.
1xx	1.00	1.00	0.96	1.00	1.00	1.00	1.00	1.00	1.00	1.00	0.99	0.99	1.00	1.00	0.96	0.79	1.00	1.00	0.92	1.00	1.00	1.00	1.00	1.00	1.00	1.00	0.99	0.99	1.00	0.34
2xx	1.00	1.00	0.41	0.56	0.96	0.69	0.96	0.70	0.95	0.85	0.97	0.57	0.97	0.64	0.56	0.52	0.97	0.86	0.48	0.53	0.96	0.82	0.98	0.54	0.98	0.56	0.97	0.57	0.95	0.21
3xx	1.00	1.00	0.47	0.82	0.95	0.66	0.82	0.71	0.81	0.77	0.90	0.75	0.90	0.71	0.61	0.40	0.87	0.81	0.49	0.25	0.80	0.81	0.95	0.80	0.91	0.81	0.15	0.81	0.92	0.21
H-mean	1.00	1.00	0.45	0.61	0.97	0.71	0.96	0.72	0.95	0.85	0.97	0.62	0.97	0.68	0.59	0.54	0.96	0.87	0.52	0.55	0.95	0.83	0.98	0.59	0.98	0.61	0.67	0.62	0.95	0.22

Table 2: Resuts on the 2007 tests only.

Comparison to 2004 tests

Table 3 provides results on the very easy tests of 2004. As expected, systems perform very well.

algo	refalign		edna		aflood		aroma		ASMOV		CIDER		DSSim		GeRoMe		Lily		MapPSO		RiMOM		SAMBO		SAMBOdtf		SPIDER		TaxoMap
test	Prec.	Rec.	Prec.	Rec.	Prec.	Rec.	Prec.	Rec.	Prec.	Rec.	Prec.	Rec.	Prec.	Rec.	Prec.	Rec.	Prec.	Rec.	Prec.	Rec.	Prec.	Rec.	Prec.	Rec.	Prec.	Rec.	Prec.	Rec.	Prec.	Rec.
1xx	1.00	1.00	0.96	1.00	1.00	1.00	1.00	1.00	1.00	1.00	0.99	0.99	1.00	1.00	0.96	0.79	1.00	1.00	0.92	1.00	1.00	1.00	1.00	1.00	1.00	1.00	0.99	0.99	1.00	0.34
2xx	1.00	1.00	0.67	0.74	0.98	0.92	0.99	0.94	0.99	0.98	0.98	0.82	0.98	0.89	0.93	0.72	0.99	0.98	0.66	0.72	1.00	0.98	0.99	0.75	0.99	0.77	0.98	0.82	0.94	0.24
3xx	1.00	1.00	0.47	0.82	0.95	0.66	0.82	0.71	0.81	0.77	0.90	0.75	0.90	0.71	0.61	0.40	0.87	0.81	0.49	0.25	0.80	0.81	0.95	0.80	0.91	0.81	0.15	0.81	0.92	0.21
H-mean	1.00	1.00	0.67	0.80	0.98	0.90	0.97	0.92	0.97	0.96	0.97	0.84	0.97	0.88	0.90	0.69	0.98	0.96	0.70	0.71	0.97	0.96	0.98	0.80	0.98	0.81	0.57	0.85	0.95	0.25

Table 3: Resuts on the 2004 tests only.

Precision/recall graphs

This year again the apparently best algorithms provided their results with confidence measures. It is thus possible to draw precision/recall graphs in order to compare them.

We provide in Figure 1 the precision and recall graphs of this year. They are only relevant for the results of participants who provided confidence measures different from 1 or 0 (see the table in the Summary page). This graph has been drawn with only technical adaptation of the technique used in TREC. Moreover, due to lack of time, these graphs have been computed by averaging the graphs of each of the tests (instead to pure precision and recall). They do not feature the curves of previous years since the test sets have been changed.

These results and those displayed in Figure 2 single out a group of systems, ASMOV, Lily, and RiMOM which seem to perform these tests at the highest level of quality. So this confirm the lead that we see on raw results.

Like the two previous years, there is a gap between these systems and their followers. The gap between these systems and the next ones (AROMA, DSSim, and Anchor-Flood) has reformed. It was filled last year by Falcon and OLA and Prior which did not participate this year.

Triangle

The triangular display shows the same results as the other representations.

Full results

n/a: result alignment not provided or not readable
NaN: division per zero, likely due to empty alignent.