This page presents the results of the OAEI 2009 Library Track. The evaluation was done according to two scenarios. From a methodological perspective, it is very similar to what was done for the evaluation of the previous 2008 Library Track, which featured a "thesaurus merging" evaluation (based on a post-hoc gold standard) and a "re-indexing" one (assessing the use of mappings for translating subject annotations from one thesaurus to another). The main difference is that in the TELplus case, the first evaluation method has been replaced by comparing to an already existing partial reference alignment, avoiding to assess the results of participants manually.
For each scenario, this page presents the raw results that were obtained. For further details, we refer the reader to this report and to the slides presented at the Ontology Matching workshop.
The statistics for the results sent by the unique TaxoMap participant are:
| Type of relation | LCSH-RAMEAU | RAMEAU-SWD | LCSH-SWD |
|---|---|---|---|
| exactMatch | 5074 | 1265 | 38 |
| broadMatch | 116789 | 17220 | 0 |
| narrowMatch | 48817 | 6690 | 0 |
| relatedMatch | 13205 | 1317 | 0 |
As no participant used the training set we provided, we use the complete MACS mappings as gold standard. This reference alignment comprises 87183 LCSH-RAMEAU mappings, 13723 RAMEAU-SWD mappings, and 12203 LCSH-SWD mappings.
The table below shows the results when taking into account all mappings that belong to a certain relation selection. "-" for a given relation selection means that no extra relation was provided at that level, hence the results are identical to the ones of the previous selection level.
| TaxoMap links evaluated | LCSH-RAMEAU | RAMEAU-SWD | LCSH-SWD | |||
|---|---|---|---|---|---|---|
| Precision | Coverage | Precision | Coverage | Precision | Coverage | |
| exactMatch | 72.1 | 5.7 | 27.1 | 1.4 | 44.4 | 0.03 |
| exactMatch + broadMatch | 3.6 | 6.9 | 2.3 | 1.9 | - | - |
| exactMatch + broadMatch + narrowMatch | 2.8 | 7.3 | 1.8 | 2.0 | - | - |
| all relations | 2.7 | 7.5 | 1.9 | 2.2 | - | - |
The table below shows the results obtained when selecting only the "best" available mapping for one concept (that is, the one with the highest confidence measure), and discarding the others.
| TaxoMap links evaluated | LCSH-RAMEAU | RAMEAU-SWD | LCSH-SWD | |||
|---|---|---|---|---|---|---|
| Precision | Coverage | Precision | Coverage | Precision | Coverage | |
| exactMatch | 78.7 | 5.7 | 39.5 | 1.4 | 44.4 | 0.03 |
| exactMatch + broadMatch | 22.0 | 6.0 | 13.5 | 1.6 | - | - |
| exactMatch + broadMatch + narrowMatch | 14.4 | 5.9 | 10.8 | 1.6 | - | - |
| all relations | 13.4 | 5.8 | 10.9 | 1.7 | - | - |
Those are the results for automated evaluation, using a gold standard of books indexed against both thesauri of the pair considered:
| TaxoMap links evaluated | LCSH-RAMEAU | RAMEAU-SWD | LCSH-SWD | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Precision | Recall | Jaccard | Precision | Recall | Jaccard | Precision | Recall | Jaccard | |
| exactMatch | 22.3 | 6.1 | 5.5 | 14.2 | 3.1 | 2.4 | 1.3 | 0.003 | 0.002 |
| exactMatch + broadMatch | 2.1 | 7.8 | 1.5 | 2.3 | 3.6 | 1.1 | - | - | - |
| exactMatch + broadMatch + narrowMatch | 1.2 | 9.2 | 1.0 | 0.8 | 3.9 | 0.5 | - | - | - |
| all relations | 1.1 | 9.3 | 0.9 | 0.7 | 4.0 | 0.5 | - | - | - |
The table below shows the results obtained when selecting only the "best" available mapping for one concept (that is, the one with the highest confidence measure), and discarding the others.
| TaxoMap links evaluated | LCSH-RAMEAU | RAMEAU-SWD | LCSH-SWD | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Precision | Recall | Jaccard | Precision | Recall | Jaccard | Precision | Recall | Jaccard | |
| exactMatch | 22.8 | 5.8 | 5.3 | 14.2 | 1.9 | 1.7 | 1.2 | 0.002 | 0.002 |
| exactMatch + broadMatch | 10.2 | 6.0 | 4.9 | 6.9 | 2.0 | 1.7 | - | - | - |
| exactMatch + broadMatch + narrowMatch | 7.2 | 4.5 | 3.3 | 5.9 | 1.9 | 1.5 | - | - | - |
| all relations | 6.4 | 4.0 | 2.9 | 5.8 | 1.9 | 1.5 | - | - | - |
Initial location of this page: http://www.few.vu.nl/~aisaac/oaei2009/results.html