Complex track - Taxon dataset - Evaluation

The output alignments can be downloaded here.

Evaluation measures

Differently from the last years, in 2021 we manually evaluate the quality of the generated alignments, in terms of precision.

Execution environment

The alignment systems have been executed on a Ubuntu Linux machine configured with 32GB of RAM running under a Intel Core CPU 2.00GHz x8 cores. All measurements are based on a single run.

Participants

Only two systems applied for the complex track: AMLC and AMD, among which only AMLC was able to produce results. AMD was not able to parse the files, hence it produced no result.

The track manager launched all the systems (not only those that applied for the complex track) using the complex track dataset. Three experts evaluated separately the results of AMLC, Logmap and LogmapLite systems. Among them, one expert evaluated the results of all systems. It was the first time that those experts evaluated complex alignments using EDOAL language.

Evaluation results

Table below presents the synthesis of the manual evaluation produced by the three experts. Each cell contains the number of alignments and among them the number of validated alignments.

Discussion

First remark, the experts expected to evaluate the links between taxonomic entities, i.e. taxa or scientific names, belonging to one of the proposed taxonomic registers (AgronomicTaxon, Agrovoc, DBpedia, TAXREF-LD). However, only little systems were able to find alignments between these entities. In many other cases, the aligned resources are terms from the vocabularies shared by several taxonomic registers (e.g. properties from SKOS, Dublin Core Terms or DOLCE+DnS Ultralite). This type of match is named “vocab” in table X. For instance, Logmap states that DOLCE+DnS Ultralite property hasPart (http://www.ontologydesignpatterns.org/ont/dul/DUL.owl#hasPart) is equivalent to Agrovoc property hasPart (http://aims.fao.org/aos/agrontology#hasPart) . Furthermore, some systems propose trivial alignments of vocabulary terms with themselves. For example, ALMC states that dcterm:created in Agrovoc is equivalent to dcterm:created in TAXREF-LD, which is true but useless. This type of trivial matches is named "vocab trivial" in table above.

Some complex alignments involve some SKOS properties related to creation or modification dates of a resource in the taxonomic register. This information is not relevant to identify entity alignments.

Some alignments involve entities belonging to external taxonomic registers (e.g. Geospecies or Umbold). The experts assume that some of these external entities come from some alignments that were declared in the taxonomic registers, e.g. TAXREF-LD comes with alignments with Geospecies.

A discussion between the experts emerged about this type of alignment and several questions came up:

• Why are those external entities part of the results? They may be true but mostly add noise to the results. Ideally, an alignment system should focus on the entities defined by the data source (typically within a configurable namespace).
• Some of these external URIs are not dereferenceable (they cannot be resolved with a sheer web browser). Finding information about them requires looking for downloadable ontologies which makes validation cumbersome.
• Should the alignments that contain some of those external entities be evaluated by the experts and how? The first answer of the expert will be that those alignements should not be evaluated and removed from the results. The experts identify this problem by adding a quality “out of the scope” in their evaluation.

Only the ATM, Fine-TOM and Logmap system seems capable of proposing valid alignments between taxa. Moreover, Logmap seems capable of dealing with challenge C2, that is, align two taxa that have different reference names but where the reference name on one side is considered a synonym on the other side.

To conclude, overall the experts were surprised that so little alignments were proposed. The taxonomic registers contain thousands to hundreds of thousands of taxa each, yet several tools were not able to propose more than a handful of alignments on vocabulary terms. No valid complex alignments were proposed between taxa. Only systems based on simple alignments were able to detect some of them.

Furthermore, the experts feel like a new interface is needed to visualize the results and ease the evaluation process. This interface should help filter out trivial matches and mention the origin of the entity referenced in the alignment (external resource, shared vocabulary). Moreover, the experts had difficulties understanding the EDOAL language. We expect that next year the evaluation process will integrate some tools meant to simplify the exploitation of EDOAL files like the edoal2csv tool [1]. Regarding taxonomic registers, a new resource about taxonomy was identified by the experts: the NCBI taxonomy available on the Agroportal repository. The TAXREF-LD taxonomic register used in the complex track is an old version. A new version is available (version 13) on the web. Unfortunately this graph is huge and could not be parsed by some system. A solution is to select a subset of TAXREF-LD covering only plant taxonomy. This entails the question about whether the other large taxonomic registers (such as NCBI or DBpedia) should also be limited to plant taxonomy.

References