Ontology Alignment Evaluation Initiative - HOBBIT evaluation for OAEI 2018 Campaign

HOBBIT evaluation for OAEI 2018

Instructions for organisers

OAEI track organisers should follow the HOBBIT benchmark specification.

These benchmark definitions can be used as reference:

• The Largebio Benchmark. Contact: Ernesto Jimenez Ruiz: ernesto [dot] jimenez [dot] ruiz [at] gmail [dot] com
• The Spatial Benchmark. Contact: Nina Saveta: jsaveta [at] ics [dot] forth [dot] gr

Instructions for participants

System developers willing to participate in the HOBBIT tracks are required to follow different instructions with respect to the SEALS wrapping.

Check the following examples to submit a HOBBIT compliant system:

• Ontology matching tasks. See example of LogMap wrapping for HOBBIT. So far implements the interfaces for Conference, Anatomy, Largebio and SPIMBENCH tracks. Contact: Ernesto Jimenez Ruiz: ernesto [dot] jimenez [dot] ruiz [at] gmail [dot] com
• Link discovery task. See instructions here. Main contact: Nina Saveta: jsaveta [at] gmail [dot] com

In addition, Sven Hertling (sven [at] informatik [dot] uni-mannheim [dot] de) has implemented a framework to create SEALS and HOBBIT compliant matching systems. Check details here and/or get in touch with him. It also contains a small client which allows to evaluate the matcher locally (i.e., it downloads the benchmarks and computes the KPIs).

The standalone software library called the HOBBIT Java SDK may also enable an easier way to implement a benchmark or a system. It will also make possible to run local tests before submitting to the platform.

Participants have to follow this procedure:

1. Register your matching tool. Participants must register their intention to participate in the OAEI 2018 by filling this online form.
2. Wrap the current version of your matching tool. Follow the instruction in the examples above and push the docker images to the HOBBIT platform (a personal access token is required).
3. Test your tool with the data sets available. Test your system (locally when possible) and remotely using HOBBIT's public instance. More details in the HOBBIT wiki pages.