Submitting a data set to OAEI
The aim of this (short) document is to explain how to propose a
dataset and evaluation modalities to OAEI.
Since 2010, these rules have been made more precise.
Hereby, organiser denotes a group of person willing to propose
a data set to evaluate or an experiment in a particular OAEI campain.
OAEI denotes the people coordinating the OAEI campaigns
including its steering committee.
In practice, OAEI is the set of all organisers of a particular
campaign (so, may be you).
Data set submission
Potential organisers must send an email providing the folowing
information to OAEI:
all this information, would be preferrably put available somewhere on
the web (most of this information follows the template).
A short presentation of the dataset and evaluation in very few
lines like there are in the call for participation to campaigns;
- Name and email of the organisers;
- Explanation of what the evaluation is supposed to show (e.g., test a
particular feature of matchers);
- Description of the data sets (where do they come from, if they are
in public access -preferred- or not, their size, languages and
- Description of the evaluation modalities and which conclusion
could be derived from the eventual results;
- Potential list of participating system with indication of their
asserted willingness to participate if they have been contacted
(this can also be "all usual participants of OAEI").
In principle, this should be sent at the beginning of the year
(between January and March, the earlier the best).
Your duties as an organiser
If your evaluation proposal is accepted, you will have to:
- Set up a web page/site exposing the data set and explaining the rules
to participate. This web page should follow
the template provided by the organisers.
- Answer to the participants and update the web page with these answers
- Evaluate the participant results in due time and providing them
- Providing the final results in due times as (1) a web page/site,
(2) a section in the OAEI summary paper.
- Both web pages/sites will be relocated within the OAEI web site
at the end of the campaign. They should be maintained until
then. Alternatively, it is possible to ask OAEI to maintain the data track
web page/site but this would be at the expense of responsiveness
and at the discretion of the web site maintainer.
What OAEI will do for you
In fact, not much. OAEI is rather targeted as tool developers and you
will be also working for these tool developers. However, OAEI will:
- Maintain this web site;
- Link your web page to its campaign web site;
- Advertise the campaigns (but you can do it as well);
- Provide advise and tools from former organisers;
- Organise the final meeting (usually collocated with some other event);
- Invite you to write and edit the final reports usually published on the web;
- Bug you in order to get everybody satisfied.
- Any track which has less than two participants independent from the organisers will be declared fruitless.
- It is not forbidden that a system participates in a track that its
promoters have organised. However, fairness would require that such
systems participate at least in two other tracks.
$Id: oaei-submitting.1.html,v 1.2 2010/04/15 14:53:28 euzenat Exp $