Offical ground truth files (aka QRELs) for TREC CAR 2017

Original assessment on V1.5 data release. Qrels are translated to paragraph ids and entity ids used in V2.0.

Manual QRELs (graded)
   manual asessment through NIST assessors
   collected on pool of participant-contributed runs
   positive grades: 3: must be mentioned; 2: should be mentioned; 1: can be mentioned
   negative grades: 0: roughly on topic; -1: non-relevant; -2: trash
   

Lenient QRELs (graded)
   based on the manual QRELs by counting "roughly on topic" as a positive grade. 
   (this is a super lenient assessmnet, anything below the bar really should not be retrieved)
   positive grades: 5: MUST; 4: SHOULD; 3: COULD; 2: TOPIC
   negative grades: 0: non-relevant; -2: trash

Automatic QRELs: See benchmarkY1-test release.

Evaluation scores are generated with trec_eval using this command:
  trec_eval -c -m all_trec $qrel $run
  Notice that `-c` ensures that omitted queries count against you. 
  Official metric(s) for TREC CAR are Rprec (R-Precision), map (mean-average precision), recip_rank (mean reciprocal rank)


This data set is part of the TREC CAR dataset version v2.0.
The included TREC CAR data sets by Laura Dietz, Ben Gamari available at trec-car.cs.unh.edu are provided under a <a rel=license href=http://creativecommons.org/licenses/by-sa/3.0/deed.en_US>Creative Commons Attribution-ShareAlike 3.0 Unported License</a>. The data is based on content extracted from www.Wikipedia.org that is licensed under the Creative Commons Attribution-ShareAlike 3.0 Unported License.

Direct questions to Laura Dietz: dietz@cs.unh.edu
