Trec

Qrels

Functions and classes for dealing with English qrel files. The specification can be viewed at: http://trec.nist.gov/data/qrels_eng/

Usage:

>>> sample_qrels = '1 0 AP880212-0161 0\n1 0 AP880216-0139 1\n1 0 AP880216-0169 0'
>>> qrels = loads(sample_qrels)
>>> qrels.topic
['1', '1', '1']
>>> [x.topic for x in qrels['1']]
['1', '1', '1']

Harry Scells Mar 2017

class irkit.trec.qrels.Qrel(topic: str, iteration: int, document_num: str, relevancy: int)

A line in a qrels file conforming to the specification at: http://trec.nist.gov/data/qrels_eng/

class irkit.trec.qrels.Qrels(qrels: typing.List[irkit.trec.qrels.Qrel])

A python representation of a qrels file. It is a list of Qrel objects.

dump(fp: _io.TextIOWrapper) → None

Dump the qrels to a file

Parameters:fp – A File pointer
dumps() → str

Dump the qrels to a string.

Returns:Formatted qrels
irkit.trec.qrels.load(qrels: _io.TextIOWrapper) → irkit.trec.qrels.Qrels

Load qrels from a file.

Parameters:qrels – File pointer
Returns:Qrels object
irkit.trec.qrels.loads(qrels: str) → irkit.trec.qrels.Qrels

Load qrels from a string.

Parameters:qrels – Some string representation of qrels
Returns:Qrels object

Runs

Functions and classes for dealing with trec_eval run files. For a description on how run files look, see: http://faculty.washington.edu/levow/courses/ling573_SPR2011/hw/trec_eval_desc.htm

Usage:

>>> sample_run = '''351   0  DOC1  1   100   run-name\n351   0  DOC2  2   50   run-name'''
>>> runs = loads(sample_run)
>>> runs.dumps()
'351    Q0      DOC1    1       100     run-name
351     Q0      DOC2    2       50      run-name'
>>> [str(run) for run in runs.runs]
['351   Q0      DOC1    1       100     run-name', '351 Q0      DOC2    2       50      run-name']
>>> [str(run) for run in runs['351']]
['351   Q0      DOC1    1       100     run-name', '351 Q0      DOC2    2       50      run-name']
>>> runs.rank
['1', '2']
>>> TrecEvalRuns(runs['351']).rank
['1', '2']

Harry Scells May 2017

class irkit.trec.run.TrecEvalRun(topic: str, q: int, doc_id: str, rank: int, score: float, run_id: str)

TrecEvalRun is a container class for a line in a trec_eval run file.

class irkit.trec.run.TrecEvalRuns(runs: typing.List[irkit.trec.run.TrecEvalRun])

TrecEvalRuns is a wrapper around a TrecEvalRun which is just a container class for a line in a trec_eval run file. This class contains some convenience functions for dealing with runs, such as getting a list of the runs but topic id or slicing by column.

dump(fp: _io.TextIOWrapper) → None

Dump the qrels to a file

Parameters:fp – A File pointer
dumps() → str

Dump the qrels to a string.

Returns:Formatted qrels
irkit.trec.run.load(runs: _io.TextIOWrapper) → irkit.trec.run.TrecEvalRuns

Load a trec_eval run file.

Parameters:runs – A file pointer containing runs.
Returns:TrecEvalRuns
irkit.trec.run.loads(runs: str) → irkit.trec.run.TrecEvalRuns

Load a trec_eval run file from a string.

Parameters:runs – A string containing runs.
Returns:TrecEvalRuns

Results

Functions and classes for dealing with trec_eval results files.

Harry Scells Mar 2017

class irkit.trec.results.TrecEvalResults(run_id: str, results: typing.Dict, queries: typing.Dict)

An object that stores the results output by trec_eval.

irkit.trec.results.load(trec_result_file: _io.TextIOWrapper) → irkit.trec.results.TrecEvalResults

Load trec_eval results from a file.

Parameters:trec_result_file – File pointer
Returns:Qrels object
irkit.trec.results.loads(trec_results: str) → irkit.trec.results.TrecEvalResults

Load trec_eval results from a string.

Parameters:trec_results – Some string representation of trec results
Returns:TrecEvalResults object