Terrier Indexing¶
PyTerrier has a number of useful classes for creating Terrier indices, which can be used for retrieval, query expansion, etc. There are four indexer classes:
You can create an index from TREC-formatted files, from a TREC test collection, using TRECCollectionIndexer.
For indexing TXT, PDF, Microsoft Word files, etc files you can use FilesIndexer.
For indexing Pandas Dataframe you can use DFIndexer.
For any abitrary iterable dictionaries, you can use IterDictIndexer.
There are also different types of indexing supported in Terrier that are exposed in PyTerrier.
We explain both the indexing types and the indexer classes below, with examples. Further worked examples of indexing are provided in the example indexing notebook.
Index Types¶
- class pyterrier.index.IndexingType(value)[source]¶
This enum is used to determine the type of index built by Terrier. The default is CLASSIC.
- CLASSIC = 1¶
A classical indexing regime, which also creates a direct index structure, useful for query expansion
- SINGLEPASS = 2¶
A single-pass indexing regime, which builds an inverted index directly. No direct index structure is created. Typically is faster than classical indexing.
- MEMORY = 3¶
An in-memory index. No direct index is created.
Indexer Classes¶
TRECCollectionIndexer¶
- class pyterrier.TRECCollectionIndexer(index_path, blocks=False, overwrite=False, type=IndexingType.CLASSIC, collection='trec', verbose=False, meta={'docno': 20}, meta_reverse=['docno'], meta_tags={})[source]¶
Use this Indexer if you wish to index a TREC formatted collection
Init method
- Parameters
index_path (str) – Directory to store index. Ignored for IndexingType.MEMORY.
blocks (bool) – Create indexer with blocks if true, else without blocks. Default is False.
overwrite (bool) – If index already present at index_path, True would overwrite it, False throws an Exception. Default is False.
type (IndexingType) – the specific indexing procedure to use. Default is IndexingType.CLASSIC.
collection (Class name, or Class instance, or one of "trec", "trecweb", "warc") –
meta (Dict[str,int]) – What metadata for each document to record in the index, and what length to reserve. Defaults to {“docno” : 20}.
meta_reverse (List[str]) – What metadata shoudl we be able to resolve back to a docid. Defaults to [“docno”].
meta_tags (Dict[str,str]) – For collections formed using tagged data (e.g. HTML), which tags correspond to which metadata. This is useful for recording the text of documents for use in neural rankers - see Working with Document Texts.
Example indexing the TREC WT2G corpus:
import pyterrier as pt
pt.init()
# list of filenames to index
files = pt.io.find_files("/path/to/WT2G/wt2g-corpus/")
# build the index
indexer = pt.TRECCollectionIndexer("./wt2g_index", verbose=True, blocks=False)
indexref = indexer.index(files)
# load the index, print the statistics
index = pt.IndexFactory.of(indexref)
print(index.getCollectionStatistics().toString())
FilesIndexer¶
- class pyterrier.FilesIndexer(index_path, *args, meta={'docno': 20, 'filename': 512}, meta_reverse=['docno'], meta_tags={}, **kwargs)[source]¶
Use this Indexer if you wish to index a pdf, docx, txt etc files
- Parameters
index_path (str) – Directory to store index. Ignored for IndexingType.MEMORY.
blocks (bool) – Create indexer with blocks if true, else without blocks. Default is False.
type (IndexingType) – the specific indexing procedure to use. Default is IndexingType.CLASSIC.
meta (Dict[str,int]) – What metadata for each document to record in the index, and what length to reserve. Defaults to {“docno” : 20, “filename” : 512}.
meta_reverse (List[str]) – What metadata shoudl we be able to resolve back to a docid. Defaults to [“docno”],
meta_tags (Dict[str,str]) – For collections formed using tagged data (e.g. HTML), which tags correspond to which metadata. Defaults to empty. This is useful for recording the text of documents for use in neural rankers - see Working with Document Texts.
Init method
- Parameters
index_path (str) – Directory to store index. Ignored for IndexingType.MEMORY.
blocks (bool) – Create indexer with blocks if true, else without blocks. Default is False.
overwrite (bool) – If index already present at index_path, True would overwrite it, False throws an Exception. Default is False.
verbose (bool) – Provide progess bars if possible. Default is False.
type (IndexingType) – the specific indexing procedure to use. Default is IndexingType.CLASSIC.
DFIndexer¶
- class pyterrier.DFIndexer(index_path, *args, blocks=False, overwrite=False, verbose=False, meta_reverse=['docno'], type=IndexingType.CLASSIC, **kwargs)[source]¶
Use this Indexer if you wish to index a pandas.Dataframe
Init method
- Parameters
index_path (str) – Directory to store index. Ignored for IndexingType.MEMORY.
blocks (bool) – Create indexer with blocks if true, else without blocks. Default is False.
overwrite (bool) – If index already present at index_path, True would overwrite it, False throws an Exception. Default is False.
verbose (bool) – Provide progess bars if possible. Default is False.
type (IndexingType) – the specific indexing procedure to use. Default is IndexingType.CLASSIC.
- index(text, *args, **kwargs)[source]¶
Index the specified
- Parameters
text (pd.Series) – A pandas.Series(a column) where each row is the body of text for each document
*args – Either a pandas.Dataframe or pandas.Series. If a Dataframe: All columns(including text) will be passed as metadata If a Series: The Series name will be the name of the metadata field and the body will be the metadata content
**kwargs – Either a list, a tuple or a pandas.Series The name of the keyword argument will be the name of the metadata field and the keyword argument contents will be the metadata content
Example indexing a dataframe:
# define an example dataframe of documents
import pandas as pd
df = pd.DataFrame({
'docno':
['1', '2', '3'],
'url':
['url1', 'url2', 'url3'],
'text':
['He ran out of money, so he had to stop playing',
'The waves were crashing on the shore; it was a',
'The body may perhaps compensates for the loss']
})
# index the text, record the docnos as metadata
pd_indexer = pt.DFIndexer("./pd_index")
indexref = pd_indexer.index(df["text"], df["docno"])
IterDictIndexer¶
- class pyterrier.IterDictIndexer(index_path, *args, meta={'docno': 20}, meta_reverse=['docno'], threads=1, **kwargs)¶
Use this Indexer if you wish to index an iter of dicts (possibly with multiple fields). This version is optimized by using multiple threads and POSIX fifos to tranfer data, which ends up being much faster.
- Parameters
index_path (str) – Directory to store index. Ignored for IndexingType.MEMORY.
meta (Dict[str,int]) – What metadata for each document to record in the index, and what length to reserve. Defaults to {“docno” : 20}.
meta_reverse (List[str]) – What metadata shoudl we be able to resolve back to a docid. Defaults to [“docno”],
- index(it, fields=('text',), meta=None, meta_lengths=None)¶
Index the specified iter of dicts with the (optional) specified fields
- Parameters
it (iter[dict]) – an iter of document dict to be indexed
fields (list[str]) – keys to be indexed as fields
meta (list[str]) – keys to be considered as metdata
meta_lengths (list[int]) – length of metadata, defaults to 512 characters
Examples using IterDictIndexer
An iterdict can just be a list of dictionaries:
docs = [ { 'docno' : 'doc1', 'text' : 'a b c' } ]
iter_indexer = pt.IterDictIndexer("./index", meta=['docno', 'text'], meta_lengths=[20, 4096])
indexref1 = iter_indexer.index(docs)
A dataframe can also be used, virtue of its .to_dict()
method:
df = pd.DataFrame([['doc1', 'a b c']], columns=['docno', 'text'])
iter_indexer = pt.IterDictIndexer("./index")
indexref2 = indexer.index(df.to_dict(orient="records"))
However, the main power of using IterDictIndexer is for processing indefinite iterables, such as those returned by generator functions. For example, the tsv file of the MSMARCO Passage Ranking corpus can be indexed as follows:
dataset = pt.get_dataset("trec-deep-learning-passages")
def msmarco_generate():
with pt.io.autoopen(dataset.get_corpus()[0], 'rt') as corpusfile:
for l in corpusfile:
docno, passage = l.split("\t")
yield {'docno' : docno, 'text' : passage}
iter_indexer = pt.IterDictIndexer("./passage_index", meta=['docno', 'text'], meta_lengths=[20, 4096])
indexref3 = iter_indexer.index(msmarco_generate())
IterDictIndexer can be used in connection with Indexing Pipelines.
Similarly, indexing of JSONL files is similarly a few lines of Python:
def iter_file(filename):
import json
with open(filename, 'rt') as file:
for l in file:
# assumes that each line contains 'docno', 'text' attributes
# yields a dictionary for each json line
yield json.loads(l)
indexref4 = pt.IterDictIndexer("./index", meta=['docno', 'text'], meta_lengths=[20, 4096]).index(iter_file("/path/to/file.jsonl"))
NB: Use pt.io.autoopen()
as a drop-in replacement for open()
that supports files compressed by gzip etc.
Indexing TREC-formatted files using IterDictIndexer
If you have TREC-formatted files that you wish to use with an IterDictIndexer-like indexer, pt.index.treccollection2textgen()
can be used
as a helper function to aid in parsing such files.
- pyterrier.index.treccollection2textgen(files, meta=['docno'], meta_tags={'text': 'ELSE'}, verbose=False, num_docs=None, tag_text_length=4096)[source]¶
Creates a generator of dictionaries on parsing TREC formatted files. This is useful for parsing TREC-formatted corpora in indexers like IterDictIndexer, or similar indexers in other plugins (e.g. ColBERTIndexer).
- Parameters
files (-) – list of files to parse in TREC format.
meta (-) – list of attributes to expose in the dictionaries as metadata.
meta_tags (-) – mapping of TREC tags as metadata.
tag_text_length (-) – maximium length of metadata. Defaults to 4096.
verbose (-) – set to true to show a TQDM progress bar. Defaults to True.
num_docs (-) – a hint for TQDM to size the progress bar based on document counts rather than file count.
Example:
files = pt.io.find_files("/path/to/Disk45") gen = pt.index.treccollection2textgen(files) index = pt.IterDictIndexer("./index45").index(gen)
Example using Indexing Pipelines:
files = pt.io.find_files("/path/to/Disk45")
gen = pt.index.treccollection2textgen(files)
indexer = pt.text.sliding() >> pt.IterDictIndexer("./index45")
index = indexer.index(gen)
Threading
On UNIX-based systems, IterDictIndexer can also perform multi-threaded indexing:
iter_indexer = pt.IterDictIndexer("./passage_index_8", meta=['docno', 'text'], meta_lengths=[20, 4096], threads=8)
indexref6 = iter_indexer.index(msmarco_generate())
Note that the resulting index ordering with multiple threads is non-deterministic; if you need
deterministic behavior you must index in single-threaded mode. Furthermore, indexing can only go
as quickly as the document iterator, so to take full advantage of multi-threaded indexing, you
will need to keep the iterator function light-weight. Many datasets provide a fast corpus iteration
function (get_corpus_iter()
), see more information in the Importing Datasets.
Indexing Configuration¶
Our aim is to expose all conventional Terrier indexing configuration through PyTerrier, for instance as constructor arguments to the Indexer classes. However, as Terrier is a legacy platform, some changes will take time to integrate into Terrier. Moreover, the manner of the configuration needed varies a little between the Indexer classes. In the following, we list common indexing configurations, and how to apply them when indexing using PyTerrier, noting any differences betweeen the Indexer classes.
stemming configuation or stopwords: the default Terrier indexing configuration is to use a term pipeline of Stopwords,PorterStemmer. You can change the term pipeline configuration using the “termpipeline” property:
indexer.setProperty("termpipelines", "")
Note that any subsequent indexing or retrieval operation would also require the “termpipeline” property to be suitably updated. See the org.terrier.terms package for a list of the available term pipeline objects provided by Terrier.
languages and tokenisation: Similarly, the choice of tokeniser is controlled by the “tokeniser” property. EnglishTokeniser is the default tokeniser. Other tokenisers are listed in org.terrier.indexing.tokenisation package. For instance, use indexer.setProperty(“tokeniser”, “UTFTokeniser”) when indexing non-English text.
positions (aka blocks) - all indexers expose a blocks boolean constructor argument to allow position information to be recoreded in the index. Defaults to False, i.e. positions are not recorded.
fields - fields refers to storing the frequency of a terms occurrence in different parts of a document, e.g. title vs body vs anchor text. In the IterDictIndexer, fields are set in the index() method; otherwise the “FieldTags.process” property must be set. See the Terrier indexing documentation on fields for more information.
changing the tags parsed by TREC Collection - use the relevant properties listed in the Terrier indexing documentation.
metaindex configuration: metadata refers to the arbitrary strings associated to each document recorded in a Terrier index. These can range from the “docno” attribute of each document, as used to support experimentation, to other attributes such as the URL of the documents, or even the raw text of the document. Indeed, storing the raw text of each document is a trick often used when applying additional re-rankers such as BERT (see pyterrier_bert for more information on integrating PyTerrier with BERT-based re-rankers). Indexers now expose meta and meta_tags constructor kwarg to make this easier.
reverse metaindex configuration: on occasion, there is a need to lookup up documents in a Terrier index based on their metadata, e.g. “docno”. The meta_reverse constructor kwarg allows meta keys that support reverse lookup to be specified.