Query Rewriting & Expansion

Query rewriting refers to changing the formulation of the query in order to improve the effectiveness of the search ranking. PyTerrier supplies a number of query rewriting transformers designed to work with BatchRetrieve.

Firstly, we differentiate between two forms of query rewriting:

  • Q -> Q: this rewrites the query, for instance by adding/removing extra query terms. Examples might be a WordNet- or Word2Vec-based QE; The input dataframes contain only [“qid”, “docno”] columns. The output dataframes contain [“qid”, “query”, “query_0”] columns, where “query” contains the reformulated query, and “query_0” contains the previous formulation of the query.

  • R -> Q: these class of transformers rewrite a query by making use of an associated set of documents. This is typically exemplifed by pseudo-relevance feedback. Similarly the output dataframes contain [“qid”, “query”, “query_0”] columns.

The previous formulation of the query can be restored using pt.rewrite.reset(), discussed below.

SequentialDependence

This class implements Metzler and Croft’s sequential dependence model, designed to boost the scores of documents where the query terms occur in close proximity. Application of this transformer rewrites each input query such that:

  • pairs of adjacent query terms are added as #1 and #uw8 complex query terms, with a low weight.

  • the full query is added as #uw12 complex query term, with a low weight.

  • all terms are weighted by a proximity model, either Dirichlet LM or pBiL2.

For example, the query pyterrier IR platform would become pyterrier IR platform #1(pyterrier IR) #1(IR platform) #uw8(pyterrier IR) #uw8(IR platform) #uw12(pyterrier IR platform). NB: Acutally, we have simplified the rewritten query - in practice, we also (a) set the weight of the proximity terms to be low using a #combine() operator and (b) set a proximity term weighting model.

This transfomer is only compatible with BatchRetrieve, as Terrier supports the #1 and #uwN complex query terms operators. The Terrier index must have blocks (positional information) recorded in the index.

class pyterrier.rewrite.SequentialDependence(verbose=0, remove_stopwords=True, prox_model=None, **kwargs)[source]

Implements the sequential dependence model, which Terrier supports using its Indri/Galagoo compatible matchop query language. The rewritten query is derived using the Terrier class DependenceModelPreProcess.

This transformer changes the query. It must be followed by a Terrier Retrieve() transformer. The original query is saved in the “query_0” column, which can be restored using pt.rewrite.reset().

transform(topics_and_res)

Abstract method for all transformations. Typically takes as input a Pandas DataFrame, and also returns one.

Example:

sdm = pt.rewrite.SequentialDependence()
dph = pt.BatchRetrieve(index, wmodel="DPH")
pipeline = sdm >> dph
References:
  • A Markov Random Field Model for Term Dependencies. Donald Metzler and W. Bruce Croft. In Proceedings of SIGIR 2005.

  • Incorporating Term Dependency in the DFR Framework. Jie Peng, Craig Macdonald, Ben He, Vassilis Plachouras, Iadh Ounis. In Proceedings of SIGIR 2007. July 2007. Amsterdam, the Netherlands. 2007.

Bo1QueryExpansion

This class applies the Bo1 Divergence from Randomess query expansion model to rewrite the query based on the occurences of terms in the feedback documents provided for each query. In this way, it takes in a dataframe with columns [“qid”, “query”, “docno”, “score”, “rank”] and returns a dataframe with [“qid”, “query”].

class pyterrier.rewrite.Bo1QueryExpansion(*args, **kwargs)[source]

Applies the Bo1 query expansion model from the Divergence from Randomness Framework, as provided by Terrier. It must be followed by a Terrier Retrieve() transformer. The original query is saved in the “query_0” column, which can be restored using pt.rewrite.reset().

Instance Attributes:
  • fb_terms(int): number of feedback terms. Defaults to 10

  • fb_docs(int): number of feedback documents. Defaults to 3

Parameters:
  • index_like – the Terrier index to use.

  • fb_terms (int) – number of terms to add to the query. Terrier’s default setting is 10 expansion terms.

  • fb_docs (int) – number of feedback documents to consider. Terrier’s default setting is 3 feedback documents.

transform(topics_and_res)

Abstract method for all transformations. Typically takes as input a Pandas DataFrame, and also returns one.

Example:

bo1 = pt.rewrite.Bo1QueryExpansion(index)
dph = pt.BatchRetrieve(index, wmodel="DPH")
pipelineQE = dph >> bo1 >> dph

View the expansion terms:

pipelineDisplay = dph >> bo1
pipelineDisplay.search("chemical reactions")
# will return a dataframe with ['qid', 'query', 'query_0'] columns
# the reformulated query can be found in the 'query' column,
# while the original query is in the 'query_0' columns

Alternative Formulations

Note that it is also possible to configure BatchRetrieve to perform QE directly using controls, which will result in identical retrieval effectiveness:

pipelineQE = pt.BatchRetrieve(index, wmodel="DPH", controls={"qemodel" : "Bo1", "qe" : "on"})

However, using pt.rewrite.Bo1QueryExpansion is preferable as:

  • the semantics of retrieve >> rewrite >> retrieve are clearly visible.

  • the complex control configuration of Terrier need not be learned.

  • the rewritten query is visible outside, and not hidden inside Terrier.

References:
  • Amati, Giambattista (2003) Probability models for information retrieval based on divergence from randomness. PhD thesis, University of Glasgow.

KLQueryExpansion

Similar to Bo1, this class deploys a Divergence from Randomess query expansion model based on Kullback Leibler divergence.

class pyterrier.rewrite.KLQueryExpansion(*args, **kwargs)[source]

Applies the KL query expansion model from the Divergence from Randomness Framework, as provided by Terrier. This transformer must be followed by a Terrier Retrieve() transformer. The original query is saved in the “query_0” column, which can be restored using pt.rewrite.reset().

Instance Attributes:
  • fb_terms(int): number of feedback terms. Defaults to 10

  • fb_docs(int): number of feedback documents. Defaults to 3

Parameters:
  • index_like – the Terrier index to use

  • fb_terms (int) – number of terms to add to the query. Terrier’s default setting is 10 expansion terms.

  • fb_docs (int) – number of feedback documents to consider. Terrier’s default setting is 3 feedback documents.

transform(topics_and_res)

Abstract method for all transformations. Typically takes as input a Pandas DataFrame, and also returns one.

References:
  • Amati, Giambattista (2003) Probability models for information retrieval based on divergence from randomness. PhD thesis, University of Glasgow.

RM3

class pyterrier.rewrite.RM3(*args, fb_terms=10, fb_docs=3, fb_lambda=0.6, **kwargs)[source]

Performs query expansion using RM3 relevance models. RM3 relies on an external Terrier plugin, terrier-prf. You should start PyTerrier with pt.init(boot_packages=[“com.github.terrierteam:terrier-prf:-SNAPSHOT”]).

This transformer must be followed by a Terrier Retrieve() transformer. The original query is saved in the “query_0” column, which can be restored using pt.rewrite.reset().

Instance Attributes:
  • fb_terms(int): number of feedback terms. Defaults to 10

  • fb_docs(int): number of feedback documents. Defaults to 3

  • fb_lambda(float): lambda in RM3, i.e. importance of relevance model viz feedback model. Defaults to 0.6.

Example:

bm25 = pt.BatchRetrieve(index, wmodel="BM25")
rm3_pipe = bm25 >> pt.rewrite.RM3(index) >> bm25
pt.Experiment([bm25, rm3_pipe],
            dataset.get_topics(),
            dataset.get_qrels(),
            ["map"]
            )
Parameters:
  • index_like – the Terrier index to use

  • fb_terms (int) – number of terms to add to the query. Terrier’s default setting is 10 expansion terms.

  • fb_docs (int) – number of feedback documents to consider. Terrier’s default setting is 3 feedback documents.

  • fb_lambda (float) – lambda in RM3, i.e. importance of relevance model viz feedback model. Defaults to 0.6.

transform(queries_and_docs)[source]

Abstract method for all transformations. Typically takes as input a Pandas DataFrame, and also returns one.

References:
  • Nasreen Abdul-Jaleel, James Allan, W Bruce Croft, Fernando Diaz, Leah Larkey, Xiaoyan Li, Mark D Smucker, and Courtney Wade. UMass at TREC 2004: Novelty and HARD. In Proceedings of TREC 2004.

AxiomaticQE

class pyterrier.rewrite.AxiomaticQE(*args, fb_terms=10, fb_docs=3, **kwargs)[source]

Performs query expansion using axiomatic query expansion. This class relies on an external Terrier plugin, terrier-prf. You should start PyTerrier with pt.init(boot_packages=[“com.github.terrierteam:terrier-prf:-SNAPSHOT”]).

This transformer must be followed by a Terrier Retrieve() transformer. The original query is saved in the “query_0” column, which can be restored using pt.rewrite.reset().

Instance Attributes:
  • fb_terms(int): number of feedback terms. Defaults to 10

  • fb_docs(int): number of feedback documents. Defaults to 3

Parameters:
  • index_like – the Terrier index to use

  • fb_terms (int) – number of terms to add to the query. Terrier’s default setting is 10 expansion terms.

  • fb_docs (int) – number of feedback documents to consider. Terrier’s default setting is 3 feedback documents.

transform(queries_and_docs)[source]

Abstract method for all transformations. Typically takes as input a Pandas DataFrame, and also returns one.

References:
  • Hui Fang, Chang Zhai.: Semantic term matching in axiomatic approaches to information retrieval. In: Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 115–122. SIGIR 2006. ACM, New York (2006).

  • Peilin Yang and Jimmy Lin, Reproducing and Generalizing Semantic Term Matching in Axiomatic Information Retrieval. In Proceedings of ECIR 2019.

Combining Query Formulations

pyterrier.rewrite.linear(weightCurrent, weightPrevious, format='terrierql', **kwargs)[source]

Applied to make a linear combination of the current and previous query formulation. The implementation is tied to the underlying query language used by the retrieval/re-ranker transformers. Two of Terrier’s query language formats are supported by the format kwarg, namely “terrierql” and “matchoptql”. Their exact respective formats are detailed in the Terrier documentation.

Return type:

Transformer

Parameters:
  • weightCurrent (float) – weight to apply to the current query formulation.

  • weightPrevious (float) – weight to apply to the previous query formulation.

  • format (str) – which query language to use to rewrite the queries, one of “terrierql” or “matchopql”.

Example:

pipeTQL = pt.apply.query(lambda row: "az") >> pt.rewrite.linear(0.75, 0.25, format="terrierql")
pipeMQL = pt.apply.query(lambda row: "az") >> pt.rewrite.linear(0.75, 0.25, format="matchopql")
pipeT.search("a")
pipeM.search("a")

Example outputs of pipeTQL and pipeMQL corresponding to the query “a” above:

  • Terrier QL output: “(az)^0.750000 (a)^0.250000”

  • MatchOp QL output: “#combine:0:0.750000:1:0.250000(#combine(az) #combine(a))”

Resetting the Query Formulation

The application of any query rewriting operation, including the apply transformer, pt.apply.query(), will return a dataframe that includes the input formulation of the query in the query_0 column, and the new reformulation in the query column. The previous query reformulation can be obtained by inclusion of a reset transformer in the pipeline.

pyterrier.rewrite.reset()[source]

Undoes a previous query rewriting operation. This results in the query formulation stored in the “query_0” attribute being moved to the “query” attribute, and, if present, the “query_1” being moved to “query_0” and so on. This transformation is useful if you have rewritten the query for the purposes of one retrieval stage, but wish a subquent transformer to be applies on the original formulation.

Internally, this function applies pt.model.pop_queries().

Example:

firststage = pt.rewrite.SDM() >> pt.BatchRetrieve(index, wmodel="DPH")
secondstage = pyterrier_bert.cedr.CEDRPipeline()
fullranker = firststage >> pt.rewrite.reset() >> secondstage
Return type:

Transformer

Tokenising the Query

Sometimes your query can include symbols that aren’t compatible with how your retriever parses the query. In this case, a custom tokeniser can be applied as part of the retrieval pipeline.

pyterrier.rewrite.tokenise(tokeniser='english', matchop=False)[source]

Applies tokenisation to the query. By default, queries obtained from pt.get_dataset().get_topics() are normally tokenised.

Return type:

Transformer

Parameters:
  • tokeniser (Union[str,TerrierTokeniser,FunctionType]) – Defines what tokeniser should be used - either a Java tokeniser name in Terrier, a TerrierTokeniser instance, or a function that takes a str as input and returns a list of str.

  • matchop (bool) – Whether query terms should be wrapped in matchops, to ensure they can be parsed by a Terrier BatchRetrieve transformer.

Example - use default tokeniser:

pipe = pt.rewrite.tokenise() >> pt.BatchRetrieve()
pipe.search("Question with 'capitals' and other stuff?")

Example - roll your own tokeniser:

poortokenisation = pt.rewrite.tokenise(lambda query: query.split(" ")) >> pt.BatchRetrieve()

Example - for non-English languages, tokenise on standard UTF non-alphanumeric characters:

utftokenised = pt.rewrite.tokenise(pt.TerrierTokeniser.utf)) >> pt.BatchRetrieve()
utftokenised = pt.rewrite.tokenise("utf")) >> pt.BatchRetrieve()

Example - tokenising queries using a HuggingFace tokenizer

# this assumes the index was created in a pretokenised manner
br = pt.BatchRetrieve(indexref)
tok = AutoTokenizer.from_pretrained("bert-base-uncased")
query_toks = pt.rewrite.tokenise(tok.tokenize, matchop=True)
retr_pipe = query_toks >> br

Stashing the Documents

Sometimes you want to apply a query rewriting function as a re-ranker, but your rewriting function uses a different document ranking. In this case, you can use pt.rewrite.stash_results() to stash the retrieved documents for each query, so they can be recovered and re-ranked later using your rewritten query formulation.

pyterrier.rewrite.stash_results(clear=True)[source]

Stashes (saves) the current retrieved documents for each query into the column “stashed_results_0”. This means that they can be restored later by using pt.rewrite.reset_results(). thereby converting a retrieved documents dataframe into one of queries.

Args: clear(bool): whether to drop the document and retrieved document related columns. Defaults to True.

Return type:

Transformer

pyterrier.rewrite.reset_results()[source]

Applies a transformer that undoes a pt.rewrite.stash_results() transformer, thereby restoring the ranked documents.

Return type:

Transformer

Example: Query Expansion as a re-ranker

Some papers advocate for the use of query expansion (PRF) as a re-ranker. This can be attained in PyTerrier through use of stash_results() and reset_results():

# index: the corpus you are ranking

dph = pt.BatchRetrieve(index)
Pipe = dph
    >> pt.rewrite.stash_results(clear=False)
    >> pt.rewrite.RM3(index)
    >> pt.rewrite.reset_results()
    >> dph

Summary of dataframe types:

output of

dataframe contents

actual columns

dph

R

qid, query, docno, score

stash_results

R + “stashed_results_0”

qid, query, docno, score, stashed_results_0

RM3

Q + “stashed_results_0”

qid, query, query_0, stashed_results_0

reset_results

R

qid, query, docno, score, query_0

dph

R

qid, query, docno, score, query_0

Indeed, as we need RM3 to have the initial ranking of documents as input, we use clear=False as the kwarg to stash_results().

Example: Collection Enrichment as a re-ranker:

# index: the corpus you are ranking
# wiki_index: index of Wikipedia, used for enrichment

dph = pt.BatchRetrieve(index)
Pipe = dph
    >> pt.rewrite.stash_results()
    >> pt.BatchRetrieve(wiki_index)
    >> pt.rewrite.RM3(wiki_index)
    >> pt.rewrite.reset_results()
    >> dph

In general, collection enrichment describes conducting a PRF query expansion process on an external corpus (often Wikipedia), before applying the reformulated query to the main corpus. Collection enrichment can be used for improving a first pass retrieval (pt.BatchRetrieve(wiki_index) >> pt.rewrite.RM3(wiki_index) >> pt.BatchRetrieve(main_index)). Instead, the particular example shown above applies collection enrichment as a re-ranker.

Summary of dataframe types:

output of

dataframe contents

actual columns

dph

R

qid, query, docno, score

stash_results

Q + “stashed_results_0”

qid, query, saved_docs_0

BatchRetrieve

R + “stashed_results_0”

qid, query, docno, score, stashed_results_0

RM3

Q + “stashed_results_0”

qid, query, query_0, stashed_results_0

reset_results

R

qid, query, docno, score, query_0

dph

R

qid, query, docno, score, query_0

In this example, we have a BatchRetrieve instance executed on the wiki_index before RM3, so we clear the document ranking columns when using stash_results().