SuitEval API Refernece

suiteeval.suite

Abstract suite interface.

class suiteeval.suite.base.Suite(*args, **kwargs)

Bases: ABC

Abstract base class for a set of related evaluations across one or more datasets.

Subclasses (or classes created via SuiteMeta.register()) must populate:

_datasets

Either a dict[str, str] mapping display name → IRDS dataset ID, or a list[str] of IRDS dataset IDs.

_dataset_ids

Normalized mapping of display name → IRDS dataset ID (filled in by registration helpers).

_metadata

Optional per-dataset or global metadata.

_measures

A list of ir_measures.Measure or a mapping from dataset name to such a list. When not provided, defaults are derived from metadata or IRDS documentation; ultimately falling back to _default_measures.

_default_measures

Fallback measures when nothing else is configured.

_query_field

Optional topic field name to use when fetching topics.

Evaluation runs through a chain of small, overridable steps. In call order:

Hook	Responsibility
resolve_config()	Split kwargs into a RunConfig
iter_corpus_groups()	Group datasets by shared corpus
select_members()	Pick the datasets to evaluate in a group
build_context()	Build the shared per-corpus context
iter_pipeline_batches()	Decide how pipelines are batched
wrap_pipeline()	Decorate each pipeline (filters, etc.)
prepare_topics_qrels()	Fetch and normalise topics/qrels
measures_for()	Choose metrics for a dataset
has_cached_run() /
load_cached_run() /
run_file_path()	Reuse run files written by a previous call
run_experiment()	The pyterrier.Experiment() call itself
annotate_results()	Tag result rows with their dataset
release_pipelines()	Free memory between batches
postprocess_results()	Aggregate the concatenated results

Notes

Instances are singletons per subclass (enforced by SuiteMeta).

iter_corpus_groups()

Group the suite’s datasets by the corpus they share.

Membership is decided by ir_datasets.docs_parent_id(), so datasets built on the same document collection are indexed once. Override to impose a different grouping.

Return type:

Iterator[Tuple[str, Dataset, list[Tuple[Any, Any]]]]

Yields:

tuple[str, pyterrier.datasets.Dataset, list[tuple[Any, Any]]] – (corpus_id, corpus_dataset, [(display_key, dataset_ref), ...]).

select_members(members, subset)

Choose which members of a corpus group to evaluate.

Return type:

list[Tuple[Any, Any]]

Parameters:

• members (Sequence[Tuple[Any, Any]]) – (display_key, dataset_ref) pairs sharing one corpus.

• subset (str | None) – Optional display name to restrict evaluation to.

Returns:

The members to evaluate; empty skips the group.

Return type:

list[tuple[Any, Any]]

property datasets: Generator[Tuple[str, Dataset], None, None]

Iterate over declared datasets yielding display name and PyTerrier dataset.

Yields:

tuple[str, pyterrier.datasets.Dataset] – Pairs of (name, dataset object).

Raises:

ValueError – If _datasets has an invalid type.

static parse_measures(measures)

Convert a list of measure strings or ir_measures.Measure objects into a flat list[Measure].

Return type:

list[Measure]

Parameters:

measures (Sequence[str | Measure]) – A sequence containing measure strings (e.g., "nDCG@10") and/or ir_measures.Measure instances.

Returns:

Parsed measure objects.

Return type:

list[Measure]

Raises:

ValueError – If a string entry cannot be parsed by either ir_measures.parse_measure() or ir_measures.parse_trec_measure(), or if an entry has an invalid type.

coerce_measures(metadata)

Populate self._measures by aggregating available sources in priority order:

1. Global metadata['official_measures'] if present.

2. Per-dataset metadata[name]['official_measures'] if present.

3. IRDS documentation official_measures for each dataset (when available).

If no measures are discovered, falls back to _default_measures.

Parameters:

metadata (dict[str, Any]) – The suite metadata dictionary as configured at construction time.

Returns:

None

Return type:

None

get_measures(dataset)

Resolve the measures configured for a given dataset name.

Return type:

list[Measure]

Parameters:

dataset (str) – Dataset display name as used in this suite.

Returns:

The list configured for this dataset (or the suite-wide

list if a single list is maintained). Falls back to _default_measures when the dataset is unknown.

Return type:

list[Measure]

measures_for(dataset_name, eval_metrics=None)

Decide which metrics to evaluate for one dataset.

Return type:

Sequence[Any]

Parameters:

• dataset_name (str) – Dataset display name.

• eval_metrics (Sequence[Any] | None) – Explicit metrics supplied by the caller, if any.

Returns:

eval_metrics when given, else the suite’s configuration.

Return type:

Sequence[Any]

wrap_pipeline(pipeline, context)

Decorate a single pipeline before it is evaluated.

This is the single seam for suite-specific pipeline modifications such as appending a result filter. It is applied by both sequential and grouped coercion, so overriding it alone covers every execution mode.

Return type:

Transformer

Parameters:

• pipeline (Transformer) – The pipeline produced by a generator.

• context (DatasetContext) – The shared context for the corpus being evaluated.

Returns:

The pipeline to evaluate. The default returns it unchanged.

Return type:

Transformer

coerce_pipelines_sequential(context, pipeline_generators)

Yield pipelines lazily, one at a time, without materializing the full set.

Use this when you want to minimize memory/VRAM footprint and you do not require joint analysis across all systems at once (e.g., significance testing).

Return type:

Iterator[Tuple[Transformer, str | None]]

Parameters:

• context (DatasetContext) – The shared DatasetContext for the current corpus group.

• pipeline_generators (Callable[[DatasetContext], Any] | Sequence[Callable[[DatasetContext], Any]]) – Callable or sequence of callables that produce either: * a single pyterrier.Transformer, * a sequence of transformers, * a tuple (pipelines, name_or_names) where names may be a single label applied to all pipelines or a sequence aligned with pipelines.

Yields:

tuple[Transformer, Optional[str]] – The pipeline and an optional display name.

Raises:

ValueError – If a generator yields an invalid structure.

coerce_pipelines_grouped(context, pipeline_generators)

Materialize all pipelines (and optional names) into lists.

Use this when downstream evaluation requires access to the full set of systems simultaneously (e.g., significance tests).

Return type:

Tuple[list[Transformer], list[str] | None]

Parameters:

• context (DatasetContext) – The shared DatasetContext for the current corpus group.

• pipeline_generators (Callable[[DatasetContext], Any] | Sequence[Callable[[DatasetContext], Any]]) – Callable or sequence of callables following the same conventions as in coerce_pipelines_sequential().

Returns:

A list of pipelines and, if provided, a list of corresponding names. If no names were supplied, returns None for the second element.

Return type:

tuple[list[Transformer], Optional[list[str]]]

Raises:

ValueError – If the generators produce no pipelines or an invalid structure.

iter_pipeline_batches(context, pipeline_generators, grouped)

Yield the units of work that are evaluated together.

Grouped mode yields a single batch holding every pipeline, so that pyterrier.Experiment() can run significance tests across systems. Sequential mode yields one single-pipeline batch at a time, so that each pipeline can be released before the next is built.

Return type:

Iterator[list[Tuple[Transformer, str | None]]]

Parameters:

• context (DatasetContext) – The shared DatasetContext for the current corpus group.

• pipeline_generators (Callable[[DatasetContext], Any] | Sequence[Callable[[DatasetContext], Any]]) – Callable or sequence of callables producing pipelines.

• grouped (bool) – Whether all pipelines must be materialized together.

Yields:

list[tuple[Transformer, Optional[str]]] – One batch of named pipelines.

index_dir_for(index_dir, corpus_id)

Directory holding the shared index for one corpus.

Return type:

str

Parameters:

• index_dir (str)

• corpus_id (str)

save_dir_for(save_dir, dataset_name)

Directory holding the run files for one dataset.

Return type:

str

Parameters:

• save_dir (str)

• dataset_name (str)

run_file_path(save_dir, dataset_name, pipeline_name)

Path of the run file for one pipeline on one dataset.

Return type:

str

Parameters:

• save_dir (str)

• dataset_name (str)

• pipeline_name (str)

has_cached_run(save_dir, dataset_name, pipeline_name, save_mode)

Whether a previously written run can be replayed instead of re-run.

Return type:

bool

Parameters:

• save_dir (str) – Root run-file directory.

• dataset_name (str) – Dataset display name.

• pipeline_name (str) – Pipeline display name.

• save_mode (str) – PyTerrier save mode; "overwrite" always re-runs.

Returns:

True when a reusable run file exists.

Return type:

bool

load_cached_run(filepath)

Load a gzipped TREC run file into a transformer that replays it.

Return type:

Transformer

Parameters:

filepath (str) – Path returned by run_file_path().

Returns:

A transformer yielding the stored ranking.

Return type:

Transformer

prepare_save_dir(save_dir, dataset_name)

Create and return the run-file directory for one dataset.

Return type:

str

Parameters:

• save_dir (str)

• dataset_name (str)

build_context(corpus_id, corpus_ds, config)

Build the context shared by every dataset in a corpus group.

Indexing happens once per context, so overriding this is the way to inject a custom context type or a bespoke index location.

Return type:

DatasetContext

Parameters:

• corpus_id (str) – The shared corpus identifier.

• corpus_ds (Dataset) – The PyTerrier dataset for that corpus.

• config (RunConfig) – The resolved run configuration.

Returns:

The context handed to pipeline generators.

Return type:

DatasetContext

prepare_topics_qrels(dataset, dataset_name)

Fetch topics and qrels for one dataset, with identifier columns as strings.

Return type:

Tuple[DataFrame, DataFrame]

Parameters:

• dataset (Dataset) – A pyterrier.datasets.Dataset instance.

• dataset_name (str) – Dataset display name.

Returns:

(topics, qrels).

Return type:

tuple[pandas.DataFrame, pandas.DataFrame]

run_experiment(pipelines, names, topics, qrels, dataset_name, config, **experiment_kwargs)

Evaluate pipelines against one dataset.

The single point where pyterrier.Experiment() is called; override to swap the evaluation backend or inject extra arguments.

Return type:

DataFrame

Parameters:

• pipelines (Sequence[Transformer]) – Pipelines to evaluate together.

• names (Sequence[str | None]) – Display names aligned with pipelines; None entries are labelled positionally.

• topics (DataFrame) – Topics frame.

• qrels (DataFrame) – Qrels frame.

• dataset_name (str) – Dataset display name, used to resolve metrics.

• config (RunConfig) – The resolved run configuration.

• **experiment_kwargs (Any) – Extra arguments forwarded to the experiment.

Returns:

The raw experiment results.

Return type:

pandas.DataFrame

evaluate_batch(batch, topics, qrels, dataset_name, config)

Evaluate one batch of pipelines against one dataset.

Pipelines with a reusable run file are replayed from disk one by one; the remainder are evaluated together in a single experiment so that cross-system tests still see the full set.

Return type:

Iterator[DataFrame]

Parameters:

• batch (Sequence[Tuple[Transformer, str | None]]) – (pipeline, name) pairs to evaluate.

• topics (DataFrame) – Topics frame.

• qrels (DataFrame) – Qrels frame.

• dataset_name (str) – Dataset display name.

• config (RunConfig) – The resolved run configuration.

Yields:

pandas.DataFrame – One frame per experiment performed.

annotate_results(results, dataset_name, corpus_id)

Tag a result frame with the dataset it came from.

Return type:

DataFrame

Parameters:

• results (DataFrame) – Raw frame returned by run_experiment().

• dataset_name (str) – Dataset display name.

• corpus_id (str) – Identifier of the corpus the dataset belongs to.

Returns:

The annotated frame.

Return type:

pandas.DataFrame

static release_pipelines()

Best-effort memory cleanup between pipeline batches.

Calls gc.collect() and, if torch.cuda.is_available(), empties the CUDA cache. Silently ignores any exceptions (CUDA and torch are optional).

Return type:

None

static metric_columns(results)

Numeric columns of results that hold measure values.

Override alongside postprocess_results() when a suite reports columns that should not be aggregated as metrics.

Return type:

list[str]

Parameters:

results (DataFrame)

compute_overall_mean(results)

Append overall (geometric mean) rows across datasets for each system name.

This first aggregates per-dataset means over repeated runs, then computes the geometric mean across datasets for each metric and appends rows with dataset == "Overall".

Return type:

DataFrame

Parameters:

results (DataFrame) – DataFrame with at least ["dataset", "name"] and metric columns.

Returns:

The input results with additional Overall rows appended.

Return type:

pandas.DataFrame

postprocess_results(results, config)

Transform the concatenated results before they are returned.

Override to aggregate sub-datasets or reshape the table, then call super().postprocess_results(...) to keep the Overall rows.

Return type:

DataFrame

Parameters:

• results (DataFrame) – Concatenation of every annotated result frame.

• config (RunConfig) – The resolved run configuration.

Returns:

The final results table.

Return type:

pandas.DataFrame

resolve_config(eval_metrics=None, subset=None, compute_overall=True, **experiment_kwargs)

Split raw call arguments into suite settings and experiment kwargs.

index_dir and save_dir are consumed here because the suite rewrites them per corpus and per dataset respectively; everything else is forwarded to pyterrier.Experiment() untouched.

Return type:

RunConfig

Parameters:

• eval_metrics (Sequence[Any] | None) – Explicit metrics overriding the suite configuration.

• subset (str | None) – Optional dataset display name to restrict evaluation to.

• compute_overall (bool) – Whether to append geometric-mean Overall rows.

• **experiment_kwargs (Any) – Remaining arguments, including index_dir and save_dir.

Returns:

The resolved configuration for this call.

Return type:

RunConfig

run(ranking_generators, config)

Evaluate every pipeline against every selected dataset.

Walks corpus groups, building one shared context per corpus so that indexing happens once, and releases each pipeline batch once it has been evaluated against all datasets sharing that corpus.

Return type:

Iterator[DataFrame]

Parameters:

• ranking_generators (Callable[[DatasetContext], Any] | Sequence[Callable[[DatasetContext], Any]]) – Callable or sequence of callables producing pipelines.

• config (RunConfig) – The resolved run configuration.

Yields:

pandas.DataFrame – Annotated result frames, in evaluation order.

__call__(ranking_generators, eval_metrics=None, subset=None, compute_overall=True, **experiment_kwargs)

Run the experiment(s) for each dataset in the suite and return a results table.

If a baseline is provided in experiment_kwargs, all pipelines are materialized together (grouped mode) to enable tests that require joint access (e.g., significance). Otherwise, pipelines are streamed one-by-one to reduce memory usage (sequential mode).

Return type:

DataFrame

Parameters:

• ranking_generators (Callable[[DatasetContext], Any] | Sequence[Callable[[DatasetContext], Any]]) – Callable or sequence of callables producing pipelines per DatasetContext (same conventions as in coerce_pipelines_sequential()).

• eval_metrics (Sequence[Any] | None) – Optional explicit metrics to evaluate; defaults to the suite’s configuration for each dataset.

• subset (str | None) – Optional dataset display name to restrict evaluation to a single member.

• compute_overall (bool) – Whether to append geometric-mean Overall rows.

• **experiment_kwargs (Any) – Additional keyword arguments forwarded to pyterrier.Experiment(). If save_dir is provided, it is suffixed per dataset. If index_dir is provided, it is suffixed per corpus for index storage.

Returns:

The concatenated experiment results. When perquery is

not set, an additional Overall row is appended per system with geometric-mean aggregation across datasets.

Return type:

pandas.DataFrame

Notes

This method reuses a single index per corpus group and cleans up GPU memory between pipeline evaluations. Subclasses should prefer overriding the individual hooks documented on Suite over this method.

class suiteeval.suite.base.RunConfig(eval_metrics=None, subset=None, compute_overall=True, index_dir=None, save_dir=None, save_mode='warn', perquery=False, grouped=False, experiment_kwargs=<factory>)

Resolved settings for one __call__() invocation.

Produced by resolve_config(), which separates suite-level options from the keyword arguments forwarded verbatim to pyterrier.Experiment().

Parameters:

• eval_metrics (Sequence[Any] | None)

• subset (str | None)

• compute_overall (bool)

• index_dir (str | None)

• save_dir (str | None)

• save_mode (str)

• perquery (bool)

• grouped (bool)

• experiment_kwargs (dict[str, Any])

eval_metrics

Explicit metrics overriding the suite’s configuration.

subset

Restrict evaluation to a single dataset display name.

compute_overall

Whether to append geometric-mean Overall rows.

index_dir

Root directory for per-corpus indexes, if any.

save_dir

Root directory for per-dataset run files, if any.

save_mode

PyTerrier save mode; "overwrite" disables run-file reuse.

perquery

Whether results are reported per query.

grouped

Whether pipelines must be materialised together (significance tests).

experiment_kwargs

Remaining kwargs passed to pyterrier.Experiment().

eval_metrics: Sequence[Any] | None = None

subset: str | None = None

compute_overall: bool = True

index_dir: str | None = None

save_dir: str | None = None

save_mode: str = 'warn'

perquery: bool = False

grouped: bool = False

experiment_kwargs: dict[str, Any]

suiteeval.context

class suiteeval.context.DatasetContext(dataset, path=None)

Bases: object

Holds both a PyTerrier Dataset and a filesystem path (for indexes, caches, etc.).

Parameters:

• dataset (Dataset) – The pyterrier Dataset instance (must have _irds_id).

• path (str | None) – Optional filesystem path to use; if omitted, a temp dir will be created for you.

text_loader(fields='*')

Returns a IRDSTextLoader instance for retrieving document texts.

Parameters:

fields (List[str] | str | Literal['*']) – Fields to load; can be a list of field names, a single field name, or “*” for all fields.

Returns:

An IRDSTextLoader instance.

get_corpus_iter(**iter_kwargs)

Returns an iterator over the corpus documents.

Parameters:

**iter_kwargs – Keyword arguments passed to get_corpus_iter.

suiteeval.utility

suiteeval.utility.geometric_mean(sequence)

Compute the geometric mean of a sequence of numbers.

Return type:

float

Parameters:

sequence (Sequence[float]) – A sequence of numbers.

Returns:

The geometric mean of the sequence.

Return type:

float

suiteeval.utility.harmonic_mean(sequence)

Compute the harmonic mean of a sequence of numbers.

Return type:

float

Parameters:

sequence (Sequence[float]) – A sequence of numbers.

Returns:

The harmonic mean of the sequence.

Return type:

float