laurel.scenario_framework package

Submodules

laurel.scenario_framework.build module

Abstract base classes for Kedro scenario configuration generation.

Provides ScenarioBuilder, the abstract interface that concrete scenario builders must implement to generate a Kedro configuration file tree for a batch of scenario runs (States of the World). Each concrete builder maps a parameter grid to a directory layout that is compatible with SLURM job arrays and Kedro’s PartitionedDataset format.

The generated layout follows the pattern:

<scenario_name>/<intermediate_dirs>/task_<N>/parameters.yml
<scenario_name>/<intermediate_dirs>/task_<N>/catalog.yml

where N is the zero-based SLURM array task index. The per-task catalog.yml overrides selected PartitionedDataset entries from conf/base/catalog.yml with direct single-file or single-directory entries that point at the task-specific path. This lets Kedro determine whether an individual task has already run (via its standard “skip if output exists” logic) without touching the base catalog.

Which entries are overridden is declared in conf/base/catalog.yml itself, using the metadata.scenario_override field on each relevant entry. The base catalog therefore remains the single source of truth: adding a new scenario-specific dataset only requires annotating it there.

Key design decisions

  • ``task_N`` directory layout: giving each task its own subdirectory avoids filesystem conflicts when SLURM array jobs run concurrently, and makes it trivial for ScenarioReader to filter completed vs. pending tasks by path prefix.

  • Per-task catalog overrides: generated alongside each parameters.yml so Kedro can check individual task completion. Entries with flatten: true collapse a PartitionedDataset into the inner dataset type (e.g. pandas.FeatherDataset) pointing at the task file; entries without flatten redirect path to the task subdirectory (used for debug partitions whose nodes return dicts).

  • ``n_tasks_generated`` side-effect: ScenarioBuilder.build_configs() sets this attribute so that ScenarioBashWriter can generate the correct --array=0-N SLURM range without a separate counting step.

class laurel.scenario_framework.build.ScenarioBuilder(scen_params, all_params, catalog)[source]

Bases: ABC

Abstract base class for scenario configuration builders.

Concrete subclasses define how a parameter grid maps to Kedro configuration partitions. The framework uses the SLURM_ARRAY_TASK_ID environment variable to select the correct partition at runtime, or a local for loop for serial execution.

Class attributes:
_registry: Maps each concrete subclass name to the class itself.

Populated automatically by __init_subclass__() when a subclass is defined (i.e. when its module is imported).

n_tasks_generated: Set to the total number of tasks by

build_configs(); initially None.

__init__(scen_params, all_params, catalog)[source]

Initialise a scenario builder.

Parameters:

  • scen_params (dict) – Scenario-specific parameters dict. Must contain the key "display_name" (a human-readable scenario identifier used as the root directory name). May contain additional keys consumed by _build_param_dicts() in concrete subclasses (e.g. "builder", "n_scenarios", "seed").

  • all_params (dict) – The full Kedro parameters dict for the run, forwarded to concrete builders that need pipeline-level parameters such as base paths or default energy rates.

  • catalog (dict) – The raw conf/base/catalog.yml dict, loaded as plain YAML (no OmegaConf interpolation). _build_single_partition() emits a catalog.yml alongside each parameters.yml by calling _build_single_catalog().

Raises:

RuntimeError – If "display_name" is absent from scen_params.

build_configs()[source]

Generate the full set of Kedro configuration partition dicts.

Calls _build_param_dicts(), then wraps each (path, params) pair with _build_single_partition(). Sets n_tasks_generated as a side-effect so the bash writer can generate the correct SLURM array range.

Return type:

dict[Path, dict]

Returns:

Dict mapping each task’s config file path to its content dict. Suitable for saving as a Kedro PartitionedDataset.

n_tasks_generated: int | None = None
abstract property partition_level_names: tuple[str]

Ordered tuple of path-component names that describe each partition level.

These names correspond to the path components between the scenario root and the task_N leaf directory (e.g. ("run_name", "task_id")). ScenarioReader uses them to extract structured metadata from partition paths.

class laurel.scenario_framework.build.TestScenarioBuilder(scen_params, all_params, catalog)[source]

Bases: ScenarioBuilder

Minimal concrete builder that produces a single-task scenario partition.

Used in tests and as a reference implementation for ScenarioBuilder. Generates one partition whose output directory is <display_name>/task_0/ with no extra parameters.

partition_level_names = ('run_name', 'task_id')

laurel.scenario_framework.cmd module

Bash and SLURM script generation for batch scenario execution.

Provides ScenarioBashWriter for generating shell scripts that launch Kedro scenario arrays either locally or via SLURM, and a thin pipeline-facing wrapper generate_bash_script() used from the build_scenarios pipeline.

Key design decisions

  • Three command modes: sbatch generates a self-contained SLURM batch script with #SBATCH directives and dynamic config-directory discovery; salloc generates an interactive allocation command (the --array option is stripped since salloc does not support job arrays); local wraps the Kedro call in a shell for loop that iterates over SLURM_ARRAY_TASK_ID values, mimicking SLURM array behaviour for local testing.

  • Runtime config discovery: rather than hardcoding task paths, the generated script uses find to locate the task_$SLURM_ARRAY_TASK_ID directory under conf/scenarios/<name>/ at runtime. This keeps the script independent of the absolute project path on the HPC cluster.

  • Multiple sequential Kedro calls: cmd_line_calls.kedro in the parameters YAML accepts either a single dict or a list of dicts. Each dict produces one kedro run line; multiple calls are emitted as consecutive lines so each runs independently. The config-directory discovery block is emitted once per task regardless of how many kedro run calls follow.

  • Option building via string template: ScenarioBashWriter.build_opts() uses a simple KEY/VALUE placeholder template rather than a CLI-parsing library, keeping the output predictable and easy to inspect.

class laurel.scenario_framework.cmd.ScenarioBashWriter(name, command)[source]

Bases: object

Generates shell scripts that launch Kedro scenario arrays.

Supports three execution modes (sbatch, salloc, local) controlled at construction time. Use compile() to obtain the complete script string, then write it to disk or pass it to the build_scenarios pipeline for storage.

Parameters:

  • name (str) – Human-readable scenario name (matches display_name). Used as the SLURM job name and the config search path.

  • command (str) – Execution mode — one of "sbatch", "salloc", or "local".

__init__(name, command)[source]
build_kedro_run(kedro_calls, prefix='', n_tasks=1)[source]

Build the shell commands that discover the config dir and invoke Kedro.

For sbatch mode, produces a find command that sets conf_dir from SLURM_ARRAY_TASK_ID, followed by one kedro run line per entry in kedro_calls. For local mode, wraps the entire block in a for loop over seq 0 <n_tasks-1> that simulates a SLURM array, with each line individually indented.

The config-directory discovery block is emitted exactly once per task regardless of how many kedro run calls are requested.

Parameters:

  • kedro_calls (list[dict] | dict) – A list of dicts, each mapping Kedro CLI option names to values (e.g. [{"pipeline": "electrify_trips"}, {"pipeline": "evaluate_impacts"}]). A bare dict is accepted for backward compatibility and is treated as a single-element list. The "env" key is injected automatically into each call and should not be included. At least one entry is required.

  • prefix (str) – Shell command prefix inserted before each kedro run invocation (e.g. "uv run" or "srun"). Defaults to "".

  • n_tasks (int) – Total number of tasks in the array; used only for local mode to set the for loop range. Defaults to 1.

Return type:

str

Returns:

Multi-line shell string containing the config-discovery block and one or more kedro run invocations.

Raises:

ValueError – If kedro_calls is empty.

static build_opts(d, template)[source]

Render a dict of key-value pairs into a shell option string.

Substitutes each (key, value) pair into template by replacing the literal strings KEY and VALUE. Concatenates all rendered lines into a single string.

Parameters:

  • d (dict[slice(<class ‘str’>, str | int | float, None)]) – Ordered dict of option names to values (e.g. {"ntasks": 4, "mem": "64G"}).

  • template (str) – Format string containing the placeholders KEY and VALUE (e.g. " --KEY=VALUE" or "#SBATCH --KEY=VALUE\n").

Return type:

str

Returns:

Concatenated option string (e.g. " --ntasks=4 --mem=64G").

build_slurm_request(resources, reporting=None)[source]

Build the SLURM resource-request block for sbatch or salloc mode.

For sbatch, produces #SBATCH --key=value header lines (one per resource option) and injects job-name automatically. For salloc, produces a single salloc --key=value ... command string and strips the array option, which salloc does not support.

Parameters:

  • resources (dict) – Dict of SLURM resource options (e.g. {"ntasks": 4, "mem": "64G", "array": "0-511"}). Modified in-place to add job-name for sbatch mode.

  • reporting (dict) – Optional dict of additional SLURM options (e.g. email and output-file settings) merged into resources for sbatch mode only.

Return type:

str

Returns:

Multi-line string of #SBATCH directives (sbatch mode) or a single salloc ... command line (salloc mode).

Raises:

RuntimeError – If resources is None.

compile(params, resources=None, reporting=None, n_tasks=None)[source]

Compile all script sections into a complete shell script string.

Assembles the shebang line, optional SLURM directives, and the Kedro run block into a single multi-line string suitable for writing to a .sh file.

Parameters:

  • params (dict) –

    Dict with two required keys:

  • resources (dict) – SLURM resource dict forwarded to build_slurm_request(). Required when command is "sbatch" or "salloc"; ignored for "local".

  • reporting (dict) – Optional SLURM reporting options (email, output paths) forwarded to build_slurm_request().

  • n_tasks (int) – Total number of array tasks; forwarded to build_kedro_run() for local mode loop sizing.

Return type:

str

Returns:

Complete shell script string starting with #!/bin/bash.

laurel.scenario_framework.cmd.generate_bash_script(command, builder, cmd_params=None, resources=None, reporting=None)[source]

Generate a Bash script for running a full scenario array.

Constructs a ScenarioBashWriter from the builder’s display name and compiles the complete script. Intended to be called as a Kedro node in the build_scenarios pipeline after ScenarioBuilder.build_configs() has been run (so that n_tasks_generated is set).

Parameters:
Return type:

dict[slice(<class ‘pathlib.Path’>, <class ‘str’>, None)]

Returns:

Single-entry dict {builder.display_name: script_string} suitable for saving as a Kedro PartitionedDataset.

laurel.scenario_framework.io module

Kedro pipeline node functions for reading and writing scenario-partitioned datasets.

Deprecated since version :func:`write_scenario_partition`: and read_scenario_partition() are no longer used by the electrify_trips or evaluate_impacts pipelines. Per-task I/O is now handled by per-task catalog.yml overrides generated by _build_single_catalog(). These functions are retained for backwards compatibility with ScenarioReader analysis scripts and any external callers.

Provides two thin wrapper nodes that bridge the generic Kedro PartitionedDataset API and the scenario directory layout defined by ScenarioBuilder:

  • write_scenario_partition() — packages an arbitrary object for Kedro to save under the current task’s output directory.

  • read_scenario_partition() — loads a single partition from that directory, enabling within-pipeline re-reads of just-written data.

laurel.scenario_framework.io.read_scenario_partition(partitions, params, client=None)[source]

Load a single partition from the current scenario’s output directory.

Deprecated since version Use: per-task catalog.yml overrides generated by _build_single_catalog() instead of this wrapper node.

Filters the full PartitionedDataset dict to the one entry whose path matches params["dir"], calls its loader function, and returns the result. Raises if zero or more than one matching partition is found.

For reading multiple partitions across scenarios, use read_partitions() instead.

Parameters:

  • partitions (dict) – Full Kedro PartitionedDataset dict mapping partition path strings to zero-argument loader callables.

  • params (dict) – The "results_partition" sub-dict from the task’s Kedro parameter config. Must contain "dir" (str) — the path of the partition to load.

  • client (Client) – Unused Dask Client argument retained for pipeline compatibility (ensures Dask is started before this node runs).

Return type:

object

Returns:

The loaded dataset object returned by the partition’s loader callable.

Raises:

RuntimeError – If more than one partition matches params["dir"].

laurel.scenario_framework.io.write_scenario_partition(obj, params)[source]

Package an object for Kedro to save to the current scenario’s output directory.

Deprecated since version Use: per-task catalog.yml overrides generated by _build_single_catalog() instead of this wrapper node.

Wraps obj in a single-entry partition dict keyed by the task directory path. Kedro’s PartitionedDataset machinery uses this dict to determine where and how to serialise the object (format is controlled by catalog.yml).

Parameters:

  • obj (object) – The dataset to save (e.g. a pd.DataFrame, dict, or any object supported by the catalog entry).

  • params (dict) –

    The "results_partition" sub-dict from the task’s Kedro parameter config. Must contain:

    • "dir" (str): the task output directory path.

    • "level_names" (list[str]): partition level name metadata (not used by this function but present in the dict).

Return type:

dict[str, object]

Returns:

Single-entry dict {params["dir"]: obj} ready for Kedro to persist.

laurel.scenario_framework.nodes module

Kedro node for dynamic scenario configuration dispatch.

Provides a single node function generate_scenario_configs() that looks up the requested ScenarioBuilder subclass in the class-level registry, instantiates it, and delegates to its build_configs() method.

Usage

Import this function into any Kedro pipeline’s nodes.py alongside the builder imports that should be available for that pipeline:

# In your pipeline's nodes.py:
import laurel.scenario_builders  # noqa: F401 — registers all bundled builders
from laurel.scenario_framework.nodes import generate_scenario_configs  # noqa: F401

Any ScenarioBuilder subclass imported (anywhere) before generate_scenario_configs() is called becomes discoverable — importing the module is the registration step.

Key design decisions

  • ``__init_subclass__`` registry: builders self-register when their module is imported via __init_subclass__(). This replaces the previous globals()/inspect approach, which required the function and all builder imports to live in the same module.

  • Import-as-registration: callers signal intent by importing builder modules. A plain import laurel.scenario_builders is enough to register all bundled builders; custom builders only need to be imported once anywhere before the node runs.

laurel.scenario_framework.nodes.generate_scenario_configs(scen_params, all_params, catalog)[source]

Dispatch to the named ScenarioBuilder and return configuration partitions.

Looks up scen_params["builder"] in _registry, instantiates the matching class, calls build_configs(), and returns the resulting partitions dict together with the builder instance.

The registry is populated automatically when a ScenarioBuilder subclass module is imported. Callers are responsible for ensuring the desired builders are imported before this function is called (typically via a module-level import laurel.scenario_builders in the pipeline’s nodes.py).

Parameters:

  • scen_params (dict) –

    Scenario-specific parameter dict. Must contain:

    • "builder" (str): class name of the concrete ScenarioBuilder subclass to use.

    • "display_name" (str): human-readable name forwarded to the builder constructor.

    • Any additional keys consumed by the chosen builder’s _build_param_dicts implementation.

  • all_params (dict) – The complete Kedro parameters dict for the run, forwarded unchanged to the builder constructor.

  • catalog (dict) – The raw conf/base/catalog.yml dict, loaded as plain YAML by the base_catalog catalog entry (no OmegaConf interpolation). Each task’s partition will include a catalog.yml that overrides scenario-specific entries with task-scoped paths.

Return type:

tuple

Returns:

Two-tuple (partitions, builder) where partitions is the dict of {config_file_path: content_dict} entries returned by build_configs(), and builder is the instantiated builder (whose n_tasks_generated is set and available for downstream nodes such as generate_bash_script()).

Raises:

NotImplementedError – If no builder matching scen_params["builder"] is found in _registry.

laurel.scenario_framework.read module

Abstract base classes for reading scenario-partitioned datasets into analysis DataFrames.

Provides ScenarioReader, the abstract interface that concrete readers must implement to collate Kedro PartitionedDataset partitions across scenario runs (States of the World) into a single combined DataFrame.

Typical post-hoc analysis workflow

  1. Instantiate a concrete reader with a list of scenario directories (or None to read all available partitions).

  2. Call ScenarioReader.read_partitions() with the Kedro partition dict to obtain a combined DataFrame labelled by scenario metadata.

  3. Use ScenarioReader.list_completed_partitions() to audit which tasks have finished before reading.

Key design decisions

  • Lazy/eager dual path: ScenarioReader.read_partitions() supports lazy=True (Dask from_map) for large post-hoc analyses that would exhaust memory if loaded eagerly, and lazy=False (pandas concat) for small result sets or interactive exploration.

  • Multi-level column handling: ScenarioReader._collate_partition_df() pads metadata column names with empty strings when the DataFrame has a multi-level column index, so that metadata columns appear at the correct level without restructuring the DataFrame.

  • ``metadata_level_names ⊆ partition_level_names`` validation: the constructor checks this invariant at instantiation time, surfacing misconfiguration before any I/O is attempted.

  • Path-based metadata extraction: scenario metadata (e.g. adoption-rate bin, energy-rate value) is encoded in the partition path by ScenarioBuilder; concrete readers parse it via extract_metadata() rather than storing it in the data files, keeping the files themselves schema-free.

class laurel.scenario_framework.read.ScenarioReader(dirs=None)[source]

Bases: ABC

Abstract base class for collating scenario-partitioned Kedro datasets.

Concrete subclasses must implement three abstract members:

Class attributes:
builder: The ScenarioBuilder class

(not instance) associated with this reader.

metadata_level_names: Tuple of path-level names to extract as DataFrame

metadata columns; must be a subset of builder.partition_level_names.

scenario_name: Column name for the human-readable scenario label added

by _collate_partition_df(). Defaults to "scenario".

__init__(dirs=None)[source]

Initialise the reader and validate metadata/partition name consistency.

Parameters:

dirs (list[Path] | list[str] | None) – One or more directories to restrict reading to. If a single string or Path is given, it is wrapped in a list. Pass None to read all available partitions.

Raises:

RuntimeError – If any name in metadata_level_names is not present in builder.partition_level_names.

abstract property builder: ScenarioBuilder

The ScenarioBuilder class whose partition layout this reader expects.

concat_name_components(*args, sep=', ')[source]

Join path-component strings into a formatted scenario name.

Replaces underscores with spaces in each component, then joins with sep.

Parameters:

  • *args (str) – Path component strings (e.g. "high_adoption", "task_42").

  • sep (str) – Separator inserted between components. Defaults to ", ".

Return type:

str

Returns:

Formatted scenario name string (e.g. "high adoption, task 42").

abstract extract_metadata(path)[source]

Parse scenario-identifying metadata from a partition path.

Return values must correspond in order with metadata_level_names.

Parameters:

path (Path) – Path object for a completed partition file.

Return type:

tuple

Returns:

Tuple of metadata values (strings or scalars) in the same order as metadata_level_names.

get_metadata_values(path)[source]

Extract the metadata-level path components from a partition path.

Uses the positional indices of metadata_level_names within builder.partition_level_names to select the corresponding path.parts components.

Parameters:

path (Path) – Path object for a completed partition file.

Return type:

dict[str, str]

Returns:

Dict mapping each metadata level name to its value from path.parts.

list_completed_partitions(data_partitions, config_partitions=None, incomplete=False, report_type='scenario')[source]

List completed (or incomplete) scenario partitions within self.dirs.

Compares the set of data partitions that already exist on disk against the full set of config partitions to determine which tasks have finished. When incomplete=True, returns the complement — the tasks that are still pending.

Parameters:

  • data_partitions (dict[str, object]) – Kedro PartitionedDataset dict for the output dataset (e.g. dwells_with_charging_partition). Only partitions within self.dirs are considered.

  • config_partitions (dict[str, object] | None) – Kedro PartitionedDataset dict for the generated configs (used only when incomplete=True to determine the full target set). May be None when incomplete=False.

  • incomplete (bool) – If True, return tasks that are in config_partitions but not yet in data_partitions. Defaults to False (return completed tasks).

  • report_type (str) – Format of the return values. "scenario" returns human-readable scenario names (via name_scenario()); "task" returns integer task IDs parsed from the task_N path component.

Return type:

list[str] | list[int]

Returns:

Sorted list of scenario name strings (report_type="scenario") or integer task IDs (report_type="task").

Raises:

  • RuntimeError – If no partitions are found within self.dirs.

  • NotImplementedError – If report_type is not "scenario" or "task".

abstract property metadata_level_names: tuple[str]

Ordered subset of builder.partition_level_names to use as metadata columns.

Names must exactly match entries in partition_level_names of the associated builder.

abstract name_scenario(path)[source]

Convert a partition path to a human-readable scenario label.

The helper concat_name_components() is typically useful for formatting the label from path components.

Parameters:

path (Path) – Path object for a completed partition file.

Return type:

str

Returns:

Human-readable scenario name string added to the combined DataFrame under the scenario_name column.

read_partitions(partitions, lazy=False)[source]

Load and collate scenario partitions into a single combined dataset.

Filters the partition dict to the directories in self.dirs, then for each matching partition calls the Kedro loader function, attaches metadata and scenario-name columns, and concatenates all results.

For pd.DataFrame partitions, supports both eager (pandas.concat) and lazy (dask.dataframe.from_map) execution. For dict partitions, only eager loading is supported.

Parameters:

  • partitions (dict[str, object]) – Full Kedro PartitionedDataset dict mapping path strings to zero-argument loader callables.

  • lazy (bool) – If True, return a Dask DataFrame constructed via from_map so that loading is deferred until .compute() is called. Only valid for pd.DataFrame partitions. Defaults to False.

Returns:

a pd.DataFrame (lazy=False) or dd.DataFrame (lazy=True) with additional columns for each metadata_level_names entry and for scenario_name. For dict partitions: a dict keyed by scenario name.

Return type:

Combined dataset. For pd.DataFrame partitions

Raises:

  • RuntimeError – If no partitions are found within self.dirs.

  • NotImplementedError – If lazy=True is requested for dict partitions, or if the partition data type is not supported.

scenario_name: str = 'scenario'
select_partitions(partitions)[source]

Filter a Kedro partition dict to entries under self.dirs.

Delegates to select_partitions_static() with self.dirs.

Parameters:

partitions (dict[Path, object]) – Full Kedro PartitionedDataset dict mapping path strings to loader callables.

Return type:

dict[Path, object]

Returns:

Filtered dict containing only the partitions whose paths are relative to one of the directories in self.dirs.

Raises:

RuntimeError – If no matching partitions are found.

static select_partitions_static(partitions, dirs=None)[source]

Filter a partition dict to entries whose paths are under any of dirs.

If dirs is None, all partitions are returned unchanged. Uses Path.is_relative_to to test containment so both absolute and relative paths are handled consistently.

Parameters:

  • partitions (dict[Path | str, object]) – Kedro PartitionedDataset dict (path → loader).

  • dirs (list[Path] | None) – List of directories to restrict to, or None to return all partitions.

Return type:

dict[Path, object]

Returns:

Filtered dict of matching {Path: loader} entries.

Raises:

RuntimeError – If no partitions match any of the given directories.

class laurel.scenario_framework.read.TestScenarioReader(dirs=None)[source]

Bases: ScenarioReader

Minimal concrete reader for the single-partition test scenario.

Pairs with TestScenarioBuilder. Extracts no metadata and labels every partition "Test".

builder

alias of TestScenarioBuilder

extract_metadata(path)[source]

Return an empty tuple (no metadata levels defined for test scenarios).

Parameters:

path (Path) – Partition path (unused).

Return type:

tuple

Returns:

Empty tuple ().

metadata_level_names: tuple[str] = ()
name_scenario(path)[source]

Return the fixed label "Test" for every partition.

Parameters:

path (Path) – Partition path (unused).

Return type:

str

Returns:

The string "Test".

Module contents