laurel.datasets package

Submodules

laurel.datasets.directory_partitioned module

DirectoryPartitionedDataset for handling partitioned datasets where each partition is a directory

class laurel.datasets.directory_partitioned.DirectoryPartitionedDataset(*, path, dataset, credentials=None, load_args=None, fs_args=None, overwrite=False, save_lazily=True, metadata=None)

Bases: PartitionedDataset

DirectoryPartitionedDataset extends PartitionedDataset to work with directory-based partitions.

This dataset treats directories (not individual files) as partitions, which is useful for datasets like Dask Parquet, GeoPandas Parquet, or any other format that stores data as directories containing multiple files.

Unlike the standard PartitionedDataset which treats each file as a partition, this dataset treats each directory as a partition, making it suitable for: - Dask Parquet datasets (directories with part.*.parquet files) - GeoPandas Parquet datasets - Any custom dataset that stores partitioned data as directories

Example usage in catalog.yml:

# With Dask Parquet
my_dask_partitioned_data:
  type: laurel.datasets.directory_partitioned.DirectoryPartitionedDataset
  path: data/07_model_output/my_dask_data
  dataset:
    type: dask.ParquetDataset
    save_args:
      write_index: False
      engine: pyarrow
    load_args:
      engine: pyarrow

# With GeoPandas Parquet
my_geo_partitioned_data:
  type: laurel.datasets.directory_partitioned.DirectoryPartitionedDataset
  path: data/07_model_output/my_geo_data
  dataset:
    type: laurel.datasets.geoparquet.GeoParquetDataset
    save_args:
      write_index: False
    load_args:
      engine: pyarrow

The dataset will create/read directory structures like:

data/07_model_output/my_data/
├── partition1/
│   ├── part.0.parquet
│   ├── part.1.parquet
│   └── _metadata
└── partition2/
    ├── part.0.parquet
    ├── part.1.parquet
    └── _metadata

On load, returns a dict with partition IDs as keys and lazy load functions as values, exactly like the standard PartitionedDataset.

On save, expects a dict with partition IDs as keys and data as values.

__init__(*, path, dataset, credentials=None, load_args=None, fs_args=None, overwrite=False, save_lazily=True, metadata=None)

Creates a new instance of DirectoryPartitionedDataset.

Parameters:

• path (str) – Path to the directory containing partitioned data directories.

• dataset (str | type | dict[str, Any]) –

  Underlying dataset definition for each partition directory. This can be any dataset type that can handle directory-based data. Accepted formats are:

  1. object of a class that inherits from AbstractDataset

  2. a string representing a fully qualified class name to such class

  3. a dictionary with ‘type’ key pointing to a string from b), other keys are passed to the dataset initializer.

• credentials (dict[str, Any] | None) – Protocol-specific options passed to fsspec.filesystem.

• load_args (dict[str, Any] | None) – Keyword arguments passed to filesystem find() method.

• fs_args (dict[str, Any] | None) – Extra arguments for underlying filesystem class constructor.

• overwrite (bool) – If True, existing partitions will be removed before saving.

• save_lazily (bool) – Enable/disable lazy saving (default True).

• metadata (dict[str, Any] | None) – Any arbitrary metadata (ignored by Kedro).

load()

Loads data by delegation to the provided load method.

Return type:

dict[str, Callable[[], Any]]

Returns:

Data returned by the provided load method.

Raises:

DatasetError – When underlying load method raises error.

save(data)

Saves data by delegation to the provided save method.

Parameters:

data (dict[str, Any]) – the value to be saved by provided save method.

Raises:

• DatasetError – when underlying save method raises error.

• FileNotFoundError – when save method got file instead of dir, on Windows.

• NotADirectoryError – when save method got file instead of dir, on Unix.

Return type:

None

laurel.datasets.geodatabase module

class laurel.datasets.geodatabase.GeoDataBaseDataset(filepath, load_args=None, save_args=None)

Bases: AbstractDataset[GeoDataFrame, GeoDataFrame]

GeoDataBaseDataset loads / saves geographic data from a given filepath as geopandas.GeoDataFrame using the GeoDataBase format.

Example:

GeoDataBaseDataset(filepath='/geo/file/path.gpkg')

DEFAULT_LOAD_ARGS: dict[str, Any] = {}

DEFAULT_SAVE_ARGS: dict[str, Any] = {}

__init__(filepath, load_args=None, save_args=None)

Creates a new instance of GeoDataBaseDataset pointing to a concrete GeoDataBase file on a specific filesystem.

Parameters:

• filepath (str) – Filepath in POSIX format to a GeoDataBase file prefixed with a protocol like s3://. If prefix is not provided, file protocol (local filesystem) will be used. The prefix should be any protocol supported by fsspec.

• load_args (dict[str, Any]) – GeoPandas options for loading GeoDataBase files. Here you can find all available arguments: https://geopandas.org/en/stable/docs/reference/api/geopandas.read_file.html All defaults are preserved.

• save_args (dict[str, Any]) – GeoPandas options for saving GeoDataBase files. Here you can find all available arguments: https://geopandas.org/en/stable/docs/reference/api/geopandas.GeoDataFrame.to_file.html All defaults are preserved, but “index”, which is set to False.

• metadata – Any Any arbitrary metadata. This is ignored by Kedro, but may be consumed by users or external plugins.

load()

Loads data from the GeoDataBase file.

Return type:

GeoDataFrame

Returns:

Data from the GeoDataBase file as a geopandas.GeoDataFrame

save(gdf)

Saves geographic data to the specified filepath.

Return type:

None

laurel.datasets.geopackage module

class laurel.datasets.geopackage.GeoPackageDataset(filepath, load_args=None, save_args=None)

Bases: AbstractDataset[GeoDataFrame, GeoDataFrame]

GeoPackageDataset loads / saves geographic data from a given filepath as geopandas.GeoDataFrame using the GeoPackage format.

Example:

GeoPackageDataset(filepath='/geo/file/path.gpkg')

DEFAULT_LOAD_ARGS: dict[str, Any] = {}

DEFAULT_SAVE_ARGS: dict[str, Any] = {'driver': 'GPKG', 'index': False, 'layer': 'default_layer'}

__init__(filepath, load_args=None, save_args=None)

Creates a new instance of GeoPackageDataset pointing to a concrete GeoPackage file on a specific filesystem.

Parameters:

• filepath (str) – Filepath in POSIX format to a GeoPackage file prefixed with a protocol like s3://. If prefix is not provided, file protocol (local filesystem) will be used. The prefix should be any protocol supported by fsspec.

• load_args (dict[str, Any]) – GeoPandas options for loading GeoPackage files. Here you can find all available arguments: https://geopandas.org/en/stable/docs/reference/api/geopandas.read_file.html All defaults are preserved.

• save_args (dict[str, Any]) – GeoPandas options for saving GeoPackage files. Here you can find all available arguments: https://geopandas.org/en/stable/docs/reference/api/geopandas.GeoDataFrame.to_file.html All defaults are preserved, but “index”, which is set to False.

• metadata – Any Any arbitrary metadata. This is ignored by Kedro, but may be consumed by users or external plugins.

load()

Loads data from the GeoPackage file.

Return type:

GeoDataFrame

Returns:

Data from the GeoPackage file as a geopandas.GeoDataFrame

save(gdf)

Saves geographic data to the specified filepath.

If the “layer” argument is a string, then the GeoDataFrame will be saved to a layer with the name given by “layer” using the currently set geometry.

If the “layer” argument is a dictionary, then the GeoDataFrame will be saved to a layer with the name given by the key of layer and the geometry in the column of the GeoDataFrame indexed by the value of the dict.

Return type:

None

laurel.datasets.geoparquet module

ParquetDataset is a dataset used to load and save data to parquet files using Dask dataframe

class laurel.datasets.geoparquet.GeoParquetDataset(*, filepath, load_args=None, save_args=None, credentials=None, fs_args=None, metadata=None)

Bases: AbstractDataset[GeoDataFrame, GeoDataFrame]

ParquetDataset loads and saves data to parquet file(s). It uses Dask remote data services to handle the corresponding load and save operations: https://docs.dask.org/en/stable/how-to/connect-to-remote-data.html

Example usage for the YAML API:

cars:
    type: dask.ParquetDataset
    filepath: s3://bucket_name/path/to/folder
    save_args:
    compression: GZIP
    credentials:
    client_kwargs:
        aws_access_key_id: YOUR_KEY
        aws_secret_access_key: YOUR_SECRET

Example usage for the Python API:

import dask.dataframe as dd
import pandas as pd
from kedro_datasets.dask import ParquetDataset
import numpy as np

data = pd.DataFrame({"col1": [1, 2], "col2": [4, 5], "col3": [6, 7]})
ddf = dd.from_pandas(data, npartitions=2)

dataset = ParquetDataset(
    filepath=tmp_path / "path/to/folder", save_args={"compression": "GZIP"}
)
dataset.save(ddf)
reloaded = dataset.load()

assert np.array_equal(ddf.compute(), reloaded.compute())

The output schema can also be explicitly specified using Triad. This is processed to map specific columns to PyArrow field types or schema. For instance:

parquet_dataset:
  type: dask.ParquetDataset
  filepath: "s3://bucket_name/path/to/folder"
  credentials:
    client_kwargs:
      aws_access_key_id: YOUR_KEY
      aws_secret_access_key: "YOUR SECRET"
  save_args:
    compression: GZIP
    schema:
      col1: [int32]
      col2: [int32]
      col3: [[int32]]

DEFAULT_LOAD_ARGS: dict[str, Any] = {}

DEFAULT_SAVE_ARGS: dict[str, Any] = {'write_index': False}

__init__(*, filepath, load_args=None, save_args=None, credentials=None, fs_args=None, metadata=None)

Creates a new instance of ParquetDataset pointing to concrete parquet files.

Parameters:

• filepath (str) – Filepath in POSIX format to a parquet file parquet collection or the directory of a multipart parquet.

• load_args (dict[str, Any] | None) – Additional loading options dask.dataframe.read_parquet: https://docs.dask.org/en/stable/generated/dask.dataframe.read_parquet.html

• save_args (dict[str, Any] | None) – Additional saving options for dask.dataframe.to_parquet: https://docs.dask.org/en/stable/generated/dask.dataframe.to_parquet.html

• credentials (dict[str, Any] | None) – Credentials required to get access to the underlying filesystem. E.g. for GCSFileSystem it should look like {"token": None}.

• fs_args (dict[str, Any] | None) – Optional parameters to the backend file system driver: https://docs.dask.org/en/stable/how-to/connect-to-remote-data.html#optional-parameters

• metadata (dict[str, Any] | None) – Any arbitrary metadata. This is ignored by Kedro, but may be consumed by users or external plugins.

property fs_args: dict[str, Any]

Property of optional file system parameters.

Returns:

A dictionary of backend file system parameters, including credentials.

load()

Loads data by delegation to the provided load method.

Return type:

GeoDataFrame

Returns:

Data returned by the provided load method.

Raises:

DatasetError – When underlying load method raises error.

save(data)

Saves data by delegation to the provided save method.

Parameters:

data (GeoDataFrame) – the value to be saved by provided save method.

Raises:

• DatasetError – when underlying save method raises error.

• FileNotFoundError – when save method got file instead of dir, on Windows.

• NotADirectoryError – when save method got file instead of dir, on Unix.

Return type:

None

laurel.datasets.shapefile module

class laurel.datasets.shapefile.ShapefileDataset(filepath, load_args=None, save_args=None)

Bases: AbstractDataset[GeoDataFrame, GeoDataFrame]

ShapefileDataset loads / saves geographic data from a given filepath as geopandas.GeoDataFrame using the Shapefile format.

Example:

ShapefileDataset(filepath='/geo/file/path.gpkg')

DEFAULT_LOAD_ARGS: dict[str, Any] = {}

DEFAULT_SAVE_ARGS: dict[str, Any] = {}

__init__(filepath, load_args=None, save_args=None)

Creates a new instance of ShapefileDataset pointing to a concrete Shapefile file on a specific filesystem.

Parameters:

• filepath (str) – Filepath in POSIX format to a Shapefile file prefixed with a protocol like s3://. If prefix is not provided, file protocol (local filesystem) will be used. The prefix should be any protocol supported by fsspec.

• load_args (dict[str, Any]) – GeoPandas options for loading Shapefile files. Here you can find all available arguments: https://geopandas.org/en/stable/docs/reference/api/geopandas.read_file.html All defaults are preserved.

• save_args (dict[str, Any]) – GeoPandas options for saving Shapefile files. Here you can find all available arguments: https://geopandas.org/en/stable/docs/reference/api/geopandas.GeoDataFrame.to_file.html All defaults are preserved, but “index”, which is set to False.

• metadata – Any Any arbitrary metadata. This is ignored by Kedro, but may be consumed by users or external plugins.

load()

Loads data from the Shapefile file.

Return type:

GeoDataFrame

Returns:

Data from the Shapefile file as a geopandas.GeoDataFrame

save(gdf)

Saves geographic data to the specified filepath.

Return type:

None

Module contents