datakiln.extraction.strategies module

Execution strategies and parallel orchestration for the extraction pipeline.

datakiln.extraction.strategies.compile_oracle_float(type_, compiler, **kwargs)[source]

Compile generic SQLAlchemy Float for Oracle dialect.

Pandas to_sql maps float64 columns to Float(precision=53). Oracle FLOAT types use binary precision instead of decimal precision, causing SQLAlchemy 2.0+ to raise an ArgumentError. Defaulting to plain ‘FLOAT’ allows Oracle to use its default binary precision (126).

datakiln.extraction.strategies.compile_oracle_values(element, compiler, **kwargs)[source]

Translate a Values clause to Oracle JSON_TABLE query.

class datakiln.extraction.strategies.HashingReader(fp)[source]

Bases: object

File-like wrapper that calculates MD5 hash on the fly while reading.

read(size=-1)[source]

Read data and update hasher.

seek(offset, whence=0)[source]

Delegate seek to the underlying file pointer.

tell()[source]

Delegate tell to the underlying file pointer.

class datakiln.extraction.strategies.ExecutionStrategy(db_params: Dict, directory: Path)[source]

Bases: ABC

Abstract base class for extraction strategies.

property engine

Returns the shared database engine.

session(bind: bool | None = True) → Iterator[Session][source]

Returns a managed database session.

If a session is already active for this thread, it is yielded. Otherwise, a new pooled session is created and registered as active.

Parameters:

bind – Whether to use the shared engine. If False, a new engine is created using Session.create().

build_queries(task: ExtractMeta) → Select[source]

Build the extraction SA Select statement.

Parameters:

task (ExtractMeta) – Extraction task metadata.

Returns:

SA Select statement.

Return type:

Select

property is_read_only: bool

Returns whether the strategy is read-only.

abstractmethod process_filter(filter_config: FilterConfig, session: Session, task: ExtractMeta) → List[Any][source]

Determine how to process a filter for this strategy.

abstractmethod execute(task: ExtractMeta, export_to: Path | None = None)[source]

Execute the extraction task.

abstractmethod get_cohort(cohort: str) → str[source]

Retrieve or create a cohort representation.

abstractmethod drop_cohort(cohort: str)[source]

Drop a specific cohort table.

property tmp_dir: Path

Folder of the files that are not exported.

drop(name: str)[source]

Drop the temporary data of the task with this temporary name, which is its file.

cleanup()[source]

Drop all remaining temporary data and cohorts, and the folder of the files that are not exported.

drop_cohorts()[source]

Drop all remaining cohort tables.

class datakiln.extraction.strategies.TempTableStrategy(db_params: Dict, directory: Path)[source]

Bases: ExecutionStrategy

Strategy that uses temporary tables (requires write permissions).

has_index(session: Session, schema: str | None, table: str, columns: List[str]) → bool[source]

Evaluate the existence of an index with the specified columns.

ensure_index(session: Session, table_parts: List[str], columns: List[str])[source]

Create a covering index on the reference table if non-existent.

process_filter(filter_config: FilterConfig, session: Session, task: ExtractMeta) → List[Any][source]

Determine how to process a filter for this strategy.

execute(task: ExtractMeta, export_to: Path | None = None)[source]

Execute the extraction task.

drop(name: str)[source]

Drop a temporary table, and its file.

get_cohort(cohort: str) → str[source]

Retrieve or create a cohort table.

drop_cohort(cohort: str)[source]

Drop a specific cohort table.

class datakiln.extraction.strategies.JoinChain(filter_config: FilterConfig, engine)[source]

Bases: object

Join chain for filtering on read-only mode.

classmethod build(filter_config: FilterConfig, engine, task: ExtractMeta) → JoinChain[source]

Build and return a fully resolved chain for filter_config.

Parameters:
  • filter_config (FilterConfig) – Filter configuration describing the join chain.

  • engine – SQLAlchemy engine used to reflect remote tables.

  • task (ExtractMeta) – Extraction task metadata.

Returns:

Populated JoinChain ready to execute seed chunks.

Return type:

JoinChain

execute(chunk: List, session) → List[source]

Execute a seed chunk against the built chain.

Parameters:
  • chunk (list) – Batch of seed tuples to look up.

  • session (Session) – Active SQLAlchemy session.

Returns:

All matching rows from the remote tables.

Return type:

list

class datakiln.extraction.strategies.StreamStrategy(db_params: Dict, directory: Path)[source]

Bases: ExecutionStrategy

Strategy that streams data directly from source (read-only).

property is_read_only: bool

Returns whether the strategy is read-only.

process_filter(filter_config: FilterConfig, session: Session, task: ExtractMeta) → List[Any][source]

Determine how to process a filter for this strategy.

execute(task: ExtractMeta, export_to: Path | None = None)[source]

Execute extraction by streaming and filtering in memory.

drop_cohort(cohort: str)[source]

Clear cohort from memory (no DDL in stream mode).

get_cohort(cohort: str) → Any[source]

Register the cohort but do not load data yet.

get_cohort_set(filter_config: FilterConfig, task: ExtractMeta) → List[Tuple[str, ...]][source]

Load cohort data into memory.

Parameters:
Returns:

List of cohort records.

Return type:

list[tuple[str, …]]

class datakiln.extraction.strategies.ParallelExecutor(strategy: ExecutionStrategy, graph: ExtractGraph, max_workers: int = 4)[source]

Bases: object

Orchestrates parallel execution with reference counting for cleanup.

execute(data_dir: Path)[source]

Run tasks in parallel using TopologicalSorter interactive mode.

Parameters:

data_dir (Path) – Directory of the data of the tasks to export.