datakiln.extraction.strategies module¶
Execution strategies and parallel orchestration for the extraction pipeline.
- datakiln.extraction.strategies.compile_oracle_float(type_, compiler, **kwargs)[source]¶
Compile generic SQLAlchemy Float for Oracle dialect.
Pandas to_sql maps float64 columns to Float(precision=53). Oracle FLOAT types use binary precision instead of decimal precision, causing SQLAlchemy 2.0+ to raise an ArgumentError. Defaulting to plain ‘FLOAT’ allows Oracle to use its default binary precision (126).
- datakiln.extraction.strategies.compile_oracle_values(element, compiler, **kwargs)[source]¶
Translate a Values clause to Oracle JSON_TABLE query.
- class datakiln.extraction.strategies.HashingReader(fp)[source]¶
Bases:
objectFile-like wrapper that calculates MD5 hash on the fly while reading.
- class datakiln.extraction.strategies.ExecutionStrategy(db_params: Dict, directory: Path)[source]¶
Bases:
ABCAbstract base class for extraction strategies.
- property engine¶
Returns the shared database engine.
- session(bind: bool | None = True) Iterator[Session][source]¶
Returns a managed database session.
If a session is already active for this thread, it is yielded. Otherwise, a new pooled session is created and registered as active.
- Parameters:
bind – Whether to use the shared engine. If False, a new engine is created using Session.create().
- build_queries(task: ExtractMeta) Select[source]¶
Build the extraction SA Select statement.
- Parameters:
task (ExtractMeta) – Extraction task metadata.
- Returns:
SA Select statement.
- Return type:
Select
- property is_read_only: bool¶
Returns whether the strategy is read-only.
- abstractmethod process_filter(filter_config: FilterConfig, session: Session, task: ExtractMeta) List[Any][source]¶
Determine how to process a filter for this strategy.
- abstractmethod execute(task: ExtractMeta, export_to: Path | None = None)[source]¶
Execute the extraction task.
- property tmp_dir: Path¶
Folder of the files that are not exported.
- drop(name: str)[source]¶
Drop the temporary data of the task with this temporary name, which is its file.
- class datakiln.extraction.strategies.TempTableStrategy(db_params: Dict, directory: Path)[source]¶
Bases:
ExecutionStrategyStrategy that uses temporary tables (requires write permissions).
- has_index(session: Session, schema: str | None, table: str, columns: List[str]) bool[source]¶
Evaluate the existence of an index with the specified columns.
- ensure_index(session: Session, table_parts: List[str], columns: List[str])[source]¶
Create a covering index on the reference table if non-existent.
- process_filter(filter_config: FilterConfig, session: Session, task: ExtractMeta) List[Any][source]¶
Determine how to process a filter for this strategy.
- execute(task: ExtractMeta, export_to: Path | None = None)[source]¶
Execute the extraction task.
- class datakiln.extraction.strategies.JoinChain(filter_config: FilterConfig, engine)[source]¶
Bases:
objectJoin chain for filtering on read-only mode.
- classmethod build(filter_config: FilterConfig, engine, task: ExtractMeta) JoinChain[source]¶
Build and return a fully resolved chain for filter_config.
- Parameters:
filter_config (FilterConfig) – Filter configuration describing the join chain.
engine – SQLAlchemy engine used to reflect remote tables.
task (ExtractMeta) – Extraction task metadata.
- Returns:
Populated
JoinChainready to execute seed chunks.- Return type:
- class datakiln.extraction.strategies.StreamStrategy(db_params: Dict, directory: Path)[source]¶
Bases:
ExecutionStrategyStrategy that streams data directly from source (read-only).
- property is_read_only: bool¶
Returns whether the strategy is read-only.
- process_filter(filter_config: FilterConfig, session: Session, task: ExtractMeta) List[Any][source]¶
Determine how to process a filter for this strategy.
- execute(task: ExtractMeta, export_to: Path | None = None)[source]¶
Execute extraction by streaming and filtering in memory.
- get_cohort_set(filter_config: FilterConfig, task: ExtractMeta) List[Tuple[str, ...]][source]¶
Load cohort data into memory.
- Parameters:
filter_config (FilterConfig) – Filter configuration.
task (ExtractMeta) – Extraction task metadata.
- Returns:
List of cohort records.
- Return type:
list[tuple[str, …]]
- class datakiln.extraction.strategies.ParallelExecutor(strategy: ExecutionStrategy, graph: ExtractGraph, max_workers: int = 4)[source]¶
Bases:
objectOrchestrates parallel execution with reference counting for cleanup.