datakiln.extraction.models module

Configuration domain models and task graphs for DBMS extraction pipeline.

datakiln.extraction.models.alias_column(column: str, default: str) → str[source]

Impute table alias for column reference.

Parameters:
  • column (str) – Column name or template string.

  • default (str) – Default table alias.

Returns:

Aliased column name.

Return type:

str

datakiln.extraction.models.get_table(name: str, alias: str | None = None) → Any[source]

Helper to create a Table object, resolving schema-qualification.

Parameters:
  • name (str) – Table name (potentially schema-qualified).

  • alias (str, optional) – Optional table alias name.

Returns:

SA Table or Alias object.

Return type:

Table | Alias

class datakiln.extraction.models.ReferenceConfig(columns: List[str], schema: str | None = None, table: str | None = None, operators: List[str] | None = None, alias: str | None = 'ref')[source]

Bases: object

Target reference table configuration.

Parameters:
  • columns (list[str]) – Reference column names.

  • schema (str, optional) – Reference database schema.

  • table (str, optional) – Reference table name.

  • operators (list[str], optional) – Comparison operators.

  • alias (str, optional) – Table alias.

property full_name: str

Returns the full table name (schema.table).

property patterns: List[Any]

Precompiled regex patterns for reference columns.

class datakiln.extraction.models.ColumnPair(left: str, right: str)[source]

Bases: object

Base class for a single condition in a JoinConfig pair list.

static bare(col: str) → str[source]

Strip the alias prefix from a qualified column name.

Parameters:

col (str) – Qualified or bare column name (e.g. 'alias.column').

Returns:

The bare column name after the last dot.

Return type:

str

static extract_columns(expr: str, alias: str) → List[str][source]

Extract all column names associated with an alias.

Parameters:
  • expr (str) – SQL column expression (e.g. 'COALESCE(src.id, ref.pat_id)').

  • alias (str) – Table alias (e.g. 'ref').

Returns:

List of extracted bare column names.

Return type:

list[str]

classmethod classify(left: str, right: str, cohort_alias: str) → ColumnPair[source]

Classify a processed column pair by inspecting its syntax.

Parameters:
  • left (str) – Processed left-side column.

  • right (str) – Processed right-side column.

  • cohort_alias (str) – Alias of the root reference (local cohort table).

Returns:

Appropriate ColumnPair subclass instance.

Return type:

ColumnPair

to_onclause(cohort_alias: str, ref_alias: str) → str[source]

Return the SQL ON-clause fragment for this pair.

Parameters:
  • cohort_alias (str) – Alias of the local cohort table.

  • ref_alias (str) – Alias of the remote join table.

Returns:

SQL equality fragment for use in a JOIN ON clause.

Return type:

str

class datakiln.extraction.models.SeedPair(left: str, right: str)[source]

Bases: ColumnPair

Links a local cohort column to a remote join-table column.

At query time the local cohort column is matched against the remote table via a VALUES construct, passing seed IDs across database boundaries.

property local: str

Bare cohort column name: ‘alias.col’ -> ‘col’.

property remote: str

Bare remote column name: ‘alias.col’ -> ‘col’.

to_onclause(cohort_alias: str, ref_alias: str) → str[source]
Inheritdoc:

class datakiln.extraction.models.ConstantFilter(left: str, right: str)[source]

Bases: ColumnPair

A constant equality constraint on a remote column.

Applied as a WHERE predicate in StreamStrategy and as an ON-clause fragment in TempTableStrategy. Normalised so the constant value is always stored in left.

property value: str

The literal constant: e.g. ‘15’.

property remote: str

Bare remote column name: ‘alias.col’ -> ‘col’.

to_onclause(cohort_alias: str, ref_alias: str) → str[source]
Inheritdoc:

class datakiln.extraction.models.CrossPair(left: str, right: str)[source]

Bases: ColumnPair

An equality condition between two remote aliased columns.

Applied as part of the JOIN ON clause when joining remote tables to one another (e.g. hx.pat_id = enc.pat_id).

to_onclause(cohort_alias: str, ref_alias: str) → str[source]
Inheritdoc:

class datakiln.extraction.models.JoinConfig(columns: List[str], reference: Dict[str, Any], reference_alias: str = 'ref')[source]

Bases: object

Intermediate join configuration for reference chains.

Parameters:
  • columns (list[str]) – Source columns for the join.

  • reference (dict[str, Any]) – Reference table configuration.

property columns: List[str]

Left-side column references, other than the constants.

seed_pairs() → Iterator[SeedPair][source]

Pairs that link a local cohort column to a remote column.

const_filters() → Iterator[ConstantFilter][source]

Pairs that apply a constant equality constraint.

cross_pairs() → Iterator[CrossPair][source]

Pairs that join two remote aliased columns.

to_join(cohorts: Dict[str, str])[source]

Return the resolved table name to join against.

Prefers the temporary table created during this run if it exists, otherwise falls back to the permanent schema-qualified table name.

Parameters:

cohorts (dict[str, str]) – Mapping of task name -> created temp table name, as maintained by the active ExecutionStrategy.

Returns:

Resolved table name string (used to build SA TableClause).

Return type:

str

to_onclause() → Any[source]

Build a SQLAlchemy ON clause for this join as a text expression.

Returns:

SA text clause for the JOIN ON condition.

Return type:

ClauseElement

class datakiln.extraction.models.FilterConfig(columns: List[str], reference: Dict[str, Any], cohort: str | None = None, join: List[Dict[str, Any]] | None = None)[source]

Bases: object

Filter configuration mapping a task to a cohort or reference table.

Parameters:
  • columns (list[str]) – Target columns to filter.

  • reference (ReferenceConfig, optional) – Target reference configuration.

  • cohort (str, optional) – Source cohort identifier.

  • join (list[JoinConfig], optional) – List of intermediate join chains.

property is_sql_cohort: bool

Evaluates whether the filter targets an explicit SQL cohort file.

property resolved_cohort: str | None

Return the resolved cohort table name.

property cohort_columns: List[str]

Return the target columns to fetch from the cohort table.

to_exists(strategy: ExecutionStrategy) → Any[source]

Generate a SQLAlchemy EXISTS subquery for this filter configuration.

Parameters:

strategy (ExecutionStrategy) – Execution strategy (used to resolve cohort aliases).

Returns:

SA exists() expression.

Return type:

Exists

to_where(values: List[Tuple]) → Any[source]

Generate a SQLAlchemy WHERE clause from pre-fetched cohort data.

Parameters:

values (list[tuple]) – List of value tuples from the resolved cohort.

Returns:

SA ClauseElement.

Return type:

ClauseElement

class datakiln.extraction.models.ExtractMeta(table: str, schema: str, filters: List[Dict | List[Dict]] = None, encoding: str = 'utf-8', chunk_size: int = 10000, **kwargs)[source]

Bases: object

Extraction metadata.

property full_name: str

Returns the full table name (schema.table).

property temp_name: str

Returns the temporary table name used during extraction.

property requires: Set[str]

Retrieve dependencies from filters.

Returns:

Set of dependency task names.

Return type:

set[str]

classmethod from_json(path: Path) → ExtractMeta[source]

Load extraction metadata from JSON.

Parameters:

path (Path) – Path to the extraction metadata file.

Returns:

Extraction metadata.

Return type:

ExtractMeta

class datakiln.extraction.models.ExtractGraph(tasks: Iterable[ExtractMeta], export: Set[ExtractMeta] | None = None)[source]

Bases: DependencyGraph

Directed Acyclic Graph for extraction tasks.

Dependencies that are not tasks (e.g. cohorts and tables that are not extracted) are kept, so they can be orchestrated by the executor.

classmethod select(tasks: List[ExtractMeta], include: List[str] | None = None) → ExtractGraph[source]

Graph of the tasks to export, and the tasks they need.

Parameters:
  • tasks (list[ExtractMeta]) – Extraction tasks.

  • include (list[str]) – Names of the tasks to export. All by default.

Raises:

ValueError – If any of the names is not a task.

property tasks: Dict[str, ExtractMeta]

Extraction tasks by name.