datakiln.extraction.models module¶
Configuration domain models and task graphs for DBMS extraction pipeline.
- datakiln.extraction.models.alias_column(column: str, default: str) str[source]¶
Impute table alias for column reference.
- Parameters:
column (str) – Column name or template string.
default (str) – Default table alias.
- Returns:
Aliased column name.
- Return type:
str
- datakiln.extraction.models.get_table(name: str, alias: str | None = None) Any[source]¶
Helper to create a Table object, resolving schema-qualification.
- Parameters:
name (str) – Table name (potentially schema-qualified).
alias (str, optional) – Optional table alias name.
- Returns:
SA Table or Alias object.
- Return type:
Table | Alias
- class datakiln.extraction.models.ReferenceConfig(columns: List[str], schema: str | None = None, table: str | None = None, operators: List[str] | None = None, alias: str | None = 'ref')[source]¶
Bases:
objectTarget reference table configuration.
- Parameters:
columns (list[str]) – Reference column names.
schema (str, optional) – Reference database schema.
table (str, optional) – Reference table name.
operators (list[str], optional) – Comparison operators.
alias (str, optional) – Table alias.
- property full_name: str¶
Returns the full table name (schema.table).
- property patterns: List[Any]¶
Precompiled regex patterns for reference columns.
- class datakiln.extraction.models.ColumnPair(left: str, right: str)[source]¶
Bases:
objectBase class for a single condition in a JoinConfig pair list.
- static bare(col: str) str[source]¶
Strip the alias prefix from a qualified column name.
- Parameters:
col (str) – Qualified or bare column name (e.g.
'alias.column').- Returns:
The bare column name after the last dot.
- Return type:
str
- static extract_columns(expr: str, alias: str) List[str][source]¶
Extract all column names associated with an alias.
- Parameters:
expr (str) – SQL column expression (e.g.
'COALESCE(src.id, ref.pat_id)').alias (str) – Table alias (e.g.
'ref').
- Returns:
List of extracted bare column names.
- Return type:
list[str]
- classmethod classify(left: str, right: str, cohort_alias: str) ColumnPair[source]¶
Classify a processed column pair by inspecting its syntax.
- Parameters:
left (str) – Processed left-side column.
right (str) – Processed right-side column.
cohort_alias (str) – Alias of the root reference (local cohort table).
- Returns:
Appropriate
ColumnPairsubclass instance.- Return type:
- to_onclause(cohort_alias: str, ref_alias: str) str[source]¶
Return the SQL ON-clause fragment for this pair.
- Parameters:
cohort_alias (str) – Alias of the local cohort table.
ref_alias (str) – Alias of the remote join table.
- Returns:
SQL equality fragment for use in a JOIN ON clause.
- Return type:
str
- class datakiln.extraction.models.SeedPair(left: str, right: str)[source]¶
Bases:
ColumnPairLinks a local cohort column to a remote join-table column.
At query time the local cohort column is matched against the remote table via a VALUES construct, passing seed IDs across database boundaries.
- property local: str¶
Bare cohort column name: ‘alias.col’ -> ‘col’.
- property remote: str¶
Bare remote column name: ‘alias.col’ -> ‘col’.
- class datakiln.extraction.models.ConstantFilter(left: str, right: str)[source]¶
Bases:
ColumnPairA constant equality constraint on a remote column.
Applied as a WHERE predicate in StreamStrategy and as an ON-clause fragment in TempTableStrategy. Normalised so the constant value is always stored in
left.- property value: str¶
The literal constant: e.g. ‘15’.
- property remote: str¶
Bare remote column name: ‘alias.col’ -> ‘col’.
- class datakiln.extraction.models.CrossPair(left: str, right: str)[source]¶
Bases:
ColumnPairAn equality condition between two remote aliased columns.
Applied as part of the JOIN ON clause when joining remote tables to one another (e.g.
hx.pat_id = enc.pat_id).
- class datakiln.extraction.models.JoinConfig(columns: List[str], reference: Dict[str, Any], reference_alias: str = 'ref')[source]¶
Bases:
objectIntermediate join configuration for reference chains.
- Parameters:
columns (list[str]) – Source columns for the join.
reference (dict[str, Any]) – Reference table configuration.
- property columns: List[str]¶
Left-side column references, other than the constants.
- const_filters() Iterator[ConstantFilter][source]¶
Pairs that apply a constant equality constraint.
- to_join(cohorts: Dict[str, str])[source]¶
Return the resolved table name to join against.
Prefers the temporary table created during this run if it exists, otherwise falls back to the permanent schema-qualified table name.
- Parameters:
cohorts (dict[str, str]) – Mapping of task name -> created temp table name, as maintained by the active ExecutionStrategy.
- Returns:
Resolved table name string (used to build SA TableClause).
- Return type:
str
- class datakiln.extraction.models.FilterConfig(columns: List[str], reference: Dict[str, Any], cohort: str | None = None, join: List[Dict[str, Any]] | None = None)[source]¶
Bases:
objectFilter configuration mapping a task to a cohort or reference table.
- Parameters:
columns (list[str]) – Target columns to filter.
reference (ReferenceConfig, optional) – Target reference configuration.
cohort (str, optional) – Source cohort identifier.
join (list[JoinConfig], optional) – List of intermediate join chains.
- property is_sql_cohort: bool¶
Evaluates whether the filter targets an explicit SQL cohort file.
- property resolved_cohort: str | None¶
Return the resolved cohort table name.
- property cohort_columns: List[str]¶
Return the target columns to fetch from the cohort table.
- to_exists(strategy: ExecutionStrategy) Any[source]¶
Generate a SQLAlchemy EXISTS subquery for this filter configuration.
- Parameters:
strategy (ExecutionStrategy) – Execution strategy (used to resolve cohort aliases).
- Returns:
SA exists() expression.
- Return type:
Exists
- class datakiln.extraction.models.ExtractMeta(table: str, schema: str, filters: List[Dict | List[Dict]] = None, encoding: str = 'utf-8', chunk_size: int = 10000, **kwargs)[source]¶
Bases:
objectExtraction metadata.
- property full_name: str¶
Returns the full table name (schema.table).
- property temp_name: str¶
Returns the temporary table name used during extraction.
- property requires: Set[str]¶
Retrieve dependencies from filters.
- Returns:
Set of dependency task names.
- Return type:
set[str]
- classmethod from_json(path: Path) ExtractMeta[source]¶
Load extraction metadata from JSON.
- Parameters:
path (Path) – Path to the extraction metadata file.
- Returns:
Extraction metadata.
- Return type:
- class datakiln.extraction.models.ExtractGraph(tasks: Iterable[ExtractMeta], export: Set[ExtractMeta] | None = None)[source]¶
Bases:
DependencyGraphDirected Acyclic Graph for extraction tasks.
Dependencies that are not tasks (e.g. cohorts and tables that are not extracted) are kept, so they can be orchestrated by the executor.
- classmethod select(tasks: List[ExtractMeta], include: List[str] | None = None) ExtractGraph[source]¶
Graph of the tasks to export, and the tasks they need.
- Parameters:
tasks (list[ExtractMeta]) – Extraction tasks.
include (list[str]) – Names of the tasks to export. All by default.
- Raises:
ValueError – If any of the names is not a task.
- property tasks: Dict[str, ExtractMeta]¶
Extraction tasks by name.