datakiln.metadata.graph module

Dependency graphs for metadata objects.

class datakiln.metadata.graph.Unresolved(full_name: str)[source]

Bases: object

Name that nodes require, which is not a node. Two with the same name are the same dependency.

full_name: str
class datakiln.metadata.graph.DependencyGraph(nodes: List, keep_unresolved: bool = False, aliases: Dict[str, Set] | None = None)[source]

Bases: object

Directed Acyclic Graph for nodes that depend on one another.

Nodes list the names of the nodes they requires, which the graph turns into the nodes themselves. Required names that are not nodes (e.g. tables loaded elsewhere) are ignored, unless keep_unresolved is set, in which case they become dependencies of the nodes that are Unresolved objects, which are not part of the nodes.

Nodes have a full_name, which tells them apart in errors.

nodes: Dict[str, object]
unresolved: Set[Unresolved]
dependencies(node: object) → Set[source]

Nodes that a node needs directly.

Parameters:

node (object) – Node of the graph.

Returns:

The nodes it needs, which include the dependencies that are not nodes if they are kept.

Return type:

set[object]

property topological_sorter: TopologicalSorter

Returns a prepared TopologicalSorter for the graph, which gives the nodes that can run as the ones they need are done.

Raises:

ValueError – If the dependencies contain a cycle.

find(names: Iterable[str]) → Set[source]

Nodes by name.

Parameters:

names (Iterable[str]) – Names of nodes of the graph.

Returns:

The nodes.

Return type:

set[object]

Raises:

ValueError – If any of the names is not a node.

requirements(nodes: Iterable) → Set[source]

The specified nodes, with the nodes they need, directly or indirectly. Names that are not nodes are left out.

Parameters:

nodes (Iterable[object]) – Nodes of the graph.

Returns:

The nodes and what they need.

Return type:

set[object]

affected(nodes: Iterable) → Set[source]

Nodes that need any of the specified nodes, directly or indirectly.

Parameters:

nodes (Iterable[object]) – Nodes of the graph.

Returns:

The nodes that need them.

Return type:

set[object]

sort() → List[source]

Perform topological sort to determine execution order.

Returns:

Nodes ordered so that dependencies come first. Dependencies that are not nodes are not returned.

Return type:

list

Raises:

ValueError – If the dependencies contain a cycle.

class datakiln.metadata.graph.DatabaseGraph(tables: List, scripts: List[ScriptMeta], external: List | None = None)[source]

Bases: DependencyGraph

Dependency graph for the tables (metadata) and the scripts (derive) of a database.

property schemas: Set[str]

Schemas of the tables and scripts, in lower case, other than the ones of the external objects. A function is in none.

property tables: List

Tables to build, ordered so that referenced tables come first.

property scripts: List[ScriptMeta]

Scripts to build, ordered so that the scripts creating the objects they use come first.