datakiln.metadata.table module

Table metadata.

class datakiln.metadata.table.Base(**kwargs: Any)[source]

Bases: DeclarativeBase

Base ORM Class.

metadata: ClassVar[MetaData] = MetaData()

Refers to the _schema.MetaData collection that will be used for new _schema.Table objects.

See also

orm_declarative_metadata

registry: ClassVar[registry] = <sqlalchemy.orm.decl_api.registry object>

Refers to the _orm.registry in use where new _orm.Mapper objects will be associated.

class datakiln.metadata.table.Artifact(**kwargs)[source]

Bases: Base

Artifact metadata.

path_: Mapped[str]
name_: Mapped[str]
checksum_: Mapped[str]
table: Mapped[Metadata]
class datakiln.metadata.table.Object(**kwargs)[source]

Bases: Base

Object built in the database, from the metadata of a table or from a script. The metadata of the kinds of object are in their own tables.

name_: Mapped[str]
kind_: Mapped[str]
checksum_: Mapped[str]
property schema

Schema the object is in, if it is in one.

property drops

Statements that drop the object.

class datakiln.metadata.table.Metadata(**kwargs)[source]

Bases: Object

Table built from its metadata.

name_: Mapped[str]
artifact: Mapped[Artifact]
property schema

Schema the object is in, if it is in one.

property drops

Statements that drop the object.

checksum_: Mapped[str]
kind_: Mapped[str]
class datakiln.metadata.table.Script(**kwargs)[source]

Bases: Object

Derive script, which creates one object.

name_: Mapped[str]
drop_: Mapped[str]
property schema

Schema the object is in, if it is in one.

property drops

Statements that drop the object.

checksum_: Mapped[str]
kind_: Mapped[str]
class datakiln.metadata.table.TableBase(name, schema, columns, na_values=None, encoding='utf-8', delimiter=',', checksum=None, artifact=None)[source]

Bases: object

Base metadata for table.

property columns

Metadata for the columns, in order.

Return type:

list[ColumnMeta]

create(session)[source]

Create statement.

Parameters:

session (datakiln.metadata.database.Session) – Database Session.

Returns:

None

class datakiln.metadata.table.StageMeta(**kwargs)[source]

Bases: TableBase

Metadata for table.

create(session)[source]

Create statement.

Parameters:

session (datakiln.metadata.database.Session) – Database Session.

Returns:

None

load(session)[source]

Stage data.

Parameters:

session (datakiln.metadata.database.Session) – Database Session.

Returns:

None

class datakiln.metadata.table.TableMeta(constraints=None, indexes=None, unpivot=None, location=None, external=False, **kwargs)[source]

Bases: TableBase

Metadata for table.

property full_name

Returns the full table name (schema.table), as the database folds it, since it is created without quotes.

property requires

Objects (schema.table) referenced by foreign keys.

property stage_meta

Metadata for stage table.

Returns:

Table metadata.

Return type:

TableMeta

Returns:

None

create(session)[source]

Create statement.

Parameters:

session (datakiln.metadata.database.Session) – Database Session.

Returns:

None

unpivot(session)[source]

Unpivot a table.

Parameters:

session (datakiln.metadata.database.Session) – Database Session.

Returns:

None

resolve_columns(session)[source]

Resolve NULL values and apply description/comment.

Parameters:

session (datakiln.metadata.database.Session) – Database Session.

Returns:

None

enforce_fks(session)[source]

Enforce foreign key constraints.

Parameters:

session (datakiln.metadata.database.Session) – Database Session.

Returns:

None

create_indexes(conn)[source]

Create table indexes.

Parameters:

conn (sqlalchemy.Engine) – SQLAlchemy connection or session.

Returns:

None

create_uniques(conn)[source]

Create table unique indexes.

Parameters:

conn (sqlalchemy.Engine) – SQLAlchemy connection or session.

Returns:

None

enforce_pk(conn)[source]

Enforce primary key.

Parameters:

conn (sqlalchemy.Engine) – SQLAlchemy connection or session.

Returns:

None

load(session, batch_size=1000000)[source]

Load data from stage.

Parameters:

session (datakiln.metadata.database.Session) – Database Session.

Returns:

None

create_metadata(session)[source]
Parameters:

session (datakiln.metadata.database.Session) – Database Session.

pull()[source]

Fetch the data file, if it is not in the folder of the data files. It is pulled from the default DVC remote, if there is one. Otherwise, a file that DVC imported from a URL (dvc import-url) is downloaded from there.

Returns:

None

record(session)[source]

Record of the table in the metadata, if it was built.

Parameters:

session (datakiln.metadata.database.Session) – Database Session.

Return type:

datakiln.metadata.table.Metadata

needs_update(session)[source]

Whether the table was not built, or its metadata or file changed since it was.

Parameters:

session (datakiln.metadata.database.Session) – Database Session.

Return type:

bool

classmethod order_columns(columns, order)[source]

Sort table columns in the order presented in a CSV file to be used in a COPY statement.

Parameters:
  • columns (list[ColumnMeta]) – Columns to be ordered.

  • order (list[str]) – Column names in the order they are to be returned.

Returns:

Ordered list of columns.

Return type:

list[ColumnMeta]

classmethod from_json(meta_path, data_files=None, external=False)[source]

Load metadata.

Parameters:
  • meta_path (str) – Path to the metadata

  • data_files (datakiln.metadata.files.DataFiles) – Data files of the tables, unless it is external.

  • external (bool) – Whether another repo builds the table.

Returns:

Table metadata.

Return type:

TableMeta