datakiln.metadata.table module¶
Table metadata.
- class datakiln.metadata.table.Base(**kwargs: Any)[source]¶
Bases:
DeclarativeBaseBase ORM Class.
- metadata: ClassVar[MetaData] = MetaData()¶
Refers to the
_schema.MetaDatacollection that will be used for new_schema.Tableobjects.See also
orm_declarative_metadata
- registry: ClassVar[registry] = <sqlalchemy.orm.decl_api.registry object>¶
Refers to the
_orm.registryin use where new_orm.Mapperobjects will be associated.
- class datakiln.metadata.table.Artifact(**kwargs)[source]¶
Bases:
BaseArtifact metadata.
- path_: Mapped[str]¶
- name_: Mapped[str]¶
- checksum_: Mapped[str]¶
- class datakiln.metadata.table.Object(**kwargs)[source]¶
Bases:
BaseObject built in the database, from the metadata of a table or from a script. The metadata of the kinds of object are in their own tables.
- name_: Mapped[str]¶
- kind_: Mapped[str]¶
- checksum_: Mapped[str]¶
- property schema¶
Schema the object is in, if it is in one.
- property drops¶
Statements that drop the object.
- class datakiln.metadata.table.Metadata(**kwargs)[source]¶
Bases:
ObjectTable built from its metadata.
- name_: Mapped[str]¶
- property schema¶
Schema the object is in, if it is in one.
- property drops¶
Statements that drop the object.
- checksum_: Mapped[str]¶
- kind_: Mapped[str]¶
- class datakiln.metadata.table.Script(**kwargs)[source]¶
Bases:
ObjectDerive script, which creates one object.
- name_: Mapped[str]¶
- drop_: Mapped[str]¶
- property schema¶
Schema the object is in, if it is in one.
- property drops¶
Statements that drop the object.
- checksum_: Mapped[str]¶
- kind_: Mapped[str]¶
- class datakiln.metadata.table.TableBase(name, schema, columns, na_values=None, encoding='utf-8', delimiter=',', checksum=None, artifact=None)[source]¶
Bases:
objectBase metadata for table.
- property columns¶
Metadata for the columns, in order.
- Return type:
list[ColumnMeta]
- create(session)[source]¶
Create statement.
- Parameters:
session (datakiln.metadata.database.Session) – Database Session.
- Returns:
None
- class datakiln.metadata.table.StageMeta(**kwargs)[source]¶
Bases:
TableBaseMetadata for table.
- create(session)[source]¶
Create statement.
- Parameters:
session (datakiln.metadata.database.Session) – Database Session.
- Returns:
None
- load(session)[source]¶
Stage data.
- Parameters:
session (datakiln.metadata.database.Session) – Database Session.
- Returns:
None
- class datakiln.metadata.table.TableMeta(constraints=None, indexes=None, unpivot=None, location=None, external=False, **kwargs)[source]¶
Bases:
TableBaseMetadata for table.
- property full_name¶
Returns the full table name (schema.table), as the database folds it, since it is created without quotes.
- property requires¶
Objects (schema.table) referenced by foreign keys.
- property stage_meta¶
Metadata for stage table.
- Returns:
Table metadata.
- Return type:
- Returns:
None
- create(session)[source]¶
Create statement.
- Parameters:
session (datakiln.metadata.database.Session) – Database Session.
- Returns:
None
- unpivot(session)[source]¶
Unpivot a table.
- Parameters:
session (datakiln.metadata.database.Session) – Database Session.
- Returns:
None
- resolve_columns(session)[source]¶
Resolve NULL values and apply description/comment.
- Parameters:
session (datakiln.metadata.database.Session) – Database Session.
- Returns:
None
- enforce_fks(session)[source]¶
Enforce foreign key constraints.
- Parameters:
session (datakiln.metadata.database.Session) – Database Session.
- Returns:
None
- create_indexes(conn)[source]¶
Create table indexes.
- Parameters:
conn (sqlalchemy.Engine) – SQLAlchemy connection or session.
- Returns:
None
- create_uniques(conn)[source]¶
Create table unique indexes.
- Parameters:
conn (sqlalchemy.Engine) – SQLAlchemy connection or session.
- Returns:
None
- enforce_pk(conn)[source]¶
Enforce primary key.
- Parameters:
conn (sqlalchemy.Engine) – SQLAlchemy connection or session.
- Returns:
None
- load(session, batch_size=1000000)[source]¶
Load data from stage.
- Parameters:
session (datakiln.metadata.database.Session) – Database Session.
- Returns:
None
- create_metadata(session)[source]¶
- Parameters:
session (datakiln.metadata.database.Session) – Database Session.
- pull()[source]¶
Fetch the data file, if it is not in the folder of the data files. It is pulled from the default DVC remote, if there is one. Otherwise, a file that DVC imported from a URL (
dvc import-url) is downloaded from there.- Returns:
None
- record(session)[source]¶
Record of the table in the metadata, if it was built.
- Parameters:
session (datakiln.metadata.database.Session) – Database Session.
- Return type:
- needs_update(session)[source]¶
Whether the table was not built, or its metadata or file changed since it was.
- Parameters:
session (datakiln.metadata.database.Session) – Database Session.
- Return type:
bool
- classmethod order_columns(columns, order)[source]¶
Sort table columns in the order presented in a CSV file to be used in a COPY statement.
- Parameters:
columns (list[ColumnMeta]) – Columns to be ordered.
order (list[str]) – Column names in the order they are to be returned.
- Returns:
Ordered list of columns.
- Return type:
list[ColumnMeta]
- classmethod from_json(meta_path, data_files=None, external=False)[source]¶
Load metadata.
- Parameters:
meta_path (str) – Path to the metadata
data_files (datakiln.metadata.files.DataFiles) – Data files of the tables, unless it is external.
external (bool) – Whether another repo builds the table.
- Returns:
Table metadata.
- Return type: