datakiln.metadata.database module¶
Database metadata.
- class datakiln.metadata.database.DatabaseMeta(directory, name)[source]¶
Bases:
objectDatabase metadata.
- property tables¶
Retrieve database tables, in the order of their files.
- Returns:
Table metadata.
- Return type:
- property scripts¶
Retrieve the derive scripts.
- Returns:
Script metadata.
- Return type:
- property dialect¶
SQL dialect, which is set in the environment of the database.
- property external¶
Retrieve the tables and scripts of the repos in the dependencies folder, which those repos build. They have no data files here.
- Returns:
Tables and scripts, marked as external.
- Return type:
list
- property graph¶
Dependency graph of the tables (metadata) and scripts (derive), which gives them in the order they depend on one another. The objects of the dependencies folder are in it, so they can be used, but they are not built.
- Returns:
Dependency graph.
- Return type:
- grant(session, users=None)[source]¶
Grants read on the objects of the schemas of the tables and scripts, which excludes the stage and the metadata, to the database role, and the role to the users.
- Parameters:
session (datakiln.metadata.database.Session) – Database Session.
users (list[str]) – Users to give the role to.
- create_role(session)[source]¶
Create database role.
- Parameters:
session (datakiln.metadata.database.Session) – Database Session.
- execute_directory(directory, session, **kwargs)[source]¶
Executes SQL scripts in a given directory.
- Parameters:
directory (str) – Path for metadata directory.
session (datakiln.metadata.database.Session) – Database Session.
ignore (str) – List of files to ignore.
- load_stale(session)[source]¶
Load what was built before and is no longer in the repo, because its metadata or script was removed or ignored.
- Parameters:
session (datakiln.metadata.database.Session) – Database Session.
- Raises:
ValueError – If what is in the repo requires any of them.
- forget_dependents(session, objects)[source]¶
Deletes the records of the tables and scripts that need any of the objects. Rebuilding an object drops what is built on it, the foreign keys of the tables and the views of the scripts, so the next run has to build them again.
- Parameters:
session (datakiln.metadata.database.Session) – Database Session.
objects (list[datakiln.metadata.table.TableMeta]) – Tables or scripts.
- execute_derive(session, force=False, record=True, include=None, **kwargs)[source]¶
Executes the derive scripts, each one after the scripts that create the objects it uses. Only the scripts that changed, or that were not built since what they use was rebuilt, are executed.
- Parameters:
session (datakiln.metadata.database.Session) – Database Session.
force (bool) – Execute all scripts, even if they are up to date.
record (bool) – Whether to save the scripts executed to the metadata, which is what tells later runs they are up to date.
include (set[datakiln.metadata.script.ScriptMeta]) – Scripts to execute, if not all of them.
- prune(session, drop=False)[source]¶
Drops what was built before and is no longer in the repo, because its metadata or script was removed or ignored, and the schemas that are left empty.
- Parameters:
session (datakiln.metadata.database.Session) – Database Session.
drop (bool) – Whether to drop them. If not, they are only listed.
- create_metadata(session)[source]¶
- Parameters:
session (datakiln.metadata.database.Session) – Database Session.
- datakiln.metadata.database.before_cursor_execute(conn, cursor, statement, parameters, context, executemany)[source]¶
Format SQL query before execution.
- datakiln.metadata.database.set_oracle_nls(dbapi_connection, _connection_record)[source]¶
Set standard NLS date/timestamp formats on Oracle connections.
- class datakiln.metadata.database.Session(**kwargs)[source]¶
Bases:
SessionManages persistence operations for ORM-mapped objects.
- execute(statement, *args, buffer_size=1, max_statements=1, validate=True, **kwargs)[source]¶
Executes a SQL statement construct and returns a CursorResult.
- Parameters:
statement (str) – Statement to be executed.
buffer_size (int) – Buffer size. The statements will only be sent to the database if the number of statements in the buffer exceeds the value of this parameter.
max_statements (int) – Maximum number of statements to allow per execute call.
**kwargs (dict) – SQLAlchemy execute parameters.
- classmethod create(user, host, port, database, dialect, password=None, isolation_level='READ COMMITTED')[source]¶
Connect to database using SQLAlchemy.
- Parameters:
user (str) – Name of the database user.
host (str) – Hostname of the database server.
port (int) – Network port of the database server.
database (str) – Name of the database.
dialect (str) – Dialect of the database.
password (str) – User’s password.
isolation_level (str) – Transaction isolation level.
- Returns:
Database session.
- Return type: