datakiln.extraction.exporters module

Data and metadata export logic for the extraction pipeline.

class datakiln.extraction.exporters.MetadataExtractor[source]

Bases: object

Class to handle metadata extraction and generation.

static get_metadata(session: Session, schema: str, tables: List[str]) → MetaData[source]

Generate column metadata from DBMS.

static generate_column_meta(column: Column, dialect: str = None, compile_type: bool = True) → Dict[source]

Generate column metadata from DBMS.

static generate_table_meta(meta_dir: Path, table: Table, dialect: str = None) → Dict[source]

Generate table metadata from DBMS.

class datakiln.extraction.exporters.HashingFileWriter(fp)[source]

Bases: object

A file-like writer that calculates the MD5 hash on-the-fly.

write(b)[source]

Write bytes to the file and update the hash.

flush()[source]

Flush the file buffer.

close()[source]

Close the file.

class datakiln.extraction.exporters.DataExporter[source]

Bases: object

Class to handle data export logic.

static stream_to_csv(session: Session, query: Select, export_path: Path, table_name: str = 'unknown', insert_to: Table | None = None, chunk_size: int = 10000, abort: Event | None = None) → str[source]

Streams a single SQL query into a compressed CSV.

Returns the MD5 checksum of the generated compressed CSV file.