CamZalewski
a1a621ddfd
KDB returns mixed-typed columns (most commonly: temporal nulls of varying precision -- 0Np, 0Nz, 0Nd -- interleaved) as q general lists. The faithful Arrow projection emits a `Union` of the per-element DataTypes. Polars and most DataFrame consumers reject `Union` outright, making such columns unusable downstream without a re-roundtrip dance. Add `HeterogeneousListMode::CoalesceTemporals` (off by default) on `ProjectionOptions`. When set, `project_heterogeneous_list` checks whether every arm is temporal (Timestamp, Date32, Date64, Time32, Time64, Duration) and, if so, casts each child to `Timestamp(Nanosecond, None)` via `arrow_cast::cast`, concatenates, and emits a flat array. Any non-temporal arm or cast error falls back to the existing Union path, so the flag is safe to enable globally. Plumbed through the Python `DecodeOptions` API as `with_coalesce_temporals(bool)` with matching getter and pyi stub. The default stays `False`; users opt in when they know the consumer (Polars) can't handle Union and accept the lossy precision promotion. Tests cover (a) default-Union, (b) all-temporal coalesce, and (c) non-temporal fallback to Union.
27 lines
905 B
Rust
27 lines
905 B
Rust
//! Arrow interop layer for qroissant.
|
|
//!
|
|
//! Converts decoded q `Value` trees (from `qroissant-core`) into Apache Arrow
|
|
//! arrays and record batches. PyO3 and PyCapsule handling live in
|
|
//! `qroissant-python`; this crate is intentionally free of Python dependencies.
|
|
|
|
pub mod error;
|
|
pub mod ingestion;
|
|
pub mod metadata;
|
|
pub mod options;
|
|
pub mod projection;
|
|
|
|
pub use error::IngestionError;
|
|
pub use ingestion::ingest_array;
|
|
pub use ingestion::ingest_record_batch;
|
|
pub use ingestion::ingest_record_batch_reader;
|
|
pub use options::HeterogeneousListMode;
|
|
pub use options::ListProjection;
|
|
pub use options::ProjectionOptions;
|
|
pub use options::StringProjection;
|
|
pub use options::SymbolProjection;
|
|
pub use options::UnionMode;
|
|
pub use projection::ArrayExport;
|
|
pub use projection::BatchExport;
|
|
pub use projection::project;
|
|
pub use projection::project_table;
|
|
pub use qroissant_core::HEADER_LEN as QIPC_HEADER_LEN;
|