114 lines
4.3 KiB
Markdown
114 lines
4.3 KiB
Markdown
# Building qroissant wheels
|
|
|
|
This project ships as a [maturin](https://www.maturin.rs/) / PyO3 mixed
|
|
Rust+Python package. The native extension uses `#![feature(portable_simd)]` so
|
|
**nightly Rust is required**, pinned via `rust-toolchain.toml`.
|
|
|
|
All builds run inside a Docker image (`scripts/Dockerfile.build`) so the host
|
|
needs only Docker and Python 3 — no Rust, no MSVC SDK, no mingw to install.
|
|
|
|
## Prerequisites
|
|
|
|
- Docker (tested with 29.x)
|
|
- Python 3.11+ on the host, **only** if you want to install/test the produced
|
|
Linux wheel locally via `scripts/build.sh check`.
|
|
|
|
## One-time setup
|
|
|
|
```sh
|
|
scripts/build.sh image
|
|
```
|
|
|
|
Builds `qroissant-build:latest`. Pulls `ghcr.io/rust-cross/cargo-xwin`,
|
|
installs the nightly toolchain + `x86_64-pc-windows-msvc` target + maturin.
|
|
~5 min on a cold cache, then it's a no-op.
|
|
|
|
## Build commands
|
|
|
|
```sh
|
|
scripts/build.sh linux # -> dist-linux/qroissant-*-manylinux_*_x86_64.whl
|
|
scripts/build.sh windows # -> dist-windows/qroissant-*-win_amd64.whl
|
|
scripts/build.sh all # image + linux + windows
|
|
scripts/build.sh check # install latest linux wheel into .venv, import-smoke,
|
|
# then `zipfile -l` the windows wheel
|
|
scripts/build.sh clean # rm dist-linux dist-windows
|
|
scripts/build.sh clean-cache # also drop the cargo/target/xwin Docker volumes
|
|
```
|
|
|
|
The wheels target **abi3-py311**, so a single artifact per platform covers
|
|
CPython 3.11, 3.12, 3.13, and onward.
|
|
|
|
## What lives where
|
|
|
|
| Artifact | Path |
|
|
| --- | --- |
|
|
| Build image definition | `scripts/Dockerfile.build` |
|
|
| Wrapper script | `scripts/build.sh` |
|
|
| Toolchain pin | `rust-toolchain.toml` |
|
|
| Linux wheels | `dist-linux/` (gitignored) |
|
|
| Windows wheels | `dist-windows/` (gitignored) |
|
|
| Cargo registry cache | Docker volume `qroissant-cargo-registry` |
|
|
| Cargo target dir | Docker volume `qroissant-target` |
|
|
| cargo-xwin MSVC cache | Docker volume `qroissant-xwin` |
|
|
|
|
The three Docker volumes persist between runs so reruns only rebuild changed
|
|
crates; nuke them with `scripts/build.sh clean-cache` if anything looks stale.
|
|
|
|
## Debug symbols on Windows
|
|
|
|
The release profile sets `debug = "line-tables-only"`, so a `_native.pdb` is
|
|
produced alongside the wheel. `scripts/build.sh windows` copies it into
|
|
`dist-windows/`. To get resolved frames in a Rust panic backtrace on a Windows
|
|
machine:
|
|
|
|
1. `pip install qroissant-*-win_amd64.whl`
|
|
2. Locate the installed extension, e.g.
|
|
`Lib\site-packages\qroissant\_native.pyd`
|
|
3. Drop `_native.pdb` next to it (same directory, same basename).
|
|
|
|
Symbol info is line-tables-only, so frames resolve to `file:line` but local
|
|
variables and types aren't included. That's enough for backtraces and is
|
|
~30 MB; full debug info would be much larger.
|
|
|
|
## How Windows cross-compile works
|
|
|
|
The Windows path uses [cargo-xwin](https://github.com/rust-cross/cargo-xwin),
|
|
which downloads the Microsoft CRT + Windows SDK headers (cached in the xwin
|
|
volume; license auto-accepted via `XWIN_ACCEPT_LICENSE=1`). PyO3's
|
|
`generate-import-lib` feature — enabled in
|
|
`crates/qroissant-python/Cargo.toml` — lets us produce the `python3.dll`
|
|
import library at build time, so no Windows Python install is required on the
|
|
host.
|
|
|
|
## Quick wheel sanity checks
|
|
|
|
```sh
|
|
# Listing
|
|
python3 -m zipfile -l dist-windows/qroissant-*-win_amd64.whl
|
|
|
|
# Validate the dist-info
|
|
pipx run twine check dist-linux/*.whl dist-windows/*.whl
|
|
```
|
|
|
|
## Iterating on Python code
|
|
|
|
For Python-side changes (no Rust touched), the fastest loop is still
|
|
`maturin develop` against a local rustup install. The Docker flow above is
|
|
optimized for producing wheels, not for inner-loop iteration. If you have a
|
|
nightly rustup toolchain on the host, you can do:
|
|
|
|
```sh
|
|
python3 -m venv .venv && .venv/bin/pip install 'maturin>=1.8,<2.0'
|
|
.venv/bin/maturin develop --release
|
|
.venv/bin/python -c 'import qroissant'
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
- **"manylinux_2_34" wheel won't install on old distros** — the image base is
|
|
Debian 13 (glibc 2.38). For broader compatibility, build inside a
|
|
manylinux2014 image or run `auditwheel repair` against the produced wheel.
|
|
- **"failed to generate python3.dll import library"** — means `llvm-dlltool`
|
|
is missing inside the image. Rebuild it with `scripts/build.sh image`.
|
|
- **xwin download is slow / fails** — Microsoft's CDN occasionally rate-limits.
|
|
Drop the cache (`docker volume rm qroissant-xwin`) and retry.
|