Quibble

CLI reference

All commands are quibble <command> [options]. python -m quibble <command> is equivalent. Run quibble <command> --help for generated options, except doctor, which prints diagnostics. Square-bracketed options and placeholder names below are syntax illustrations, not commands to paste verbatim. See getting started for executable examples.

Settings can come from .env, environment variables, or flags; flags win. See Configuration.

quibble run

Run one backend on one dataset, or a sweep of many.

Single run:

quibble run --backend lightgbm --dataset credit-g --model-seed 0 --split-seed 0

Sweep:

quibble run --sweep --backends histgb,lightgbm --datasets credit-g,phoneme --cv 5x2 --model-seeds 0,1,2
quibble run --sweep --backends all --datasets breast_cancer       # 'all' excludes the internal 'weak'

Behaviour worth knowing:

• Finished runs are skipped. If their saved device, preprocessing mode, or CV scheme differs from the current settings, quibble prints a WARNING (they are still skipped; use --force or a different --out).
• On cuda, GPU-capable backends run one at a time even when --jobs > 1.
• A backend refusing a dataset, or crashing, produces a result file with refused or error; the sweep carries on.

quibble analyze

Confidence intervals, calibration, slices and disagreement for one dataset. Needs results from at least two backends for the disagreement part, and CV results for meaningful intervals.

quibble analyze --dataset credit-g [--backends a,b,c] [--model-seed 0] [--metric auc] [--min-slice-n 30]

Writes to <out>/<dataset>/analysis/. Backends are aligned on the splits they all completed; splits missing for any usable backend are dropped and listed; entirely unusable backends are excluded. See Analysis.

quibble swap

quibble swap --incumbent histgb --candidate lightgbm --dataset credit-g [--model-seed 0] [--thresholds limits.json]

Prints a table of checks and an overall verdict. Exit code: 0 = PASS, 1 = FAIL, 2 = INCONCLUSIVE (also 1 if the incumbent has no usable runs). See Swap checks.

quibble report

quibble report [--datasets a,b] [--swap histgb:lightgbm]... [--thresholds limits.json] [--report-dir DIR]

Writes <out>/report/report.md and figures. --swap can be repeated and is evaluated for every dataset. Without --datasets, every dataset folder with results is included. See Reports.

quibble compare

Quick single-split summary: mean metrics per backend, refusals and errors, and pairwise disagreement for one split id. Useful for a fast look; use analyze for anything statistical.

quibble compare --dataset credit-g [--split-seed 0]

quibble datasets

quibble datasets --list
quibble datasets --verify [--names credit-g,wine]

--list lists built-in, OpenML, and discovered local names without loading tables. Verification loads each selected dataset (downloading OpenML entries if needed), prints rows, features, task, categorical share, missing share, majority-class share and whether the OpenML name matches what the registry expects (NAME MISMATCH means the id is wrong). Writes datasets_meta.json in the current directory. Network errors are reported per dataset. Verification exits 1 if any dataset fails or a name mismatches, otherwise 0. Without --names, all registered and discovered local datasets are verified. See Datasets and Testing your own data.

Local names work with run, benchmark, analyze, swap, compare, and report:

python -m quibble benchmark --datasets payments --backends histgb,lightgbm --device cpu --smoke

Create data\payments\dataset.json and the input table first. No direct file-path/target CLI syntax is implemented; the manifest declares target/task and column rules.

quibble doctor

Prints Python, platform, CPU, RAM, git hash, installed package versions, and torch/CUDA/VRAM. Warns when free VRAM is low (another process may be holding the GPU).

quibble demo

quibble demo [--out results] [--cv 3x2] [--jobs 2] [--backends histgb,lightgbm,weak] [--force]

End-to-end demonstration on breast_cancer, wine, and diabetes_reg; forces QUIBBLE_DEVICE=cpu and model seed 0. The default backends run offline without PyTorch. --backends accepts a nonempty comma-separated list of registered names; optional backends need their packages in the chosen interpreter and may download weights.

python -m quibble demo --backends histgb,lightgbm,tabfm,kumo_tabular --jobs 1

The report checks swaps from HistGB to the other selected backends, or from the first selected backend if HistGB is absent. The weak-candidate sanity message only appears when both HistGB and Weak are selected. A single backend produces a report without swap checks.

Saved results, including errors, are skipped. --force reruns every selected cell, which lets you retry cached import failures after switching environments. Prefer a separate --out only when intentionally isolating a comparison. By default, demo uses the same QUIBBLE_OUT_DIR (or results) as run and report. Existing backend results in that directory also appear in its report. Dated snapshots preserve overwritten outputs and regenerated reports; <out>/report/history.md links to them.

quibble benchmark

Device-aware preset comparison with report generation:

python -m quibble benchmark --preset foundations --smoke
python -m quibble benchmark --preset foundations --device cuda --force
python -m quibble benchmark --backends histgb,autogluon --cv holdout --keep-models

Presets: trees selects HistGB/LightGBM/CatBoost; foundations (default) selects HistGB/LightGBM/TabFM/KumoTabular; all selects all public backends. --backends overrides the preset. --datasets defaults to wine, --cv to 3x2, and --model-seeds to 0. Device and parallelism follow .env unless overridden with --device, --jobs, or --threads. --out, --force, and --keep-models work as with run.

--smoke fixes holdout/seed 0, caps training/test rows at 128/32, and creates a unique <out>/smoke/<id>/ directory. Smoke reports interpret the bounded setup measurements but do not provide deployment recommendations or swap verdicts. Downloads and authentication may still be needed. Preset reports exclude incompatible cached backends, show participation and failure status, and include AutoGluon validation leaderboard links when available.

quibble web

Start the localhost experiment workbench for scenario planning, reviewed synthetic data, existing datasets, benchmark jobs, and reports:

quibble web --open
quibble web --login --device-code --port 8876

Flags: --port (default 8765), --open (open browser), --login (explicit Azure CLI login before startup), --device-code (use device-code login). Only 127.0.0.1 is bound. See the complete workbench guide for configuration, consent, generation assumptions, stored files, and local-only limits. The header's About link opens a separate page at /about, including the running version in its content and privacy/validation boundaries. All pages share navigation; Blog and Docs work locally, and GitHub links to the source repository. Browser saving continues directly to configuration; starting the experiment remains a separate explicit action.

quibble generate

Print an editable local example or generate a dataset from a reviewed specification:

quibble generate --example
quibble generate --spec reviewed-spec.json

Choose exactly one of --example or --spec. Generation is deterministic for the specification and software environment, never executes supplied code, and never overwrites a dataset folder. See the specification guide.

quibble auth

Check the optional planner's configuration or authenticate using Entra ID:

quibble auth
quibble auth --device-code
quibble auth --check

--check acquires an AAD token without login; --device-code selects explicit device-code login for Azure CLI credentials. Key mode checks only configuration presence. Neither establishes live provider access. See provider setup.

Exit codes

run and demo can return 0 after recording model-level errors; inspect records/report participation. Use benchmark when failed or missing requested fits must produce a non-zero aggregate status. Argparse usage errors also return 2.