Why Quibble?
Two models can get the same score and make different mistakes. Quibble shows which predictions change, who gets them right, and what each method costs, so you can judge whether a change is worth making.
What to do next
Open the workbench, choose a ready-made dataset, review its rows, then run a smoke comparison. No planning provider is needed. Or design your own synthetic dataset to explore a specific question.
This is the public documentation site. Experiments run in your own local Quibble environment.
What Quibble does
Quibble compares machine-learning methods for tabular classification and regression on the same held-out rows. Reports show prediction quality, uncertainty, disagreements, calibration, slices, execution costs, errors, and refusals. Configurable swap checks produce PASS, FAIL, or INCONCLUSIVE evidence.
It is a comparison and verification workbench, not an automatic model tuner, deployment platform, or guarantee that a model is suitable for a real decision.
Three steps, two ways to choose data
- Choose data. Review an existing dataset and continue, or design a synthetic table, preview it, confirm review, then Save and continue. Four bundled grocery and retail clothing examples work without a planning provider. Classification includes binary and multiclass datasets; new synthetic designs support binary classification or regression.
- Configure & run. Choose methods, device policy, and a smoke or full comparison. Smoke is a bounded setup check, not benchmark evidence. Full uses the configured evaluation splits.
- Results. Inspect the report and logs. Past experiments reopens saved evidence without creating another run.
Creating a variant leaves its original unchanged. Save and continue uses the saved variant; Discard changes and continue uses the original rows. Both open experiment configuration, not an automatic run.
Methods and compute
Adapters include HistGB, LightGBM, CatBoost, AutoGluon, TabPFN, TabFM, and KumoTabular. Availability depends on installed packages, model access, task support, and resources. Some methods download pretrained weights; package and model licenses can differ from Quibble's MIT license.
CPU pins selected methods to CPU. CUDA requires an available GPU and excludes incompatible methods. Auto resolves once to CUDA or CPU. There is no silent CPU fallback after GPU failure. Data preparation, metrics, and reporting still use CPU.
What the evidence means
Synthetic experiments test a recipe's assumptions, not real business performance or causal effects. Fresh rows reduce direct reuse of familiar benchmark examples but do not prove an unseen task or eliminate pretraining exposure. Repeat across generation seeds and confirm useful findings on representative real data.
A successful smoke run does not validate every task, dataset, library version, or hardware configuration. Timings and memory measurements depend on the machine. Review coverage, refusals, uncertainty, and thresholds before interpreting a swap verdict.
Local operation and provider calls
The workbench binds to localhost and is intended for one user. It is not a hosted or multi-user service. Keep the server running while experiments execute; closing a tab does not cancel a run.
Existing dataset rows stay local in the evaluation pipeline. Optional scenario planning sends your approved description and generation controls to your configured provider. Do not enter sensitive information. Invalid output can trigger one correction request; charges can apply to both requests. Credentials remain in the server environment.
Reports, logs, source metadata, and disputed-row files can contain local paths or data-derived information. Review them before sharing. Original data, model downloads, and provider services have their own terms and data-handling requirements.
Documentation and project status
Version 0.0.1 · Pre-release · MIT license
Source: ushakrishnan/quibble on GitHub. Docs and Blog are available through this site. Start with the documentation for installation, the workbench guide, CLI reference, troubleshooting, and validation status.
Read the Predicting from Tables blog series: Same Accuracy, Different Mistakes and Making Prediction Comparisons Worth Trusting. Articles and diagrams open locally; no public hosting is required.
Quibble is an experimental pre-release. The package name is quibble-ml; the Python import and command are quibble. See the repository's MIT license and issue tracker. Optional libraries, datasets, and checkpoints have their own terms.