Quibble

Datasets and data locations

Quibble supports three scikit-learn built-in datasets, fourteen OpenML entries, four bundled synthetic examples, and named local CSV/Parquet datasets. Public source entries are defined in the dataset loader. Named local CSV/Parquet datasets are also supported; see Testing your own data for folder/schema requirements and complete commands. Registration means a name is supported, not that it has been downloaded, verified, or evaluated.

What is stored where, and when

Paths below assume the local Windows .venv, default cache root .cache, and results root results. Cache paths resolve from the project root; relative results paths resolve from the working directory. Run commands from Quibble's root. Settings can override these locations; see configuration.

Installing requirements installs software and bundled sample data. It does not automatically fetch every OpenML table or every pretrained checkpoint. Downloaded weights are distinct from fitted-model artifacts and from the input datasets. Package-manager caches (pip/uv) are outside this runtime-cache policy.

Built in (offline)

These are official scikit-learn datasets, not files manually added by Quibble. They need no network after scikit-learn is installed and are used by the default demo and tests. The benchmark presets default to wine; select other datasets with --datasets. Wine contains chemical measurements and cultivar labels. It is not the separate OpenML wine_quality entry, and diabetes_reg is not the OpenML classification entry diabetes.

For a Windows checkout, the installed data directory is:

<checkout>\.venv\Lib\site-packages\sklearn\datasets\data\

For example, the original Wine table is wine_data.csv in that directory. Descriptions, provenance, and citation information are in the adjacent descr directory: breast_cancer.rst, wine_data.rst, and diabetes.rst. Source filenames describe the installed scikit-learn layout, not a stable Quibble storage API. Use the loaders rather than parsing the installed CSVs: headers and target layouts differ, and load_diabetes applies its default feature scaling.

Locate the installed directories and inspect the complete Wine table without downloading:

python -c "from pathlib import Path; import sklearn.datasets as d; p=Path(d.__file__).parent; print('Data:', p/'data'); print('Descriptions:', p/'descr')"
python -c "from sklearn.datasets import load_wine; d=load_wine(as_frame=True); print(d.frame.to_string(index=False))"

Bundled synthetic examples

Each package folder contains data.csv, dataset.json, and generation.json. These are already-created rows from Quibble's bounded generator, not live provider output. CLI commands and the browser can reuse them offline. A local dataset folder with the same name takes precedence; it is never overwritten or silently replaced by the bundled copy. Changing a recipe requires a newly named, reviewed variant. See the workbench guide for generation assumptions and limits. The examples are fictional, not evidence of real grocery or clothing performance.

The browser uses one task-filtered selector with groups for built-in benchmarks, bundled examples, and saved datasets. Classification includes binary and multiclass; regression is separate. All entries are usable existing data, not readiness levels.

OpenML registry

Defined as name: (OpenML id, task, expected OpenML name) in the loader. The IDs are the requested source identifiers, not a claim that the entries were validated.

[!WARNING] The registry has not been comprehensively validated against current OpenML metadata. Run quibble datasets --verify --names ... before using an OpenML entry. NAME MISMATCH means the returned name differs from the registry; ERROR means loading failed. Inspect the output as well as the exit code: verification reports failures per dataset and returns 1 for any load failure or name mismatch.

Datasets are downloaded on first use with scikit-learn's fetch_openml and cached in QUIBBLE_DATA_HOME. When blank, that follows QUIBBLE_CACHE_DIR/openml, normally .cache\openml. The current .env may explicitly set the same location. Scikit-learn manages nested metadata and data files; do not assume there will be one friendly <name>.csv per dataset. Quibble does not redistribute these source tables. Only selected OpenML entries are loaded; registering fourteen entries does not download all fourteen.

Actual download sources, destinations, and triggers

The following paths are relative to the source checkout's root for the default local setup. Cache overrides in .env change the destinations, not the upstream model identity.

Bundled datasets: installed with scikit-learn

These pages identify the original dataset and its references. Quibble obtains the bundled copy from the installed scikit-learn package, not by visiting those pages during a run. The installer gets that package from your configured package index, normally PyPI's scikit-learn distribution, or your trusted mirror.

OpenML tables: downloaded when a selected entry is needed

Every OpenML entry above has its own source link. Quibble calls fetch_openml(data_id=...); scikit-learn requests metadata from OpenML, then follows the returned data-file URL. For example, adult requests dataset 1590, whose metadata is available at the OpenML API. The metadata's file URL determines the actual download; it is not a Hugging Face model.

All fourteen entries use .cache\openml\ by default, with scikit-learn-managed subdirectories. The trigger is the first run, smoke run, verification, or analysis/report that needs that entry and lacks the cached data. A later request reuses available cached files. datasets --verify without --names requests all registered entries, so it can download all fourteen, plus verify discovered local datasets. Source links identify requested IDs; verification is still required to confirm they match the intended dataset names.

Foundation checkpoints: downloaded when the selected model loads

Sources below were checked against the installed tabfm 1.0.1, structured-data-models 0.2.1, and tabpfn 9.2.0 loaders. Package upgrades, task selection, or upstream version settings can change checkpoint filenames or revisions.

For Kumo's default small size, the installed loader requests small/classifier.pt or small/regressor.pt, plus config.json. Changing size selects the corresponding medium/ or large/ files. TabFM selects a task-specific subfolder from its repository; Quibble uses its PyTorch loader, not Google's separate JAX model repository.

The installed TabPFN default is version 3.5, with default checkpoint tabpfn-v3.5-20260909.safetensors; multiclass handling may select a variant. TabPFN supports other model versions and may use different repositories when configured. Its older v2 loader also contains a legacy Google Cloud Storage fallback at https://storage.googleapis.com/tabpfn-v2-model-files/05152025; that is not the documented default v3.5 source.

Hugging Face model directories contain blobs, refs, and snapshots\<revision> rather than a single flat weights file. The hub resolves model-page URLs to download endpoints and may use redirects or content-addressed transfer services. The repository links above identify the upstream model; they are not a promise of one permanent CDN URL. Cached weights can be reused, but the library may still check remote metadata unless configured for offline use. A newly selected task, size, revision, or missing file can require another download. Availability, model licences, and authentication are controlled by the upstream providers.

HistGB, LightGBM, CatBoost, and Weak learn local models and do not request these foundation checkpoints. AutoGluon trains local model families; if a chosen preset/model family uses pretrained dependencies, its own library can request additional assets. Quibble does not maintain a fixed exhaustive checkpoint URL list for every AutoGluon preset.

What a first invocation actually downloads

# Only uses bundled datasets and tree models; no OpenML tables or foundation weights.
python -m quibble demo

# Loads bundled Wine; downloads uncached TabFM/Kumo weights, then writes isolated smoke evidence.
python -m quibble benchmark --preset foundations --smoke

# Requests only these OpenML tables; does not train models or load foundation checkpoints.
python -m quibble datasets --verify --names credit-g,adult

In the second example, checkpoints stay in .cache, while metrics/predictions/report go to results\smoke\<unique-id>\. Full benchmark results go to results\<dataset>\. Existing successful or failed run records may cause fits to be skipped; --force retries the fit but does not inherently force a new weight download.

To see effective destinations and files already present:

python -c "from quibble.cache import prepare_caches; print('\n'.join(str(k)+': '+str(v) for k,v in prepare_caches().items()))"
Get-ChildItem .\.cache\huggingface\hub -Directory -Filter 'models--*'
Get-ChildItem .\.cache\tabpfn -Recurse -File
Get-ChildItem .\.cache\openml -Recurse -File

The listing examples assume default cache paths; use the printed paths for overrides. Empty cache directories do not establish that any model or dataset was downloaded.

How and when a command uses data

1. A run selects a registered dataset and loads its features and target. Built-ins come from the installed package; OpenML may need network access if cache files are absent.
2. Holdout uses approximately 70% training and 30% test rows. KxR uses K-fold splitting repeated R times. Classification is stratified when class counts permit.
3. Each backend receives the same split's training and test rows. Model seeds vary model randomness; CV seed and split ID determine row assignments.
4. Ordinary runs keep the full split. Smoke checks select at most 128 training and 32 test rows after splitting, deterministically, preserving classification labels where possible. The same sampled rows are used across backends. These are setup checks, not full-data evidence.
5. Models fit or condition on training rows and predict held-out test rows. Tree models learn trees; foundation models load pretrained weights and use the labelled training context. Quibble does not pretrain those foundation checkpoints from scratch.
6. Results save predictions, true test labels, and original test-row positions. The complete feature table and a separate training-feature CSV are not copied into each result directory.
7. Analysis and reporting read saved predictions, but also reload the original dataset for row traits and slices. They do not retrain models; an absent OpenML cache can still require network access during report generation.

Saved run records, including errors, are reused unless --force is supplied. Download caches and completed-run caches are separate: reusing downloaded weights does not skip a new fit, and skipping a completed fit does not prove its original source data is still available. --force reruns selected cells after archiving existing results; it does not mean every source dataset or model weight is downloaded again.

Registered, downloaded, verified, and actually evaluated

These are separate states:

This guide is the supported inventory, not a live list of successful experiments. Check the current report's participation table and each record's run_type, row counts, timestamp, settings, and device fields to establish what was actually evaluated. invocation.json describes the latest sweep request, not the entire project's run history. Use dated snapshots for older evidence. Smoke success does not establish full-benchmark success.

List current Wine records and their recorded outcomes from the repository root:

Get-ChildItem .\results\wine\*_ss*_ms*.json | ForEach-Object {
    $r = Get-Content -Raw -LiteralPath $_.FullName | ConvertFrom-Json
    [pscustomobject]@{
        Backend = $r.backend
        Split = $r.split_seed
        Type = $r.run_type
        TrainRows = $r.n_train
        TestRows = $r.n_test
        Error = $r.error
        Refused = $r.refused
    }
}

For a smoke experiment, inspect the corresponding directory under results\smoke\<id>\wine instead. Missing folders are not evidence of failure unless that dataset was actually requested.

Verification

python -m quibble datasets --verify --names breast_cancer,wine,diabetes_reg  # offline
python -m quibble datasets --verify --names credit-g,adult                 # may download
python -m quibble datasets --verify                                       # all registered; may download all OpenML entries

For each dataset it prints and saves (to datasets_meta.json in the current directory):

Use the metadata to describe your benchmark (and to make sure it spans sizes, categorical share, missingness and imbalance).

Reading the saved Wine predictions

Use the row-index array to connect predictions back to the original source features:

from pathlib import Path
import numpy as np
from quibble.datasets import load

directory = Path("results") / "wine"  # Or the wine directory inside one smoke run.
X, y, task = load("wine")
idx = np.load(directory / "idx_test_ss0.npy")
truth = np.load(directory / "y_test_ss0.npy")
probabilities = np.load(directory / "histgb_ss0_ms0.pred.npy")
rows = X.iloc[idx].copy()
rows["target"] = truth
rows["predicted_class"] = probabilities.argmax(axis=1)
print(rows)

Check the matching run JSON for errors before reading an array: an unsuccessful forced retry can leave an older prediction file on disk. The analysis loader ignores failed records. For classification, each probability column corresponds to the encoded class label. For regression, the prediction array is a one-dimensional vector. See results format for all array and JSON fields, and reports for interpretation and model-artifact locations.

Loading rules

• Classification targets are label-encoded (sorted order of the original labels) to 0..k-1.
• Regression targets are converted to floats.
• Features are passed through untouched except for the index reset; categorical columns keep their object/category dtype and are handled by each backend's preprocessing rule.
• Row identity is the 0-based position in the loaded table; that is what the row column in disagreement CSVs refers to.

Choosing datasets for a benchmark

Aim for variety rather than convenience: some small (under 5,000 rows), some medium, at least one imbalanced, one with many missing values, one categorical-heavy, and two or three regression sets. Beware of datasets that every model solves perfectly: they hide differences.

Adding a dataset

For your own file, create data\<name>\dataset.json plus CSV/Parquet as described in Testing your own data; no code changes are required. For another public OpenML entry, add its id, task, and exact expected name to OPENML, then run quibble datasets --verify --names yourname.

Licences

Each OpenML dataset has its own licence and citation requirements. Check them before publishing results derived from a dataset, and cite sources in your write-up.