Quibble

Experiment workbench

Describe a prediction problem, review an explicit generation recipe, preview the data, and run a comparison without switching between scripts.

The optional language-model connection drafts a specification, not prediction results. Quibble's local generation code creates the rows and outcomes. Existing prediction methods then train and are evaluated on held-out outcomes. This does not benchmark ChatGPT as a predictor.

Why generate a fresh dataset

Familiar public benchmark tables may have appeared in a pretrained predictor's training data. Quibble cannot inspect every predictor's pretraining corpus or prove that a benchmark is unseen. Newly sampled rows reduce direct reuse of those benchmark examples and make the input and outcome rules inspectable.

That is an experimental control, not a guarantee of no contamination. A predictor may already know similar task structures or generating rules. A language-model planner can also propose familiar assumptions, and a synthetic process may favor some methods. Changing only a dataset's name does not create new examples; the same specification and seed reproduce the same rows.

Use the generator to ask controlled questions: what changes when noise increases, an input disappears, or irrelevant fields are added? Repeat across independently generated tables and several task structures. Keep development variants separate from confirmatory evaluation, preserve the recipe and hashes, and test useful findings on representative real data. Synthetic tests complement real-data evaluation; they do not replace it.

The browser adds an interactive route through these steps. It uses the existing benchmark runner and report pipeline; CLI commands remain available for scripting and settings outside the UI's deliberately bounded controls.

Start locally

Use the existing environment from Getting started:

& .\.venv\Scripts\Activate.ps1
python -m quibble web --open

Open http://127.0.0.1:8765. Choose another port with --port 8876. The server binds only to 127.0.0.1, not a network interface. The default address opens About. Choose Open workbench in its What to do next section, or Workbench in the shared header, to open /workbench.

The prominent Quibble heading identifies the product; the smaller subtitle describes its purpose. Every page shares Workbench / Blog / About / GitHub / Docs navigation, with the current page highlighted. These are product pages, not experiment steps. Blog, About, and Docs open separately from the workbench so its draft stays intact. It explains workflow, methods, devices, privacy, limitations, and the running version. Its Predicting from Tables blog series link opens the local reader at /blog; both articles, diagrams, and curated evidence work without public hosting. Docs and Blog work locally; GitHub links to the source repository. Version and license details appear in About's content, not the shared header. Read About Quibble and the documentation home locally meanwhile.

The local Docs reader uses a documentation-site layout: grouped sidebar navigation, the current guide highlighted, breadcrumbs, an On this page section index, and Previous/Next guide links. Blog articles use the same layout and shared product header, but their sidebar and Previous/Next links stay within the blog series, not the docs. On narrow screens the navigation stacks above the article and can be collapsed using Documentation navigation. Heading links work without JavaScript.

Circled i icons beside fields and prediction methods explain their purpose, limits, and trade-offs. Hover or focus an icon with the keyboard to read its help; click or tap to keep it open. Press Escape, tap the icon again, or click elsewhere to dismiss it.

The unified Dataset selector includes already-created grocery and retail clothing tables without an account, provider call, or regeneration. Both domains have binary classification and regression examples, each with 1,000 rows and eight predictors. Saved recipes, input hashes, and explanations accompany all four tables.

One starting point, two data paths

The top navigation has two views: Experiment workflow and Past experiments. Full-width navigation cards describe both entry points. The active card is marked Selected and colored; the other stays neutral. Cards stack on narrow screens. Only the workflow view shows its three-step progress navigation: Choose data -> Configure & run -> Results. Past experiments is not a fourth step. Switching back to the workflow restores its last step. Start at Choose data:

• Use existing dataset shows ready-made and other saved datasets. Choose one, inspect its actual rows and assumptions, confirm review, then choose Continue to experiment. No specification creation is required.
• Design new dataset shows the scenario and recipe editor, not the existing-data controls. Propose a design, preview, confirm review, then choose Save and continue to save the dataset and open experiment configuration.

Preview generation stays disabled after a successful preview until the recipe changes. An unchanged saved recipe also needs no generation: discard the variant to use its existing rows, or change the name or recipe to enable a new preview.

The run screen shows the chosen dataset as a summary, not another dataset picker. Use Change dataset to return to the data step. Starting a run opens Results; bookmarked jobs also reopen there. Past experiments opens saved report selection without requiring a new data choice.

For a new design, enter your own description and choose Propose a scenario with my provider. Starting a proposal clears the previous recipe and preview, so a failed request cannot leave an old example looking like the new design. Invalid provider specifications get one validation-guided correction request, then an explicit error if still invalid. Both requests may incur provider charges; malformed specifications are never saved.

The selected outcome type takes precedence over scenario wording. For an inventory coverage yes/no question, select classification. Selecting regression asks for a continuous synthetic score instead; the planner must disclose that reinterpretation and its meaning in the assumptions. The score can be negative and is not automatically demand units or a calibrated coverage probability. Classification assumptions should explain labels 0 and 1.

Select a ready-made dataset to inspect its saved recipe, sample rows, and collapsed explanation in the existing-data path. Review and continue; saving or generating it again is unnecessary. Changing the outcome filter clears the previous selection and review. The single dataset list shows only datasets matching its outcome filter, with classification entries labeled binary or multiclass. Groups identify bundled synthetic examples, built-in benchmarks, and other saved datasets. They are all usable existing data, not different readiness levels. Existing local folders override bundled names. Choose Design new dataset for a blank design. Its outcome selector is separate from the ready-made filter; custom scenario text is preserved on task changes.

The bundled names are grocery_sample, grocery_regression, retail_clothing_binary, and retail_clothing_regression. CLI dataset commands can use them too. An existing local folder with the same name takes precedence and is never overwritten. Changing controls does not change saved rows; use Create variant from this recipe to create a reviewed variant. These examples demonstrate task shapes, not calibrated domain simulations or four independent sources of real-world evidence. The existing-data path hides creation controls instead of showing an unrelated editor. Choose Design new dataset to design a new dataset. To copy an existing recipe, expand Advanced: create a dataset variant in the saved-data review and choose Create variant from this recipe. The optional action appears only when a recipe exists; datasets without one do not show variant controls. Built-in benchmarks show a plain source note instead of empty metadata JSON. Imported datasets show their available source details without an empty generation entry. While editing a variant, Discard changes and continue sits beside Save and continue. Saving uses the newly saved variant; discarding abandons the unsaved recipe and uses the original saved rows. Both open experiment configuration without starting a run. Discarding never modifies or deletes saved files. After saving a variant, the discard action is no longer shown.

Expand About this dataset after a proposal, preview, or save to read its scenario, target interpretation, input descriptions, and assumptions. This explanation comes from the current recipe, not another provider request. Editing the recipe hides the old explanation until a validated preview refreshes it. Reviewing a saved generated dataset also provides a collapsed explanation from its saved recipe.

Repository requirements include the planning/authentication SDKs. A minimal package install can add them with python -m pip install -e ".[web]"; local generation and the local example do not need those SDKs.

Configure your planning provider

Copy the template only if you do not already have settings:

if (-not (Test-Path .env)) { Copy-Item .env.example .env }

The planner supports the OpenAI-compatible Chat Completions protocol with JSON-object responses. Choose a provider, its inference base URL, a model/deployment name, and authentication. Azure Foundry is one option, not a requirement.

Restart the workbench after editing .env. The settings are server-side; credentials are not entered into or stored in the browser.

These are configuration examples, not claims that every service/model has been tested. Native Anthropic, Gemini, Bedrock, or other non-compatible protocols need an adapter or compatible gateway; changing a URL does not translate an incompatible API. Your chosen model must support this protocol and JSON-object output. No automatic provider fallback occurs.

OpenAI or another compatible hosted service

Set these entries in .env:

LLM_BASE_URL=https://api.openai.com/v1/
LLM_MODEL=YOUR-MODEL
LLM_AUTH_MODE=api_key
LLM_API_KEY=YOUR-PROVIDER-KEY
LLM_PROVIDER_NAME=My planning provider

For another compatible service, replace the URL, model, and provider-specific key. Remote endpoints require HTTPS; URL credentials, query strings, and fragments are rejected. Do not put keys in URLs.

Standard OPENAI_BASE_URL, OPENAI_MODEL, and OPENAI_API_KEY names also work; the base URL defaults to OpenAI when no endpoint is configured. The LLM_* settings take precedence. A custom LLM_BASE_URL outside api.openai.com requires its own LLM_API_KEY: it never implicitly receives an unrelated OpenAI or old Azure key. When using aliases for a non-OpenAI compatible service, explicitly pair OPENAI_BASE_URL with that service's OPENAI_API_KEY.

Microsoft Entra ID

For an Azure deployment, configure:

LLM_BASE_URL=https://YOUR-RESOURCE.openai.azure.com/openai/v1/
LLM_MODEL=YOUR-DEPLOYMENT
LLM_AUTH_MODE=aad
LLM_AAD_CREDENTIAL=cli
LLM_TOKEN_SCOPE=https://ai.azure.com/.default
AZURE_TENANT_ID=
AZURE_SUBSCRIPTION_ID=

Use the inference base URL, not a Foundry portal or /api/projects/... management URL. An Azure deployment name is not necessarily the model family's name. Install Azure CLI if needed and sign in explicitly:

python -m quibble auth
python -m quibble auth --check
python -m quibble web --open

For browserless/device-code login use python -m quibble auth --device-code. Alternatively, python -m quibble web --login --open combines explicit login and startup; add --device-code if required. Optional tenant and subscription settings select the intended account context.

cli uses AzureCliCredential. default uses noninteractive DefaultAzureCredential, allowing an existing environment or managed-identity setup. The auth login helper is only for cli; for default, establish credentials externally and use auth --check. Token acquisition does not prove that inference authorization works. The identity also needs the inference role appropriate to the resource, such as Cognitive Services OpenAI User for an Azure OpenAI resource. Follow your organization's role-assignment policy.

Azure API key

Keep the Azure URL/deployment and use this instead of AAD if your resource permits key access:

LLM_AUTH_MODE=api_key
LLM_API_KEY=YOUR-RESOURCE-KEY

Do not commit keys. auth --check in key mode checks configuration presence, not a live request. Planning errors are surfaced without printing provider response bodies or credentials. Real environment variables take precedence over .env.

Local compatible service

Start and configure the local service separately, including any model download:

LLM_BASE_URL=http://127.0.0.1:11434/v1/
LLM_MODEL=YOUR-INSTALLED-MODEL
LLM_AUTH_MODE=none
LLM_PROVIDER_NAME=Local service

HTTP and unauthenticated mode are restricted to loopback addresses. In none mode, the SDK uses a non-secret placeholder, never a stored OpenAI/Azure key. Entra tokens are not sent to loopback services. A local service must still implement the required API/output shape; Quibble does not install or start it.

Older FOUNDRY_PROJECT_ENDPOINT, AZURE_AI_MODEL_DEPLOYMENT_NAME, FOUNDRY_AUTH_MODE, and FOUNDRY_API_KEY configurations remain supported when no generic/OpenAI endpoint is set. They retain the old AAD default; key maps to api_key. Prefer the provider-neutral names for new configuration.

See Microsoft's Azure v1 API guide and keyless authentication guidance. Mocked OpenAI/custom-provider/AAD/local-service contracts have automated coverage; live provider access must be verified with your own configuration.

Design and review

1. Describe the prediction decision, what a row represents, what information is available at prediction time, and what outcome should be predicted. Do not enter sensitive data.
2. Set rows, predictor fields, task, and generation seed. The target is an additional column: 8 predictors means 9 total columns.
3. For an arbitrary natural-language scenario, explicitly consent to sending that text and controls to your configured provider, then choose Propose a scenario with my provider. This may incur provider charges. No existing dataset rows, private references, or local documents are sent.
4. Review the JSON specification, particularly assumptions, timing, field ranges, weights, interactions, noise, and the task. The draft is a hypothesis, not a factual description of the industry.
5. Preview. Quibble generates the whole table, displays the first 20 rows, and summarizes target distribution and missing values.
6. Edit if needed and preview again. Confirm review and choose Save and continue. Saving requires the current specification's preview from this server session; edits invalidate it.
7. Configuration opens with the saved dataset already chosen. Choose prediction methods and device policy. Return to Choose data if you need the saved CSV download.
8. Run a smoke check, inspect failures and the interpreted report, then explicitly confirm a full comparison if appropriate.

The grocery example is fictional. It uses independent decisions rather than repeated store histories. Do not treat its results as real demand forecasts or deployment evidence.

Names, saved datasets, and variants

Ready-made datasets already have names; the planning provider proposes a name for custom scenarios. For a new design or editable variant, change New dataset name without editing JSON, or edit the JSON's name directly. Both stay synchronized. Changing the name requires another preview and review confirmation. Names must pass the generator's portable-name validation.

To reuse data, select it in Dataset, then choose Review selected dataset. The browser shows the actual saved first 20 rows, counts, outcome summary, missingness, and available source/generation metadata. Confirm review and choose Continue to experiment without generating or modifying rows. Source fingerprints are rechecked on confirmation; if the source changes, review it again. Existing benchmark validation still applies.

For generated data, Create variant from this recipe under Advanced: create a dataset variant loads the saved recipe into the editor. Choose a new unused name, make any changes, preview, confirm, and choose Save and continue, or choose Discard changes and continue to use the original. This generates a new table from the reviewed recipe; it does not rename or overwrite the original. Provenance identifies the original source fingerprint. Imported tables and built-in datasets without generation metadata can be used as-is, but cannot be regenerated through a missing recipe.

There is no overwrite operation. Existing data and previous experiment evidence remain unchanged. Selecting a different dataset clears the current review.

Specification and generation mathematics

Version 1 accepts only known schema keys. Start from the UI or python -m quibble generate --example; unsupported keys or executable code are rejected.

Supported feature distributions:

• uniform: continuous samples over the chosen bounds.
• integer: equally likely integers, inclusive bounds.
• normal: midpoint mean, standard deviation one sixth of the range, clipped to bounds.
• categorical: equally likely listed strings, at least two unique categories. Numeric bounds are placeholders; category order encodes evenly spaced values from -1 to 1. Reordering categories changes their outcome effects.

All features are independently sampled. This version does not simulate correlated inputs, time series, panel data, exact category proportions, or arbitrary business rules. Numeric features are rescaled from their bounds to -1 through 1.

Effects transform each encoded value: linear leaves it unchanged, quadratic squares it, threshold tests whether it exceeds zero, and none contributes zero (weight must also be zero). A weighted sum plus the listed products forms a score. The latent outcome is intercept + scale * score + Normal(0, noise).

• Regression uses that latent outcome directly. It is continuous and may be negative; it does not promise nonnegative integer demand or bounded business quantities.
• Binary classification samples a Bernoulli outcome with logistic probability of the latent value. Even zero Gaussian noise does not remove Bernoulli randomness.

Feature missingness is independently applied after outcome generation. Hidden pre-missing values, latent outcomes, and oracle probabilities are not predictor columns. This separation prevents exposing generator internals as features; it does not automatically detect semantic leakage in arbitrary datasets or validate domain realism.

Limits are 50 to 50,000 rows, 1 to 100 predictors, at most 2,000,000 predictor cells, and seeds from 0 to 4,294,967,295. Missingness is limited to 0.5 per field. Every generated classification class must realize at least 10 rows; regression must be nonconstant. Violations fail explicitly instead of silently resampling or changing requested counts. Balance is stochastic, not guaranteed exact.

Files and reproducibility

Saved datasets use QUIBBLE_DATA_DIR (default data):

data/<dataset-name>/
  data.csv          generated table, target last
  dataset.json      existing local-data loader manifest
  generation.json   reviewed specification, seed, generator version, hash, package versions

Provider-assisted records additionally preserve endpoint, requested/reported model, response identifier, token usage when provided, and planning instructions. Edits are represented by the saved final specification; these fields identify the drafting request, not provider approval of subsequent edits. Preserve all three files. Generation metadata is included in source fingerprints; reports label the experiment synthetic. Changing the CSV without its generation record is rejected. Save a newly designed variant under a new name. Existing folders are never overwritten.

A specification and matching software environment reproduce the local table. Hosted model drafts are not guaranteed reproducible. Generation seeds differ from split seeds and prediction-method seeds; the UI currently fixes model seed 0. Repeat comparisons across fresh generation seeds before generalizing.

For CLI use, save a reviewed example/specification:

python -m quibble generate --example
# Save the JSON as reviewed-spec.json using a UTF-8 editor, then:
python -m quibble generate --spec reviewed-spec.json
python -m quibble datasets --verify --names grocery_sample

Use the actual saved name. Existing local-data commands can evaluate the same dataset.

Experiments and reports

The UI launches the existing benchmark command in a subprocess, not arbitrary scripts. Choose one to four public prediction methods and cpu, cuda, or auto. Installing an SDK does not download all model weights. Selected pretrained methods may need their own packages, authentication, and model downloads.

Smoke uses the established 128-training/32-test maximum and is not a swap approval. Full runs use holdout, 3x1, or 3x2, fixed model seed 0, and one fit worker. One experiment can run at a time; each gets an isolated output directory.

Read what Full does and what the results establish for the split, training-only preprocessing, fitting, held-out prediction, analysis, and reporting sequence. The page also explains why fast fit() does not imply fast inference, and why these results do not establish deployment readiness.

CPU pins all selected methods to CPU. CUDA requires availability; Auto chooses CUDA when available, otherwise CPU, and pins that choice once per experiment. CUDA selections disable incompatible methods with explanations: HistGB/Weak have no CUDA implementation, this LightGBM adapter is CPU-only, and AutoGluon can mix CPU components into its ensemble. Select at least one enabled method. API/CLI selections are filtered under the same policy, with exclusions recorded; an empty eligible selection is rejected. Models are not substituted. Data preparation, metrics, and report generation still run on CPU. GPU failure never triggers a CPU retry. Existing reports retain their original device records.

results/web/<job-id>/
  job.json
  run.log
  output/report/report.md                 full report
  output/smoke/<smoke-id>/report/report.md  smoke report

The browser shows execution logs, sanitized formatted Markdown with inline PNG figures, and a Markdown download. The underlying reports retain interpretation, source fingerprints, participation/failure details, and synthetic limitations. A report can exist for a failed or partial job; only subprocess exit code 0 marks a job completed. Download/copy the associated figures as well when moving a Markdown report.

Headings, tables, lists, emphasis, and code blocks render in the browser. Embedded HTML is treated as text and the generated HTML is sanitized. Only the job's known report figures load as images; external images are not fetched. External HTTP/HTTPS links and fragment links are allowed. Other local artifact/history links are inactive in the browser; use the original downloaded Markdown and results directory for those files.

Jobs and datasets persist on disk. Draft/preview permissions are session-local and are lost on server restart. Previous unfinished jobs are marked interrupted; restart does not resume them. There is no cancellation button. Normal shutdown waits for the active experiment; a forced process termination may leave child work running.

After Start experiment, a running indicator shows the dataset, mode, and experiment ID while the browser checks status every 2.5 seconds. Another experiment cannot start while one is active. The indicator is an activity signal, not a percentage or time estimate. Open Execution log to see completed fits and diagnostics.

The report dropdown starts with no experiment selected and an empty report viewer. Choose an experiment to load its status and report. Starting a new experiment or opening a bookmarked experiment link selects that specific run automatically. Choosing the empty dropdown entry clears the report details again.

You can close the browser tab without stopping the server-side experiment. Keep the Quibble terminal/server running. Bookmark the experiment link under the result status, or return to the same web address and select the dataset/mode/experiment ID in Current and previous experiments. The browser reconnects to running work and shows the saved report when it finishes. The saved-files path is displayed alongside the link. After a server restart, use the same output directory and port/address to reopen completed reports. Unfinished jobs are marked interrupted, not automatically resumed. If the status connection is lost, the UI says so rather than claiming completion; use Refresh after restoring the server connection.

On Windows, keep data/output roots and dataset names short. Deeply nested working directories can exceed filesystem path limits during dated result snapshots. Select a shorter QUIBBLE_OUT_DIR before starting the server if needed; such failures remain failed jobs with diagnostic logs, not completed comparisons.

Local-only boundary

Host checks, same-origin POST checks, a per-start request token, bounded JSON, fixed routes, and safe text rendering reduce accidental browser-to-localhost access. The request token is not user authentication. Do not expose this port through a proxy, tunnel, shared machine, or public network. Local processes and users can access the same filesystem. One workbench instance owns each output root.

Deployment would require authentication, authorization, tenant/job isolation, quotas, secret management, and controlled storage. Those are not provided here.