Human in the Stream, LLC
All Projects
Sean McIntyre · 6 min read

miDataworks: Datasets for Detectors and Training

The data plane of miSuite. miDataworks imports, curates, labels, checks and publishes datasets, with every row traceable to where it came from and every version reproducible from its recipe. It builds the detector sets miStudio trains probes on and the training sets miForge consumes, and it refuses to hand over data that would make a detector look better than it is.

Part of miSuite, and fully usable on its own.

miDataworks makes the data. A detector is only as honest as the data it was trained and tested on, and most of the ways a detector fools its builder are data problems: a column that gives the label away, near-duplicates that sit on both sides of a train/test split, labels from a model nobody checked, wording that separates the classes without any of the meaning. miDataworks exists to catch those problems before they become a published number.

It imports datasets, curates them through explained and reproducible steps, labels them with a classifier or a judge model, checks labelers against human labels, and publishes versions to the Hugging Face Hub. It needs no graphics card of its own; labeling and generation call whatever model endpoints you configure.

Sources, versions and recipes

  • Import from Hugging Face or from uploaded files. A Hub import is pinned to the exact commit it read. Before importing, miDataworks can search the Hub by topic and license and profile a whole split: row counts, label shares and lengths.
  • Versions are immutable. A version is built from sources by a recipe, a list of operator steps, and is identified by the hash of that recipe. Every dropped row records why it left. A version can be rebuilt and checked against its own record.
  • Operators do the work. Native operators cover filtering, sampling, splitting, balancing and deduplication. Two open-source engines plug in as operators: Data Designer, which generates and transforms rows from templates, and Data-Juicer, with its library of cleaning and filtering operators. Each runs in its own isolated worker. Every step previews on a sample before it runs, and a template is checked against Data Designer’s own rules before a build is attempted.

Curation that explains itself

  • Shortcut audit. After any build with a label column, miDataworks measures how well each other column predicts the label on its own. It exists because a format column once predicted a humor label 88.5% of the time and nothing said so. A text baseline (bag-of-words and length-only classifiers on held-out folds) tells you how far the wording alone gets.
  • Leakage checks find near-duplicate rows that cross split boundaries, and a repair plan removes them.
  • Balancers and splitters that state their own side effects. A label balancer, for instance, carries a mandatory warning on the version’s lineage.
  • Contamination and diversity reports, including embedding-based clustering when an embeddings endpoint is configured.

Labeling with endpoints you choose

Labeling is configured as four endpoint roles, not a list of servers: a classifier, a judge, a generation model and an embeddings model. Each role points at any compatible server: miLLM, a Text Embeddings Inference (TEI) server, any OpenAI-compatible endpoint, or an external classification API. Keys are encrypted at rest, shown only masked, and sent only to the server they were entered for.

  • Rubrics and templates define the question a judge answers. Rubrics are versioned, carry a behavior key, and are published as the midataworks.rubric/v1 schema so miStudio can import one as a judge prompt.
  • Every label records who produced it, with which model, revision, question and thresholds.
  • Calibration and human review. Before a labeler’s output is trusted, miDataworks compares it with human labels and records a verdict. Review queues record every human decision on a row without ever editing the row.
  • Generation and steered pairs. Generated rows can fill gaps where real data is thin, without letting a model grade its own output. Minimal pairs (a row and a counterpart edited just enough to flip the label, verified by an independent judge) are built as a chain of recorded steps. With miLLM present, pairs can be made by steering the model, and the steering that was actually applied is recorded.

Detector sets: the handoff to miStudio

A detector set turns versions into the four kinds of data a probe needs: training rows, an in-distribution test set, one or more out-of-distribution evaluation sets, and calibration negatives. Before anything is sent, nine checks run, each green, a note, or refused with a reason and a next step. They cover complete roles, valid label mappings, the shortcut audit, leakage across roles, length match between calibration and monitored text, synthetic rows in held-out roles, labeler calibration records, the basis for calling a row negative, and whether the wording alone separates the labels.

The send publishes the versions, has miStudio download and verify them, registers each role as a view, and compares miStudio’s counts with its own. When miStudio has trained and evaluated probes, miDataworks reads the evidence back, so every figure on its screen names the set and sample it came from. In the suite’s personal-data walkthrough, this is the check that refused an agent’s first set over 189 near-duplicate pairs.

Publishing and export

Publishing to the Hub runs in three visible steps: build the files, run the checks, push. Export writes training files to disk instead: sets shaped for the Hugging Face Transformer Reinforcement Learning (TRL) library, reward bundles, and sets for miForge. Every published or exported version carries a manifest in the midataworks.dataset-version/v1 contract, which miStudio and miForge read.

Agent control, with approvals

miDataworks runs its own Model Context Protocol (MCP) server with about 190 tools in 11 categories, covering every screen. Agents can read, plan, preview and dry-run freely. These actions wait for a person in miDataworks’ Approvals view: pushing to the Hub, labeling more than 5,000 rows of a version in 24 hours, deleting a version, loading a model on miLLM, writing a secret or key, and changing a runtime setting. An agent cannot approve its own request. Permanent deletion of archived records is operator-only and has no MCP tool.

Standalone first

miDataworks works fully without miStudio, miLLM or miForge. Labeling uses whatever endpoints you configure, and detector sets and training sets can be published to the Hub or exported as files for any consumer. When a sibling is present, miDataworks detects it at runtime and enables the matching features: model leases on miLLM, detector-set sends and evidence read-back from miStudio, and set delivery to miForge. When one is absent, those features refuse clearly and name what is missing.

Status

miDataworks is at version 0.1.0. Under the suite’s versioning, an application stays at 0.1.0 until the operator declares it has reached its first feature-complete milestone, which is marked 0.5.0. Every planned feature is built and in use, as the personal-data run shows, and several live acceptance checks remain open. The work on monitoring behaviors added a behavior key and a session-data caveat to its datasets; see miSuite.

Explore miDataworks