miSuite: The Mechanistic Interpretability Suite
Four open-source applications that each work alone and work better together — miDataworks makes the data, miStudio looks inside models and trains detectors, miLLM serves models with steering and live monitoring, and miForge makes the weights. Their Model Context Protocol (MCP) servers let an AI agent run the whole loop, with a person approving anything that uses a graphics card, publishes, or deletes.
miSuite is a set of four applications for people who want to know what a language model is doing inside, and to act on what they find. Each one is a complete product with its own interface, database, manual and source repository. None of them needs the others to run. When they are installed side by side, they detect each other and hand work across through portable documents, so a dataset curated in one becomes a detector trained in another and a monitor running in a third.
The suite is built for technical practitioners: machine-learning engineers, interpretability researchers, and people who run open models on their own hardware. Everything runs locally. Everything is open source.
The four applications
| Application | What it does |
|---|---|
| miDataworks | Makes the data: imports, curates, labels, checks and publishes datasets, and builds the detector sets and training sets the other applications consume. |
| miStudio | Looks inside models: trains sparse autoencoders, labels and clusters features, discovers and validates circuits, reads the model with J-Lens, and trains probe monitors with honest evidence about how well they work. |
| miLLM | Serves models behind an OpenAI-compatible application programming interface (API), and while it serves, steers them with features, clusters and circuits and watches them with probe monitors. |
| miForge | Makes weights: creates a model from scratch, changes an existing one, or trains an adapter beside it, one tested checkpoint at a time. |
A sentence from the suite’s own documentation draws the lines: miDataworks makes data, miLLM serves models, miStudio interprets models and trains detectors, and miForge makes weights. None does another’s job.
Standalone first
Every application works alone, and that is a design rule rather than an aspiration. miForge trains and checkpoints with nothing else installed. miDataworks labels with any OpenAI-compatible endpoint, not only miLLM. miLLM serves and steers using sparse autoencoders (SAEs) downloaded from Hugging Face, whether or not miStudio trained them.
Integrations are additive. Each application looks for its siblings at runtime, and when one is absent it says so by name and refuses the dependent action, instead of failing quietly or pretending the work was done. No application imports another’s code.
What connects them: portable documents
The applications talk through published contracts, not shared code. Each contract is a versioned JSON Schema (JavaScript Object Notation schema) file owned by the application that produces it. Every consumer keeps a byte-identical copy, and a test in the consumer’s repository fails if the copies ever differ.
| Contract | Produced by | Consumed by | What it carries |
|---|---|---|---|
midataworks.dataset-version/v1 | miDataworks | miStudio, miForge | A dataset version: its splits, labels, provenance and caveats |
midataworks.rubric/v1 | miDataworks | miStudio | A labeling rubric, importable as a judge prompt |
mistudio.probe-definition/v1 | miStudio | miLLM | A trained probe, its threshold, its evidence rung and test vectors that prove a consumer scores it identically |
mistudio.circuit-definition/v1, mistudio.cluster-definition/v1 | miStudio | miLLM | A discovered circuit or feature cluster, with its evidence, ready to steer with |
| Checkpoint and adapter manifests | miForge | miLLM | What a set of weights is, where it came from and how it scored |
| Scoring API | miLLM | miForge | How a served model scores text, pinned as a generated schema |
Because the documents are self-describing, they also work outside the suite: a miForge checkpoint is an ordinary Hugging Face model directory, and a probe definition can be read by anything that implements its schema.
The loop
Read left to right, the suite follows the life of a question about a model:
- Data. miDataworks imports a dataset, curates it, labels it with a classifier or a judge model, and checks it for the problems that make detectors look better than they are: shortcut columns, near-duplicates leaking across splits, and labels that the wording alone gives away.
- Interpretability. miStudio trains SAEs on the model’s activations, labels and clusters the features, discovers circuits across layers, and trains small linear probes that detect a behavior in the model’s internal state. Each result carries an evidence rung that says exactly how far it has been tested.
- Serving. miLLM serves the model to ordinary clients through an OpenAI-compatible API. While it serves, it can steer with the features, clusters and circuits miStudio found, and score every request with the probes miStudio trained.
- Weights. miForge changes the model itself, or trains an adapter beside it, testing each checkpoint against a domain set and a retention floor. A checkpoint can go back into miLLM, either by a normal load or by updating the served weights in place.
Each step is optional. A researcher who only wants SAE steering in Open WebUI needs miLLM and nothing else.
Agentic control: MCP servers with a human in the loop
Each application exposes its work as tools over MCP, the open standard that lets an AI agent call software through a typed interface. The scale is large enough to run real work end to end:
- miStudio: about 280 tools in 20 categories, including a family that drives miLLM (loading models, importing probes and circuits, steering, scoring) and a family that reaches into miDataworks.
- miDataworks: about 190 tools in 11 categories, covering every screen of the application.
- miForge: 90 tools in 10 categories.
Each server can be reached over an encrypted web endpoint that always requires a bearer token, so an agent harness does not need to run inside the cluster.
Agents can read anything, plan, validate and dry-run freely. Actions with consequences wait for a person. Starting work on a graphics processing unit (GPU), pushing to the Hugging Face Hub, changing settings or keys, loading a model that would evict the one being served, and deleting things all file an approval request that appears in the application’s own interface. The agent receives a pending status and polls it. An agent cannot approve its own request; the applications refuse it. Permanent purges are operator-only and have no MCP tool at all.
A worked example: a personal-data detector, end to end
On 2026-10-20 an AI agent was given one task: build a detector for personally identifiable information (PII) in text, using only the suite’s MCP servers. What follows is what happened, including the parts that did not work.
Finding data. The agent chose nvidia/Nemotron-PII, a public synthetic dataset licensed Creative Commons Attribution 4.0 (CC-BY-4.0), and imported 100,000 documents into miDataworks pinned to an exact revision. It built two segment sets: a US set (4,120 training, 1,370 calibration and 1,370 test rows, each balanced between PII and no PII) and an international set of 1,150 rows held back entirely. The international set is the important one. It tests whether the detector works on data it never saw anything like.
Checking the data. miDataworks refused the agent’s first detector set. Its leakage check found 189 near-duplicate pairs crossing the training and evaluation splits, mostly boilerplate. The agent rebuilt from scratch, pooling both sets and removing near-duplicates before splitting. Four superseded versions later the check was green, with zero pairs. The agent also measured how far simple text baselines get on their own, because a detector is only interesting if it beats them: bag-of-words reached an area under the receiver operating characteristic curve (AUROC, where 0.5 is chance and 1.0 is perfect) of 0.952, character n-grams 0.967, and text length alone 0.505.
Two approvals. Sending the detector set to miStudio meant publishing it to the Hugging Face Hub, so the agent’s request waited until the operator approved a private push. miDataworks then published, miStudio downloaded and verified both versions, and four evaluation views were registered with counts matching exactly. Training probes needs a GPU, so that request waited for a second approval.
Training. miStudio trained three probes on LFM2.5-1.2B-Instruct, reading layer 5 with mean, max and attention pooling. The run took 3 minutes 51 seconds. The attention probe is the one to use as a monitor:
| US test set | International set (never seen) | |
|---|---|---|
| Attention probe AUROC | 0.994 | 0.998 |
| Caught at a 1% false-positive budget | 97% | 96% |
| Judge (Qwen2.5-7B-Instruct) AUROC | 0.909 | 0.921 |
| Judge catches, at about 4% false positives | 86% | 88% |
The judge used the pii-presence/v1 prompt from miStudio’s judge-prompt library and scored the same rows. With that comparison recorded, the attention probe reached rung 3 of the evidence ladder: detects on unseen tasks, compared with a judge.
What the numbers do not say. These results are strong, and they are also easy to over-read:
- The signal is largely surface-level. A probe at layer 0, almost at the input embeddings, already scores 0.994. Digits,
@signs and field words carry most of the information. This is evidence of a good detector, not of an abstract concept of “personal data” inside the model. - The data is synthetic. Real documents are messier.
- It has not run live yet. These are offline evaluations. Arming the probe in miLLM against real traffic is the next step, and that step also waits for the operator.
- “Compared with” does not mean “beats”. Here the probe scored higher than the judge. On miStudio’s earlier reference run, for a different behavior, the judge won on all five evaluation sets, 0.874 to 0.794. The rung records that a comparison happened, not who won.
What the agent could not do, and what changed. The agent kept a log of every point where the MCP servers were not enough. There were twelve, ranked by how badly each blocked an MCP-only workflow. It had no way to search the Hub for datasets or profile a whole split, so it used the Hub API and local scripts. A template validator said “valid” for templates that the data-generation engine then refused, and it took reading the engine’s source to learn the rules. A failed preview reported “running” and then disappeared. There was no single call to ask whether a GPU was free, and no read-only way to check whether a miStudio approval had been granted. And the servers were only reachable from inside the cluster.
Every one of those became work. miDataworks gained Hub search, source profiling, a build dry run, the engine’s own template validation, stored preview failures, a text baseline in its shortcut audit, a leakage-repair plan and column projection. miStudio gained an approval-status tool, a GPU preflight and a list of active jobs across every job kind. The servers gained authenticated external endpoints. The agent found the edges of what it could do, and the suite grew to meet them.
In progress: monitoring behaviors in sessions
The suite’s current direction extends that one detector to many behaviors at once. Some of it has landed in the code; the rest is being built now.
Landed:
- Judge prompts are data. miStudio keeps a versioned library of judge prompts. A prompt is immutable once used, carries a behavior key such as
pii, and is added through the interface, over MCP, or by importing a miDataworks rubric, with no code change. - Behavior keys tie prompts, probes, datasets and runs together, so a judge only counts as evidence for a probe of the same behavior.
- Sessions in miLLM. While a probe is armed, miLLM groups the requests of one conversation into a session and records which turn each verdict belongs to. Recording conversation text is opt-in, off by default, kept for seven days, deletable per session and never broadcast.
- Per-model probe limits measured on hardware. How many probes a model can carry depends on the model and the card, so miLLM measures each probe’s cost per forward pass on the card it runs on and admits probes against a time budget.
Being built:
- A behavior catalogue in miStudio that records each behavior’s definition, its data, its probes in their roles, its judge prompts and its operating point.
- Judge spot-checks of live sessions. miStudio samples sessions from miLLM, stratified into fired, near-threshold and not fired, has a judge read the same window the probe reported, and reports agreement with a confidence interval, the disagreeing turns, and the plain statement that the judge is not ground truth.
Alerts and automatic actions are deliberately out of scope for now. A verdict is evidence for a person, not a gate, until thresholds have been proven on real traffic.
Open source
- miStudio: github.com/hitsainet/miStudio · manual
- miLLM: github.com/hitsainet/miLLM · manual
- miDataworks: github.com/hitsainet/miDataworks · manual
- miForge: github.com/hitsainet/miForge · manual
- Videos: YouTube @miSuite-hitsai
For where the research side of the suite is heading, the September direction piece is still a good read: miStudio + miLLM: Direction of the Mechanistic AI Suite. It predates miDataworks and miForge, and is kept as written.