SoulBoxFT: the 986 MB therapist I trained and shipped to the edge
Therapy is a language problem before it is a clinical one.
More than a billion people live in states where the therapy they can reach is not the therapy they can speak. Hindi, Marathi, and Telugu are the first languages of nearly 400 million people, and the mental-health apps that actually reach them default to English, or to a translated English that loses the emotional register a person panics in. You do not tell a chatbot about your worst week in your second language. You stay quiet, or you scroll past the paywall.
I cannot fix the therapist shortage. What I could do was test a narrower, more technical bet: whether a small multilingual CBT assistant, trained on disciplined synthetic data and protected by a hard guardrail, can run on hardware that costs less than a dinner. No cloud GPU. No API spend. No $500K inference bill at the end.
That project became SoulBoxFT. This is the honest version of how it went.
The bet: small models, disciplined data, edge hardware
The default move in 2026 is to rent a frontier model, feed it a system prompt, and call it a product. That works until it does not. It does not work when the user is on a 4 GB device in a city where the network is a rumor. It does not work when every token is a metered cost. And it does not work when you care about sovereignty: whose cloud is your worst day stored in?
SoulBoxFT is the opposite bet. A 1.5B parameter model, quantized to 986 MB, served entirely on-device. The logic: for a narrow, structured, low-temperature task like delivering a CBT exercise, a small model that has been trained on exactly the right data beats a large model that has been prompted at it. The frontier model is a brilliant generalist. A 1.5B fine-tune is a focused specialist that fits in your pocket.
Data is the discipline
The entire project rises or falls on 270 rows. That number looks like a joke next to the megadatasets everyone quotes, so let me explain why it is not.
I distilled the data from a 7B teacher (Qwen2.5-7B-Instruct-4bit via MLX) rather than scraping it, because synthetic distillation lets you enforce constraints a scrape cannot: every row is on-topic, every row is in the target language, and every row is structurally clean. Then I stopped trusting the teacher. For each prompt I sampled multiple candidate completions and kept the ones that agreed with themselves, best-of-K self-consistency. When seven drafts of the same response say the same thing, that is signal. When they disagree, that is noise, and noise does not get shipped.
Then the six gates, applied to every row before it touched a trainer:
- Purity. Is this actually CBT, not generic advice? No off-topic drift.
- Language. Is this real Hindi, Marathi, or Telugu, with correct script and natural register, not machine-translated mush?
- Loops. Does the response terminate, or does it circle back on itself?
- Echolalia. Did the model just parrot the user’s words back instead of responding?
- Leak. Did the teacher leak its own instruction-following artifacts into the “therapy”?
- Length. Is it scannable and human-sized, not a wall of model-speak?
Twenty-seven rows survived into the single-turn set, plus thirty three-turn conversations to teach the model that a session is a dialogue, not a monologue. Volume is a strategy for people who cannot afford quality control. I could afford it.
Training without a data center
The fine-tune runs natively on Apple Silicon with MLX and MPS. No cloud GPU, no API spend, no nvidia-smi anywhere in the pipeline. The flagship is a DoRA (weight-decomposed low-rank adaptation) fine-tune of Qwen2.5-1.5B-Instruct. DoRA decomposes the update into magnitude and direction components, which learns faster and more stably than plain LoRA at the same parameter budget. For the tightest targets there is a 0.5B ultra-light tier that trades quality for 397 MB of memory.
The honest footnote belongs here: 0.5B is below the 1.5B flagship in quality, and I say so in the results. The useful version of this project is the 1.5B.
Export and the edge
Training produced a merged SafeTensors model. Deployment required the artifact to run on hardware that has never heard of a datacenter, so I exported to GGUF through llama.cpp, first f16 as an unquantized reference, then Q4_K_M at 986 MB. That is the deployable file. It runs on the Orange Pi Zero 3, an ARM board with 1 to 4 GB of RAM that costs less than a pizza delivery, and the same GGUF runs in a browser tab through WebLLM. The deployment notes walk through both targets.
A mental-health assistant that runs on a $60 board changes the economics of the conversation. The marginal cost of a session approaches zero. The data stays on the device. The “cloud dependency” column of the pitch deck becomes a row of dashes.
The safety layer is not optional
This is where the project stops being a demo and becomes something I will defend in a review.
Every input is screened before it reaches the model, and every output is screened after it leaves. The guardrail blocks crisis, medical, and harmful prompts at the door, then filters model output, then runs a self-correcting loop that regenerates on failure and rejects echo. The measured numbers on the stress suite: crisis recall 15/15, medical recall 8/8, harmful recall 3/3, false positive rate 0.0. I want to be precise about what recall 1.0 means: on my test set, nothing dangerous got through. It does not mean nothing dangerous can ever get through, which is why the guardrail ships as a mandatory layer, not a suggested one.
And the other mandatory sentence: this model is trained on synthetic data and is not a substitute for professional mental-health care. It is a research artifact with a guardrail, and production deployment without the guardrail is not a thing that exists. I wrote that into the model card and the README so nobody can miss it.
The honest part
I tried DPO on top of the SFT and dropped it after evaluation. The paper-clean reward framing did not survive contact with the synthetic data, and keeping a stage in the pipeline because it looks impressive is how pipelines rot. The changelog says so in plain terms.
I also want to be honest about the ceiling. 270 gated rows teach a narrow, careful behavior; they do not build a general therapist. The model is a structured CBT exercise machine with a safety layer, not an intelligence that understands a human being. Anyone who tells you otherwise is selling you a demo.
The numbers
| Property | Value |
|---|---|
| Base model | Qwen2.5-1.5B-Instruct (flagship) / 0.5B (ultra-light) |
| Fine-tuning | DoRA, native MLX on Apple Silicon |
| Data | 270 gated single-turn rows + 30 three-turn conversations |
| Export | GGUF Q4_K_M, 986 MB flagship / 397 MB ultra-light |
| Hardware | Orange Pi Zero 3 (ARM, 1-4 GB) / browser via WebLLM |
| Safety | Guardrail recall 1.0 (crisis 15/15, medical 8/8, harmful 3/3), FPR 0.0 |
| License | Apache 2.0 |
Everything is published under Apache 2.0: the 1.5B model, the 0.5B tier, and the dataset with its splits and limitations documented.
What this means
SoulBoxFT is one project, but it is a repeatable pattern: distil with a teacher, gate the data harder than you gate the code, fine-tune on the hardware you already own, export to an artifact a $60 board can serve, and put a hard guardrail in front of anything that touches a person. That pattern is not specific to therapy. It is how a lot of useful AI is going to get built, not in a datacenter, but in a laptop, for people the datacenter forgot.
If you are scoping a similar bet, a small multilingual model with a real safety layer and an edge deployment story, my calendar is open. The senior hand ships.
The model is on Hugging Face. The honest caveats are in the card. The guardrail is not optional.