Your self-hosted StackChan robot assistant — kid-minded by default, hackable by design, private by architecture.

⚠️ Heads up: this is not a stable project yet. Dotty is buggy, frequently broken, and actively changing day-to-day. End-to-end behaviour works on the maintainer's hardware but regressions land all the time, the API and config surface shifts without notice, and a fresh deploy on someone else's gear has not been verified. Treat this as a hobby-grade work-in-progress, not a polished product. Bugs, PRs, and "this didn't work for me" issues all very welcome.

Known rough edges: face emoji rendering is missing visual differentiation for 4 of 9 emotions (sad / surprise / love / laughing); sound-direction localizer has a hardware-AEC-related left-bias on M5Stack CoreS3 (energy detection works, direction is unreliable); kid-voice ASR on the SenseVoice CPU default still garbles some short utterances (WhisperLocal, auto-selected on GPU hosts, handles high-pitched kid speech better).

Where it's at: the first stretch of this was vibe-coded pretty hard — I moved fast, chased ideas, and let the scope sprawl to prove out the whole end-to-end loop. That phase is over. I'm now deliberately pulling it back: deleting half-built features, trimming the docs, and hardening what's left so the core is solid rather than sprawling. Fewer things, done properly.

Dotty is a fully self-hosted voice stack for the M5Stack StackChan desktop robot. Open-source firmware on the robot, xiaozhi-esp32-server for voice I/O, and a local pi coding agent as the brain. ASR, TTS, and session state all run on your own hardware. The LLM is pluggable — the shipped default runs a small fast model for plain conversation and escalates hard questions to a more capable model, with llama-swap as the recommended local backend. Swap in Ollama for the simpler single-binary option, or point at OpenRouter / any OpenAI-compatible API if you'd rather use the cloud.

Out of the box, Dotty ships in Kid Mode — the persona and per-turn prompt sandwich steer responses toward age-appropriate topics, while a thin blocked-words filter replaces matching spoken turns before TTS. The filter is a bypassable backstop, not a safety guarantee or substitute for supervision. Disable Kid Mode for a general-purpose assistant.

I didn't like the idea of a camera and microphone running in my house unless I could (1) self-host the whole stack end-to-end and (2) understand the whole stack end-to-end. Off-the-shelf voice assistants fail both tests — audio leaves the house, the model is opaque, and you're trusting a vendor's privacy posture forever.

So Dotty is the version that passes: every component runs on hardware I own, every seam is documented and swappable, and the only thing that can leave the LAN is whatever LLM call I explicitly route out (and even that swaps to a local model with a config change). It's also meant to be fun — a friendly desktop robot for the kids, and an interesting hobby project to keep building on.

Yes — Dotty is built with AI assistance, and it says so out loud. Coding agents help write the code, draft the docs, and triage issues — a deliberate choice, not a secret. The rule: anything an AI agent authors is acknowledged as such (model-named Co-Authored-By: trailers, AI-assisted PR notes, marked AI-drafted docs), with a human reviewing and accountable for everything that lands.

Full policy: AI_TRANSPARENCY.md.

• Kid Mode (on by default) — age-appropriate responses via persona + per-turn prompt steering, backed by a thin blocked-words filter on TTS-bound output (#157). It remains bypassable and is not a substitute for supervision. Toggle off for general-purpose use. See docs/kid-mode.md.
• Local ASR — FunASR SenseVoiceSmall by default, no cloud transcription. WhisperLocal auto-selects on GPU hosts (better kid-speech accuracy); SenseVoiceOnnx is a lighter low-RAM option.
• Local or cloud TTS — Piper (offline) or EdgeTTS (cloud). Swap with a config change.
• Streaming responses — the bridge streams LLM output to the voice pipeline for lower perceived latency.
• Emoji expressions — every response starts with an emoji that the firmware maps to a face animation (smile, laugh, sad, surprise, thinking, angry, love, sleepy, neutral).
• Voice tools — the pi agent can search its memory, escalate hard questions to a bigger model, take a photo, and play songs, all mid-conversation.
• States, toggles & LEDs — a six-state mutex (idle / talk / story_time / security / sleep / dance) plus two orthogonal toggles (kid_mode, smart_mode), all owned by the firmware StateManager and surfaced on the 12-pixel LED ring. Shipped on the active firmware fork (commit d78118b, 2026-04-27); the firmware/firmware/ submodule pin in this repo lags, so flash from the active fork to get it. See "States, Toggles & LEDs" below and docs/modes.md.
• Vision (camera) — the robot's built-in camera can capture images for multimodal LLM queries.
• Privacy LEDs — hardware-bound mic (green) and camera (red) indicators on the LED ring. They light from the codec/camera enable signals via RAII guards, so a misbehaving server or model can't capture with the lights off.
• Calendar context — optional calendar integration feeds upcoming events into the conversation context.
• Hackable — every seam is swappable: LLM, TTS, ASR, agent framework. Fork it, rip out what you don't want, wire in your own.

Behaviour is a six-state mutex (idle / talk / story_time / security / sleep / dance) plus two orthogonal toggles (kid_mode, smart_mode), all owned by the firmware StateManager (shipped on the active fork in commit d78118b, 2026-04-27; bench checks tracked in #38). Voice phrases, camera edges, and dashboard controls all flow through it.

Note: the firmware/firmware/ submodule pin in this repo deliberately lags the active fork — flashing from the submodule won't give you Phase 4 yet. See the "Firmware iteration" section in CLAUDE.md and the submodule-pin caveat in docs/modes.md.

The 12-pixel LED ring shows the current state at a glance. Left ring 0-5 is the state arc — all six pixels paint the state colour, matching the dashboard's state buttons:

On the right ring, indices 8-9 are toggle pips for kid_mode (salmon pink) and smart_mode (orange), and index 11 (bottom) lights red while you have the turn (LISTENING). The idle → talk transition fires on face_detected from the firmware; VLM identity recognition runs in parallel and feeds the LLM context.

Heads up: that right-ring layout is the active-fork StateManager. On the firmware submodule pinned in this repo, pixels 6 and 11 instead drive the privacy LEDs — 6 = mic (green), 11 = camera (red) — and the StateManager pips arrive once the submodule catches up to the active fork.

Full state taxonomy, colour palette, transition diagram, and per-state backing architecture: docs/modes.md.

The dashboard service serves a web dashboard at http://<XIAOZHI_HOST>:8081/ui — host status, mode toggles (Kid Mode / Smart Mode), state switcher, perception card (face / identity), emoji presets, and a live event log (turns, perception events, errors). Light and dark themes follow the system preference. It runs as a small FastAPI service (bridge.py) on your own hardware — no external service ever sees your data.

• Hardware: M5Stack StackChan (CoreS3 + servo kit), firmware built from m5stack/StackChan.
• Brain: a pi coding agent running in the dotty-pi container. It runs qwen3.5:4b on local llama-swap for the conversation loop and escalates hard questions to qwen3.6:27b-think (also on llama-swap) via its think_hard tool. xiaozhi-server's PiVoiceLLM provider hands each voice turn to the agent. See docs/brain.md.
• Voice I/O: xiaozhi-esp32-server on Docker (any Linux Docker host).

The stack is the device plus four server-side pieces — xiaozhi-server (voice I/O), dotty-pi (the pi agent brain), dotty-behaviour (perception, ambient behaviour, and the proactive greeter), and the admin dashboard service. The four server pieces run as Docker containers on a single Docker host, alongside a local model backend (llama-swap, or Ollama for the simpler single-binary option).

Then:

• docs/quickstart.md — 15-minute happy path: flash, configure, first turn. Includes placeholder substitution table, deployment layout, endpoints, reboot survival, and common ops snippets.
• docs/troubleshooting.md — symptom-first lookup for common (and obscure) failure modes.

Questions, bugs, or "this didn't work for me" → open a GitHub issue. That's the best way to reach me. Occasional updates also go up on r/stackchan.

For what the stack is underneath — hardware specs, protocol docs, model facts, and features we aren't using — see docs/:

• docs/architecture.md — end-to-end data flow, topology, deployment files, admin surface, perception bus, threat model.
• docs/hardware.md — M5Stack StackChan body + firmware lineage + on-device MCP tool catalog.
• docs/voice-pipeline.md — xiaozhi-esp32-server internals, FunASR/SenseVoice, VAD, TTS.
• docs/brain.md — model matrix, the pi agent runtime, and how voice turns reach it.
• docs/protocols.md — Xiaozhi WS framing, MCP-over-WS, pi RPC, the dashboard HTTP API, emotion channel.
• docs/modes.md — behavioural mode taxonomy + LED contract + transition diagram (with shipped-vs-planned breakdown).
• docs/latent-capabilities.md — features upstream supports that we aren't using yet.
• docs/references.md — canonical upstream URLs, model cards, licenses.

• xiaozhi-esp32-server: https://github.com/xinnan-tech/xiaozhi-esp32-server
• xiaozhi-esp32 firmware (upstream): https://github.com/78/xiaozhi-esp32
• StackChan (hardware + open firmware): https://github.com/m5stack/StackChan
• Emotion protocol: https://xiaozhi.dev/en/docs/development/emotion/