See your tones. Hear what you actually said.

Dấu is an open-source practice lab that makes Vietnamese tones visible by drawing a learner's pitch over a DSP-validated reference target. It opens with the six meanings of ma, carries those tone shapes into connected dialogues, and uses deterministic signal processing to grade while a coach explains the physical correction and the meaning a wrong mark can create.

Demo video: Dấu - See your Vietnamese tones | OpenAI Build Week 2026

git clone https://github.com/roberthuynh/dau-tones.git
cd dau-tones
./dev.sh

Wait for READY http://localhost:5173, then open that URL. The script installs an uv-managed Python 3.11 environment and locked npm dependencies, warms the corpus-validation pYIN runtime, and starts FastAPI on port 8000 plus Vite on port 5173. Node.js 22+ and either uv or curl are the only host requirements.

Start in 1 Tone Shapes. The top rail puts ma, mà, má, mả, mã, and mạ in one lesson; select a meaning, listen to Thầy Minh, mirror Cô Dấu, then record one clear syllable. Phương, phường, and phượng follow under More words. No microphone is needed for the analyzer receipt: the committed sample controls cover correct má, má flattened into ma, and the signature Phương to phường mistake.

Continue to 2 Dialogue Practice for four linked scenes: Meet the family, Family dinner, At the phở shop, and Around the ward. The course contains 26 alternating turns and 13 substantial learner replies, preserves the scene, turn, and focus word in the URL, and links each changed token back to Tone Shapes. All 52 Northern and Southern scene WAVs passed lexical and signal checks, all seven story illustrations are committed, and one wrong-tone replay per scene closes the no-key feedback loop.

Cold-start receipt, 2026-07-19, commit 5e04444: a fresh public clone on Apple Silicon macOS with OPENAI_API_KEY explicitly unset printed READY in about 43 seconds after installing its own Python 3.11.15 runtime and locked dependencies. Health reported local DSP ready with the honest Northern and Southern four-family profiles; the check loaded all 19 words and six ma forms, streamed a validated target plus a Dialogue WAV, and passed all three offline Playwright stories in 8.7 seconds across all six release viewports.

No OpenAI key is required for browser grading, available validated target playback, committed word and scene art, deterministic coaching, analyzer demos, all 52 correct dialogue utterances, learner replay, or the four committed wrong-tone scene fixtures. With a key, server-only AI coaching and live Dialogue transcription, explanations, and optional mistake art turn on automatically:

cp .env.example .env.local
# Put OPENAI_API_KEY in .env.local. Never expose it through a VITE_ variable.
./dev.sh

The same monorepo is live at dau.huynhrobert.com as one Vercel project: Vite owns /, FastAPI owns /api, and the OpenAI secret remains scoped to Python. Live learner pitch grading stays in a browser Worker. Validated target and Dialogue WAVs are copied to web/public/audio/ and served with immutable static URLs, so practice playback never waits for a Python function. The native pYIN, SciPy, LLVM, and PyAV stack remains server-side for corpus validation, evaluation, and compatibility analyzer receipts, using Vercel Large Functions.

GitHub Actions runs only for pull requests and manual dispatches, preserving the free-plan quota while still providing a complete release gate. The same lint, type, coverage, audit, asset-validation, build, and offline end-to-end checks run locally before each published milestone.

The current gate runs Node 22 and Python 3.11, rejects secrets and unapproved model IDs, applies Ruff plus strict mypy to product and generation code, and requires 75% combined branch-aware API coverage. It also enforces web coverage floors of 65% statements, 50% branches, 65% functions, and 70% lines; audits both npm lockfiles and locked Python production dependencies; validates every inventory, classifier, evaluation, audio, image, and prompt hash; rebuilds the Vite app; and completes the offline Tone Shapes and Dialogue journeys at all six release viewports.

1. Open Tone Shapes, choose má, and press Listen + watch. Point out that the validated native contour traces left to right while Cô Dấu mirrors the physical rise.
2. Run ✓ correct má for the green acoustic-family verdict, measured local coaching, and visible next-drill reason. Then run má → ma · ghost to reveal mother versus ghost with both illustrations.
3. Run Phương → phường · ward for the signature name mistake and its level-versus-falling coaching cue.
4. Open Dialogue Practice, choose No key or no Vietnamese?, and show the ghost-at-dinner transcript diff, literal scene art, learner/correct replay, and the link back to Tone Shapes.
5. Press Practice again to prove the learner can repeat the same line without losing the scene, focus word, or accent; then use Continue scene to advance the story.
6. End on the Evaluation tables: the browser DSP currently supports 91.7% held-out acoustic-family accuracy. The native-speaker mả/mã contrast also shows the architectural point directly: transcription wrote both as mã, while DSP separated the dipping hỏi contour from ngã's 58 ms voicing break. The complete benchmark remains gated on two Southern phone references.

flowchart LR
    Mic["Microphone"] --> Decode["Main thread<br/>Web Audio decode"]
    Decode -->|"transfer PCM"| Worker["Web Worker<br/>browser-yin-v2 F0 + energy"]
    Profile["Static classifier profile<br/>36 validated templates<br/>version + corpus hash"] --> Worker
    Worker --> Grade["0.65 constrained DTW<br/>+ 0.35 feature distance<br/>+ close-pair cues"]
    Grade --> Shapes["1 Tone Shapes<br/>curve + semantic verdict"]
    Curriculum["Bundled curriculum<br/>six ma forms + 4 scenes"] --> Shapes
    Curriculum --> Dialogue["2 Dialogue Practice<br/>26 turns / 13 replies"]
    Static["Immutable static assets<br/>targets + Dialogue WAVs + art"] --> Shapes
    Static --> Dialogue
    Shapes --> Rules["Instant deterministic coach"]
    Shapes -. "async keyed refinement" .-> BotID["Vercel Basic BotID<br/>human-session verification"]
    Dialogue -. "keyed transcript + reveal" .-> BotID
    BotID --> API["FastAPI<br/>validation + safe errors"]
    API --> Guard["Upstash Redis<br/>quotas + leases + kill switch + cache"]
    Guard --> GPT
    Guard --> ASR
    Guard --> Images
    GPT["gpt-5.6-sol<br/>coach + next drill + explanation"]
    ASR["gpt-4o-transcribe<br/>Vietnamese token diff"]
    Images["gpt-image-2<br/>optional mistake scene"]
    Speech["gpt-realtime-2.1-mini<br/>bounded Dialogue speech"] --> Static
    Realtime["gpt-realtime-2.1<br/>5 target candidates"] --> Validation["Server authority<br/>librosa pYIN + lexical gates"]
    Validation --> Profile
    Validation --> Static

The latency boundary is deliberate. The main thread decodes the recording, transfers PCM to a Web Worker, and stays free to animate the interface while the Worker extracts pitch and grades. The verdict, first coaching instruction, and next-practice reason require no network request. FastAPI lazy-imports OpenAI and the heavy Python analysis stack only for keyed or compatibility routes. /api/analyze remains available for reproducible receipts, but it is not on the live microphone path.

Paid model calls cross two independent production gates. Vercel Basic BotID verifies that a protected POST came from the Dấu browser, then FastAPI uses atomic Upstash counters for per-client and global budgets, short concurrency leases, sanitized caches, and the dau:ai:enabled emergency switch. The Redis resource uses Upstash's free plan in Singapore with automatic upgrades disabled. If BotID or Redis cannot make a safe decision in production, the paid call fails closed while browser DSP, deterministic coaching, committed art/audio, learner replay, and offline Dialogue fixtures continue to work.

The credentialed admin script is the only kill-switch control; there is no public admin route. With the production Upstash variables present in the shell, run uv run --project api python -m scripts.ai_admin status, disable, or enable. The single persisted key is dau:ai:enabled, and model acquisition fails closed when it is disabled.

Raw learner audio, transcripts, prompts, request bodies, network addresses, and OpenAI keys are never written to the cache or structured logs. Responses calls set store=false; GPT receives bounded acoustic or curated mismatch facts rather than learner audio. OpenAI API data is not used for training by default. Transcription has no application-state or abuse-monitoring retention; Responses and image generation can retain abuse-monitoring data for up to 30 days, while Dấu itself persists neither recordings nor transcripts. Before the first microphone permission request, a versioned disclosure explains that Tone Shapes stays in the browser and that keyed Dialogue audio is sent only to gpt-4o-transcribe; the Privacy control reopens that explanation at any time.

1. gpt-realtime-2.1 speaks five Thầy Minh reference candidates for every word in Northern and Southern Vietnamese. The product calls the male reference teacher Thầy Minh; the provider voice ID stays only in generation receipts. gpt-4o-transcribe checks lexical identity, then the server DSP rejects acoustically invalid candidates before one take can become ground truth.
2. librosa.pyin extracts authoritative reference and evaluation F0, preserves voicing and RMS evidence, fills valid gaps, converts pitch to speaker-relative semitones, and resamples to 64 points. The production browser path removes octave spikes and computes the same contour and feature contract in a Worker.
3. The browser classifier ranks accent-conditioned templates with 0.65 × constrained DTW + 0.35 × robust feature distance. Energy and voicing cues break close hỏi/ngã, sắc/ngã, and huyền/nặng cases; a bounded trajectory-plausibility cue prevents a strong late recovery from being absorbed by a falling template. Signal confidence and class confidence are separate; weak audio or a weak margin abstains instead of asserting an accidental meaning.
4. gpt-5.6-sol runs only on FastAPI for structured coaching, next-drill selection, and Dialogue meaning explanations. The deterministic coach returns observation, one physical correction, next_word, and rationale immediately; GPT refinement can replace it asynchronously without delaying the verdict.
5. gpt-4o-transcribe runs server-side for keyed Dialogue transcription. gpt-realtime-2.1-mini generates one bounded Thầy Minh or learner-model utterance per scene turn, and gpt-image-2 generates committed story art plus optional live mistake art. Static generation is build-time; the browser never receives an OpenAI key.

Cô Dấu is the visual teacher, not the reference voice. In Tone Shapes she occupies the right teaching rail with a large face, lips, mouth close-up, throat cue, and contour-driven chin motion. In Dialogue Practice she demonstrates the selected focus word's contour throughout the line rather than assigning one guessed tone to the whole sentence; exact word-level timing is a future media-receipt improvement. Thầy Minh supplies the male reference speech and the partner lines.

The semantic layer is independent from the acoustic score. It emits six explicit states: exact_correct, family_correct, family_ambiguous, wrong_known_word, wrong_no_known_word, and uncertain. Only a supported exact or family assertion may name an accidental meaning. A same-family ambiguity is amber; low signal or a weak class margin asks for another take.

Audio language models are poor judges of pitch shape, so the DSP judges and the LLM coaches. Pitch grading is deterministic and inspectable; GPT-5.6 handles concrete instruction, drill choice, and meaning. The native-speaker contrast fixture already demonstrates the boundary: gpt-4o-transcribe preserved the syllable but normalized Northern mả to mã, while the shared pitch/voicing pipeline distinguished all three labeled hỏi/ngã pairs. The committed Stage 6 harness will measure the full audio-model comparison after the two Southern fallbacks complete the corpus.

Dialogue Practice is a course, not a live conversation session. Partner lines start only after an explicit user gesture, each learner reply receives an NFC-aware token diff, and changed words link back to Tone Shapes. Its four scenes are Meet the family, Family dinner, At the phở shop, and Around the ward. Optional wrong-scene art appears after the text diff and is keyed by a versioned prompt hash. Roadmap: a live Vietnamese conversation mode can build on these single-utterance pieces later.

Stage 0 is deliberately an all-or-nothing manifest gate. Thầy Minh produced five isolated takes and up to five carrier-phrase takes per word and accent; gpt-4o-transcribe checked lexical identity, then the shared DSP checked signal quality and expected contour. The current receipt accepts 36 of 38 pairs. A native Hà Nội speaker supplied Northern mả, mã, and phở: the importer promoted the best validated mả and phở takes and committed the independently validated mả/mã contrast as provenance. Only Southern mả and Southern phượng remain. No failed take is shipped as ground truth, and targets/manifest.json remains withheld until the complete 38-target audit passes.

Import the two remaining Southern replacements from the repository root:

uv run --project api python -m scripts.import_phone_targets \
  south/ma-grave=<PATH_TO_SOUTHERN_MA_GRAVE_WAV> \
  south/phuong-phoenix=<PATH_TO_SOUTHERN_PHUONG_PHOENIX_WAV>

For an ASR-ambiguous, native-speaker-labeled minimal pair, the importer also accepts an explicit contrast witness such as --contrast north/ma-grave=ma-code=<PATH>. Both recordings must pass their own DSP tone gate, use the same base syllable, differ in tone, and be different audio. If any lexical, signal, contour, hash, or inventory check fails, no invalid target is promoted.

The static browser profile is dau-browser-dsp-2.0.0, bound to partial-corpus SHA-256 ad0234e68352db66d00128d9417c5da1b6b480711f4bcf06d938cd3a60148ea0. Northern now has all 19 references; Southern has 17. The global receipt remains corpus_complete: false, so both accents stay in four-family mode until the two Southern imports pass and the complete evaluation gates decide whether Northern can promote to six-tone grading.

Active model IDs live in one API config module:

This task is the build log and scored Codex artifact. The repository is pushed as verified stages land so the history records the product being made, not a final code dump.

Estimated build-time OpenAI spend recorded in the ignored ledger is $15.07, below the $45 hard stop. The additional four-scene course, including 52 validated utterances, retries, lexical checks, and seven generated images, added about $3.54 to the earlier $11.53 receipt.

The browser-profile receipt below is a mixed-source reference regression, not learner-population accuracy. It uses 34 accepted generated references plus two native-speaker Hà Nội references and remains provisional while the Southern corpus is incomplete. Final product metrics still come from committed artifacts generated by python -m api.eval after the two Southern fallbacks pass.

Run the receipt after the validated target corpus is present:

PYTHONPATH=api api/.venv/bin/python -m api.eval

The final evaluator fits scales, confidence temperature, and abstention inside grouped leave-one-word-out folds. Northern six-way grading turns on only when accuracy is at least 0.80, macro recall at least 0.75, every-tone recall at least 0.60, hỏi/ngã mutual confusion at most 0.20, and every tone has at least three held-out words. Southern remains four-family scoring while all six forms and curves stay visible.

Provisional grouped held-out exact-tone matrix over the 36 accepted mixed-source references. Rows are intended tones; columns are top-1 detected tones.

The complete 38-target confusion matrices remain pending the two validated Southern phone imports. No score above is presented as learner-population accuracy.

Dấu is released under the MIT License. Be Vietnam Pro is self-hosted under its SIL Open Font License, included beside the font files.