Lastivka: Teaching Gemma to Speak Ukrainian — an Offline Voice Tutor in One Model

Lastivka (Ластівка) — an offline voice tutor in your pocket

Gemma 4 E2B that speaks Ukrainian & English — fully offline, private, no subscription.

The problem (a personal one)

When the war started, my family fled Ukraine to the United States — my wife, my mother-in-law, my 15-year-old daughter, my five-year-old son, and me. Overnight, everyday life became a language exam you never studied for: a form at the clinic, a teacher's note, a word on a street sign. The hardest question was almost never "what does it mean" — translation apps do that. It was "how do I actually say this?" And that question always arrived at the worst moment: offline, in a waiting room, on a phone with no data plan, with a child asking now.

Existing tools fail exactly here. They need connectivity, they gate good voices behind subscriptions, and they ship your family's questions to someone else's server. A refugee's phone is often the only computer in the household — the assistant has to live on it.

The idea

Lastivka is a voice assistant that runs fully offline on a phone. You ask, in plain language — typed or spoken — "Say it slowly: Привіт, як справи?" and it answers out loud. No account, no network, no data leaving the device.

The technical bet that makes this possible: instead of bolting a separate TTS engine onto an LLM, we teach one Gemma 4 model to generate speech itself. Gemma 4 E2B is natively multimodal and accepts audio input, so the same model can eventually listen to the user's pronunciation and coach it — but the core capability built and proven here is language understanding and speech generation inside a single set of weights. The only extra component is a tiny streaming audio decoder.

Architecture: how Gemma 4 becomes a speaker

Gemma 4's tokenizer reserves a large block of unused vocabulary slots (<unusedN>). We map a 25 Hz LSCodec audio codebook (1024 entries) directly onto 1024 of these slots. Speech becomes just more tokens. Gemma 4 generates them autoregressively in the same stream as text, and a lightweight LSCodec decoder (a few MB, streaming-capable — it can start sounding before generation finishes) turns those tokens back into waveform.

So the runtime is one model + one small decoder. No cloud TTS, no second LLM.

Speech is exposed through a dedicated channel so text reasoning and audio never bleed into each other:

User: Скажи голосом швидко: Привіт я Ластівка
Model: <|channel>voice:uk:fast
       <unused65><unused556><unused225>…
       <channel|>

The channel encodes language and speaking speed (voice:uk:fast, voice:en:slow, …). For a learner, slow is not a nicety — it is the feature. The instruction wrapper is sampled across Ukrainian, English, and Russian phrasings so the model responds to however a real bilingual household actually talks.

Training Gemma 4 to emit audio without forgetting how to be Gemma is the real engineering. The recipe:

LoRA (r=32) on attention + MLP, plus embed_tokens as a trainable module so the new audio rows can actually learn.
A gradient mask that lets only the 1024 audio embedding rows update — the text/structural embeddings stay frozen, which prevented an early failure where the model lost its text↔audio switch and emitted garbled mixed output.
Manual lm_head re-tying to the trainable embedding copy. Gemma 4 ties input/output embeddings; the LoRA wrapper silently broke that tie, so audio tokens never learned to be produced. Re-tying was the single highest-leverage fix of the project.
Capability rehearsal: ~30% of training is ordinary Q&A so the model stays a helpful assistant, not a parrot. Lastivka still thinks — it just can also speak.

The dataset

Everything was built from openly-licensed sources, fully reproducible:

Real speech — Ukrainian: Yehor/opentts-uk (studio, 5 named speakers, Apache-2.0); English: parler-tts/libritts_r_filtered (LibriTTS-R, ~1.1k speakers, CC-BY-4.0). VAD-trimmed and LUFS-normalised, encoded to LSCodec.
Synthetic speech — prompts written by an LLM, voiced with a multi-speaker model (10+ voices per language — single-voice synth bakes one timbre into the codes; I learned that the hard way). Roughly a quarter of the Ukrainian synthetic set deliberately code-mixes English IT/loanwords (Slack, vscode, deploy) — that is exactly the register a working refugee family lives in and the studio corpus never covers.
Capability rehearsal — self-distilled Gemma Q&A for anti-forgetting.

The final v3 manifest is ~535k examples with long-form clips (up to ~30 s → 750+ codes), at 1024-token context. A verified collapse filter (drop clips >15 s with code-entropy H<6.5; 7,931 rows removed, 0 violations remaining on re-scan) keeps the encoder's rare silence/garbage outputs out of training — proven on the data, not asserted.

Challenges along the way

Catastrophic forgetting & the embedding tie. Documented above — the lm_head re-tie plus the audio-only gradient mask turned an unintelligible model into one whose words are clearly recognisable in both languages (~98% of generated tokens are valid audio at the end of training, consistent across runs).

Cost. I built an autonomous bid-host orchestrator for vast.ai: provision the cheapest interruptible GPU, fix the stale-SSH-key issue, bootstrap, resume from the last Hugging Face checkpoint, watch liveness, and migrate on eviction — unattended. A full run survived ~30 host migrations at ~$0.10–0.62/h with quality identical to an uninterrupted run. (This orchestrator deserves its own article — coming next in this series.)

On-device quantization — the hardest part, and the real breakthrough. Fitting the fine-tuned model into an 8 GB phone (iPhone 15 Pro class) without wrecking either text or speech took four recipes:

Recipe	Size	Outcome
uniform INT4 (channelwise)	2.8 GB	fast, but multilingual word-salad in text + audio dropouts (INT4 destroyed embed/lm_head) ❌
uniform INT8	5.1 GB	bf16-quality but won't fit RAM → ~1 token/min, dead ❌
v6-mixed (embed/lm_head INT8 + decoder INT4 channelwise)	4.05 GB	working: clean text, clean TTS — but large, residual artifacts ✅
v7-all-blockwise-32 (embed and FC all INT4, blockwise-32)	3.04 GB	🏆 perfectly clean text (zero stray `<unused>`, no salad), clean TTS, −1.1 GB, comfortably fits an 8 GB iPhone

The key insight: "INT4 breaks embeddings/lm_head" is only true for channelwise quantization. Blockwise-32 — an independent scale per 32 weights — recovers the precision those layers need, so the entire model can stay INT4 and still be both small and high-quality. That is the result that makes a single-model offline voice assistant actually shippable on a phone. The export goes to LiteRT-LM with weight patching (validated end-to-end on-device), plus a quantization-aware training tail so the model is trained to survive its own deployment quant rather than merely hoping it does.

What's public today

The full fine-tune — merged Gemma 4 E2B weights + LoRA adapter on Hugging Face.
The dataset & reproducible pipeline — manifest + every data-prep step.
The quantized on-device model — a single .litertlm file that already speaks.
A console demo — point it at the .litertlm, type скажи голосом: Привіт, and it produces speech. No GPU, no network. This is the exact artifact that will run inside the phone app.

The honest claim: the deployment path is done. The app is packaging, not research risk.

Current limits — and what's next

Full transparency about where the project stands:

The method is proven, the polish is not. Words are clearly recognisable in both languages, but pronunciation is not yet studio-grade. The two known levers are fine-tuning the audio encoder and scaling the speech data — every data increase so far moved quality up and the loss curves kept converging, so this is an engineering budget question, not a research question.
Training is paused on compute budget. The whole project ran on cheap interruptible GPUs; the next quality jump needs a longer run than the current budget allows. If you run open-model research grants or GPU programs (TPU Research Cloud and similar) — this project is fully open-data and reproducible, exactly what those programs exist for.
Next: the on-device app (iOS + Android) wrapping this exact model; using Gemma 4's audio input so Lastivka can hear a learner say a word and correct them; and additional language pairs trained the same way, one model per pair, all offline.

Why these choices were right

A separate ASR→LLM→TTS pipeline would be three models, three failure modes, and hundreds of megabytes of TTS — impossible to keep offline on a refugee's only device. Folding speech into Gemma 4's own token stream collapses that to one model + a few-MB streaming decoder, keeps every question on-device by construction, and inherits Gemma 4's reasoning for free. Ukrainian↔English is the first pair because it is my family's; the method is language-agnostic — the next pair is just data.

I built this because my family needed it. I'm sharing it so the next family that lands somewhere new with a single phone and no signal does not have to.