I'd like to present MichiAI, a speech LLM designed for full-duplex interaction, allowing it to listen and speak simultaneously just like a human, while needing only a fraction of compute for training and inference. In this post I'll explain the speech generation component of MichiAI.

I am presenting this as a proof-of-concept. The GPT backbone is small by design to validate the architecture quickly, without a massive cluster of H100s.
It's not like I have that lying around in my garage anyway :)

Requirements

• Multimodal Input: It cannot be audio-only. The ability to prompt via text as well as audio is essential. Almost all existing LLM applications leverage RAG and text prompting, which is impossible with pure audio models.
• Coherence: The audio modality cannot degrade the overall coherence of the model.
• Low Time-to-First-Audio: Anything over 500ms breaks the flow of conversation. To feel natural, I targeted sub-100ms response times.
• Compatibility with Existing Text LLMs: Pretraining text models is prohibitively expensive. Since many high-quality open-weight LLMs already exist, utilizing their pretrained text knowledge is a massive benefit.
• End-to-End Processing: The model must "hear" the user and itself directly rather than relying on transcripts. Speech conveys more than just text; prosody, subtle timbre changes, and emotions provide valuable context for the model to interpret.
• Full Duplex: It must handle interjections and backchanneling implicitly, learned directly from the dataset.
• Voice Cloning: The model should be capable of cloning a voice and its specific speaking style from a short audio prompt.
• Paralinguistics: It should support paralinguistic effects like breathing or laughing implicitly.

State of the Art

The Model Pipeline Approach

The standard approach uses three distinct models: ASR -> LLM -> TTS.

Although popular and easy to implement, this architecture suffers from several significant issues:

1. Latency: The LLM requires the full input text to be transcribed before it can begin processing.
2. Turn Detection: Detecting turns is problematic. Most solutions use a simple VAD (Voice Activity Detection) model with a 1-2 second delay. This is naive, fails to model natural conversational flow, and adds to the overall latency.
3. Information Loss: By converting audio entirely to the text domain, you lose all of paralinguistic information that the model could otherwise utilize.
4. Short Context Windows: ASR models usually have very short context windows (e.g., a 30s for Whisper). This causes prediction quality to suffer in longer interactions.
   Example: If you introduce a specific name early on, the ASR model will forget it later, leading to transcription errors.
5. TTS Limitations: Standard TTS models typically require at least a full sentence to model speech correctly. Streaming TTS models often produce lower-quality output because text and audio generation are disjointed. The TTS model might "anticipate" different text than what the LLM provides, leading to discrepancies. The common solution is to generate speech in small "chunks" and stitch the audio together. This unfortunately introduces artifacts and increases latency because the TTS lacks the context of previous generations. Some TTS models might require a lookahead text buffer, also increasing latency.

Current Speech-to-Speech Models and Their Problems

Examples include Gemini Pro, GPT-Realtime (size unknown, but likely massive), Moshi (7B), Hertz-dev (8.5B), Sesame cms (8B), and Qwen Omni (7B).

There are a few key issues with these existing architectures:

1. Slow Decoding and High Inference Cost: These models generally rely on RVQ (Residual Vector Quantization) audio embeddings. For models like Moshi and Sesame cms, the system must predict 32 codebook entries for every 80ms audio frame. To decode a single spoken word, the model might require as many as 400 forward passes. Reducing codebook entries to increase speed drastically reduces audio quality.
2. Coherence Degradation: Adding speech context into the LLM input inherently introduces degradation into the model if not handled carefully. Recent research (Cuervo & Marxer, 2024) indicates that audio models with this architecture scale up to three orders of magnitude more slowly than pure text models. This suggests that to achieve the same level of syntactic and semantic proficiency, speech models require roughly three orders of magnitude more compute or data than text-based models.
3. Modality Fusion Issues: Despite being trained on speech input, existing models struggle to utilize the audio modality effectively. They often fail the "Am I whispering?" or "Am I screaming?" test, indicating they are processing the text content while ignoring the vocal delivery.

My Model

I opted for continuous rather than quantized (RVQ) embeddings to represent audio frames. While RVQ architectures are bottlenecked by the need to predict multiple discrete codebook entries sequentially, continuous embeddings allow the model to predict the same information in a single pass. This massive reduction in inference overhead is what makes real-time performance feasible on consumer hardware. Additionally, avoiding quantization artifacts results in much higher audio fidelity and allows for finer control over the latent space.

I employed a Rectified Flow Matching head to predict these continuous embeddings.
This approach ensures the generation is not only high-quality and diverse but also computationally efficient.

The audio embeddings are decoded into a raw audio waveform using a HiFi-GAN vocoder.
I designed the vocoder to be lightweight and causal to allow streaming which is critical for speech language models. Causal models are notoriously harder to train than non-causal ones and take more time to converge, so I'm not quite at ground-truth parity yet. The only way to improve this is to train it for more iterations to reach that level.

The 'Listen' head functions as a multi-modal encoder, mapping raw audio into continuous embeddings while simultaneously generating corresponding text tokens.

Predicted audio and text embeddings are looped back into the model. The system operates in full-duplex mode; at any time, the speak head and listen head can add more text embeddings and corresponding audio embeddings to the backbone.

The model achieves a time-to-first-audio of ~75ms. This was tested on an RTX 4090 using pure Python inference without any optimizations.

Solving the Degradation Problem

I addressed degradation issues through several strategies:

1. To ensure beneficial modality fusion, I specifically shaped the audio embeddings and trained the model end-to-end, allowing the encoder to produce high-quality outputs that don't confuse the LLM backbone.
2. Unlike other speech LLMs user input isn't encoded solely as audio embeddings; text tokens are also added. This saves parameters and increases coherence since the model doesn't have to decode audio embeddings into semantic meaning. Text embeddings are information-dense compared to audio embeddings alone.
3. High-Quality Labels: A precise match between text labels and spoken speech is crucial. I fine-tuned a Whisper model on a hand-transcribed dataset to ensure high transcription precision and formatting.

Voice Cloning

1. Voice cloning is "zero-shot". The model needs only a few seconds of audio to capture the vocal timbre with good precision. To capture the specific "way of speaking" and other personal quirks, longer input audio is preferred.

Coherence degradation

So far I'm not seeing any noticeable dips in reasoning or perplexity compared to the original text-only SmolLM.
Normally for speech LLMs of that size you wouldn't get any intelligible speech at all, let alone maintain the reasoning capabilities of the underlying LLM.
These results are currently based on internal perplexity tracking and qualitative playtesting, so the next step is to run formal evals like MMLU and Hellaswag to confirm this empirically.

Summary

Pretraining Dataset

• LibriTTS (200 hours)
• LibriVox (4.8k hours)
• SmolLM-corpus, Cosmopedia-v2

Dataset Preparation

I implemented the dataset generation pipeline using Luigi and transcribed the speech using a fine-tuned Whisper Large model.

To maintain the reasoning capabilities learned during text-domain pretraining, I mixed pure text samples into the dataset. I do this not only to preserve text-only comprehension but also because I want the model to handle "mixed" prompting. For example, a prompt might be half pure text and half audio. Furthermore, I want the model to be capable of generating and accepting text-only input for downstream applications. This approach is also helpful for Function Calling or Chain of Thought (CoT) generation.

For example, the model might say: "Hmm, let me think about it..." [CoT in pure text or mixed text/audio to prevent awkward silence] "...The answer is..."

For this sample, the model was provided with a pure text knowledge base (RAG input), then prompted via text to respond.
The model successfully synthesized the external information to generate a factually grounded audio response, demonstrating its ability to bridge text-based retrieval with spoken output.

Homographs

Model correctly pronounces homographs based on the context. For example, the word "lives" is pronounced differently in "Cats have nine lives" vs "She lives in New York".

Numbers, abbreviations, and units

Model learns to pronounce numbers, abbreviations, and units correctly based on the context.
Input text normalization is not needed as the model learns how to pronounce them directly from the dataset.

Demo

(Coming soon) Due to limited compute resources, I cannot host a live demo at this time.

Roadmap

• Larger LLM backbone.
• Train on a conversational dataset.
• Launch a Hugging Face Space.
• Integrate a multilingual dataset.

If you would like to contribute in any way or have questions, please contact me using the form here.