Quality improvements

Training on a conversational dataset plus architecture improvements led to a significant increase in speech naturalness and quality. Previously, I couldn't completely eliminate a slight metallic buzz in the audio. Inspired by the Mimi decoder, I switched to pure adversarial training, which solved the issue. Big shoutout to Kyutai Labs for their amazing work. The output audio is now very close to ground-truth quality.

Dataset preparation

Many speech models are trained on datasets that filter out "noise" like breaths, mouth clicks, and laughter. This often produces speech that sounds overly smoothed out, unnatural and audiobook like. I decided to embrace those natural audio artifacts and design the model and training pipeline to handle and leverage them. The result is a surprisingly realistic voice. The model transfers its pretrained knowledge really well so I didn't need a massive dataset this time as well.

Training

As with previous iterations, the model was trained strictly using a zero-coherence loss regimen. This preserved the backbone's language capabilities while learning from the conversational speech dataset. This time I selected the 135M SmoLLM backbone to accelerate development iteration times. While small, the zero-coherence loss allowed me to do meaningful development. Hardware-wise, training was primarily done on 1x RTX 4090, with 2x A6000s utilized for specific runs.

Optimization

The initial inference implementation had high Python overhead, which caused some issues with realtime streaming. I refactored and optimized the codebase so the model can be compiled end-to-end without graph breaks. This made streaming inference much smoother and increased GPU utilization.

Single speaker samples

I provide a few samples of the base model in action. Speakers are sourced from the LibriTTS dataset.

Two speakers samples

In these examples, input audio is fed to the model frame by frame, and the output is generated in real-time. The first speaker is generated using ElevenLabs TTS, while the second speaker's audio is generated entirely by MichiAI. The output audio speakers are also sourced from the LibriTTS dataset.

Smooth Replying

The model doesn't rely on hardcoded Voice Activity Detection (VAD) rules like waiting for 1-2 seconds of silence before responding. Because it operates in full-duplex mode, it decides naturally when to output speech. This behavior was learned organically from the dataset. In natural human speech, people often begin responding just as the other person is finishing up when it's obvious that the other speaker is yielding the floor.
This is called "overlap" and it helps conversations flow more smoothly. MichiAI mirrors this behavior seamlessly.

Backchanneling

The model naturally generates subtle backchanneling responses (such as "uh-huh," "yeah," or "I see") while the user is speaking. Rather than acting as full conversational turns, these short cues serve a couple of key functions:

• Signaling active listening and engagement without interrupting.
• Providing immediate, non-intrusive feedback during ongoing speech.
• "Soft launching" turn transitions, making conversational shifts smoother and less abrupt.

Humans backchannel subconsciously to maintain conversational flow. MichiAI learned this behavior organically from the training data without explicit prompts, turn-taking heuristics, or hardcoded rules. Ultra-low latency is critical here, as even a minor delay destroys the illusion and creates an awkward conversational experience.

Some existing systems attempt to simulate backchanneling using heuristic rules and TTS models. However, because standard TTS lacks conversational context, those implementations often feel out of place and also are limited to a hardcoded set of words. MichiAI handles backchanneling dynamically, producing both single-word acknowledgments and short conversational phrases.

Non-Verbal Sounds & Expressions

MichiAI can naturally express laughter, sighs, and gasps although it's difficult to bring them up in the base model.

Multi-Speaker Support

The underlying architecture inherently supports more than two speakers. However, I don't have a high-quality multi-speaker dataset to test this.

Function Calling

Similarly, the architecture fully supports function-calling. Since function calling relies on text only data, curating a dataset for this should be much easier than preparing audio data.

What's Next

Because this is a base model, it is quite untamed especially for such a small backbone. While it can maintain a simple conversation, it needs fine-tuning to be used in any downstream application.
The samples above were generated by feeding static audio files into the pipeline, but the system is built for live interaction. The next step is building a web client so people can test and speak to the model directly in real time.

Contacts

Thank you to everyone who has reached out regarding the project! My current bandwidth is a bit constrained so I may be slow to respond, but I'm catching up and will reply to everyone as soon as possible.