I'd love to know how this handles proper subtitle formatting. I'm in the process of learning many languages, and being able to cross check my own understanding with film and video would be fantastic.
Hi, I am making an app for language learning which generates subtitles to help with this exact problem. Its called SubSmith and free to try, would love some feedback on it!
That's really impressive! It says it's available in GBoard on Android but I don't see it yet. I assume like most Google features it will be a rolling release over the course of several months, can't wait to try it some day.
It is confusing. As far as I can tell "rambler for Android" is a named subfeature of gboard and only on Pixel 10+ and newer Samsung phones. From a user standpoint it "just" seems like the transcription has gotten better - not an app you use or a feature that you enable.
I am not sure if "Word Error Rate" captures what has always been wrong with transcription. My biggest complaint is that it inserts sentence breaks in random places, then fails to evaluate the result, even though it is obviously wrong. Then I have to go fix it which can be harder than having just typed it myself, due to the difficulty of positioning the Android cursor, the fact that it automatically capitalizes if you delete a capital letter, etc. And much of the time I fail to notice the errors until later.
have another model do a pass to clean it up, saw a demo of local STT where someone did this, can fix a lot of things, especially with gotchas for the STT model in a clean-transcript.md
I think the model can even evaluate itself. If it looks afterward at an output like "do you. Want to get lunch?" in the absence of affirmative evidence that the user wanted it that way, it should be able to see that it goofed.
Curious if this suffers from the same hallucination problems as Chirp or not. For example, if you pass chirp some audio with noise or even no audio, it will barf text at you like "I don't know. I don't know. I don't know." until a request timeout fires after like 10 minutes. It's... really bad. For this reason, we've moved back to Whisper for timestamp accuracy and then Gemini Flash for transcription correction.
I've seen Whisper do this exact thing as well. And also repeating a few words over and over near the end, words that were said but not completely at the end. It seems to overwrite the last few sentences with that so actual content is lost. It's really annoying especially because Whisper is really quite good otherwise. Sometimes a friend sends me audio messages from a noisy car and I can't make it all out but whisper gets it mostly right.
I eval'd this pretty heavily with no audio or just noise, as for us hallucinating a response is very bad. It works great in this scenario. There are some bugs, e.g. randomly exactly 20 seconds of silence will lead to a 403, but I'm sure these aspects will be improved over time. I didn't get any hallucinations though.
I’ve noticed the some of YouTube’s auto-generated captions will sometimes hallucinate “Heat” during long music segments; probably an older model though.
Yes, that's sort of our strategy as well: for us accuracy (and following a style guide, and extracting on-screen text from video) is important. So we do a first pass with whisper to get the word timestamps, and the pass the same audio (chunked in case of longer ones) to Gemini (pro, not this model), to transcribe without timestamps. After that you can transfer the timestamps from whisper to the timestamp-less Gemini text, by looking for runs of identical words in both and fudging timestamps for the other words I'm between a bit. Works quite well, and I'll be tesing this model to see if it can replace Whisper.
I've been testing it on Pixel 11 Pro and I mostly dislike it. It is convenient when you have something long to say without thinking about it first.
But the main issue is when you want to say something precise with specific wording it might "simplify" it and break the meaning.
Something like "I hesitated to check it, I should have verified" => "I should have verified" (The "I hesitated..." is removed but I said it because I wanted to let the person know that I thought about it earlier)
I tried the sentence few times and it always removed the first part.
I’ve tested at least 20 STT models in a benchmark I’ve set up with German, Italian and English voices from meetings in my company.
The voices contained very industry specific words, the languages changed from one sentence to another, sometimes words in a language were mentioned while a discussion was in another.
The only local model that satisfies me is Voxtral Mini 3b, the only paid API that is slightly better is eleven labs. Yes, Voxtral might not reach the best score in the benchmarks, but to me, it just solves a problem. It might not be the best in terms of speed... but that’s not a problem for me.
Happy to test this new model from Google but I’m not sure I’d go with that instead of something that can run so easily in my machine.
Of course not, but it’s easily replicable just by mixing different languages conversations together, adding a word here and there of some very specific German jargon.
Agree about Voxtral being the only model — local or cloud — that handles multilingual conversation really well. I‘m not sure what they do differently, but mixed-language sentences and industry terms don‘t seem to faze it where other model begin to struggle.
I love it but it misses the business specific words when in different language. Sometimes it stretches them out to fit unrelated words in the language of the rest of the conversation. I miss its speed though.
Do you use it on a desktop? Mac by any chance? What's your setup?
I've been looking to find a simple and fast dictation app for English but almost everything I've tried (from Handy to many apps, eg some with Whisper in their names, after the model I assume) just don't work well. Apple's offering is worse than those though. I even tried with local enhancement models.
I am biased as a developer on the project but you should give Epilude [1] a go if you want local dictation on a Mac.
We’ve fine-tuned open-weight models to make them better (in our benchmarks) at cleaning up and formatting what you say so you don’t have to edit what you dictate.
This problem is legitimately hard and needs high cognitive abilities. Even the biggest generalist models struggle with memes and lingo salad that sound immediately intuitive for an out-of-the-loop human, and I'm talking about text comprehension. Modern models are optimized for decision making and are worse in that than old ones optimized for creative writing, but those also struggled.
You should probably not expect a small STT model like Voxtral or Parakeet to do any better, unless it's laser focused on that area in particular and sucks in everything else.
What i want is a model that outputs its predictions and their scores along with the text it choose. So I could flag something it's getting consistently wrong, like mis-predicting a technical term or name, or acronym, and say replace it with my correction, and have the ui be able to smartly replace that in the whole text so far and future parts. even better would be the ability to feed this back into the model for future runs.
Good news: most of these models can include a prompt that steers the transcription; if you use frequently a word you just invented, add it there and it will be transcribed correctly more likely.
It's interesting that the results can be so different depending on the person, the use case and even the microphone used. I use dictation a lot, so I try to stay up to date with the latest models as much as I can. So far, for my needs, nothing could beat Whisper Large v3. I keep hearing that Nvidia Parakeet models are better, but they just don't work as well for me, even though they are unquestionably faster.
Things change a lot if you need to speak to the model in multiple languages. There are very few models out there that can automatically detect the language spoken and produce correct output.
I tried the larger Voxtral models, but they didn't work for me at all. When I spoke to them in Polish, they produced output in Russian or Ukrainian.
For now, I settled on creating my own plugin for TypeWhisper, which runs Whisper Large on the GPU and does it much faster than pretty much anything else out there. But I'm still hoping that something better will come along, as Whisper Large is quite old at this point.
I wonder if cross training models specifically on people who combine languages (like Spanglish) would help with this. Surely must be patterns in what words people choose to use in each language
I did the math based on our data from customers in https://whispermemos.com/ and it seems to be $0.12 per hour of input audio, with our average tokens/sec.
Scribe is $0.22.
If the accuracy is close to Scribe, I think it's a good deal.
I've been using Wispr Flow on my pixel 9 and I like it. It does the things that Rambler says it's supposed to do but it's available on my Pixel 9 unlike rambler that's currently only available for pixel 11.
I'd certainly use Gemini API a lot more if I could figure out how to give them money and they give me tokens. The whole tier nonsense is just insane hoop jumping compared to their competition.
I would also like to see benchmark for translation. I'm looking for live translated subtitles so my Japanese wife can enjoy any show with out waiting months for official VOD streams to release them.
Still no real time diarization beyond 3 people (and even then experimental) when others do it very well, like Soniox and Deepgram. For something like meeting notes this is critical. Not sure what the issue is to implement it, maybe that's not Google's use case in mind and rather it's about personal Rambling as the feature on Pixels shows, which uses this model.
Yes, I specifically avoided mentioning that because they don't need to diarize from audio directly as they know exactly which participant is speaking at all times.
"Function calling: The model can delegate complex tasks (such as image generation and file analysis) to other Gemini models via function calls. Currently available in the Gemini macOS app."
This confused the heck out of me because it makes it sound like the STT model can make function calls in order to execute arbitrary tasks, which wouldn't make any sense. The developer docs (https://ai.google.dev/gemini-api/docs/models/gemini-3.5-tran...) confirm that the Gemini 3.5 Transcribe model cannot in fact make function calls. I guess the blog post is just using very confusing wording to describe how their consumer assistant/chatbot app can both take audio input via Gemini 3.5 Transcribe and then call other things as needed. Or maybe that bullet point was meant to be for a different model announcement and somebody made an editing error.
Maybe I missed something, but isn’t it impossible to detect whether something is written by AI or not? A human on a bad day can write like AI while an AI on a good day can write like a human.
No, that's not correct for any reasonable definition of "impossible." Look up pangram's accuracy ratings. It's not perfect, but it's pretty good. LLMs in fact leave very distinguishing traces of their logit distributions in the text they write. It's one of the reasons why it's so easy for humans to also smell them.
It is possible to trick pangram - they bias toward a low false positive and a higher false negative - but it is not true that it is essentially random.
> In preliminary testing, Mantzarlis found Pangram was more likely to misclassify AI-generated text as human-authored when it rhymed, repeated itself, and when it used archaic language. He then built an adversarial set of 588 AI-generated text samples tailored to these weaknesses. When he used Pangram to evaluate them, the tool falsely labelled AI text as human 86% of the time.
> “I don't think that Pangram is bad,” Mantzarlis said. “I think actually Pangram at scale is probably a pretty solid tool. That said, I am extremely worried about it being used in individual cases.”
Why? Does TFA rhyme, repeat itself, use archaic language or otherwise looks adversarial? It doesn't, so we can assume Pangram's usual false positive/negative rates apply. Yeah there's a chance it's wrong, and they'll need to catch up to new models, but my instinct can be wrong too and I don't stop using it to filter what I read; at least Pangram's accuracy can be measured.
> Why? Does TFA rhyme, repeat itself, use archaic language or otherwise looks adversarial? It doesn't, so we can assume Pangram's usual false positive/negative rates apply.
No it literally does not apply is the point of the article. Please read what it is about and what it says instead of asking for spoon-feeding.
> but my instinct can be wrong too and I don't stop using it to filter what I read
Again, your instinct is not something that matters to anyone other than you. But you are presenting Pangram as fact and doing on a moral crusade (I WILL NOT READ ANYTHING PANGRAM SAYS AS AI). You can also have an instinct that "THIS IS WRONG" and go on crusade but you will naturally understand your foundation is not solid at all.
Lastly, if your instincts serve you well why are you outsourcing yourself to another instinct? Is it for yourself or to say to others "LOOK AI CONTENT LOOK AI CONTENT!!!"? Is that purely to serve your interests of filtering what you read or are you using it in the wrong way here?
The text you linked simply says that for individual analysis instead of bulk one, there will be false positives. I already acknowledged that, and I still need some filter anyway whether you want me to have one or not. Mistaking your blog posts for an AI under a fairly low false positive rate is a sacrifice I'm willing to make; I'm not grading college students here.
Also, I'm not presenting Pangram as anything, much less said what you just claimed I said. You might be mistaking who you're talking to in this thread, either way you clearly aren't debating in good faith.
What’s impossible is what a normie would understand that it does based on their marketing and home page.
It’s trivially defeated though, and anything with a false positive rate shouldn’t be used by any serious institution on a decision making basis. For general stats, sure. For trying to punish and individual, no thank you.
Checking the following with another AI checker and it says 100% AI generated. (https://originality.ai/). We should run each check on multiple checkers if you want to provide a substantive claim that something is AI-written or not. This habit of just putting text in a "AI text checker" and then treating whatever it says as the truth is absolutely one of the worst things to emerge in recent times. You should check out the CEO of Pangram who uses this to go on witch hunts on X and even though the app claims "our results do not reflect reality 100% and can be false", he always uses his stupid checks as "LOOK THERE IS PROOF YOU ARE AN AI WRITER" and he points it at legit journalists etc. This whole company is honestly such a piece of shit that I as a human writer think is going to kill art more than AI writing does. I just hope people see it for the nonsense it is sooner than later.
---
Experience smart transcription and advanced dictation
In addition to the Gemini API in the Google AI Studio and Gemini Enterprise Agent Platform, 3.5 Transcribe goes further than standard speech-to-text to make working across Google feel more natural and intuitive. By bringing context-aware understanding directly into everyday surfaces like Gboard, Antigravity, the Gemini app, and Chrome, it captures nuances, intent, and inline edits with ease.
On Gboard on Android, through the new Rambler feature, 3.5 Transcribe transforms spoken thoughts into well-formatted text, filtering out filler words. You can also use your voice to make edits, correct misspellings, and change the writing style.
On Google Antigravity, 3.5 Transcribe pairs screen context and chat history, with your permission, to ensure pinpoint transcription accuracy across file names, agent thoughts, and active documents.
In Google AI Studio, you can access 3.5 Transcribe in Build mode to vibe code apps with your voice on the fly.
In the Gemini app on macOS, 3.5 Transcribe not only transcribes your free natural speech into clean formatted text, but also enables voice commands that can pair seamlessly with screen context to power complex workflows. By calling on other Gemini models in the background to handle the heavy lifting, the model makes it effortless to summarize local files, repurpose text across apps, or generate images right at your cursor—using just your voice.
Coming soon to Chrome, you’ll be able to talk to type in any web field — making it effortless to dictate replies, draft posts, or prompt Gemini in Chrome more naturally and easily with your voice.
I'm looking at originality.ai and I'm suspicious. Their roundup of studies is largely ones from 2023 (and I don't think any AI detectors from 2023 actually worked), with a few newer ones that nonetheless don't include Pangram among the detectors that they're comparing (which I consider highly suspicious and possible evidence of cherry-picking). A quick internet search surfaces various anecdotes suggesting that originality.ai has a high rate of false positives compared to other detectors (and in particular doesn't highly prioritize making false positives rare).
I'm open to the idea of different norms for citing AI detectors, but someone needs to propose what they should be and they need to make sense.
Remember that function calling doesn’t mean that the model is actually calling a function for you: what it does it interpret the intent, then return back to the application the function and arguments, then the application runs the function with those arguments.
Think voice control for your device. You speak, and it returns back instructions ( a function and arguments) for your device to be execute.
STT models just turn audio into text, though; they don't interpret what the text means or figure out how to translate it into other representations like function calls. You would use a general-purpose model for that.
I personally tested all the STT models for my real-time translator (https://fliptalk.ai). From language detection and accuracy in a noisy environment to the most important point: latency.
At the moment, Soniox STT v5 is definitely the best, and I'm impressed by its performance. It's good that Google released Gemini-3.5-Transcribe, and it beats every other model on accuracy, but it definitely needs a bit more work on latency, which is the most important factor for STT apps.
For me, Gemini-3.5-Transcribe actually has slightly lower latency. Kudos to Soniox for both paying their competitor and letting them win. But yes, Soniox is much cheaper.
Interesting, I uploaded a voice recording from a meeting I had recorded with a relatively cheap microphone.
Soniox came out really good. OpenAI started getting some things very wrong and even introduced some German. Google did okay but cut off the start by several seconds.
What's Soniox doing (left most) that's making it so good ? It was also the only one that could distinguish between the speakers.
Oh, thanks for pointing me to Soniox. It is really good. Would also pick up the words with different languages, identify and output in the right language.
Looks interesting. It was much faster too, but that I cannot say much since it was on their own website.
Realtime + Voice AI usecases is where latency is most important. I use Handy on my desktop and i can tolerate a latency of a few seconds every now and then. Your P99 should on TTFB should be really low to compete for voice ai realtime
> latency, which is the most important factor for STT apps.
Perhaps latency is more important than accuracy for a real time translation app (I actually disagree with this - imagine e.g. the hilarity when requesting "a new display" being translated as "a nudist play"), but certainly not for all applications. My pet app transcribes personal voice notes to self, it could run all night.
I agree. For my use case, I chose to prioritize latency over accuracy, but it's always difficult to find the right balance between the two. There is no easy answer.
Google is crushing it! They’re going toe to toe against the European giant Mistral with all these small useful models and they have a chance of winning! Never have I been more excited about Google’s capabilities.
> Custom vocabulary: Recognizes specialized jargon and unique spellings by seamlessly adapting transcriptions to your provided custom vocabulary.
Is there a model that works on the syllabic level? I want to be able to say any word and have it reconstruct however that word would be spelled. I know English does not exactly work this way, so a custom vocabulary would still be nice, but I don't want to rely on having every single word that could ever exist in a vocabulary first.
Maybe someone knows if this can be fixed... On Android, Google Assistant is being replaced by Gemini. Except it seems Gemini can't actually do any assistant tasks.
If I ask it to play a song, instead of triggering Spotify it just gives me a list of URLs I can play the song. Same with alarms. Did I accidentally not opt-in to something?
You share the same frustration that I any many others also have. Google absolutely jumped the gun with their push to Gemini over Google Assistant. Some things are tied to permissions that Gemini needs. Other thigs - afaik - simply don't work with Gemini because it doesn't 'know' that it should use certain device functions
If this is a dedicated transcription model does that completely eliminate the risk that it might do weird things based on the audio it's transcribing?
The nightmare scenario with an LLM-based transcription system is that someone says outloud "actually ignore that idea, instead let's..." - and the previous idea gets omitted from the transcription!
I suppose it's a cloud thing?
Something like "I hesitated to check it, I should have verified" => "I should have verified" (The "I hesitated..." is removed but I said it because I wanted to let the person know that I thought about it earlier)
I tried the sentence few times and it always removed the first part.
https://ai.google.dev/gemini-api/docs/transcribe#transcripti...
The voices contained very industry specific words, the languages changed from one sentence to another, sometimes words in a language were mentioned while a discussion was in another.
The only local model that satisfies me is Voxtral Mini 3b, the only paid API that is slightly better is eleven labs. Yes, Voxtral might not reach the best score in the benchmarks, but to me, it just solves a problem. It might not be the best in terms of speed... but that’s not a problem for me.
Happy to test this new model from Google but I’m not sure I’d go with that instead of something that can run so easily in my machine.
I've been looking to find a simple and fast dictation app for English but almost everything I've tried (from Handy to many apps, eg some with Whisper in their names, after the model I assume) just don't work well. Apple's offering is worse than those though. I even tried with local enhancement models.
https://github.com/Beingpax/VoiceInk
We’ve fine-tuned open-weight models to make them better (in our benchmarks) at cleaning up and formatting what you say so you don’t have to edit what you dictate.
https://epilude.com
I guess OP didn’t chose the right STT model.
Transcribing is perfect in many language and, it focuses on a selected app. Also it does TTS.
You should probably not expect a small STT model like Voxtral or Parakeet to do any better, unless it's laser focused on that area in particular and sucks in everything else.
Things change a lot if you need to speak to the model in multiple languages. There are very few models out there that can automatically detect the language spoken and produce correct output.
I tried the larger Voxtral models, but they didn't work for me at all. When I spoke to them in Polish, they produced output in Russian or Ukrainian.
For now, I settled on creating my own plugin for TypeWhisper, which runs Whisper Large on the GPU and does it much faster than pretty much anything else out there. But I'm still hoping that something better will come along, as Whisper Large is quite old at this point.
Transcribing is perfect in many language and, it focuses on a selected app. Also it does TTS.
Using a combi of the large Whisper and koroko/supersonic for TTS. Plugs into dev environments via MCP and hooks (Claude, Codex).
Scribe is $0.22.
If the accuracy is close to Scribe, I think it's a good deal.
It doesn't reach the frontier in either latency or accuracy for ai multilingual conversations.
I would also like to see benchmark for translation. I'm looking for live translated subtitles so my Japanese wife can enjoy any show with out waiting months for official VOD streams to release them.
That would be interesting when they also own Google Meet.
This confused the heck out of me because it makes it sound like the STT model can make function calls in order to execute arbitrary tasks, which wouldn't make any sense. The developer docs (https://ai.google.dev/gemini-api/docs/models/gemini-3.5-tran...) confirm that the Gemini 3.5 Transcribe model cannot in fact make function calls. I guess the blog post is just using very confusing wording to describe how their consumer assistant/chatbot app can both take audio input via Gemini 3.5 Transcribe and then call other things as needed. Or maybe that bullet point was meant to be for a different model announcement and somebody made an editing error.
It is possible to trick pangram - they bias toward a low false positive and a higher false negative - but it is not true that it is essentially random.
> “I don't think that Pangram is bad,” Mantzarlis said. “I think actually Pangram at scale is probably a pretty solid tool. That said, I am extremely worried about it being used in individual cases.”
https://reutersinstitute.politics.ox.ac.uk/news/human-wrote-...
Using it for an individual article to fully determine if its AI or not is "impossible" because you're not even using the tool properly.
No it literally does not apply is the point of the article. Please read what it is about and what it says instead of asking for spoon-feeding.
> but my instinct can be wrong too and I don't stop using it to filter what I read
Again, your instinct is not something that matters to anyone other than you. But you are presenting Pangram as fact and doing on a moral crusade (I WILL NOT READ ANYTHING PANGRAM SAYS AS AI). You can also have an instinct that "THIS IS WRONG" and go on crusade but you will naturally understand your foundation is not solid at all.
Lastly, if your instincts serve you well why are you outsourcing yourself to another instinct? Is it for yourself or to say to others "LOOK AI CONTENT LOOK AI CONTENT!!!"? Is that purely to serve your interests of filtering what you read or are you using it in the wrong way here?
Also, I'm not presenting Pangram as anything, much less said what you just claimed I said. You might be mistaking who you're talking to in this thread, either way you clearly aren't debating in good faith.
It’s trivially defeated though, and anything with a false positive rate shouldn’t be used by any serious institution on a decision making basis. For general stats, sure. For trying to punish and individual, no thank you.
---
Experience smart transcription and advanced dictation
In addition to the Gemini API in the Google AI Studio and Gemini Enterprise Agent Platform, 3.5 Transcribe goes further than standard speech-to-text to make working across Google feel more natural and intuitive. By bringing context-aware understanding directly into everyday surfaces like Gboard, Antigravity, the Gemini app, and Chrome, it captures nuances, intent, and inline edits with ease.
On Gboard on Android, through the new Rambler feature, 3.5 Transcribe transforms spoken thoughts into well-formatted text, filtering out filler words. You can also use your voice to make edits, correct misspellings, and change the writing style.
On Google Antigravity, 3.5 Transcribe pairs screen context and chat history, with your permission, to ensure pinpoint transcription accuracy across file names, agent thoughts, and active documents.
In Google AI Studio, you can access 3.5 Transcribe in Build mode to vibe code apps with your voice on the fly.
In the Gemini app on macOS, 3.5 Transcribe not only transcribes your free natural speech into clean formatted text, but also enables voice commands that can pair seamlessly with screen context to power complex workflows. By calling on other Gemini models in the background to handle the heavy lifting, the model makes it effortless to summarize local files, repurpose text across apps, or generate images right at your cursor—using just your voice.
Coming soon to Chrome, you’ll be able to talk to type in any web field — making it effortless to dictate replies, draft posts, or prompt Gemini in Chrome more naturally and easily with your voice.
I'm open to the idea of different norms for citing AI detectors, but someone needs to propose what they should be and they need to make sense.
Think voice control for your device. You speak, and it returns back instructions ( a function and arguments) for your device to be execute.
Gemini 3.5 Transcribe Live (Per 1M tokens in USD):
Gemini 3.5 Transcribe: [1] https://ai.google.dev/gemini-api/docs/pricing#gemini-3.5-tra...No need to go from audio to text to reasoning, just from audio to output json for running a command via adb automatically and it's working crazy good.
At the moment, Soniox STT v5 is definitely the best, and I'm impressed by its performance. It's good that Google released Gemini-3.5-Transcribe, and it beats every other model on accuracy, but it definitely needs a bit more work on latency, which is the most important factor for STT apps.
For me, Gemini-3.5-Transcribe actually has slightly lower latency. Kudos to Soniox for both paying their competitor and letting them win. But yes, Soniox is much cheaper.
Soniox came out really good. OpenAI started getting some things very wrong and even introduced some German. Google did okay but cut off the start by several seconds.
What's Soniox doing (left most) that's making it so good ? It was also the only one that could distinguish between the speakers.
I was using Cartesia, while their TTS is amazing their STT pricing has kind of irked me.
Interested in know how good soniox latency and EUD is on STT compared to Cartesia. Cartesia's is really in real world conversations
Is there a model that works on the syllabic level? I want to be able to say any word and have it reconstruct however that word would be spelled. I know English does not exactly work this way, so a custom vocabulary would still be nice, but I don't want to rely on having every single word that could ever exist in a vocabulary first.
If I ask it to play a song, instead of triggering Spotify it just gives me a list of URLs I can play the song. Same with alarms. Did I accidentally not opt-in to something?
The nightmare scenario with an LLM-based transcription system is that someone says outloud "actually ignore that idea, instead let's..." - and the previous idea gets omitted from the transcription!