Zach Anderson
Nov 07, 2024 15:59
An in depth comparability of Common-2 and OpenAI’s Whisper fashions reveals Common-2’s superior efficiency in accuracy, correct noun detection, and decreased hallucination charges.
In a complete evaluation of main Speech-to-Textual content fashions, AssemblyAI’s Common-2 has emerged as a high performer when in comparison with OpenAI’s Whisper variants, in response to a current report by AssemblyAI. The analysis targeted on real-world use instances, assessing fashions on duties important for creating correct transcripts, corresponding to correct noun recognition, alphanumeric transcription, and textual content formatting.
Mannequin Comparability
The evaluation in contrast Common-2 and its predecessor Common-1 with OpenAI’s Whisper large-v3 and Whisper turbo fashions. Every mannequin was evaluated based mostly on parameters like Phrase Error Fee (WER), Correct Noun Error Fee (PNER), and different metrics vital for Speech-to-Textual content duties.
Efficiency Metrics
Common-2 achieved the bottom Phrase Error Fee (WER) at 6.68%, marking a 3% enchancment over Common-1. Whisper fashions, whereas aggressive, had barely greater error charges, with large-v3 recording a WER of seven.88% and turbo at 7.75%.
In correct noun recognition, Common-2 demonstrated superior accuracy with a 13.87% PNER, outperforming each Whisper large-v3 and turbo. This mannequin additionally excelled in textual content formatting, attaining a U-WER of 10.04%, which signifies higher dealing with of punctuation and capitalization.
Alphanumeric and Hallucination Charges
Whisper large-v3 confirmed power in alphanumeric transcription with the bottom error charge of three.84%, barely forward of Common-2’s 4.00%. Nevertheless, Common-2’s decreased hallucination charges have been a big benefit, with a 30% discount in comparison with Whisper fashions, making it extra dependable for real-world purposes.
Conclusion
Common-2’s developments over Common-1 are evident, with enhancements in accuracy, correct noun dealing with, and formatting. Regardless of Whisper’s strengths in sure areas, its susceptibility to hallucinations poses challenges for constant efficiency.
For additional insights and detailed metrics, the complete analysis is offered by way of AssemblyAI’s official report.
Picture supply: Shutterstock
