What is Whisper? OpenAI's open-source speech-to-text model that runs on your machine

November 17, 2026

Updated:

Guide
What is Whisper? OpenAI's open-source speech-to-text model that runs on your machine

What is Whisper?

Whisper is an open-source speech recognition model released by OpenAI, used to convert audio into text, translate speech into English, and automatically detect language, including Vietnamese. According to the GitHub API (read on 04/11/2026), the openai/whisper repository was created on 16/9/2022, has about 111,900 stars, is licensed under MIT, written in Python, and the most recent code push was on 31/8/2026. According to the model card in the repository, the model was trained on 680,000 hours of audio collected from the internet, a figure announced by OpenAI and not yet verified. This post answers: what Whisper can do, how many model sizes exist, what hardware is needed, how to install it, what its limitations and risks are, and when a solo worker should use it.

Whisper diagram: audio file goes through ffmpeg into log-Mel spectrum, Transformer model encodes then decodes into text, English translation or language code; beside it is a table of six model sizes from tiny to turbo and required VRAM
Whisper takes audio, processes each 30-second window and returns text. Diagram: OPCAIOS.

What is Whisper and who created it?

Whisper is OpenAI's multitask speech recognition model, released in September 2022 with source code and weights under the MIT license. A single model that does three things: transcribe speech, translate speech into English, and detect language, replacing multiple steps of traditional speech processing pipelines.

Technically, the repository's README describes Whisper as a sequence-to-sequence Transformer model. The encoder reads the audio spectrum, the decoder generates a sequence of tokens, and "special tokens" tell the model what task it is doing: transcribing, translating, or just identifying the language. Because of this training approach, Whisper also learns to detect whether there is speech in an audio segment. According to the model card (read on 04/11/2026), the training data consists of 680,000 hours, about 65% English, with the rest spread across 98 other languages. OpenAI also published the paper "Robust Speech Recognition via Large-Scale Weak Supervision" on arXiv (code 2212.04356). Release milestones according to the model card: original version September 2022, large-v2 December 2022, large-v3 November 2023, and turbo September 2024. The most recent release on GitHub has the tag v20250625 (26/6/2025).

What can Whisper do?

Whisper can do three main things: transcribe speech from audio files into text, translate speech in other languages into English, and detect the language being spoken. Users choose one of six model sizes depending on their hardware and required accuracy. The table below summarizes according to the README (read on 04/11/2026).

Feature Description from README
Transcribe Reads entire file, processes each 30-second window, returns text with timestamps by segment
Translate to English Use the --task translate option with multilingual model; turbo version does not support translation
Language detection detect_language function returns probability for each language
Six model sizes tiny (39M parameters, ~1 GB VRAM), base (74M, ~1 GB), small (244M, ~2 GB), medium (769M, ~5 GB), large (1550M, ~10 GB), turbo (809M, ~6 GB)
English-only versions tiny.en, base.en, small.en, medium.en: more accurate for English at smaller sizes
Command line and Python whisper command and import whisper library
Input formats Any format that ffmpeg can read: mp3, wav, flac, m4a, including video

The README notes that turbo is an optimized version of large-v3, about 8 times faster than large with slightly reduced accuracy; speed measured on A100 GPU with English, your machine will differ. For Vietnamese, we have not yet measured it ourselves.

Is Whisper free? License and hidden costs

Whisper is free: both the source code and model weights are under the MIT license, usable for commercial purposes. The real cost lies in hardware and time: large models need GPUs with sufficient VRAM, while running on CPU is much slower. The table below summarizes it (read on 04/11/2026, may change).

Item Details
Software fee 0 đồng, MIT license for code and weights
GPU Not mandatory but nearly needed for medium and above; VRAM from ~1 GB (tiny) to ~10 GB (large) per README
Server rental If you have no GPU, rent GPU servers by the hour; price decided by providers, we don't list it because we haven't read any official pricing table in this post
API key Not needed: Whisper runs entirely locally, no calls to OpenAI
Download size Model weights downloaded on first run, large version around several GB
Supporting software ffmpeg (free), Python 3.8 to 3.11, PyTorch; may need Rust if tiktoken prebuilt is missing

OpenAI also sells transcription service via API; we haven't read their pricing page so we don't mention it.

How to install Whisper and test run it?

Installing Whisper requires Python, ffmpeg, and one pip command. According to the README (read on 04/11/2026), the process has four steps.

  1. First, install ffmpeg because Whisper uses it to read audio files. On macOS:
brew install ffmpeg

On Ubuntu or Debian: sudo apt update && sudo apt install ffmpeg. On Windows you can use Chocolatey (choco install ffmpeg) or Scoop.

  1. Next, install the Python package from PyPI:
pip install -U openai-whisper

The README states the code is expected to run with Python 3.8 to 3.11 and recent PyTorch. If pip reports a tiktoken-related error, install Rust per official instructions then add ~/.cargo/bin to PATH; if setuptools_rust is missing, run pip install setuptools-rust.

  1. Then, transcribe your first file using the command line. The default model is turbo:
whisper audio.mp3 --model turbo

For Vietnamese speech, you should specify the language so the model doesn't have to guess: add --language Vietnamese. To translate to English, use a multilingual model like medium or large with --task translate; turbo will return the original language because it wasn't trained to translate.

  1. Finally, if you need to call it from Python code, the README gives a three-line example: whisper.load_model("turbo"), model.transcribe("audio.mp3") then read result["text"]. The first run will download model weights to your machine.

Example (hypothetical): a solo podcaster records 40 minutes each week, runs whisper ep12.m4a --model small --language Vietnamese on a laptop without a discrete GPU, waits several tens of minutes, then uses the transcript to write episode description and blog post. Actual time depends on the machine; we haven't measured it.

Limitations, risks and legal notes when using Whisper

Whisper has three main limitations per OpenAI's own model card: it can "hallucinate" text not in the audio, accuracy varies across languages, and it's not suitable for real-time transcription out of the box. Users need to review the transcript before publishing or using it as evidence.

More specifically, the model card (read on 04/11/2026) states: output text may contain sentences the model makes up, especially in silent sections or noise; error rates are higher for languages with less training data; errors can vary by accent, gender, age; and the model sometimes repeats a phrase many times, which beam search and temperature adjustment only partly reduce. OpenAI also lists out-of-scope uses: transcribing someone else's recording without their consent, inferring human characteristics from voice, and use in high-risk decisions where errors could cause serious harm.

On personal data: the plus is Whisper runs locally, audio never leaves your machine, unlike cloud transcription services. But meeting recordings, customer call recordings are still personal data of others. Ask permission before recording and transcribing, store files with control, and delete when no longer needed. This section is for reference, not legal advice, and we haven't checked specific Vietnamese law or GDPR text.

The project releases infrequently (latest release 6/2025 per GitHub), so don't expect new features soon.

Whisper versus other options

Original Whisper is the reference model; many popular tools actually run Whisper weights themselves but optimized. The table below is a general description, not our own measurements.

Option How it differs from original Whisper Good for
Original Whisper (openai/whisper) Reference implementation in PyTorch, easy to understand, easy to modify People who want to try quickly, Python programmers
faster-whisper, whisper.cpp Same weights, runs faster or runs on CPU and Mac People who need speed or don't have NVIDIA GPU
OpenAI transcription API Pay per minute, no need for powerful machine, audio sent to server People who want simplicity, willing to send data
Commercial transcription service (with interface) Has editing, speaker separation, collaboration People who don't know how to code
Other voice models (open source) Different architecture, Vietnamese quality needs your own testing People who want to compare

In this same series of posts, we also write about Voice Pro and AI Video Transcriber, two tools with interfaces built on transcription models.

When should you use Whisper?

Use Whisper when you need to transcribe Vietnamese or English in large quantities, want data to stay on your machine, and are willing to manually fix some parts. Don't use it when you need real-time subtitles or there's no one to review the transcript.

Situation Recommendation
Transcribe your own podcasts, YouTube videos to write articles Use Whisper size small or medium, specify --language Vietnamese
Internal meeting notes, want to keep data on your machine Use Whisper, ask participants for permission first
Real-time subtitles for livestream Not suitable; model card says Whisper is not ready for real-time
Transcribe hundreds of hours, no GPU Consider faster-whisper, whisper.cpp or rent a GPU server
Don't know how to code, need an interface Use an app built on Whisper or a commercial service
Transcript used as legal or medical evidence Not suitable; model card classifies it as out of scope

OPCAIOS experience

We added Whisper to the Repo list in the Utilities section on opcaios.org after reading the README and checking that the repository is still being maintained; we haven't tested it yet, so this section outlines the criteria we use to consider it, not an evaluation after using it.

The first criterion is whether the repository is still being updated: the most recent code push was 31/8/2026 according to GitHub API, the latest release in June/2025, meaning the project is in maintenance status rather than active development; for a model that's already stable, this is acceptable. The second criterion is the license: MIT for both code and weights, suitable for incorporating into internal processes or products. The third criterion is hidden costs: no software fees or API keys, but needs GPU or waiting time; we do use a rented GPU server for our OmniVoice video generation workflow, so Whisper could run on it when we need transcription. The fourth criterion is whether we can replace tools we're already using: currently we transcribe using services built into AI tools; Whisper is attractive because it keeps audio on the machine. Finally, there's risk: no issues with platform terms, only risks like hallucinations and personal data as mentioned.

Read more: What is Ollama? to run language models locally on your machine, Turn YouTube video into 10 blog posts, What is n8n? to automate transcription steps, 6 best speech-to-text tools in 2026.

Frequently Asked Questions

Does Whisper support Vietnamese?

Yes. Whisper is a multilingual model, Vietnamese is in the language list in the tokenizer.py file of the repository, and the model card publishes error rates by language on Common Voice 15 and Fleurs. However OpenAI clearly states that accuracy is uneven across languages and lower for low-resource languages. With regional Vietnamese accents, specialized terminology, or background noise, you should use medium size or larger, specify --language Vietnamese, and review the transcript. We have not measured Vietnamese quality numerically.

Does Whisper require a GPU?

Not required, but highly recommended if using medium, large, or turbo sizes. The README states VRAM needs around 1 GB for tiny and base, 2 GB for small, 5 GB for medium, 6 GB for turbo, and 10 GB for large. Running on CPU is possible with smaller sizes, just much slower. If your machine doesn't have an NVIDIA GPU, you can rent a GPU server by the hour, or use optimized versions like whisper.cpp which run better on CPU and Mac machines. Actual processing time depends on your machine and language; the README only gives relative numbers measured on an A100.

Does Whisper send my audio to OpenAI?

No. The openai/whisper repository runs entirely on your machine: model weights are downloaded once, then all computation happens locally, no API key is needed, and there are no calls to OpenAI servers. This is the main difference from OpenAI's paid transcription API, where audio files are sent to the cloud. If you use Whisper through a third-party application, check where that application sends data, since their policies are not covered by Whisper's MIT license.

How does Whisper differ from faster-whisper and whisper.cpp?

Original Whisper is the PyTorch reference implementation released by OpenAI. faster-whisper and whisper.cpp are community projects using the same weights but rewriting the model execution to be faster, use less memory, or run on CPU and Apple Silicon. For transcription quality, they share the same foundation so they should be equivalent in principle, but segmentation and default parameters may differ. This is a general description, not our measurement; if you need speed, try both on your own files.

Can someone without coding knowledge use Whisper?

Yes, at the command line level: after installing ffmpeg and running pip install -U openai-whisper, you just need one command whisper file.mp3 --language Vietnamese to get a transcript in txt, srt, or vtt format. If the command line is still a barrier, choose an application with a UI built on Whisper, or ask an AI agent like Claude Code to install and run it for you, then review the result. This approach, using AI to operate technical tools without writing code yourself, is what we teach in OPCAIOS's AI application training course.

Source: GitHub openai/whisper (README, model-card.md, API, release page, read 04/11/2026); not read/not verified: Whisper introduction blog post on openai.com (page returns 403 error when accessed), OpenAI transcription API pricing page, arXiv paper 2212.04356; star count and pricing may change. Experience is from OPCAIOS.

Translated from Vietnamese with AI. Read the original.

Want to apply this in your business?

Book a free 30-minute call (Vietnam time, in English or Vietnamese).

Book a call