Daves Additions

The way I eventually implemented this was a wrapper around the actual model call rather than around the whole whisper software. This basically means that we still keep the whisper chunking and feature extraction and I have added a reprocessing loop if a Hallucination is detected using the n-gram and compression rate thresholds as we have discussed. Now the only pre-processing done in the pipeline is VAD, which requires saving the result as a tempfile, see examples. There is already a simular reprocessing loop that seems to work well in the faster-whisper software. This mainly takes place in the fuction generate_with_fallback.Their reprocessing here works like so:

• By deafult temperature is set to a list of temperatures like [0.0,0.2,0.4,0.6,0.8,1.0].
• If a compression rate above the threshold is detected then it will increase the temperature to the next value in that list until an acceptable string is returned.
• This is simular logic to our reprocessing, but is not deterministic and so we need to explicitly call temperature=0.0

To maintain this functionality if needed I have placed the wrapper around this function meaning it will still occur if tempreature is not set explicitly. If we set temperature to 0.0 none of this reprocessing takes place and the text from the first pass is returned. Then we use our traffic light recovery to fix the hallu.

• In generate_segments the model tries to generate a sensible translation by calling generate_with_fallback.
• As described above when temp=0 this returns the first result without any reprocessing.
• Then if enable_hallucination_recovery is True i.e. we have the recovery pipeline turned on it will use the function detect_hallucination to check the metrics against some thresholds, which can be passed as parameters.
• If detect_hallucination returns True then we run _attempt_hallucination_recovery which will continue to attempt to generate text without any hallucinations iterating through the discussed lights.

Config options added (added to WhisperModel class)

enable_hallucination_recovery - bool max_beam_size - int min_beam_size - int ngram_threshold - float compression_threshold - float red_light - bool yellow_light - bool green_light - bool

Utility functions (added to WhisperModel class)

detect_hallucination - Takes the full translated text from the model and checks the n-gram and compression score, if either of them pass their respective thresholds then returns True, a hallu is detected, else returns False.

convert_audio_to_features - Some of the reprocessing models like convtasnet work on audio features and so cannot use the extracted features from whisper directly - this function converts them back into spectrograms for whisper after reprocessing.

get_audio_chunk_for_segment - Get the audio from the features.

get_segment_duration - I found that after chunking or VAD there can be very short segments left behind that always cause problems. This function just returns the legnth of a clip in seconds rather than frames as I found that more intuative to tune.

Recovery Functions (added to WhisperModel class)

These function are use to recover a hallucination and are called in the following order if their light colour is turned on. All functions will try the recovery and then recall generate_with_segments with the new settings or audio.

_recovery_increase_beam_size - Add two the current beam_size but max out at 7. The value can be tuned aswell as the init value. I start with 5.

_recovery_decrease_beam_size - Subtract two from the current beam_size but stop at 3. The minimum value can be tuned but note that I added a decrease to 1 as I found that to work quite often.

_recovery_drop_beam_size - Drops the beam_size to the min value of 1.

_recovery_vad_cleaning - This isnt how we orignally concieved of the idea which was to essentially tune the VAD model already used by whisper. Since VAD only happens at the start before feature extraction we need to just run the VAD again but with more sensitive parameters than what is used for preprocessing. Currently it just uses the silero library rather than running the VAD script from the preprocessing package but this could be changed if needed.

_recovery_audio_enhancement - Try to denoise the model using a given enhancement model which is set by deafault to convtasnet. Calls stuff from the preprocessing pipeline package to handle this. It was causing some memory issues so I added some code to clean cahce up. You might not have these on your end?

_recovery_speaker_separation - The separation testing did not go to well. I assume that the models are not good enough to work on real data and they add too many artifacts or just stright up ruin the audio. Also some speakers are only speaking for a second or two which means more VAD would be needed which is only going to add more processing time. Probs not somthing we want in the main pipeline but worth retaining the code for spercific cases?

_merge_transcriptions - Only needed with separation enabled. Since we process and translate multiple speaker tracks we get multiple transcripts which need to be mereged. We can maintain timestamps by using diarization.

_check_for_speaker_overlap - This is only relevant with separation turned on. Checks for more than 1 speaker with a significant amount of overlapping speech using a diarization model. Allows us to skip the separation step in some cases.

_attempt_hallucination_recovery - Runs through all the above recovery stratagies and checking for hallucinations in their results. Returns the orignal value if all fail. A possible alternative would be to return the best output from all the above.

Faster Whisper transcription with CTranslate2

faster-whisper is a reimplementation of OpenAI's Whisper model using CTranslate2, which is a fast inference engine for Transformer models.

This implementation is up to 4 times faster than openai/whisper for the same accuracy while using less memory. The efficiency can be further improved with 8-bit quantization on both CPU and GPU.

Benchmark

Whisper

For reference, here's the time and memory usage that are required to transcribe 13 minutes of audio using different implementations:

• openai/whisper@v20240930
• whisper.cpp@v1.7.2
• transformers@v4.46.3
• faster-whisper@v1.1.0

Large-v2 model on GPU

Implementation	Precision	Beam size	Time	VRAM Usage
openai/whisper	fp16	5	2m23s	4708MB
whisper.cpp (Flash Attention)	fp16	5	1m05s	4127MB
transformers (SDPA)1	fp16	5	1m52s	4960MB
faster-whisper	fp16	5	1m03s	4525MB
faster-whisper (batch_size=8)	fp16	5	17s	6090MB
faster-whisper	int8	5	59s	2926MB
faster-whisper (batch_size=8)	int8	5	16s	4500MB

distil-whisper-large-v3 model on GPU

Implementation	Precision	Beam size	Time	YT Commons WER
transformers (SDPA) (batch_size=16)	fp16	5	46m12s	14.801
faster-whisper (batch_size=16)	fp16	5	25m50s	13.527

GPU Benchmarks are Executed with CUDA 12.4 on a NVIDIA RTX 3070 Ti 8GB.

Small model on CPU

Implementation	Precision	Beam size	Time	RAM Usage
openai/whisper	fp32	5	6m58s	2335MB
whisper.cpp	fp32	5	2m05s	1049MB
whisper.cpp (OpenVINO)	fp32	5	1m45s	1642MB
faster-whisper	fp32	5	2m37s	2257MB
faster-whisper (batch_size=8)	fp32	5	1m06s	4230MB
faster-whisper	int8	5	1m42s	1477MB
faster-whisper (batch_size=8)	int8	5	51s	3608MB

Executed with 8 threads on an Intel Core i7-12700K.

Requirements

• Python 3.9 or greater

Unlike openai-whisper, FFmpeg does not need to be installed on the system. The audio is decoded with the Python library PyAV which bundles the FFmpeg libraries in its package.

GPU

GPU execution requires the following NVIDIA libraries to be installed:

• cuBLAS for CUDA 12
• cuDNN 9 for CUDA 12

Note: The latest versions of ctranslate2 only support CUDA 12 and cuDNN 9. For CUDA 11 and cuDNN 8, the current workaround is downgrading to the 3.24.0 version of ctranslate2, for CUDA 12 and cuDNN 8, downgrade to the 4.4.0 version of ctranslate2, (This can be done with pip install --force-reinstall ctranslate2==4.4.0 or specifying the version in a requirements.txt).

There are multiple ways to install the NVIDIA libraries mentioned above. The recommended way is described in the official NVIDIA documentation, but we also suggest other installation methods below.

Other installation methods (click to expand)

Note: For all these methods below, keep in mind the above note regarding CUDA versions. Depending on your setup, you may need to install the CUDA 11 versions of libraries that correspond to the CUDA 12 libraries listed in the instructions below.

Use Docker

The libraries (cuBLAS, cuDNN) are installed in this official NVIDIA CUDA Docker images: nvidia/cuda:12.3.2-cudnn9-runtime-ubuntu22.04.

Install with pip (Linux only)

On Linux these libraries can be installed with pip. Note that LD_LIBRARY_PATH must be set before launching Python.

pip install nvidia-cublas-cu12 nvidia-cudnn-cu12==9.*

export LD_LIBRARY_PATH=`python3 -c 'import os; import nvidia.cublas.lib; import nvidia.cudnn.lib; print(os.path.dirname(nvidia.cublas.lib.__file__) + ":" + os.path.dirname(nvidia.cudnn.lib.__file__))'`

Download the libraries from Purfview's repository (Windows & Linux)

Purfview's whisper-standalone-win provides the required NVIDIA libraries for Windows & Linux in a single archive. Decompress the archive and place the libraries in a directory included in the PATH.

Installation

The module can be installed from PyPI:

pip install faster-whisper

Other installation methods (click to expand)

Install the master branch

pip install --force-reinstall "faster-whisper @ https://github.com/SYSTRAN/faster-whisper/archive/refs/heads/master.tar.gz"

Install a specific commit

pip install --force-reinstall "faster-whisper @ https://github.com/SYSTRAN/faster-whisper/archive/a4f1cc8f11433e454c3934442b5e1a4ed5e865c3.tar.gz"

Usage

Faster-whisper

from faster_whisper import WhisperModel

model_size = "large-v3"

# Run on GPU with FP16
model = WhisperModel(model_size, device="cuda", compute_type="float16")

# or run on GPU with INT8
# model = WhisperModel(model_size, device="cuda", compute_type="int8_float16")
# or run on CPU with INT8
# model = WhisperModel(model_size, device="cpu", compute_type="int8")

segments, info = model.transcribe("audio.mp3", beam_size=5)

print("Detected language '%s' with probability %f" % (info.language, info.language_probability))

for segment in segments:
    print("[%.2fs -> %.2fs] %s" % (segment.start, segment.end, segment.text))

Warning: segments is a generator so the transcription only starts when you iterate over it. The transcription can be run to completion by gathering the segments in a list or a for loop:

segments, _ = model.transcribe("audio.mp3")
segments = list(segments)  # The transcription will actually run here.

Batched Transcription

The following code snippet illustrates how to run batched transcription on an example audio file. BatchedInferencePipeline.transcribe is a drop-in replacement for WhisperModel.transcribe

from faster_whisper import WhisperModel, BatchedInferencePipeline

model = WhisperModel("turbo", device="cuda", compute_type="float16")
batched_model = BatchedInferencePipeline(model=model)
segments, info = batched_model.transcribe("audio.mp3", batch_size=16)

for segment in segments:
    print("[%.2fs -> %.2fs] %s" % (segment.start, segment.end, segment.text))

Faster Distil-Whisper

The Distil-Whisper checkpoints are compatible with the Faster-Whisper package. In particular, the latest distil-large-v3 checkpoint is intrinsically designed to work with the Faster-Whisper transcription algorithm. The following code snippet demonstrates how to run inference with distil-large-v3 on a specified audio file:

from faster_whisper import WhisperModel

model_size = "distil-large-v3"

model = WhisperModel(model_size, device="cuda", compute_type="float16")
segments, info = model.transcribe("audio.mp3", beam_size=5, language="en", condition_on_previous_text=False)

for segment in segments:
    print("[%.2fs -> %.2fs] %s" % (segment.start, segment.end, segment.text))

For more information about the distil-large-v3 model, refer to the original model card.

Word-level timestamps

segments, _ = model.transcribe("audio.mp3", word_timestamps=True)

for segment in segments:
    for word in segment.words:
        print("[%.2fs -> %.2fs] %s" % (word.start, word.end, word.word))

VAD filter

The library integrates the Silero VAD model to filter out parts of the audio without speech:

segments, _ = model.transcribe("audio.mp3", vad_filter=True)

The default behavior is conservative and only removes silence longer than 2 seconds. See the available VAD parameters and default values in the source code. They can be customized with the dictionary argument vad_parameters:

segments, _ = model.transcribe(
    "audio.mp3",
    vad_filter=True,
    vad_parameters=dict(min_silence_duration_ms=500),
)

Vad filter is enabled by default for batched transcription.

Logging

The library logging level can be configured like this:

import logging

logging.basicConfig()
logging.getLogger("faster_whisper").setLevel(logging.DEBUG)

Going further

See more model and transcription options in the WhisperModel class implementation.

Community integrations

Here is a non exhaustive list of open-source projects using faster-whisper. Feel free to add your project to the list!

• speaches is an OpenAI compatible server using faster-whisper. It's easily deployable with Docker, works with OpenAI SDKs/CLI, supports streaming, and live transcription.
• WhisperX is an award-winning Python library that offers speaker diarization and accurate word-level timestamps using wav2vec2 alignment
• whisper-ctranslate2 is a command line client based on faster-whisper and compatible with the original client from openai/whisper.
• whisper-diarize is a speaker diarization tool that is based on faster-whisper and NVIDIA NeMo.
• whisper-standalone-win Standalone CLI executables of faster-whisper for Windows, Linux & macOS.
• asr-sd-pipeline provides a scalable, modular, end to end multi-speaker speech to text solution implemented using AzureML pipelines.
• Open-Lyrics is a Python library that transcribes voice files using faster-whisper, and translates/polishes the resulting text into .lrc files in the desired language using OpenAI-GPT.
• wscribe is a flexible transcript generation tool supporting faster-whisper, it can export word level transcript and the exported transcript then can be edited with wscribe-editor
• aTrain is a graphical user interface implementation of faster-whisper developed at the BANDAS-Center at the University of Graz for transcription and diarization in Windows (Windows Store App) and Linux.
• Whisper-Streaming implements real-time mode for offline Whisper-like speech-to-text models with faster-whisper as the most recommended back-end. It implements a streaming policy with self-adaptive latency based on the actual source complexity, and demonstrates the state of the art.
• WhisperLive is a nearly-live implementation of OpenAI's Whisper which uses faster-whisper as the backend to transcribe audio in real-time.
• Faster-Whisper-Transcriber is a simple but reliable voice transcriber that provides a user-friendly interface.
• Open-dubbing is open dubbing is an AI dubbing system which uses machine learning models to automatically translate and synchronize audio dialogue into different languages.
• Whisper-FastAPI whisper-fastapi is a very simple script that provides an API backend compatible with OpenAI, HomeAssistant, and Konele (Android voice typing) formats.

Model conversion

When loading a model from its size such as WhisperModel("large-v3"), the corresponding CTranslate2 model is automatically downloaded from the Hugging Face Hub.

We also provide a script to convert any Whisper models compatible with the Transformers library. They could be the original OpenAI models or user fine-tuned models.

For example the command below converts the original "large-v3" Whisper model and saves the weights in FP16:

pip install transformers[torch]>=4.23

ct2-transformers-converter --model openai/whisper-large-v3 --output_dir whisper-large-v3-ct2
--copy_files tokenizer.json preprocessor_config.json --quantization float16

• The option --model accepts a model name on the Hub or a path to a model directory.
• If the option --copy_files tokenizer.json is not used, the tokenizer configuration is automatically downloaded when the model is loaded later.

Models can also be converted from the code. See the conversion API.

Load a converted model

1. Directly load the model from a local directory:

model = faster_whisper.WhisperModel("whisper-large-v3-ct2")

2. Upload your model to the Hugging Face Hub and load it from its name:

model = faster_whisper.WhisperModel("username/whisper-large-v3-ct2")

Comparing performance against other implementations

If you are comparing the performance against other Whisper implementations, you should make sure to run the comparison with similar settings. In particular:

• Verify that the same transcription options are used, especially the same beam size. For example in openai/whisper, model.transcribe uses a default beam size of 1 but here we use a default beam size of 5.
• Transcription speed is closely affected by the number of words in the transcript, so ensure that other implementations have a similar WER (Word Error Rate) to this one.
• When running on CPU, make sure to set the same number of threads. Many frameworks will read the environment variable OMP_NUM_THREADS, which can be set when running your script:

OMP_NUM_THREADS=4 python3 my_script.py

Footnotes

1. transformers OOM for any batch size > 1 ↩