Tipps und Tricks für Entwickler und IT-Interessierte
Welcome to the fascinating world of Whisper AI, OpenAI’s groundbreaking speech recognition system. As we delve deeper into the digital age, the ability to accurately transcribe and understand human speech has become invaluable. From powering virtual assistants to enhancing accessibility in technology, speech-to-text solutions are reshaping our interaction with devices.
This article will explore the intricate architecture of Whisper AI, guide you through its usage via command line, Python, and even demonstrate how to integrate it into a Flask application.
Whether you’re a developer, a tech enthusiast, or simply curious, join me in unraveling the capabilities of this remarkable technology.
Whisper AI stands as a testament to the advancements in artificial intelligence, specifically in speech recognition. Developed by OpenAI, this system is designed not just to transcribe speech but to understand it, accommodating various accents, dialects, and even noisy backgrounds.
Whisper is a general-purpose speech recognition model. It is trained on a large dataset of diverse audio and is also a multi-task model that can perform multilingual speech recognition as well as speech translation and language identification.
The development of Whisper AI marks a significant milestone, showcasing OpenAI’s commitment to pushing the boundaries of AI and machine learning. In a world where effective communication is key, Whisper AI is a step towards bridging gaps and making technology more accessible to all.
At the heart of Whisper AI lies a sophisticated neural network, likely based on the Transformer model, renowned for its effectiveness in processing sequential data like speech.
These models are trained on vast datasets, enabling the system to recognize and interpret a wide array of speech patterns. What sets Whisper AI apart is its multilingual capabilities, proficiently handling different languages and dialects.
This architectural marvel not only enhances accuracy but also ensures inclusivity in speech recognition technology.
To embark on your journey with Whisper AI, a Python environment is a prerequisite. The system’s robustness requires adequate hardware specifications to function optimally.
Installation is straightforward: Whisper AI’s official documentation or https://github.com/openai/whisper provides comprehensive guides to get you started.
So, first: setup a python. virtual environment:
$ python -m venv venv $ . venv/bin/activate $ which python <Your current directoy>/venv/bin/python
Next step: install required tool for whisper:
$ sudo apt update && sudo apt install ffmpeg
And, at last: install whisper:
$ pip install git+https://github.com/openai/whisper.git
And your done.
Whisper AI shines in its simplicity of use, particularly via the command line. After a basic setup, transcribing an audio file is as simple as running a command
For instance, whisper youraudiofile.mp3
could yield a text transcription of the recorded speech. This ease of use makes Whisper AI an attractive option for quick transcription tasks, and the command line interface provides a straightforward way for anyone to harness its power.
Run the following commmand to transcribe your audio file:
$ whisper <your audio file> --model medium --threads 16
See some samples in the following Section and in Section 9.
Python enthusiasts can rejoice in the seamless integration of Whisper AI with Python scripts.
Imagine a script that takes an audio file and uses Whisper AI to transcribe it – this can be as concise as a few lines of code.
import whisper model = whisper.load_model("base") result = model.transcribe("audio.mp3") print(result["text"])
The API’s intuitive design means that with basic Python knowledge, you can create scripts to automate transcription tasks, analyze speech data, or even build more complex speech-to-text applications.
import whisper model = whisper.load_model("base") # load audio and pad/trim it to fit 30 seconds audio = whisper.load_audio("audio.mp3") audio = whisper.pad_or_trim(audio) # make log-Mel spectrogram and move to the same device as the model mel = whisper.log_mel_spectrogram(audio).to(model.device) # detect the spoken language _, probs = model.detect_language(mel) print(f"Detected language: {max(probs, key=probs.get)}") # decode the audio options = whisper.DecodingOptions() result = whisper.decode(model, mel, options) # print the recognized text print(result.text)
Integrating Whisper AI into a Flask application opens a realm of possibilities. A simple Flask server can receive audio files and return transcriptions, all powered by Whisper AI.
This setup is ideal for creating web applications that require speech-to-text capabilities. From voice-commanded actions to uploading and transcribing audio files, the combination of Flask and Whisper AI paves the way for innovative web-based speech recognition solutions.
Here is a short code for a flask app:
from flask import Flask, abort, request from flask_cors import CORS from tempfile import NamedTemporaryFile import whisper import torch torch.cuda.is_available() DEVICE = "cuda" if torch.cuda.is_available() else "cpu" model = whisper.load_model("base", device=DEVICE) app = Flask(__name__) CORS(app) @app.route("/") def hello(): return "Whisper Hello World!" @app.route('/whisper', methods=['POST']) def handler(): if not request.files: abort(400) results = [] for filename, handle in request.files.items(): temp = NamedTemporaryFile() handle.save(temp) result = model.transcribe(temp.name) results.append({ 'filename': filename, 'transcript': result['text'], }) return {'results': results}
For those looking to push the boundaries, Whisper AI offers advanced features and customization options.
Adapting the system to recognize specific terminologies or accent nuances can significantly enhance its utility in specialized fields. This level of customization ensures that Whisper AI remains a versatile tool, adaptable to various professional and personal needs.
As with any AI technology, Whisper AI brings with it ethical considerations.
The paramount concern is privacy and the security of data processed by the system. Additionally, while Whisper AI is a remarkable technology, it is not without limitations. Potential biases in language models and challenges in understanding heavily accented or distorted speech are areas that require ongoing refinement.
Addressing these concerns is crucial for the responsible development and deployment of speech recognition technologies.
First, get your audio file to transcribed. I choose a famous speech from Abraham Lincoln and get it from Youtube.
$ yt-dlp -S res,ext:mp4:m4a "https://www.youtube.com/watch?v=bC4kQ2-kHZE" --output Abraham-Lincoln_Gettysburg-Address.mp4
Then, i use whisper to transcribe the audio.
$ whisper Abraham-Lincoln_Gettysburg-Address.mp4 --model medium --threads 16 --output_dir . .../.venv/python/3.11/lib/python3.11/site-packages/whisper/transcribe.py:115: UserWarning: FP16 is not supported on CPU; using FP32 instead warnings.warn("FP16 is not supported on CPU; using FP32 instead") Detecting language using up to the first 30 seconds. Use `--language` to specify the language Detected language: English [00:00.000 --> 00:19.660] 4 score and 7 years ago, our fathers brought forth on this continent a new nation, conceived [00:19.660 --> 00:28.280] in liberty and dedicated to the proposition that all men are created equal. [00:28.280 --> 00:37.600] Now we are engaged in a great civil war, testing whether that nation, or any nation so conceived [00:37.600 --> 00:43.680] and so dedicated, can long endure. [00:43.680 --> 00:48.660] We are met on a great battlefield of that war. [00:48.660 --> 00:54.800] We have come to dedicate a portion of that field as a final resting place for those who [00:54.800 --> 00:59.840] here gave their lives that that nation might live. [00:59.840 --> 01:09.600] It is altogether fitting and proper that we should do this, but in a larger sense, we [01:09.600 --> 01:18.440] cannot dedicate, we cannot consecrate, we cannot hallow this ground. [01:18.440 --> 01:25.720] The brave men, living and dead, who struggled here, have consecrated it, far above our [01:25.720 --> 01:30.640] poor power to add or detract. [01:30.640 --> 01:37.280] The world will little note, nor long remember what we say here, but it can never forget [01:37.280 --> 01:40.040] what they did here. [01:40.040 --> 01:47.440] It is for us, the living rather, to be dedicated here, to the unfinished work which they who [01:47.440 --> 01:52.360] fought here have thus far so nobly advanced. [01:52.360 --> 01:59.680] It is rather for us to be here dedicated to the great task remaining before us, that from [01:59.680 --> 02:06.220] these honored dead we take increased devotion to that cause for which they gave the last [02:06.220 --> 02:13.960] full measure of devotion, that we here highly resolve that these dead shall not have died [02:13.960 --> 02:23.520] in vain, that this nation, under God, shall have a new birth of freedom, and that government [02:23.520 --> 02:31.000] of the people, by the people, for the people, shall not perish from the earth.
Whisper AI represents a significant leap in speech recognition technology. Its implications for the future of human-computer interaction are profound. As we continue to explore and expand the capabilities of AI, tools like Whisper AI not only enhance our present but also shape our future in technology.
I encourage you to explore Whisper AI, experiment with its features, and share your experiences. The journey of discovery is just beginning.
Just image, if AI is helping you in daily tasks, you have more free time for other things:
For those eager to dive deeper, the Whisper AI documentation, available on OpenAI’s official website and GitHub, offers extensive information and tutorials. Community forums and discussions provide valuable insights and practical advice for both novice and experienced users.
ffmpeg -i source.mp4 -c:v libvpx-vp9 -b:v 0.33M -c:a libopus -b:a 96k \<br>-filter:v scale=960x540 target.webm
ffmpeg -i input.mp4 -vcodec libx265 -crf 28 output.mp4
ffmpeg -i input.mp4 -vcodec libx264 -crf 24 output.mp4
ffmpeg -i input.avi -vf scale=1280:720 output.avi
ffmpeg -i input.avi -vf scale=852×480 output.avi
Specify only one component, width or height, and set the other component to -1, for eample,
ffmpeg -i input.jpg -vf scale=852:-1 output_852.png
ffmpeg -i video.webm video.mp4
When the WebM file contains VP8 or VP9 video, you have no choice but to transcode both the video and audio.
Video conversion can be a lengthy and CPU intensive process, depending on file size, video and audio quality, video resolution, etc. but FFmpeg provides a series of presets and controls to help you optimize for quality or faster conversions.
A note on video quality
When encoding video with H.264, the video quality can be controlled using a quantizer scale (crf value, crf stands for Constant Rate Factor) which can take values from 0 to 51: 0 being lossless, 23 the default and 51 the worst possible. So the lower the value the better the quality. You can leave the default value or, for a smaller bitrate, you can raise the value:
ffmpeg -i video.webm -crf 26 video.mp4
Video presets
FFmpeg also provides several quality presets which are calibrated for a certain encoding speed to compression ratio. A slower preset will provide a better compression. The following presets are available in descending order: ultrafast, superfast, veryfast, faster, fast
, medium, slow, slower and veryslow. The default preset is medium but you can choose a faster preset:
ffmpeg -i video.webm -preset veryfast video.mp4
Placing the MOOV atom at the beginning
All MP4 files contain a moov
atom. The moov atom contains information about the length of the video. If it’s at the beginning it immediately enables a streaming video player to play and scrub the MP4 file. By default FFmpeg places the moov atom at the end of the MP4 file but it can place the mov atom at the beginning with the -movflags faststart
option like this:
ffmpeg -i video.webm -movflags faststart video.mp4
Using Levels and Profiles when encoding H.264 video with FFmpeg
To ensure the highest compatibility with older iOS or Android devices you will need to use certain encoding profiles and levels. For example a video encoded with the High Profile and Level 4.2 will work on iPhone 5S and newer but not on older iOS devices.
ffmpeg -i video.webm -movflags faststart -profile:v high -level 4.2 video.mp4
In some rare cases the .webm file will contain H.264 video and Vorbis or Opus audio(for example .webm files created using the MediaRecorder API on Chrome 52+ ). In such cases you don’t have to re-encode the video data since it’s already in the desired H.264 format (re-encoding is also not recommended since you’ll be loosing some quality in the process while consuming CPU cycles) so we’re just going to copy over the data
To copy the video data and transcode the audio in FFmpeg you use the -c:v copy
option:
ffmpeg -i video.webm -c:v copy video.mp4
export PYSPARK_DRIVER_PYTHON=jupyter export PYSPARK_DRIVER_PYTHON_OPTS='notebook'
Then, run
pyspark
Nicepage is an awesome Web Page Builder with 400+ Features and 10000+ Templates.
One of the features is to connect to Google Analytics, another is a Cookie Consent Plugin.
Unfortunately, i didn’t succeed in getting both together work. The goal was, only to load the GA code if the consent plugin allowed this.
So i created a workaround:
<script> window.dataLayer = window.dataLayer || []; function gtag(){ console.log('gtag: arguments=', arguments); dataLayer.push(arguments); } function setupGoogleTagManager(w,d,s,l,i){ console.log('google tag manager: setup with', i); w[l]=w[l]||[]; w[l].push({'gtm.start': new Date().getTime(), event:'gtm.js'}); var f=d.getElementsByTagName(s)[0], j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:''; j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl; f.parentNode.insertBefore(j,f); } function setCookie(cname, cvalue, exdays) { const d = new Date(); d.setTime(d.getTime() + (exdays*24*60*60*1000)); let expires = "expires="+ d.toUTCString(); document.cookie = cname + "=" + cvalue + ";" + expires + ";path=/"; } function getCookie(cname) { let name = cname + "="; let decodedCookie = decodeURIComponent(document.cookie); let ca = decodedCookie.split(';'); for(let i = 0; i <ca.length; i++) { let c = ca[i]; while (c.charAt(0) == ' ') { c = c.substring(1); } if (c.indexOf(name) == 0) { return c.substring(name.length, c.length); } } return ""; } function buildConsentCookie(consent) { console.log('buildConsentCookie: consent=', consent); const consentCookie = { 'functionality_storage': consent.necessary ? 'granted' : 'denied', 'security_storage': consent.necessary ? 'granted' : 'denied', 'ad_storage': consent.marketing ? 'granted' : 'denied', 'analytics_storage': consent.analytics ? 'granted' : 'denied', 'personalization': consent.preferences ? 'granted' : 'denied', }; console.log('buildConsentCookie: consentCookie=', consentCookie); return consentCookie } function setConsent(consent) { console.log('setConsent: ', consent); const consentCookie = buildConsentCookie(consent); gtag('consent', 'default', consentCookie); console.log('store consentMode: ',consentCookie); localStorage.setItem('consentMode', JSON.stringify(consentCookie)); } if(localStorage.getItem('consentMode') === null){ const consent = { necessary: false, marketing: false, analytics: false, preferences: false, }; consentCookie = buildConsentCookie(consent); gtag('consent', 'default', consentCookie); } else { consentCookie = JSON.parse(localStorage.getItem('consentMode')); gtag('consent', 'default', consentCookie); } </script>
const consent = { necessary: true, marketing: true, analytics: true, preferences: true, }; setCookie('u-gdpr-cookie', true, 180); setConsent(consent); setupGoogleTagManager(window,document,'script','dataLayer','GTM-#######');
brew install jq
When running jq, the following arguments may become handy:
Argument | Description |
---|---|
--version | Output the jq version and exit with zero. |
--sort-keys | Output the fields of each object with the keys in sorted order. |
The syntax for jq is pretty coherent:
Syntax | Description |
---|---|
, | Filters separated by a comma will produce multiple independent outputs |
? | Will ignores error if the type is unexpected |
[] | Array construction |
{} | Object construction |
+ | Concatenate or Add |
– | Difference of sets or Substract |
length | Size of selected element |
| | Pipes are used to chain commands in a similar fashion than bash |
Description | Command |
---|---|
Display all keys | jq 'keys' |
Adds + 1 to all items | jq 'map_values(.+1)' |
Delete a key | jq 'del(.foo)' |
Convert an object to array | to_entries | map([.key, .value]) |
Description | Command |
---|---|
Concatenate two fields | fieldNew=.field1+' '+.field2 |
Description | Command |
---|---|
All | jq .[] |
First | jq '.[0]' |
Range | jq '.[2:4]' |
First 3 | jq '.[:3]' |
Last 2 | jq '.[-2:]' |
Before Last | jq '.[-2]' |
Select array of int by value | jq 'map(select(. >= 2))' |
Select array of objects by value | ** jq ‘.[] | select(.id == “second”)’** |
Select by type | ** jq ‘.[] | numbers’ ** with type been arrays, objects, iterables, booleans, numbers, normals, finites, strings, nulls, values, scalars |
Description | Command |
---|---|
Add + 1 to all items | jq 'map(.+1)' |
Delete 2 items | jq 'del(.[1, 2])' |
Concatenate arrays | jq 'add' |
Flatten an array | jq 'flatten' |
Create a range of numbers | jq '[range(2;4)]' |
Display the type of each item | jq 'map(type)' |
Sort an array of basic type | jq 'sort' |
Sort an array of objects | jq 'sort_by(.foo)' |
Group by a key – opposite to flatten | jq 'group_by(.foo)' |
Minimun value of an array | jq 'min' .See also min, max, min_by(path_exp), max_by(path_exp) |
Remove duplicates | jq 'unique' or jq 'unique_by(.foo)' or jq 'unique_by(length)' |
Reverse an array | jq 'reverse' |
When checking the date in WSl, it display the wrong date
❯ date Wed Jul 26 15:28:20 CEST 2023
So setup the correct date, you could run either
❯ sudo hwclock -s
or setup a timesync daemon (described here)
Add these lines to the /etc/wsl.conf
(note you will need to run your editor with sudo privileges, e.g: sudo nano /etc/wsl.conf
):
[boot] systemd=true
And close out of the nano editor using CTRL+O to save and CTRL+X to exit. Now close your WSL distro windows and run wsl.exe --shutdown
from PowerShell, then restart your WSL instance. Upon launch you should have systemd
running.
Now install systemd-timesyncd
:
sudo apt install systemd-timesyncd sudo systemctl edit systemd-timesyncd
In the second step, when editing, add the two lines beginning with [Unit]
, as shown below:
Now start it:
sudo systemctl start systemd-timesyncd
Check status with:
timedatectl status timedatectl timesync-status
With VS Code 1.75.1 you could get this error message.
What helped in my environment was, to patch an extension file in the WSL.
You can find this file with
cd $HOME/.vscode-server find . -type f -exec grep "is not valid" {} \;
$HOME/.vscode-server/extensions/ms-python.python-2023.5.10591006/pythonFiles/lib/python/debugpy/adapter/clients.py
With VS Code 1.75.1 you could get this error message.
What helped in my environment was, to patch an extension file in the WSL.
$HOME/.vscode-server/extensions/ms-python.python-2023.5.10591006/pythonFiles/lib/python/debugpy/adapter/clients.py
Sum of Squares
dotnet new --install Fable.Template
Create an new App
dotnet new fable
Prepare environment
dotnet tool restore
Fix NodeJS SSL Error
$ENV:NODE_OPTIONS="--openssl-legacy-provider"
Install dependencies and run app
npm install
npm start