TorchAudio feasibility

[GeorgeS2019]

PyTorch audio: TorchAudio

• Audio I/O
• Audio Resampling
• Audio Data Augmentation
• Audio Feature Extractions
• Audio Feature Augmentation
• Audio Datasets
• Speech Recognition with Wav2Vec2
• Speech Command Classification with torchaudio
• Text-to-speech with torchaudio
• Text-to-speech with torchaudio: tacotron2

@kaiidams could you please check and perhaps organize what you have done could support .NET community to work towards TorchAudio using TorchSharp.

[GeorgeS2019]

@luisquintanilla I hope you could find time to look at some of the deep voice AI contributions for .NET community by @kaiidams. He is in my view a suitable candidate for one of your monthly ML.NET standup community programs.

• Voice100Godot
• TransferLearningAudio
• Voice100AndroidApp
• NeMoOnnxSharp

[luisquintanilla]

Hi @GeorgeS2019

This is great. Thanks for sharing! I'll take a look and we'd be happy to have @kaiidams on 🙂

[kaiidams]

@GeorgeS2019 Thank you for introduction.
@luisquintanilla I am working on torch.stft/istft() which is missing and will be required for TorchAudio.
kaiidams@ada6545
I'll need next weekend to polish parameter conversions.

[NiklasGustafsson]

@kaiidams PR has been merged a couple of days ago, by the way.

[GeorgeS2019]

first of many TourchAudio commits

[GeorgeS2019]

• TorchSharp: Audio Resampling on torchautio tutorial

Resampling Overview

To resample an audio waveform from one freqeuncy to another, you can use TorchSharp's torchaudio.transforms.Resample() or torchaudio.functional.resample(). transforms.Resample precomputes and caches the kernel used for resampling, while functional.resample computes it on the fly, so using torchaudio.transforms.Resample will result in a speedup when resampling multiple waveforms using the same parameters

• TorchSharp; Audio Datasets on torchautio tutorial
• TorchSharp: Speech Command Classification with torchaudio based on this PR on torchautio tutorial

• Importing the Dataset

from torchaudio.datasets import SPEECHCOMMANDS

TorchSharp

 public static torch.utils.data.Dataset<SpeechCommandsDatasetItem> SPEECHCOMMANDS

• Formatting the Data

transform = torchaudio.transforms.Resample(orig_freq=sample_rate, new_freq=new_sample_rate)

TorchSharp

var transform = torchaudio.transforms.Resample(_sample_rate, _new_sample_rate, device: device);

• Define the Network

• Training and Testing the Network

[GeorgeS2019]

• [TorchSharp: Audio Feature Extractions] based on torchaudio tutorial

torchaudio implements feature extractions commonly used in the audio domain. They are available in TorchSharp: torchaudio.functional and torchaudio.transforms.

Possible .NET Spectrogram plotting library

For .NET interactive notebook, we need this display supported in

• Plotly.NET

Which need support in

• plotly.js

  Spectrogram feature request?

[GeorgeS2019]

• TEXT-TO-SPEECH WITH TACOTRON2 using TorchSharp

[GeorgeS2019]

torchaudio.function implements features as standalone functions. They are stateless.

new TorchSharp torchaudio.function PR

[GeorgeS2019]

@kaiidams

One of many TorchSharp's TorchAudio Goals: Language is essential to human life

[GeorgeS2019]

Another TorchSharp Wav2Vec2 example
@whuanle

https://github.com/whuanle/Maomi.Torch/blob/main/examples/wav2vec2/Program.cs

[GeorgeS2019]

#1377 (comment)