Audio, video, and multimodal models

Picture it

Sound arrives as a waveform; images arrive as pixels. Encoders turn these into representations a model can combine with text. The extra modalities add their own preprocessing, timing, and memory costs.

What to understand

• Spectrograms, speech recognition, speech synthesis
• Contrastive image/text alignment, vision-language models, video sampling

Follow the references

1. 1 · Course

   Hugging Face · Audio course ↗

   Start with audio data and the task overview, then compare ASR and text-to-speech pipelines.

2. 2 · Course

   Stanford · CS231n computer vision ↗

   Find multimodal learning in the course schedule and inspect how modalities meet.