Learn · Updated
How Audio-Reactive Visuals Work
Short answer: Audio-reactive visuals extract control signals from sound, such as band levels, onsets and tempo, shape them with smoothing and envelopes, and route them to visual parameters like scale, colour, position, camera and shader values.
Audio-reactive visuals are graphics whose parameters are controlled by sound. Unlike a classic spectrum or waveform display, the audio does not have to be drawn literally. It might scale a 3D object, shift a hue, distort a shader or trigger a camera cut.
This guide covers the chain from sound to image, as used in tools such as Synesthesia, TouchDesigner and Resolume.
1. Input
Real-time systems take audio from:
- a line or microphone input, such as a feed from a DJ mixer or front-of-house desk;
- system audio capture (loopback) on the same computer;
- inter-app audio from a DAW, or an analysis plugin inside the DAW.
Offline video makers read the audio file directly and can analyse it ahead of time.
2. Analysis into control signals
The audio is reduced to a handful of control signals that change every frame:
| Signal | What it measures | Typical use |
|---|---|---|
| Level (RMS) | Overall loudness | Global intensity, brightness |
| Band levels | Energy in bass, mid, high ranges (via FFT) | Separate elements for kick, vocals, hats |
| Onsets / beats | Sudden increases in energy | Cuts, flashes, preset changes |
| Tempo (BPM) | Beat period | Looping animation in time |
| Spectral features | Brightness (centroid), noisiness, pitch class | Colour and texture changes |
How music visualisers work explains how these are computed.
3. Shaping the signal
Raw signals are jittery. Before mapping, they are shaped:
- Smoothing: averaging over recent frames, often with fast attack and slow release, so values rise with the hit and fall gracefully.
- Thresholds and gates: ignoring values below a floor so quiet passages stay calm.
- Scaling and curves: mapping the useful range to 0–1 and applying an exponential curve so small changes stay subtle and peaks stand out.
- Accumulation: adding the signal over time, so bass drives speed rather than position, which avoids shaking.
This stage separates professional-looking reactivity from flicker.
4. Mapping to parameters
Each shaped signal is routed to one or more visual parameters. Good mappings follow the music’s roles:
- low frequencies → scale, displacement, camera shake, bloom;
- mid frequencies → shape changes and colour;
- high frequencies → particles, sparkle, fine detail;
- beats → cuts, flashes, scene changes;
- tempo → loop speed.
Node-based tools such as TouchDesigner and Magic Music Visuals let you wire any signal to any parameter. Preset-driven tools such as Synesthesia ship scenes with mappings already designed. Live-coding tools such as Hydra expose FFT values as variables in code.
5. Combining audio with control data
Audio analysis can only react to what it hears, and it reacts slightly after the event. For tighter shows, visual artists add:
- MIDI and OSC from instruments, controllers or the DAW, which carry exact note and timing data;
- tempo sync via Ableton Link or MIDI clock;
- timecode for fully pre-programmed shows.
DAW-based tools such as Videosync for Ableton Live sequence visuals on the same timeline as the music.
6. Output
Live systems send visuals to a projector or LED processor, or share them with other software through Syphon (macOS), Spout (Windows) or NDI (network). Video makers render a file.
Browse audio-reactive visualisers or the live shows and VJ collection.
Frequently asked questions
What is the difference between audio-reactive and a music visualiser?
Audio-reactive describes any visual that responds to sound. A music visualiser is software built mainly for that purpose. Many audio-reactive visuals do not look like traditional visualisers, because the sound changes motion or colour rather than being drawn as bars or waves.
Should live visuals use audio analysis or MIDI?
Audio analysis works with any music and needs no setup, but it can only react to what it hears. MIDI or OSC from the performer's instruments or DAW gives exact timing and lets visuals anticipate changes. Many shows combine both.
