Learn Creative Coding (#123) - Audiovisual Synchronization
Right, this is the one I've been quietly building toward for the last five or six episodes. We've spent a lovely long stretch making sound - synths, melodies, drum machines, drifting ambient beds - and before that we spent dozens of episodes making pictures. But we mostly kept them in separate rooms. A flash on a kick here, an opacity that tracks a density curve there, little polite handshakes between the two worlds. Today the eyes and the ears finally meet properly. Today we make what you see and what you hear into one single thing moving together, so tightly locked that your brain stops treating them as two events and starts perceiving them as one. Allez, this is the good stuff - the thing that makes music videos hit, that makes a VJ set feel alive, that makes a little browser toy weirdly hypnotic. Let's lock them together :-).
Here's the perceptual fact that makes all of this worth doing, and it's genuinly wild once you notice it. When a sound and a visual happen within about 40 milliseconds of each other, your brain fuses them into a single event -- it decides they have the same cause. Miss that window by even a little and the illusion breaks: the picture feels like it's lagging the music, or floating loose on top of it, and the whole thing reads as cheap. So the entire craft of audiovisual sync is really the craft of hitting that tiny window, over and over, reliably. Everything else is decoration. Let me show you the pieces.
The core problem: two clocks that don't agree
Before any pretty code, you need to understand why sync is hard, because it's not obvious. You'd think you'd just trigger the visual in the same line as the note and be done. But you've got two completely different timing systems fighting each other. The audio engine (Web Audio, which Tone.js sits on top of) schedules sound ahead of time on a rock-solid hardware clock -- it decides "this note fires at exactly 4.512 seconds" and queues it. Meanwhile your visuals run on requestAnimationFrame, which fires roughly every 16 milliseconds whenever the browser feels like it, and is happy to stutter if the tab is busy.
// the two clocks. they do NOT tick together, and that's the whole problem.
Tone.now(); // the audio clock - precise, scheduled ahead, sample-accurate
performance.now(); // the wall clock rAF roughly follows - sloppy, can stutter
// if you trigger a visual "now" when a note plays, "now" is already a few ms stale.
So if you naively do if (noteJustPlayed) flashScreen() inside your draw loop, the flash lands one, two, sometimes five frames after the sound, because the draw loop only checked after the audio already fired. That drift is exactly what pushes you outside the 40ms window. The fix is to stop reacting to audio and start scheduling both together off the same clock.
Tone.Draw: the bridge between the two worlds
This is the single most important tool in the whole episode, so I want you to really get it. Tone.Draw lets you schedule a visual callback at a precise audio time, and it fires that callback on the very next animation frame after that time -- as close to the intended moment as the display can manage. You hand it the same time value the audio uses, and it does the bookkeeping to line the picture up with the sound.
// schedule a note AND its visual for the exact same audio-accurate moment.
const synth = new Tone.Synth().toDestination();
const loop = new Tone.Loop((time) => {
// 'time' is the precise audio moment this note sounds:
synth.triggerAttackRelease("C4", "8n", time);
// schedule the VISUAL for that same instant - Tone.Draw fires it on the
// next frame after 'time', so picture and sound land together:
Tone.Draw.schedule(() => {
flash = 1; // whatever visual event you want, fired in sync
}, time);
}, "4n").start(0);
The reason this works and a plain callback doesn't: the audio was scheduled ahead, so at the moment the loop callback runs, the sound hasn't actually played yet -- it's queued for time. Tone.Draw holds your visual and releases it at the matching frame. Both the ear and the eye get served the event at the same real-world instant. This one function is the backbone of everything below, so keep it in your pocket.
Staccato triggers, legato interpolation
Now the aesthetic heart of beat-driven animation, and it's a principle worth tattooing somewhere. Sound events are instant -- a note attacks and it's there. But if your visual just snaps to a new state on every beat and holds it, the animation looks robotic and strobey, like a broken traffic light. The trick that makes it feel musical is this: trigger hard on the beat, then interpolate softly between beats. The hit is staccato, the motion between hits is legato. You punch a value up to 1 on the kick, and then every frame you ease it back down toward 0.
// staccato trigger + legato decay = musical motion.
let pulse = 0;
// on each beat, PUNCH the value up (staccato):
const beat = new Tone.Loop((time) => {
drum.triggerAttackRelease("C1", "8n", time);
Tone.Draw.schedule(() => { pulse = 1; }, time); // hard hit, in sync
}, "4n").start(0);
// every frame, EASE it back toward zero (legato) - this is the smooth bit:
function draw(p) {
pulse *= 0.9; // exponential decay, ~lerp toward 0
const radius = 100 + pulse * 80; // circle swells on the beat, relaxes between
p.circle(p.width / 2, p.height / 2, radius);
}
That pulse *= 0.9 is our old friend from episode 16 -- easing, exponential decay toward a target -- doing the legato work between the staccato hits. This combination is the recipe for making visuals feel like they're breathing with the music instead of flickering on top of it. Fast decay (multiply by 0.8) gives a sharp punchy snap; slow decay (0.97) gives a lingering swell. That one number is your "how bouncy" dial. Makes sense, right?
Match the envelope: attack shapes the visual too
Here's a lovely refinement. Back in episode 119 we learned that every sound has an envelope -- attack, decay, sustain, release -- the shape of how it rises and falls. A snappy pluck has a fast attack; a soft pad swells in slowly. The really satisfying sync trick is to make the visual envelope match the audio envelope. A percussive sound gets a sharp visual snap; a slow-swelling pad gets a slow-growing visual bloom. When the shape of what you see matches the shape of what you hear, the fusion gets even stronger.
// map the sound's attack time to the visual's response speed.
// fast attack -> snappy visual. slow attack -> gradual visual.
function visualDecayFor(attackTime) {
// short attack (0.01s) -> fast decay (0.82). long attack (2s) -> slow decay (0.98).
return 0.82 + Math.min(attackTime, 2) * 0.08;
}
const padDecay = visualDecayFor(2.0); // slow bloom to match a soft pad
const pluckDecay = visualDecayFor(0.01); // sharp snap to match a pluck
You don't have to be scientific about it -- the point is just that a kick should hit the picture and a pad should breathe into it. Your eye already expects that correspondence because it's how the physical world works: sharp sounds come from sharp events, soft sounds from soft ones.
Frequency to height: pitch is up, pitch is down
There's a deep bit of human wiring we can lean on for free. Across almost every culture, people map high pitch to high position and low pitch to low position -- we literally call them "high" and "low" notes even though sound has no up or down. So if you place a high note near the top of the screen and a low note near the bottom, it feels correct without anyone having to think about it. Same goes for bright-vs-dark and small-vs-large. Lean into the metaphor and your visuals read as "obviously" connected to the sound.
// map a note's pitch to vertical position - high notes up, low notes down.
// Tone.Frequency lets us pull the raw Hz out of a note name.
function noteToY(noteName, p) {
const hz = Tone.Frequency(noteName).toFrequency(); // "C4" -> 261.6 Hz
// map a musical range (say 100-800 Hz) to screen height, inverted (top = high):
const t = (Math.log2(hz) - Math.log2(100)) / (Math.log2(800) - Math.log2(100));
return p.map(1 - t, 0, 1, 40, p.height - 40); // high pitch -> small y (top)
}
Notice I mapped it in log space, not linear -- because pitch perception is logarithmic (an octave is a doubling of frequency, exactly the trig-and-periodicity stuff from episode 13). Equal musical steps should be equal visual steps, and only log spacing gives you that. Get this right and a rising melody visibly climbs the screen; get it linear and the low notes all bunch up at the bottom looking wrong.
Phrase-level sync: structure needs two scales
Beats are the micro structure -- the fast pulse. But music also has macro structure: phrases, usually 4 or 8 bars, that group the beats into sentences. Great audiovisual work syncs at both scales. On every beat, a small visual event (a pulse, a flash). On every phrase boundary, a big change -- a new colour palette, a different particle behaviour, a camera move. The small changes keep it lively frame to frame; the big changes give it a sense of going somewhere.
// two scales of sync: fast pulse on the beat, big shift on the phrase boundary.
let bar = 0;
const PHRASE = 8; // 8 bars per phrase
let palette = pickPalette(0);
const structure = new Tone.Loop((time) => {
// fires once per bar; count bars and watch for phrase edges
Tone.Draw.schedule(() => {
if (bar % PHRASE === 0) {
palette = pickPalette(bar / PHRASE); // NEW palette every 8 bars (macro)
}
}, time);
bar++;
}, "1m").start(0); // "1m" = one measure/bar
This is the same layered-timescales idea we hit in the ambient episode -- independent cycles of different lengths producing something that feels alive because it's changing at more than one rate at once. Sync the twitch and the arc.
Same data, two senses: true generative audiovisual
Everything so far has sound driving visuals -- the audio leads, the picture follows. But the most elegant approach flips the hierarchy entirely: instead of one driving the other, you generate both from a single underlying source. One noise field, say, read two ways -- its value at time t sets the melody's pitch, and the same field drawn as a flow field sets the visuals. Neither is reacting to the other; they're siblings from the same parent. That's when sound and image feel truly inseparable, because they genuinely are -- one cause, two expressions.
// ONE source drives BOTH modalities. here: a slow noise walk feeds pitch AND colour.
const scale = ["C4","D4","E4","G4","A4","C5"]; // safe pentatonic (episode 120)
let t = 0;
const unified = new Tone.Loop((time) => {
const n = noise(t); // ONE value in 0..1 (Perlin, episode 12)
// read it as SOUND: pick a scale note from the noise value
const note = scale[Math.floor(n * scale.length)];
synth.triggerAttackRelease(note, "8n", time);
// read the SAME value as PICTURE: drive hue + position off it
Tone.Draw.schedule(() => {
bgHue = n * 360; // same noise -> colour
marker.x = n * width; // same noise -> position
}, time);
t += 0.05;
}, "8n").start(0);
Do you feel the difference? When the melody rises, the colour shifts with it -- not because the colour is watching the melody, but because they're both just the noise field wearing different clothes. This is my favourite way to think about the whole genre. You're not composing a picture and a soundtrack, you're composing a system, and letting it speak through both channels at once.
Latency: the last few milliseconds
I keep saying "40 milliseconds" and there's a practical wrinkle worth naming. Web Audio deliberately schedules a little bit ahead -- a lookahead buffer -- so it never runs dry and glitches. Tone.js exposes this, and if you're doing very tight work you sometimes want to nudge the visual to account for it, or reduce the lookahead when you need snappier response at the cost of a little stability.
// tune the audio scheduling lookahead. smaller = snappier sync, riskier glitches.
Tone.getContext().lookAhead = 0.05; // default is ~0.1s; drop it for tighter feel
// Tone.Draw already compensates by firing on the frame nearest the scheduled time,
// so most of the time you DON'T fight this - you just let Tone.Draw do its job.
Honestly, for 95% of what you'll build, using Tone.Draw.schedule correctly is all the latency handling you need -- it's already doing the hard part. I mention the lookahead knob mostly so you know it exists when you eventually build something where every millisecond matters, like a live instrument that has to feel instant under your fingers.
Recording the result: capture both at once
You'll want to save these things -- for a post, a loop, a little art piece. The browser can record the canvas and the audio together into one video file using MediaRecorder, by grabbing the canvas as a video stream and the audio context as an audio stream and merging them.
// record canvas + audio together into one .webm file.
const canvasStream = canvasEl.captureStream(60); // 60fps video track
const audioStream = Tone.getContext().createMediaStreamDestination();
Tone.Destination.connect(audioStream); // route all audio in
const merged = new MediaStream([
...canvasStream.getVideoTracks(),
...audioStream.stream.getAudioTracks(),
]);
const recorder = new MediaRecorder(merged, { mimeType: "video/webm" });
const chunks = [];
recorder.ondataavailable = (e) => chunks.push(e.data);
recorder.onstop = () => saveBlob(new Blob(chunks, { type: "video/webm" }));
recorder.start(); // ...call .stop() later
For a quick share this is perfect. When you want pristine quality (no dropped frames, exact timing) the pro move is to render the frames and the audio separately and stitch them with ffmpeg afterwards -- but that's a whole thing, and the live MediaRecorder route is more than good enough to get a lovely loop out the door tonight.
Your exercise: a generative audiovisual loop
Right, let's pull the entire toolbox into one thing that genuinly feels like one thing. The brief: an 8-bar loop where a constrained-random melody plays, and every note fires a matching visual, the beat drives a background pulse, and the phrase boundary shifts the palette. Sound and picture should feel inseparable -- if you muted it, the visual alone should feel slightly wrong, like it's missing half of itself. That's the target :-).
// AUDIOVISUAL LOOP: melody -> synced note-circles, beat -> pulse, phrase -> palette.
const scale = ["C4","D4","E4","G4","A4","C5"]; // pentatonic = can't sound bad
Tone.Transport.bpm.value = 100;
const synth = new Tone.PolySynth(Tone.Synth).toDestination();
const kick = new Tone.MembraneSynth().toDestination();
let pulse = 0, bar = 0, palette = pickPalette(0);
const marks = []; // visual note-events, faded over time
The note voice: on each eighth note, roll a scale note, play it, and schedule a coloured circle to appear at the pitch's height (log-mapped) exactly in sync.
// each note: sound + a circle whose HEIGHT is its pitch, fired in perfect sync.
const melody = new Tone.Loop((time) => {
if (Math.random() < 0.7) { // a bit of space - not every step
const note = scale[Math.floor(Math.random() * scale.length)];
synth.triggerAttackRelease(note, "8n", time);
Tone.Draw.schedule(() => {
marks.push({ x: width * 0.5, y: noteToY(note, p), life: 1,
col: palette[Math.floor(Math.random() * palette.length)] });
}, time);
}
}, "8n").start(0);
The beat voice punches the pulse, and the structure loop swaps the palette every 8 bars -- both fired through Tone.Draw so the picture never lags the sound.
// beat drives the pulse (staccato), phrase boundary swaps the palette (macro).
const beat = new Tone.Loop((time) => {
kick.triggerAttackRelease("C1", "8n", time);
Tone.Draw.schedule(() => { pulse = 1; }, time);
}, "4n").start(0);
const phrase = new Tone.Loop((time) => {
Tone.Draw.schedule(() => {
if (bar % 8 === 0) palette = pickPalette(bar / 8); // new colours each phrase
bar++;
}, time);
}, "1m").start(0);
Finally the p5 draw loop ties it together -- decay the pulse (legato), fade the note-marks, and paint. Notice there's no audio logic in here at all: the draw loop only ever reads state that the synced callbacks set. That separation is the secret to keeping it locked.
// draw: pure rendering. it only READS state the synced audio callbacks wrote.
function draw(p) {
pulse *= 0.9; // legato decay between beats
p.background(15, 15, 20);
p.noStroke();
// background pulse ring, swelling on each kick:
p.fill(255, 30); p.circle(p.width/2, p.height/2, 200 + pulse * 120);
// fade and draw every note-mark:
for (let i = marks.length - 1; i >= 0; i--) {
const m = marks[i];
m.life *= 0.95; // each mark fades out
p.fill(m.col[0], m.col[1], m.col[2], m.life * 255);
p.circle(m.x, m.y, 20 + (1 - m.life) * 60); // and grows as it fades
if (m.life < 0.02) marks.splice(i, 1); // reap the dead ones
}
}
Wire it all behind a "click to start" button (never forget Tone.start() -- we learned that the hard way a few episodes back), hit go, and watch. Notes climb and fall with the melody, the whole field pulses on the kick, and every eight bars the colours turn over. Mute it in your head for a second and you'll feel how naked the picture is without the sound -- that "incompleteness apart, wholeness together" is exactly what we were chasing.
Push it when you've got an evening: drive both the melody and the background from one shared noise field so they're true siblings, add a soft camera shake scaled by pulse, map the horizontal position of each mark to note velocity so louder notes drift right, or record a thirty-second loop with MediaRecorder and you've got a shareable little art piece. It's yours now, and no two loops will land the same :-).
't Komt erop neer...
- Sync is really a perceptual trick: land a visual within ~40ms of its sound and the brain fuses them into one event. Everything else is decoration on top of hitting that window reliably
- The hard part is that audio and visuals run on two different clocks -- Web Audio schedules ahead on a precise hardware clock,
requestAnimationFramefires roughly and can stutter. Reacting to audio in the draw loop always lands late Tone.Draw.schedule(fn, time)is the bridge: hand it the sametimeyour note uses and it fires the visual on the frame nearest that instant. This is the backbone of the whole episode -- schedule both, don't react- Staccato trigger + legato interpolation: punch a value hard on the beat, then ease it back every frame (episode 16). Sharp hit, smooth motion between -- the recipe for visuals that breathe instead of strobe. Match the visual's speed to the sound's attack envelope for extra fusion
- Lean on human wiring: high pitch -> high on screen (map in log space, because pitch is logarithmic - episode 13), bright, small. The metaphor reads as "correct" for free
- Sync at two scales -- a small event every beat, a big change (palette, behaviour) every phrase (8 bars). Micro keeps it lively, macro gives it an arc
- The most elegant approach: generate both modalities from one source (a shared noise field feeding pitch and colour). They're siblings from the same parent, not one chasing the other - that's true audiovisual unity
- Let Tone.Draw handle latency for you 95% of the time; the
lookAheadknob exists for when milliseconds matter. Capture the result withMediaRecordermerging canvas + audio streams into one file
So that's synchronization -- the thing this whole audio arc has quietly been building toward. You can bridge the two clocks with Tone.Draw, punch-and-ease your way to musical motion, map pitch to height the way brains already expect, sync at both the beat and the phrase, and -- best of all -- grow sound and image from a single seed so they're two faces of one system. That's a genuinly powerful place to stand: you've got a what, a when, and now a way to make the eyes and ears agree.
And that word "space" keeps tugging at me. We've been placing our visuals across a flat screen -- left, right, up, down -- but our sound has been mostly stuck in the middle of your head, mono, flat. Real spaces have depth and direction: a sound can come from behind you, off to your left, up high, moving past. What would it take to give sound an actual position in three dimensions, so a note doesn't just play but plays over there -- and to sync a moving visual to a sound that moves with it? That's the thread I want to pull next, and it opens up a whole dimension we've barely touched. Build your audiovisual loop first -- genuinly get one note landing dead-on the beat, it's a real "oh!" moment -- and then come back, because next time we give sound somewhere to be :-).
Sallukes! Thanks for reading.
X