MIRLCa-web: Live-coding with Freesound in the web browser

[Guest blog post by Nikhil Bullock]

As part of my thesis for the Sound and Music Computing Masters programme at Queen Mary University of London this summer, I implemented MIRLCa-web, a port of the SuperCollider extension MIRLCa to the browser. MIRLCa, developed by Anna Xambó Sedó (who was also my supervisor for this project together with Panagiota Anastasopoulou, a PhD student from Music Technology Group, Universitat Pompeu Fabra, who’s part of the Freesound team and author of the Broad Sound Taxonomy), is a live-coding system that makes extensive use of Freesound and its API. Live-coding is the practice of performing music by writing code, and MIRLCa provides tools to retrieve sounds from Freesound using various methods, such as picking a sound at random or filtering by the many acoustic properties provided by the Freesound API.

Given that Freesound is so large (more than 700,000 sounds at the time of writing), choosing sounds you like while trying to perform can be quite a challenge. MIRLCa tries to solve this by using a binary classifier neural network that chooses and plays only sounds that it predicts will be good. So for example, when you pick a random sound, it’s not truly a random sound, but a sound that the neural network predicts you’ll like chosen from a random selection. MIRLCa also lets you train your own neural network, meaning it’ll be tailored to your own preferences and musical taste.

MIRLCa-web takes these ideas and ports over to the browser, so you don’t have to install and set up SuperCollider. You can try it out here. At the time of writing, apart from a successful user study with five participants, the tool has been publicly tested for the first time and well-received in a workshop at the AI Soundscapes congress in Potsdam, Germany.

The workflow of MIRLCa-web is split into two phases: training mode and performance mode. During training mode (Figure 1) you listen to and rate sounds that have been pulled from Freesound. Each good or bad rating you provide is used to train the neural network. The neural network analyses the sounds based on their MFCCs (Mel-frequency cepstral coefficients), which are provided by the Freesound API. MFCCs capture the timbral properties of a sound, so they make a suitable choice as training data. Once you’ve trained the neural network, you’re then ready to use performance mode.

Performance mode (Figure 2) is where you do the live-coding, It’s in here where you can retrieve, play, and manipulate sounds by writing JavaScript. The way you interact with sounds is through Group objects, so to start making music, you first create a group:

x = Group();

Then you can retrieve sounds and add them to the Group:

// Pick 5 random "good" sounds.
for (let i = 0; i < 5; i++) {
  x.rand();
}

// Pick a "good" sound filtered by acoustic descriptors.
x.filter({ bpm: 120, roughness: "[70 TO 100]" });

As soon as the sounds are loaded, they start playing. You can manipulate the sounds within the group, for example by changing their playback rate and volume:

x.rate(3.5);
x.vol(0.5);

And add sound effects to the group, such as reverb and echo:

// Add a reverb to the group.
x.fx.add("reverb");

// Add an echo with a delay time of 0.5 seconds and feedback of 90%.
x.fx.add("delay").time(0.5).fb(0.9);

There are other effects as well, which you read more about in the guide.

You can also sequence the sounds into rhythms using the seqfunction. A sequence contains a series of xs and -s that determine when to play the sound or rest. Square brackets nest sequences, i.e. they create a sequence within a particular beat:

// Set the tempo.
setBPM(125)

// Create a sequence for the 4th sound in the group.
x.seq("x--x--[x--]", 3)

MIRLCa-web is a work in progress and we are still experimenting with different ideas and features. It’s also an open-source project (find the code here) and I welcome any suggestions, bug reports, or pull requests. Finally, I would like to thank the participants who contributed to the user study, as well as my supervisors Anna Xambó Sedó and Panagiota Anastasopoulou.

***

Nikhil Bullock is a London-based musician and researcher whose music is influenced by UK club culture and algorithmic composition. His practice explores live-coding and generative processes as tools for sonic exploration. He is currently pursuing an MSc in Sound and Music Computing at Queen Mary University of London.