Matthew Kaney at ITP

Final Project: Musical Mappings

Physical Computing

As part of David’s and my final project, we experimented with various methods for mapping position and movement of people in a space to musical output. As we explored, we were fundamentally trying to solve this list of (sometimes contradictory) challenges:

With this in mind, we partially implemented a variety of different mapping philosophies, most of which didn’t even survive our own personal testing. Not all worked for what we needed, but I think all have potential as musical interfaces. Here are approaches we tried, with a few notes:

Option 1: Position Based Pitch, No User Tracking

This is the simplest method. If there is something interesting (however your algorithm defines that) in a specific part of the screen, then play the pitch associated for that part of the screen. This can be done with just the shapes of users, or it can be done with the centers of blobs. However, this is just based on checking the on/off state of each cell every frame, so it doesn’t have a persistent memory of where people are or how they move over time.

Drawbacks: Without tracking users, we’re incredibly limited in what we can do. In particular, we have very little information over time, so we don’t have much we can do with interesting types of note attacks or decays.

Option 2: Position Based Pitch, User Tracking

Adding a sense of different users to the previous technique. This way, you can choose to only attack notes when the user moves, or enters a new cell, or something. Also, you can assign different users different instruments or voices, which we discovered was crucial for user understanding. In the end, this was the approach we took, with multiple adjustments.

Drawbacks: Because the mapping of pitches is based on screen, rather than physical, space, it’s sometimes difficult to tell when one is between pitch areas. Also, by creating a large enough pitch grid to be legible, it forces the user to move around quite a bit for a small amount of output control. We addressed this by allowing certain gestures that could adjust the pitch relative to the root pitch of the area. Also, by limiting the system to a small number of pitches, the music can be more harmonic, but it also limits the potential range, adding to the audible confusion of too many people playing notes within a constrained pitch space.

Option 3: Direction-Based Pitch

I briefly experimented with a system where movement in one direction would cause a series of ascending notes and movement in an opposing direction would cause descending notes, and vice versa. This allowed us to break up the rigid pitch grid, and the fact that the directions didn’t have to directly oppose (just mostly oppose) meant that the user could move in more interesting patterns, while running up and down the scale.

Drawbacks: Because pitch was movement-based, there was less control over when specific notes played, which confused the mapping. Also, it was difficult to play larger intervals, or specific sequences of notes, given the limitation of moving up and down relative to the user’s previously played note.

Option 4: Human Sequencer, Non-Position Based

With this system, each participant is assigned a note in a sequence. The pitch of that note could change based on position, and the length of the note could vary based on the amount of space the person occupies, but the fundamental sequence would be that Person 1’s note plays, Person 2’s note plays, and so on until it loops back to Person 1 again. By spacing out people’s contributions over time (rather than on different pitches or voices), more people can contribute to a more complex melody, without the music becoming to chaotic.

Drawbacks: The initial state (where one person enters and gets a loop of their own note, which they can then modify) is quite discoverable, I think, but also really boring. Without having three or more people, there’s little that can be done with this system. Also, by adding and removing notes to the sequence (and varying the notes’ durations), the system has a pattern, but very irregular rhythms.

Option 5: Human Sequencer, Position Based

As a variant on the previous idea, we also tried out a much more literal sequencer: we mapped the space on the floor to time and looped across, playing notes for each person the system encountered with time offsets relative to their position. To help communicate this, we added a basic beat that would play when users were in the space.

Drawbacks: This, in particular, suffered from the lack of visual feedback. It was clear that something was happening, but since the virtual “playhead” was immaterial, it was hard to tell when one’s note was about to be played. In addition, having to wait until the loop reaches the user means that exploratory gestures don’t necessarily have prompt feedback. This has potential, but it requires a user that’s more experienced/knowledgable than our intended users.