Matthew Kaney at ITP

Week 1: Run-Length Encoder


502

For this week’s assignment, I wrote a simple run-length encoder for ASCII text in Python. It counts the number of characters in a run of characters and encodes the results in another ASCII string. In addition, it also appends a special character (a slash) whenever a character is a numeric digit, so that there’s no confusion between which characters are actual data from the input file, and which are descriptions of run lengths.

I’ve written an encoder and decoder. As an example of what the output looks like, calling the function:

encode('aaaabbbaaaaaaaaaabnbbb11111kkkk')

Returns this result:

'4a3b10a1b1n3b5/14k'

And the decoder reverses this process.

Here’s the code:

# Python Run-Length Encoder
# Matthew Kaney 2015

def encode(input):
    output = ''
    lastChar = ''
    counter = 0
    for character in input:
        if character == lastChar:
            counter += 1
        else:
            if counter > 0:
                # Add an escape character ('/') for digits and slashes
                if (lastChar >= '0' and lastChar <= '9') or lastChar == '/':
                    lastChar = '/' + lastChar
                
                output += str(counter) + lastChar
                
            counter = 1
            lastChar = character
    
    # Append the last character
    if (lastChar >= '0' and lastChar <= '9') or lastChar == '/':
        lastChar = '/' + lastChar
    
    output += str(counter) + lastChar
    
    return output

def decode(input):
    output = ''
    counter = ''
    escapeCharacter = False
    
    for character in input:
        if escapeCharacter:
            output += character * int(counter)
            counter = ''
            escapeCharacter = False
        elif character >= '0' and character <= '9':
            counter += character
        elif character == '/':
            escapeCharacter = True
        else:
            output += character * int(counter)
            counter = ''
    
    return output

Computational Portrait: Max

Computational Portraiture


43

Max Project

For my computational portraiture project, I decided to create a portrait of my friend Max. Max and I both grew up in Oklahoma and moved to New York. Because of this shared experience, I’ve been thinking about the ways in which these places have shaped our identities, and wanted to explore this notion through portraiture. To begin, I captured a 3d portrait of Max using photogrammetry. I wanted a fairly realistic, solid model of him, that I could then contrast with the abstract, remembered spaces of his past.

While we created the portrait, Max and I talked for several hours about his past, discussing his memories and stories of places he lived. For many of the places we discussed, Max generously shared some photos with me. In Maya, I took these photos and manually sculpted them into a 3d model roughly corresponding to the space of the image. I build a simple Three.js-based web project that allows the user to scroll through these spaces, and by doing so, scroll through a narrative of Max’s life. Eventually I plan on editing the audio from our conversations into form where the user can navigate to specific locations and hear associated narration in Max’s voice.

Here’s a link to the project in progress. (I need to optimize the project more. For now, it takes a couple minutes to download.)

Simple ATtiny Synth, Week 2

Homemade Hardware


299

I modified last week’s synth circuit to run on an ATtiny. There are still some bugs, and some features that I’d like to include, but I’m pretty pleased with it so far. I’ve updated the schematic accordingly, and then created a new board layout. In addition to the ATtiny, I also used the low-pass filter circuit from here, which is also the inspiration for my code.

Synth_Week2_Circuit

Synth_Week2_Board

Simple Arduino Synth, Schematic

Homemade Hardware


350

I enjoyed playing with the LCD screen a lot last week, but I don’t yet feel confident enough to tackle a circuit that incorporates an LCD sans breakout board. So, for this first project, I’m taking a detour into building a basic MIDI synth.

This week I focused on getting Eagle up and running, and working out some of the details of MIDI control. The circuit uses an ATmega328 set up as a breadboard Arduino, connected to a MIDI input (this is basically the standard circuit that the MIDI folks publish, though I wound up using Tom Igoe’s schematic). The circuit responds to MIDI signals and uses the tone() command to generate appropriately-pitched beeps.

Here’s my breadboard:

Breadboard Arduino Synth

And here’s the Eagle schematic:

Arduino Synth Schematic

I have a long wishlist for this project. First, I’d love to do some more complex synthesis than just tone(), which is why I’m intrigued by this blog post about generating waveforms with an ATtiny85. Also, my MIDI support is incredibly bare bones at the moment—there are probably other channel controls and such (other than basic note on/off) that I should implement. I could also see adding an LM386 amplifier circuit to the mix to turn up the volume a bit. Also, the circuit currently runs off of the Arduino board’s 5v power, but I’ll eventually need to add a voltage regulator to the circuit as well.

We’ll see how many of those things I can include in the next couple of weeks.

Here’s the code:

#include <SoftwareSerial.h>

const byte rxPin = 2;
const byte txPin = 3;

// set up a new serial object
SoftwareSerial mySerial (rxPin, txPin);

// variables for MIDI parsing
byte currStatus = 0;
byte currDataIndex = 0;

// We'll need to store up to two different bytes of
// data for MIDI messages
byte currData[2];

byte currNote = 0;

void setup() {
  pinMode(rxPin, INPUT);
  pinMode(txPin, OUTPUT); // We never actually use this
  
  mySerial.begin(31250);
}

void loop() {
  // Pull in any available bytes
  while(mySerial.available() > 0) {
    byte newByte = mySerial.read();

    // Check if this is a status byte
    if(newByte >> 7 == 1) {
      // If it is, then pull out just the status code
      currStatus = (newByte >> 4) & 0b0111;
      currDataIndex = 0;
    } else {
      // If it's not a status byte, then it's a data byte
      currData[currDataIndex] = newByte;

      if(currDataIndex == 0) {
        // If this is the first of two data bytes, then
        // wait for the second byte
        currDataIndex = 1;
      } else if(currDataIndex == 1) {
        // If this is the second data byte, then we've
        // received a whole message. Do something now.

        // Check the message type
        if(currStatus == 0) { // Note off message
          // Only turn the tone off if the note that was
          // turned off is the same that we're currently
          // playing
          if(currNote == currData[0]) {
            noTone(8);
          }
        } else if(currStatus == 1) { // Note on

          // Check the velocity. If it's 0, then this
          // is basically a note off message.
          if(currData[1] == 0) {
            // Duplicate note off code
            if(currNote == currData[0]) {
              noTone(8);
            }
          } else {
            // If the velocity is non-zero, then we have a
            // legit note on event. Change the tone frequency.
            int freq = pow(2, (currData[0] - 69) / 12.0) * 440;
            tone(8, freq);

            // And make note of our new current note
            currNote = currData[0];
          }
        }

        // End of MIDI message. Next data byte will be the first
        // of the next message.
        currDataIndex = 0;
      }
    }
  }
}

Flappy Bird on the ATtiny85

Homemade Hardware


227

In this class, I want to explore building hardware for small LCD screens. To begin with, I used a piece of hardware that I’ve used some in the past, the Nokia 5110 screen sold by SparkFun. In order to test the capabilities of the ATtiny85 microprocessor, I decided to see how much of the iPhone game Flappy Bird I could re-implement with this processor and screen.

The Nokia 5110 screen is 84×48 pixels. The height of the screen is divided up into six rows. These rows are then divided into 84 1×8 pixel columns. Each column is represented by one byte of data, and you write data to the screen by shifting a byte at a time, filling each row left to right, and then moving down to the next row. For example, this shows how a byte with the decimal value 39 would render on the screen:

Layout of the LCD Screen

Flappy Bird Pixel Art

Because Flappy Bird is an iPhone game, I decided to use the screen in portrait orientation, effectively rendering the image in six columns, from bottom to top. I mocked up my pixel art graphics in Photoshop (shown at left). Here, I’ve tried out several possible birds, using the original character pixel art as a reference. Also, note the gray stripes indicating the divisions of the screen—by designing with this layout in mind, I was able to create graphics that rendered more efficiently on the screen. From this art, I created the graphics in code by manually typing arrays of byte literals (e.g. “0b00100111”).

Circuit Diagram

The Nokia 5110 screen has 8 pins, 5 of which should be connected to a microcontroller: SCLK (Pin 7), DN (Pin 6), and SCE (Pin 3), which are used for data transfer; D/C (Pin 5), which is used to determine whether the data is interpreted as image data or a control signal; and RST (Pin 4), which can be used to automatically reset the LCD screen. After getting the screen to work in this configuration, I decided that I didn’t need to ever reset the screen, freeing up a pin on my ATtiny to be connected to a pushbutton. The LCD screen requires 2.7-3.3VDC, so I powered the entire circuit off of a pair of AA batteries.

Overview

(more…)

Final Project: Two Poems

Reading and Writing Electronic Text


202

Finally, a pair of poems created as part of my final project:

Poem 1 (Read Live at our Final Performance)

In 2010:

i said: kidding, considering, answering
meaning, meeting, seeing, eating, being

you said: code is compromised, Restaurants used
preview fill, people come, typing is, code is compromised

i said: caring, wearing, bring, breaking, making
missing, seeing, being, feeling, dealing

you said: way do go, Doughnuts is owned, goal being
George emailed, Noah got Call, Express is

i said: computers is, Am wrapping, nobody is expecting
Chris isn’t, Kevin doesn’t want, backend is

you said: hoping, ping, playing, spelling, setting
staying, sitting, styling, trying, string, starting

i said: Here, I’m sending you the photos.
you said: okay, yeah, you want it on

In 2011:

i said: Sheriff issued, Becky fill, Chris saw
T-Rex believes, Chris did send, person was thinking giving

you said: command used, question is, numbers is
Daniel is looking, comments are, plants became

i said: putting, cutting, shutting, editing
meaning, evening, vending, sending, ending

you said: brain was, Pandora do, electricity is
email sent, notifications do, English are

i said: fair CNN, Historic Times
saddest emoticon, soda machine

you said: numbers is, Heitz has been, god had
tail is, parsers are, fact think, problem arrived

i asked: You mean the actual typing?
you said: if you stress too much you’ll die.

In 2012:

i said: specific reason, nice Christmas Boy
feeling bad, first-born child expensive, proprietary software

you said: irritating, awaiting, updating
living, leaving, leaning, learning, turning

i said: Pete look, re going, nobody was
narration is, Adobe fixed, list shared

you said: meeting, muttering, limiting
helping, yelling, telling, staring, starting

i said: disagreeing, distracting, tracking
studying, stuffing, mystifying, missing

you said: implying, importing, morning
compounding, complaining, compiling, coming

i said: Okay, you can come over now.
you asked: have you tried other browsers?

In 2013:

i said: Code signing is, paycheck is, names are
money is, Club has taken, thing seeing

you said: waking, taking, staying, sitting
hammering, hurting, irving, unnerving

i said: other hand, previous day’s stuff
physical right, task management system

you said: grammatical clarity, Happy St. Patrick’s Day
yoko ono, Progress report, lab rat

i said: people seemed, Technicolor had
commercials are, Liz said, music is going

you said: programming, rolling, ring, running
arguing, bragging, banging, banning, burning

i asked: But, what are you afraid happened?
you said: I can help you research stuff.

In 2014:

i said: Gruber doesn’t respond, show is
person checked, turnaround looking, OP is

you said: composing, compiling, camping
fucking, king, liking, looking, linking, thinking

i said: Chat seems, classes entered, people is
images posted, project is, New seems

you said: wedding, reading, rendering, ending
starting, steering, starring, string, trying, rising

i said: accounts has, listeners attached
clinic says, services thank, street including

you said: app was taken, people are, difference is
people arguing, class was, reason including

i asked: When are you going to be done?
you said: can you do me a favor

In 2015:

i said: discussing, licensing, sequencing
exciting, writing, pricing, trying, timing

you said: Christmas is, question asking, Queens doesn’t exist
records started being kept, thing looks, everyone thinks

i said: boring, morning, running, stunning
topping, dropping, operating, playing

you said: thread agree, York is visiting
emphasis added copied, kafkaesque is

i said: separate link, reading anything
Adventures Time, grad student, net neutrality

you said: map indicates, point think, cookies are messed
host are, K. Sounds, Schwab sent, decision has been

i asked: So, how do you want to play this?
you asked: Could you do something like that?

Poem 2

In 2010:

i said: Wednesday evening, version online
different exclusively-gay porn shops

you said: major characters, da da da da da da da
good game, silly thing, friends locations, birthday present box cardboard

i said: writing, trying, string, starring, storing
bugging, bring, boring, storing, scoring, caring

you said: names leads, stuff came, woman tied, error is
output buffering, Apple emailed, Chrome isn’t

i said: daily dose, new episode Douglas
fiery passion, Thanks lot, bip-bop music

you said: adding, landing, branding, boring
buying, bribing, biking, boxing, focusing

i said: I’ll talk to you tonight, promise.
you asked: take one back with you would you?

In 2011:

i said: book based, Kevin sends, Atkinson was
Liz says, Adam wants, designers decided

you said: answer is, tone intended, today is
core set, today is, solution was, TA is talking

i said: different times, sandwich home, straight women
RIcky Gervais, something different, MAME rom

you said: sticking, setting, sitting, staying
aligning, lightning, fighting, firing, flying

i said: old code, research scientist, parents blown-glass vase
Muppet franchise, funny story, blows glass

you said: overdue balance, meals week, Prismatic Pumpkin
new group, multiple star reviews, INTERNET PEOPLE

i said: What do you mean, “what text is this?”
you asked: did you use that phone number?

In 2012:

i said: addressing, pressing, depressing
falling, fling, filling, filing, floating, loading

you said: fucking STRING, right sorry, friend Steve
new tab, word chippy, case scenario

i said: Pete look, Man thought, phone upgrade hanging
Pete included, paper burned, legislature was tricked

you said: thing has happened, heh was, post saw
stuff running, messages coming, person working

i said: statement needs, tie is, someone addressing
way do, proposal want, ones are, rule is

you said: internet security, Marc Heitz
free software people, last night, new petition

i asked: What would you like to do to me?
you said: do you even have to ask

In 2013:

i said: Linux is, Times had, time visited
way ask, Stewart talked, Stewart talked, references are

you said: picking, packing, panicking, banning
morning, warning, boring, bing, buying, banging

i said: technique had, music streaming wasn’t
movies come, wife connotes, person seems, anybody explained

you said: Linux wireless sadness, internet petitions
regular Milky Way, holy shit brandi

i said: release party, communication right
good Kelly, website design service, Kansas City

you said: certain things, love Doug, robot band
new car, school shooting stuff, hundai accent

i asked: You think Emily or Simon?
you asked: Can you wait up till midnight?

In 2014:

i said: great project, pretentious white people
first people, Visual Language assignment

you said: hell got delayed, phone call is, Mattel pulled
class did, days is, account is, couples doing

i said: mailing, bailing, boiling, buying
going, outgoing, knowing, running, opening

you said: size happened, Businesses stayed, class got caught
text file, Apple has, blob sounds, class is changing

i said: trying, turing, during, driving
chatting, cheating, seeing, sing, sawing, swing

you said: boring, porting, printing, inviting
phrasing, freezing, rising, raising, raining

i said: If it’s loose, you can pull it off.
you asked: What are you going to do?

In 2015:

i said: separate link, fine promise, dubious circumstances max
Union Square, physical membership card

you said: Reilly admits, section is, cards get
something making, robot doing, post contains

i said: grocery store, Booty Don’t Lie
Ice King, right same, music industry, Steve Jobs

you said: eating, sting, sitting, splitting, selling
summarizing, amazing, mailing, making

i said: melody were, lady sitting
Romney is, White has, Club doesn’t, Pizza was named

you said: New York, regular food list, new year
fine right, happy hour today, gain media attention

i asked: How long do you think it will take?
you asked: do you have a preference?

Final Project: Code and Description

Reading and Writing Electronic Text


228

Generative text easily lends itself to conceptual, often humorous writing, a tone that I frequently adopt in my work. So for my final project, I wanted push myself towards a more personal approach to electronic text. As a source, I looked to Gmail chat messages sent between my husband and myself. We’ve sent roughly 45,000 messages back and forth, covering the six-year span of our relationship (except for several months where we used the now-defunct Google Wave.) Chatting has always been an important, and sometimes primary, form of communication, and these chats form a rich sample. Loving, angry, depressed, intimate, and often painfully banal, these chats archive the daily facts of what happened and how we felt in our respective lives.

In a previous post, I discuss how I collected the data. Once I had the data collected and parsed, the writing could begin. To begin, I wrote a series of prototype programs, each trying to randomly extract different types of information from the messages. From these loose experiments, I developed my favorites, eventually turning them into self-contained methods.

Grabbing Noun Phrases

One thing I immediately noticed what how much of our communication was basically “Are you there? What’s up?” type of messages. Reading 200 such messages back to back could be compelling poetry, but it’s not very narrative. I really wanted to tell the story of the things that happened to us, so noun phrases seemed like a good place to start. I used TextBlob to do part of speech tagging, though instead of using their built-in noun phrase extraction, I did my own, because it allowed me a bit more control over what phrases I looked for (and I was happier with the results I got).

from textblob import TextBlob

#------------ From a set of rows, get all noun phrases -----
# (Very simplified; "noun phrase" here mean x * adj + y * noun)
def getNounPhrases(chatList):
    phrases = []
    
    for row in chatList:
        newBlob = TextBlob(row[2])
        
        phrase = [];
        hasNoun = False;
        
        for tag in getTags(newBlob):
            if len(tag[0]) > 1:
                if tag[1] == 'JJ':
                    phrase.append(tag[0]);
                elif tag[1] == 'NN' or tag[1] == 'NNS' or tag[1] == 'NNP':
                    hasNoun = True;
                    phrase.append(tag[0]);
                else:
                    if hasNoun and len(phrase) > 1:
                        phrases.append(' '.join(phrase));
                    hasNoun = False;
                    phrase = [];
        
        if hasNoun and len(phrase) > 1:
            phrases.append(' '.join(phrase));
            hasNoun = False;
    
    return phrases

Running this code on things I said in 2010 yields “good inspiration, empty box, other box, half full, organic mac cheeses”, among others.

Getting Simple Sentences

From there, I realized that I could similarly search for tiny sentences, in the most basic “sentence = noun + verb” elementary grammar sense. These have even more narrative thrust than a mere list of noun phrases.

#-------------- Get Subject + Verb ---------------------------
def getTinySentences(chatList):
    phrases = []
    
    for row in chatList:
        newBlob = TextBlob(row[2])
        
        for sentence in newBlob.sentences:
            phrase = []
            
            for tag in getTags(sentence):
                # Ignore single-character "words"
                if len(tag[0]) > 1:
                    if tag[1][0:2] == 'VB':
                        # If we get a verb, check if we already
                        # have a noun. If so, add the verb to our
                        # phrase.
                        if len(phrase) > 0:
                            phrase.append(tag)
                    elif (tag[1] == 'NN' or tag[1] == 'NNS' or
                          tag[1] == 'NNP' or tag[1] == 'PNP'):
                        # Otherwise, if we have a noun or pronoun,
                        # start assembling the phrase
                        phrase = [tag]
                    else:
                        # If we get any other word, then we're no
                        # longer looking at a phrase. If we've already
                        # collected a phrase, append it to the output.
                        if len(phrase) > 1:
                            newPhrase = ''
                            
                            for word in phrase:
                                newPhrase += word[0] + ' '
                                
                            phrases.append(newPhrase[:-1])
                        
                        phrase = []
    return phrases

For example, “circle indicates, girls watch, noise got” and so on.

Getting “—ing” Words

From there, I looked at getting all gerunds (or rather, all words ending in “—ing” except for words that were variants of “thing”, because that was obnoxious to show up in a list of verbs). The code here is pretty simple:

#-------------------- Get a list of "gerunds" from a chat list -----------
def getGerunds(chatList):
    gerunds = set();

    for row in chatList:
        blob = TextBlob(row[2])
        
        for word in blob.words:
            if word.lower()[-3:] == 'ing' and word.lower()[-5:] != 'thing':
                gerunds.add(word.lower());
    
    return gerunds;

This yields “saying, walking, breaking, pointing” etc. Nice, but it got much better when I sorted by phonetic similarity, as I did in a much older project. I built a second function that sorted the output of the first function (while not allowing duplicates, hence the set in the gerund function).

# Given a word and a list of words/phrases, sort the list of words/phrases
# based on how similarly they sound to the given word. Not rhyming per se,
# but a nice combo of rhyme, alliteration, assonance, and so on.
def rankProximity(comparison, wordList):
    comparison = pronouncer.cleanWord(comparison)
    outputList = []
    
    for word in wordList:
        cleanWord = pronouncer.cleanWord(word.split()[0])
        
        if cleanWord in pronouncer.phoneticDict and comparison != cleanWord:
            matcher = difflib.SequenceMatcher(None,
                                pronouncer.phoneticDict[comparison][0],
                                pronouncer.phoneticDict[cleanWord][0])
            similarity = matcher.ratio()
            
            outputList.append([word, similarity]);
    
    outputList.sort(key=lambda x: x[1])
    outputList.reverse();
    
    return outputList;

Sorting a given list (again, me in 2010) by their proximity to the word “making” gives us: mocking and taking (at 80% match), breaking (73% match), moneymaking (71% match), and so on.

General Search and a Couple Utilities

Also, I just wanted to be able to search for specific words, and this general-purpose code fit the bill:

def chatGrep(search, chatList, maxLength=10):
    output = []
    
    for row in chatList:
        blob = TextBlob(row[2])
        
        for sentence in blob.sentences:
            if search.lower() in sentence.lower() and len(sentence.words) < maxLength:
                output.append(sentence)
    
    return output

Finally, you might notice that I use getWords() and getTags() methods several times above. Basically, TextBlob parses apostrophes and contractions out as separate tokens, which is I’m sure sometimes useful, but more trouble than it’s worth here. So, I wrote a couple methods to reassemble contractions, while still performing the useful natural language stuff that TextBlob gives us.

def getWords(sentence):
    phrase = sentence.words
    
    out = []
    
    suffixes = ["n't", "'s", "'m", "'re", "'ve", "'d", "'ll"]
    
    for word in phrase:
        if word.lower() in suffixes:
            out[-1] = out[-1] + word
        else:
            out.append(word)
            
    return out

def getTags(sentence):
    phrase = sentence.tags
    
    out = []
    
    suffixes = ["t", "s", "m", "re", "ve", "d", "ll"]
    suffix = False
    
    for word in phrase:
        if len(out) > 0 and (word[0].lower() == "'" or word[0].lower() == "n"):
            out[-1][0] = out[-1][0] + word[0]
            suffix = True
        elif suffix and word[0].lower() in suffixes:
            out[-1][0] = out[-1][0] + word[0]
        else:
            out.append([word[0], word[1]])
            suffix = False
            
    return out

Pulling it all Together

Of course, the important thing about these chats was that I knew who said them, and when. In my final poem, I wanted to sort things chronologically by year, and distinguish the things that Caleb and I said, so I wrote a simple program for returning a subset of rows, before passing that data into any of the above functions.

#------------ Basic filtered rows -------------------------
def filteredRows(person, year):
    subset = []
    
    for row in rows:
        if row[1] == person and row[0].year == year:
            subset.append(row)
    
    return subset

With all these functions, I wrote a script for assembling them. First into lines, and then into a poem:

# From a set of gerunds, make a list of a certain number of syllables, where
# each no word is repeated, and each word sounds similar to the previous.
def getGerundList(gerunds, maxSyllableLength):
    firstWord = random.choice(list(gerunds))
    
    while pronouncer.cleanWord(firstWord) not in pronouncer.phoneticDict:
        firstWord = random.choice(list(gerunds))
    
    gerundList = [firstWord]
    syllableLength = len(pronouncer.getMeter(firstWord))
    
    while syllableLength < maxSyllableLength:
        gerunds.remove(gerundList[-1])
        sortedList = rankProximity(gerundList[-1], gerunds)
        
        gerundList.append(random.choice(sortedList[:3])[0])
        syllableLength += len(pronouncer.getMeter(gerundList[-1]))
    
    return gerundList

# From a set of phrases, string together a list of phrases of the desired
# syllable length.
def getPhraseList(phrases, maxSyllableLength):
    firstPhrase = random.choice(phrases)
    
    while not pronouncer.check(firstPhrase):
        firstPhrase = random.choice(phrases)
    
    phraseList = [firstPhrase]
    syllableLength = len(pronouncer.getMeter(firstPhrase))
    
    tries = 0
    
    while syllableLength < maxSyllableLength and tries < 300:
        newPhrase = random.choice(phrases)
        
        if pronouncer.check(newPhrase):
            phraseList.append(newPhrase)
            syllableLength += len(pronouncer.getMeter(newPhrase))
            tries = 0
        
        tries += 1
    
    return phraseList

#-------------------------------------------------------------
# Generate one stanza for each year (2010-2015)
for year in range(2010, 2016):
    myRows = filteredRows('matthew', year)
    hisRows = filteredRows('caleb', year)
    
    print 'In ' + str(year) + ':\n'
    
    # Each stanza has six pairs of lines
    for i in range(0, 6):
        # The speaker alternates each line
        if i % 2 == 0:
            currRows = myRows
            currPerson = 'i'
        else:
            currRows = hisRows
            currPerson = 'you'
        
        # The speaker can say one of three types of things, randomly chosen
        path = random.randint(0, 2)
        
        if path == 0:
            gerunds = getGerunds(currRows)
            gerundList = getGerundList(gerunds, 8);
            print currPerson + ' said: ' + ', '.join(gerundList)
            gerundList = getGerundList(gerunds, 10);
            print ', '.join(gerundList)
        elif path == 1:
            tinySentences = getTinySentences(currRows)
            tinySentenceList = getPhraseList(tinySentences, 8)
            print currPerson + ' said: ' + ', '.join(tinySentenceList)
            tinySentenceList = getPhraseList(tinySentences, 10)
            print ', '.join(tinySentenceList)
        elif path == 2:
            nounPhrases = getNounPhrases(currRows)
            nounPhraseList = getPhraseList(nounPhrases, 8)
            print currPerson + ' said: ' + ', '.join(nounPhraseList)
            nounPhraseList = getPhraseList(nounPhrases, 10)
            print ', '.join(nounPhraseList)
        
        print ''
    
    #---- End each year/stanza with a pair of sentences where ----
    #--------------- each of us addresses the other --------------
    mySentences = pronouncer.findWordsWithLength(8, chatGrep('you ', myRows))
    mySentence = random.choice(mySentences)
    
    if mySentence[-1] == '?':
        verb = 'asked'
    else:
        verb = 'said'
    
    print 'i ' + verb + ': ' + str(mySentence)
    
    hisSentences = pronouncer.findWordsWithLength(7, chatGrep('you ', hisRows))
    hisSentence = random.choice(hisSentences)
    
    if hisSentence[-1] == '?':
        verb = 'asked'
    else:
        verb = 'said'
    
    print 'you ' + verb + ': ' + str(hisSentence) + '\n'

Bonus: Better Pronouncing Module Functionality

I used my CMU Pronouncing Dictionary functions on this project to get lines with certain numbers of syllables, and compare the similarity of two words. Unfortunately, this only works for words in the speaking dictionary, which was limiting, especially considering I write linguistic monstrosities like “bit-shifting/packing/masking”. So, I updated my pronouncing library to break apart such compound words and then evaluate them based on the sum of the pronunciation of each part. In the interest of full disclosure, that code is here too:

import string

# Array of all vowel sounds in the CMU speaking dictionary
vowelSounds = ["AA", "AE", "AH", "AO", "AW", "AY", "EH", "ER", "EY",
               "IH", "IY", "OW", "OY", "UH", "UW"]

# Now, import and populate the CMU speaking dictionary
cmu = open('cmudict', 'r')

phoneticDict = {}

for line in cmu:
    tokens = line.split()
    
    # Only grab the first pronunciation in the dictionary, for now
    if tokens[1] == "1":
        key = tokens[0]
        phonemes = []
        stresses = []

        # Some phonemes have a number indicating stress (e.g. "EH1").
        for phoneme in tokens[2:]:
            # If this is one of those, split the phoneme from the stress
            if not phoneme.isalpha():
                stresses.append(int(phoneme[-1]))
                phoneme = phoneme[:-1]

            phonemes.append(phoneme)

        phoneticDict[key] = (phonemes, stresses)

cmu.close()

# Convert tokens (with attached whitespace, punctuation, and irregular
# capitalization) to a clean, all-caps form
def cleanWord(word):
    newWord = word.upper()
    newWord = newWord.strip(string.punctuation)
    return newWord

# Check whether two words are both a) in the pronouncing dictionary and
# b) rhyming. Returns True or False.
def rhymingWords(firstWord, secondWord):
    firstWord = cleanWord(breakUpWords(firstWord)[-1])
    secondWord = cleanWord(breakUpWords(secondWord)[-1])
    
    if firstWord == secondWord:
        return False
    
    if firstWord in phoneticDict:
        searchSyllables, searchStresses = phoneticDict[firstWord]
    else:
        return False
    
    if secondWord in phoneticDict:
        wordSyllables, wordStresses = phoneticDict[secondWord]
    else:
        return False
    
    lastPhoneme = ''
    stressCounter = 1
    lastStress = 0

    for i in range(1, 1 + min(len(searchSyllables), len(wordSyllables))):
        if (searchSyllables[-i] == wordSyllables[-i] and
           stressCounter <= len(wordStresses) and
           stressCounter <= len(searchStresses)):
            lastPhoneme = searchSyllables[-i]

            if lastPhoneme in vowelSounds:
                lastStress = (wordStresses[-1 * stressCounter] and
                             searchStresses[-1 * stressCounter])

                stressCounter += 1
        else:
            break

    if (lastPhoneme in vowelSounds) and (lastStress > 0):
        return True
    else:
        return False

# For a given string, split the string into individual words and then return
# the meter of the entire string of words, in terms of stresses where
# 0 = no stress, 1 = primary stress, and 2 = secondary stress.
# For example, [1, 0, 1, 1, 0, 2]
def getMeter(line):
    words = breakUpWords(line)
    meter = [];
    
    for word in words:
        currWord = cleanWord(word);
        
        if currWord in phoneticDict:
            meter += phoneticDict[currWord][1]
        else:
            return False;
    
    return meter

# Get a list of words that rhyme with a certain word. The first parameter
# is the word that the results should rhyme with. The optional second
# parameter is the list of words for searching in. If no list is specified,
# the function will search the entire pronouncing dictionary.
def findWordsWithRhyme(rhyme, searchList=phoneticDict.keys()):
    result = [];
    
    for word in searchList:
        if rhymingWords(rhyme, word):
            result.append(word);
    
    return result;

# Return a list of words that have the given pattern of stresses. The first
# parameter is a list of stresses (0, 1, or 2). The optional second parameter
# is the list of words for searching in. If no list is specified, the
# function will search the entire pronouncing dictionary.
def findWordsWithMeter(meter, searchList=phoneticDict.keys()):
    result = [];
    
    for word in searchList:
        searchMeter = getMeter(word);
        
        if searchMeter == meter:
        	result.append(word);
    
    return result;

# Return a list of words with a certain syllable length. The first parameter
# is a number of syllables, and the optional second parameter is a list of
# legal words or phrases. If no list is specified, the function will
# search the entire pronouncing dictionary.
def findWordsWithLength(syllableLength, searchList=phoneticDict.keys()):
    result = [];
    
    for word in searchList:
        searchMeter = getMeter(word);
        
        if searchMeter and len(searchMeter) == syllableLength:
        	result.append(word);
    
    return result;

# Convert a phrase (multiple words, as well as compound words formed with
# hyphens or slashes) into a list of single words.
def breakUpWords(phrase):
    if cleanWord(phrase) in phoneticDict:
        return [phrase]

    input1 = phrase.split()
    output = []

    for word1 in input1:
        input2 = word1.split('-')

        for word2 in input2:
            input3 = word2.split('/')

            for word3 in input3:
                output.append(word3)
    return output

# Check whether a phrase consists of a legal (that is, in the pronouncing
# dictionary) word, or a series of legal words (either in a phrase, or in
# compound words). Returns True or False.
def check(phrase):
    words = breakUpWords(phrase)

    for word in words:
        if cleanWord(word) not in phoneticDict:
            return False

    return True

Final Project Prologue: Harvesting Data

Reading and Writing Electronic Text


270

For my final project, I needed all of the Google chats my husband Caleb and I have ever sent each other. The next post gets into how and why I made poetry with that text. This post just covers how I got the data in the first place.

Collecting the Data

I first tried to export everything using Google Takeout, Google’s cross-product data export tool. I’ve used it successfully in the past, but for whatever reason, it ran into trouble this time, and gave me a file with large swaths of missing messages and bad data. While trying to diagnose that problem, I found another solution: this Google Apps Script for automatically scraping chats out of Gmail and populating a Google Spreadsheet. Google Apps Scripts can only run for so long (and will give you an error if you try sending too many requests to Google’s servers), so I modified the script to allow me to only request a range of messages, manually re-running the script several times over the course of a couple hours.

I set up the script so that the first column of my spreadsheet is the message thread ID number (counting up chronologically as you move back in time—useful for keeping track of which messages were already requested), the second column is the date, the third and fourth columns are sender and recipient, and the last column contains the full text of the message. The end result looked like this:

Chat Spreadsheet

With my spreadsheet, I manually deleted messages with people other than Caleb, combined the messages sent from both my personal and NYU accounts, and exported all the data as a CSV.

Parsing the Data

Unfortunately, six years is a long time for Google to maintain consistent data formatting, particularly while completely overhauling their instant messaging products. This means that the new messages were formatted correctly when my script scraped them, like so:

2/27/2015 Matthew Caleb Any good burrito places in Manhattan?

But scraping older messages gave me something like this:

8/14/2011 Caleb Matthew Caleb: Toaster: black, white, or red?
me: Hey. Do you still need toaster color input?
Caleb: Yes
Caleb: If you could.
me: How about black?
Caleb: Sure
me: Sorry I didn't get back to you earlier. I went kind of sleep for a couple of hours.

…with the body of the “message” saved as: <span><span style="font-weight:bold">Caleb</span>: Toaster: black, white, or red?</span><br><span><span style="font-weight:bold">me</span>: Hey. Do you still need toaster color input?</span>, and so on.

In a few cases, for whatever reason, times were included in the message body as well:

7/26/2010 Caleb Matthew
12:27 PM me: Are you there?
 Caleb: yes
12:28 PM me: Cool. I'm having lunch now.
 Caleb: cool
  i'm trying to decide what to do for lunch
 me: Ah.
12:29 PM Caleb: how has your day been?
12:30 PM me: Pretty good. Finished up web ads, pending revisions.
 Caleb: cool

Obviously, this meant that the sender, recipient, and timestamp information associated with the chat thread was not useful. Instead, I had to use Beautiful Soup to parse out these HTML-formatted chat threads and extract the relevant data. The resulting data was a list of chat messages, each stored of a list with the format: [timestamp, recipientName, messageText]. Because all of this HTML parsing took several minutes, I only did it once, and then I used the Pickle module to store my clean data to a file.

After a lot of tinkering, this is what I came up with:

import csv
from dateutil.parser import *
from bs4 import BeautifulSoup
import pickle

# Given a row of the CSV file (which may or may not contain
# multiple threaded messages), return a list individual chat messages.
def formatRow(input):
    rows = []
    
    # The main CSV timestamp will be the default timestamp for all messages
    timestamp = parse(input[0])
    
    # The main sender will be the default sender, unless others are specified.
    # (Because GChat formats the usernames differently at different points,
    # this simple string search is enough of a check)
    if 'caleb' in input[1].lower():
        sender = 'caleb'
    elif 'matthew' in input[1].lower():
        sender = 'matthew'
    
    # Now, pull the text in the message or thread
    threadText = input[3]
    
    # Check for empty "messages"
    if str(threadText) == '':
        return []
    
    # Beautiful soup doesn't like weak breaks, so remove them
    threadText = threadText.replace('<wbr>', '')
    
    # Parse the thread text as HTML. This will both remove HTML-encoded
    # special characters and allow us to tear apart old threads that were
    # all saved in the same message.
    thread = BeautifulSoup(input[3])
    
    # I don't want to read links in my poetry, so remove them.
    for link in thread.findAll('a'):
        link.extract()
    
    for msg in thread.html.body.children:
        if msg.name == 'p':
            # If the text is stored in a <p> tag, then it's a simple message.
            rows.append([timestamp, sender, unicode(msg.contents[0])])
        else:
            # If the text is organized by <div> tags, then it has built-in
            # timestamps.
            if msg.name == 'div':
                # Update the base timestamp with this new hour and minute
                if len(msg.contents[0].contents[0].strip()) > 0:
                    newTimestamp = parse(msg.contents[0].contents[0].strip())
                    timestamp = timestamp.replace(hour=newTimestamp.hour,
                                                  minute=newTimestamp.minute)
                msg = msg.contents[1].span
            
            # If the text is contained in a span tag, then it maybe 
            # has the sender's name in bold before the message.
            if msg.name == 'span':
                # Update the message
                if msg.contents[0].name == 'span':
                    if 'caleb' in msg.span.contents[0].lower():
                        sender = 'caleb'
                    elif 'me' in msg.span.contents[0].lower():
                        sender = 'matthew'
                    content = unicode(msg.contents[1])
                else:
                    content = unicode(msg.contents[0])
                
                # Remove the colon if present
                if content[:2] == ': ':
                    content = content[2:]
                
                # Append a new row
                if len(content) > 0:
                    rows.append([timestamp, sender, content])
    return rows

#---------------------- Main Logic ------------------------

# This is the list of data that we'll want in the future.
rows = []

with open('source/RawChatArchive.csv', 'rb') as csvfile:
    messageList = csv.reader(csvfile)
    for row in messageList:
        formatted = formatRow(row)
        rows += formatted        

# The pickle module allows us to simply save a Python data structure in a
# file, no manual parsing or formatting necessary.
pickle.dump(rows, open("chatlog.p", "wb"))

Assignment 4: Pronouncing Module

Reading and Writing Electronic Text


15

For this assignment, I focused on extending the functionality of my CMU pronouncing dictionary work from earlier in the semester, and wrapping up everything in a module I called “pronouncer.py”.

Rhyme

My module can be used to test whether two words rhyme. Note that both words must be in the CMU speaking dictionary. If they’re not, then the function returns false.

>>> pronouncer.rhymingWords(‘apple’, ‘orange’)
False

You can also search the pronouncing dictionary for rhymes:

>>> pronouncer.findWordsWithRhyme(‘persuasive’)
[‘INVASIVE’, ‘PERVASIVE’, ‘SUPERABRASIVE’, ‘ABRASIVE’, ‘EVASIVE’]

Or, if you want to work within an existing corpus, you can specify a list of words to search within:

>>> pronouncer.findWordsWithRhyme(‘lemon’, [‘demon’, ‘yemen’, ‘salmon’])
[‘yemen’]

Meter

Based on the stress information in the pronouncing dictionary, the module can do incredibly simple scansion-type analysis (expressed as a list of 1s, 2s, and 0s, for primary stress, secondary stress, and no stress, respectively). Currently, the system makes no attempt to guess meter for words it doesn’t know, so this isn’t always that helpful.

>>> pronouncer.getMeter(‘To sit in solemn silence’)
[1, 1, 0, 1, 0, 1, 0]

More helpful is the ability to search for words that have a given stress. Like the rhyme functions, this can either search the entire pronouncing dictionary, or a provided word list.

>>> pronouncer.findWordsWithMeter([1, 0, 1, 0, 0])
[‘VIVISEPULTURE’, ‘NONEXECUTIVE’, ‘SUNAMERICA’, ‘GENEMEDICINE’, ‘PUBLIC-SPIRITED’, ‘SELF-DELIVERANCE’, ‘PORT-VICTORIA’, ‘OVERSHADOWING’, ‘SELF-SUFFICIENCY’, ‘PHOTOFINISHING’, ‘MOCK-HEROICALLY’]

Module Code

import string

# Array of all vowel sounds in the CMU speaking dictionary
vowelSounds = ["AA", "AE", "AH", "AO", "AW", "AY", "EH", "ER", "EY",
               "IH", "IY", "OW", "OY", "UH", "UW"]

# Now, import and populate the CMU speaking dictionary
cmu = open('cmudict', 'r')

phoneticDict = {}

for line in cmu:
    tokens = line.split()
    
    # Only grab the first pronunciation in the dictionary, for now
    if tokens[1] == "1":
        key = tokens[0]
        phonemes = []
        stresses = []

        # Some phonemes have a number indicating stress (e.g. "EH1").
        for phoneme in tokens[2:]:
            # If this is one of those, split the phoneme from the stress
            if not phoneme.isalpha():
                stresses.append(int(phoneme[-1]))
                phoneme = phoneme[:-1]

            phonemes.append(phoneme)

        phoneticDict[key] = (phonemes, stresses)

cmu.close()


# Convert tokens (with attached whitespace, punctuation, and irregular
# capitalization) to a clean, all-caps form
def cleanWord(word):
    newWord = word.upper()
    newWord = newWord.strip(string.punctuation)
    return newWord

# Check whether two words are both a) in the pronouncing dictionary and
# b) rhyming. Returns True or False.
def rhymingWords(firstWord, secondWord):
    if firstWord == secondWord:
        return False
    
    firstWord = cleanWord(firstWord)
    secondWord = cleanWord(secondWord)
    
    if firstWord in phoneticDict:
        searchSyllables, searchStresses = phoneticDict[firstWord]
    else:
        return False
    
    if secondWord in phoneticDict:
        wordSyllables, wordStresses = phoneticDict[secondWord]
    else:
        return False
    
    lastPhoneme = ''
    stressCounter = 1
    lastStress = 0

    for i in range(1, 1 + min(len(searchSyllables), len(wordSyllables))):
        if (searchSyllables[-i] == wordSyllables[-i] and
           stressCounter <= len(wordStresses) and
           stressCounter <= len(searchStresses)):
            lastPhoneme = searchSyllables[-i]

            if lastPhoneme in vowelSounds:
                lastStress = (wordStresses[-1 * stressCounter] and
                             searchStresses[-1 * stressCounter])

                stressCounter += 1
        else:
            break

    if (lastPhoneme in vowelSounds) and (lastStress > 0):
        return True
    else:
        return False

# For a given string, split the string into individual words and then return
# the meter of the entire string of words, in terms of stresses where
# 0 = no stress, 1 = primary stress, and 2 = secondary stress.
# For example, [1, 0, 1, 1, 0, 2]
def getMeter(line):
    words = line.strip().split()
    meter = [];
    
    for word in words:
        currWord = cleanWord(word);
        
        if currWord in phoneticDict:
            meter += phoneticDict[currWord][1]
    
    return meter

# Get a list of words that rhyme with a certain word. The first parameter
# is the word that the results should rhyme with. The optional second
# parameter is the list of words for searching in. If no list is specified,
# the function will search the entire pronouncing dictionary.
def findWordsWithRhyme(rhyme, searchList=phoneticDict.keys()):
    result = [];
    
    for word in searchList:
        if rhymingWords(rhyme, word):
            result.append(word);
    
    return result;

# Return a list of words that have the given pattern of stresses. The first
# parameter is a list of stresses (0, 1, or 2). The optional second parameter
# is the list of words for searching in. If no list is specified, the
# function will search the entire pronouncing dictionary.
def findWordsWithMeter(meter, searchList=phoneticDict.keys()):
    result = [];
    
    for word in searchList:
        cleaned = cleanWord(word);
        
        if cleaned in phoneticDict and phoneticDict[cleaned][1] == meter:
        	result.append(word);
    
    return result;

Midterm: The/Is/Is/The

Reading and Writing Electronic Text


310

For my midterm, I decided to focus more on my work with the CMU pronouncing dictionary, in order to create better meter and rhyme. I was still trying to figure out ways to extract interesting words from a source text. Eventually, this will have to involve some natural language parsing, I’m sure, but for now, I discovered that I could simply search for phrases that take the form “the [something]” or “is [something]”.

The Form

My poetic form is “The/Is/Is/The”, for lack of a better name. It consists of four eight-line stanzas. Each line consists of two-word phrases, starting with either “the” or “is”: In the first and third stanzas, the “the” lines are followed by “is” lines, and in the second and fourth stanzas, the “is” lines are followed by “the” lines.

I was interested in the way that the structure “is [something]” can be used as either an absolute statement or a question. This form is designed to distill a document (presumably non-fiction, in the present tense) to some of its most basic assertions. By alternating “is” lines and “the” lines, I think the poem creates some ambiguity between these two readings.

For the meter, I started with each line having three syllables, with the stress on the second, such as “the pathway is winding” (two amphibrachs, x / x | x / x). When that became too repetitive, I added two more patterns that fit within the same scheme: “the wall is unbroken” (x / | x x / x) and “the result is confusing” (x x / | x x / x). Additionally, the last line of each stanza is two syllables, and the last lines of a pair of stanzas rhyme.

Poems

Here’s a poem generated from the U.S. Senate Report on the CIA’s Detention and Interrogation Program:

the arrest
is entitled
the techniques
is familiar
the u.s
is surrounded
the purpose
is for

is detailed
the geneva
is because
the conditions
is going
the capture
is also
the lore

the first
is important
the u.s
is obstructing
the war
is created
the senate
is told

is pointed
the rectal
is no
the destruction
is going
the abu
is also
the cold

This one’s from The C Programming Language by Brian Kernighan and Dennis Ritchie:

the space
is exhausted
the result
is encountered
the return
is provided
the body
is said

is useful
the function
is entered
the standard
is stated
the object
is present
the head

the request
is divided
the result
is unequal
the result
is illegal
the function
is all

is likely
the prefix
is more
the expression
is noted
the proper
is entered
the call

And finally, a poem generated from the iTunes Terms and Conditions:

the device
is unlawful
the update
is protected
the gift
is provided
the licensed
is for

is removed
the transaction
is subject
the entire
is subject
the external
is strongly
the store

the app
is provided
the app
is acquired
the extent
is provided
the licensed
is your

is defined
the external
is granted
the licensed
is subject
the commercial
is granted
the store

The Code

import sys
import string
import random

# Array of all vowel sounds in the CMU speaking dictionary
vowelSounds = ["AA", "AE", "AH", "AO", "AW", "AY", "EH", "ER", "EY", "IH", "IY", "OW", "OY", "UH", "UW"]

# Check whether two words are both a) in the pronouncing dictionary and b) rhyming.
# It's assumed that you'll call cleanWord on each word before passing it into this function.
def rhymingWords(firstWord, secondWord):
    if firstWord == secondWord:
        return False
    
    if firstWord in phoneticDictionary:
        searchSyllables, searchStresses = phoneticDictionary[firstWord]
    else:
        return False
    
    if secondWord in phoneticDictionary:
        wordSyllables, wordStresses = phoneticDictionary[secondWord]
    else:
        return False
    
    lastPhoneme = ''
    stressCounter = 1
    lastStress = 0

    for i in range(1, 1 + min(len(searchSyllables), len(wordSyllables))):
        if searchSyllables[-i] == wordSyllables[-i] and stressCounter <= len(wordStresses) and stressCounter <= len(searchStresses):
            lastPhoneme = searchSyllables[-i]

            if lastPhoneme in vowelSounds:
                lastStress = wordStresses[-1 * stressCounter] and searchStresses[-1 * stressCounter]

                stressCounter += 1
        else:
            break

    if (lastPhoneme in vowelSounds) and (lastStress > 0):
        return True
    else:
        return False

# Convert tokens (with attached whitespace, punctuation, and irregular capitalization) to a clean, all-caps form
def cleanWord(word):
    newWord = word.upper()
    newWord = newWord.strip(string.punctuation)
    return newWord

# Given a list of words (which should have already been passed through cleanWord()), return a randomly selected
# word that has the given pattern of stresses (passed in as an array: [0, 1, 0], for example)
# 
# This function WILL shuffle your list in place. Watch out for that.
def wordWithPattern(searchList, stresses):
    random.shuffle(searchList)
    
    for word in searchList:
        if word in phoneticDictionary and phoneticDictionary[word][1] == stresses:
        	return word.lower()


# Import and populate the speaking dictionary
cmu = open('cmudict', 'r')

phoneticDictionary = {}

for line in cmu:
    tokens = line.split()
    
    # Only grab the first pronunciation in the dictionary
    if tokens[1] == "1":
        key = tokens[0]
        phonemes = []
        stresses = []

        # Some syllables have a number indicating stress (e.g. "EH1").
        # We're not using it, so strip that character off.
        for phoneme in tokens[2:]:
            if not phoneme.isalpha():
                stresses.append(int(phoneme[-1]))
                phoneme = phoneme[:-1]

            phonemes.append(phoneme)

        phoneticDictionary[key] = (phonemes, stresses)

cmu.close()

# Now start the real work ----------------------------------------------------

# First, pull in a list of all words in the file
words = []

for line in sys.stdin:
    line = line.strip()
    newWords = line.split()
    words += newWords

# Now, 
phrases = {'subject': [],
		   'predicate': []}

currType = ''

for word in words:
    if currType != '':
        # Filter out a couple of words I don't want... single letter words and articles
        if len(cleanWord(word)) > 1 and cleanWord(word) != 'AN':
            phrases[currType].append(cleanWord(word))
        currType = ''
    elif cleanWord(word) == 'THE':
        currType = 'subject'
    elif cleanWord(word) == 'IS':
        currType = 'predicate'

rhymes = set()

for subj in phrases['subject']:
	for pred in phrases['predicate']:
		if rhymingWords(subj, pred) and (phoneticDictionary[subj][1] == [1] or phoneticDictionary[subj][1] == [0, 1]) and (phoneticDictionary[pred][1] == [1] or phoneticDictionary[pred][1] == [0, 1]):
			rhymes.add((subj, pred))


# Now, print the poem

print '\n\n'

# This could be extended out to any number of pairs of stanzas
for i in range(0, 2):
    currRhyme = random.choice(list(rhymes))

    for i in range(0, 4):
    	if i == 3:
    		print 'the ' + wordWithPattern(phrases['subject'], [1, 0])
    		print 'is ' + currRhyme[1].lower()
    	elif random.random() < 0.4:
    		print 'the ' + wordWithPattern(phrases['subject'], [1, 0])
    		print 'is ' + wordWithPattern(phrases['predicate'], [1, 0])
    	elif random.random() < 0.5:
    		print 'the ' + wordWithPattern(phrases['subject'], [0, 1])
    		print 'is ' + wordWithPattern(phrases['predicate'], [0, 1, 0])
    	else:
    		print 'the ' + wordWithPattern(phrases['subject'], [1])
    		print 'is ' + wordWithPattern(phrases['predicate'], [0, 1, 0])

    print ''

    for i in range(0, 4):
    	if i == 3:
    		print 'is ' + wordWithPattern(phrases['predicate'], [1, 0])
    		print 'the ' + currRhyme[0].lower()
    	elif random.random() < 0.4:
    		print 'is ' + wordWithPattern(phrases['predicate'], [1, 0])
    		print 'the ' + wordWithPattern(phrases['subject'], [1, 0])
    	elif random.random() < 0.5:
    		print 'is ' + wordWithPattern(phrases['predicate'], [0, 1])
    		print 'the ' + wordWithPattern(phrases['subject'], [0, 1, 0])
    	else:
    		print 'is ' + wordWithPattern(phrases['predicate'], [1])
    		print 'the ' + wordWithPattern(phrases['subject'], [0, 1, 0])
    
    print ''

print '\n\n'

Assignment 3: Wiki

Reading and Writing Electronic Text


212

For this assignment, I looked into the Wikipedia API. I’m interested in making poems that are about something (at least in an incredibly vague sense of “about”), and Wikipedia is as good a resource as any for learning about a wide variety of topics (or, for computer poets, at least gleaning the vaguest impression of knowledge).

I wanted to write a program that could take articles on two topics and mash them up, generating metaphorical comparisons between the two. I thought I’d grab sentences and combine them, but Wikipedia is quite messy, littered with markup. So, instead of trying to parse out all of the markup from sentences, I decided to focus on words again. I used my CMU Pronouncing Dictionary code from the previous assignment to filter each of the two articles into a list of words found in the CMU dictionary. I then found words that only appeared in one of the two articles, and formed pairs of words based on phonetic similarity. By focusing on words that only appear in one article, I hoped to single out interesting words, while establishing a series of contrasts.

Here’s an example, contrasting “Computer” and “Poetry”:

I am Computer and you are Poetry
I am practically and you are grammatical.
And while you are plants, I am plan.
And while you are random, I am rand.

And while you are larger, I am large.
And you are not result, for you are results.
And while you are compilation, I am computational.
And while you are dramas, I am dumas.

And you are not careful, for you are carroll.
And while you are character, I am microcomputer.
And you are not dummer, for you are adam.
And while you are two, I am too.

And I am replacements while you are placement.
I am not discussed, for I am difficulties.
And I am catering while you are cutting.
I am not thus, for I am us.

And I am fail while you are fable.
I am not free, for I am frees.
And while you are tory, I am torpedo.
I am not two, for I am tool.

For the most part, the words chosen by the program are interesting and relevant to their specific topic. Wikipedia mentions or cites a lot of names, which generally don’t lead to interesting lines. (Though, I think “And you are not careful, for you are carroll” is wonderful.) And, of course, there’s no checking of part of speech or anything, so the lines are often a syntactic mess.

Here’s another, on the same topic:

I am Computer and you are Poetry
I am not departing, for I am interpreting.
I am commands and you are combined.
I am mechanical and you are classical.

And you are not bus, for you are basis.
And you are not francisco, for you are france.
And you are not imperative, for you are emotive.
And you are not serve, for you are served.

And I am multiply while you are multiple.
And while you are memory, I am mit.
I am scales and you are schools.
I am not rising, for I am devising.

I am people and you are appeal.
I am i and you are rhyme.
I am not open, for I am punch.
I am not frequent, for I am frequently.

I am not introducing, for I am producing.
I am not cultures, for I am compilers.
I am sine and you are science.
And I am slowly while you are sleep.

And, here’s my code:

import urllib
import json
import sys
import random
import difflib
import string

# Function to find the closest phonetic word
def closestWord(search_word, word_list):
    closest = ['', 0]

    for word in word_list:
        if not word == search_word:
            matcher = difflib.SequenceMatcher(None, phoneticDictionary[search_word], phoneticDictionary[word])
            similarity = matcher.ratio()

            if similarity > closest[1]:
                closest = [word, similarity]

    return closest


# Import and populate the speaking dictionary
cmu = open('cmudict', 'r')

phoneticDictionary = {}

for line in cmu:
    tokens = line.split()

    if tokens[1] == "1":
        key = tokens[0]
        phonemes = []

        # Some syllables have a number indicating stress (e.g. "EH1").
        # We're not using it, so strip that character off.
        for phoneme in tokens[2:]:
            if not phoneme.isalpha():
                phoneme = phoneme[:-1]

            phonemes.append(phoneme)

        phoneticDictionary[key] = phonemes

cmu.close()

#Treat the first two command-line arguments as Wikipedia titles, and load the
#articles
titles = [sys.argv[1], sys.argv[2]]
query = {"format": "json",
         "action": "query",
         "titles": '|'.join(titles),
         "prop": "revisions",
         "rvprop": "content"}
query_string = urllib.urlencode(query)

url = "http://en.wikipedia.org/w/api.php?" + query_string

response = urllib.urlopen(url).read()

data = json.loads(response)

#Now, grab the content for the two pages

titles = []
pageContent = []

for id in data['query']['pages']:
    print id
    words = data['query']['pages'][id]['revisions'][0]['*'].split(' ')
    
    cleanWords = []
    
    for word in words:
        word = word.upper()
        word = word.strip(string.punctuation)
        
        if word in phoneticDictionary:
            cleanWords.append(word)
    
    titles.append(data['query']['pages'][id]['title'])
    pageContent.append(cleanWords)

#Now, clean up lists more. First, remove duplicates whithin a list
pageContent[0] = list(set(pageContent[0]))
pageContent[1] = list(set(pageContent[1]))

#Second, remove all overlapping words between the two lists
for word in pageContent[0]:
    if word in pageContent[1]:
        pageContent[0].remove(word)
        pageContent[1].remove(word)

#Now, generate poem

#First line
print 'I am ' + titles[0] + ' and you are ' + titles[1]

templates = [['I am ', 0, ' and you are ', 1],
             ['And I am ', 0, ' while you are ', 1],
             ['I am not ', 1, ', for I am ', 0],
             ['And you are not ', 0, ', for you are ', 1],
             ['And while you are ', 1, ', I am ', 0]]

for i in range(1, 20):
    randoWord = random.choice(pageContent[0])
    matchWord = closestWord(randoWord, pageContent[1])[0]
    
    words = [randoWord.lower(), matchWord.lower()]
    pattern = random.choice(templates)
    print pattern[0] + words[pattern[1]] + pattern[2] + words[pattern[3]] + '.'
    #Break it up into four-line stanzas
    if i % 4 == 3:
        print ''

Final Project Research and Proposal


14

Project Research

As I’ve previously discussed, I’m interested in an exploration of computer-generated 3d virtual space—the space of much of contemporary cinema, as well as video games and Second Life. Drawing from the content of our class readings so far, I’ve been following two main threads:

Materiality of Digital Space

Clement Valla has been doing some interesting work in the intersection of virtual and realistic space. His project Postcards from Google Earth finds places where the geography of the Earth conflicts with the geography of Google Earth, leaving trees flattened and bridges draped across hill topologies. Meanwhile, this video made with A.E. Benenson describes how the spatial/representational logic of 3d space (in particular, 3d spaces captured from digital photogrammetry differ from the spatial logic of human perception, and how “glitches” in the capture process illustrate these disconnects.

In a similar vein is the project Farm (Pryor Creek, Oklahoma) by John Gerrard where he photographed a Google data center by helicopter and created a highly detailed 3d model of the building and its infrastructure. Data centers like this one (located less than an hour from where I grew up) offer one insight into the material structure of the Internet. Representations of the Internet as a purely dematerialized space—a “cloud” in current jargon—ignore the physical realities of buildings like these, and the substantial natural resources they consume.

Other levels of the materiality of virtual space are explored in a couple of books:

Mechanisms by Matthew Kirschenbaum, which describes how the forensic aspects of computers (i.e. the physical process of rewriting memory, the spatial layout of digital data on a disk, the persistence of supposedly “deleted” data, and other factors) exist as physical traces of virtual spatial production and action.

Software Takes Command by Lev Manovich, describes how the structure of software determines the types of operations that are available in the creation of digital media. He discusses Photoshop and After Effects as case studies, analyzing them in terms that are also applicable to video games, 3d modeling programs, and so on.

To this, I would add the mathematical foundations of 3d graphics. This is a wide domain, with a vast amount of scholarship. I’m only passingly familiar with the basics, but there is a lot of future potential in critique of the structures and assumptions of this mathematic approach to space.

Virtual Performance and Intervention

Joseph DeLappe has done a series of performances in online first-person shooter games, where participants on a game server use the game’s chat system to perform an episode of Friends in Quake III or the Bush/Kerry presidential debates across three different thematically-related games. The strongest performance is probably his project Dead in Iraq, where he typed the names, ages and ranks of all American soldiers killed in the Iraq war in games of America’s Army a first-person shooter developed in conjunction with the US Military as a simultaneous game and recruitment/training tool. DeLappe’s work continues the process of spatial intervention in to virtual game space, while also invoking Games as a site for theatre.

A couple of related readings:
Computers as theatre by Brenda Laurel
And “THE BRECHTIAN, ABSURDIST, AND POOR VIDEO GAME: ALTERNATIVE THEATRICAL MODELS OF SOFTWARE-BASED EXPERIENCE”, an article that extends these ideas.

Also, “Games as Art” by Celia Pearce, published in Visible Language traces various ways in which artists have used the structure of the game, with roots in Fluxus practice. She discusses the artistic production of game “mods”, which seems quite appropriate here.

Description of Work

At the moment, my idea for the work is still pretty hazy. I’m interested in the question of what an essay or documentary would look like in the terms of a video game. The subject will be an exploration of the representational limits of these virtual spaces. The “narrative” will be reasonably linear, structured as a series of interactive spaces that are each explorable, but that lead directly from one idea to another.

Possible subjects include: a history of virtual space, and it’s applications in cinema, video gaming, architecture, warfare, etc; or a set of tools that one can use for critical playing of video games.

I’ll be working with WebGL, allowing the space to be accessible by anyone with a modern web browser. So far this semester, I’ve been familiarizing myself with the Three.js javascript library for creating WebGL environments, and I’m interested in building work on this platform.

Timeline and Production Schedule

Central question or questions posed by the work

Where is a virtual space located? When we are in a virtual space, where are we? One answer is that we’re stored in a database or computer disk somewhere, our location rendered as virtual coordinates, accessible to a software system in ways that have nothing to do with the virtual space we allegedly must travel. How does this accessibility determine our relationship to the space? What ramifications does it have in a culture of mass surveillance?

Week 5 Reading Response

Ecology of Space


340

Michel Foucault casts a wide net in his discussion of heterotopias, describing a vast array of different spaces, with an equally diverse set of social roles and embedded power structures. Still, he’s clearly hit upon a rich idea in human spatial practice, one that we can see at work any time someone wants to (or is forced to) “get away from it all”. Of course, there is no “away from it all”—as Trevor Paglen has discussed, all actions, even “dark” or “invisible” actions have a physical location and leave material traces. Still, Foucault’s interest is in the way such spaces function in our psychology and society, ways in which we make certain spaces invisible through collective social decision.

The one issue that Foucault doesn’t really address is how the experience of heterotopias can be intensely personal. He describes how these spaces fulfill different roles from society to society, but doesn’t describe ways in which these spaces may change from person to person. For example, to the visitor, the museum exists as an accumulation of time in a confined space adrift from everyday life. However, what about the museum director? The curator? The janitor? A museum has a full staff who are intimately aware of the current state of the museum as a physical building and an institution whose funding, design, management and so on is subject to all the economic and cultural trends of the outside world. Do the people in power over such spaces have a self-awareness of how their space functions, or are the wrapped up in the same cultural mythologies they’re creating? In particular, I’m reminded of recent reporting on prisoner abuse in Rikers Island. In this case, prison officers have personally absorbed the ideology of the prison system to violent ends, and I wonder if that’s a behavior of the prison system specifically, or heterotopias in general.

Richard Ross’s photo project, Architecture of Authority is an exploration of such spaces, deliberately collecting images from institutions at all levels of power across the globe and then shuffling them together. What’s especially striking about many of these spaces is how much they conform to a logic of modularity and duplication. Public school classrooms, holding cells, hospital beds, hotel rooms, and so on all exist as copies, often aligned in a grid to make this especially obvious. The motives behind this design are often practical, but I think they have a deeply ideological effect. As Foucault describes, the standardization of spaces creates a sort of paradoxical situation. On the one hand the space seems anonymous and open (that is to say, if I can access one hotel room, then I understand a great deal about the spatial layout and living situations in every room), but on the other, each space is still occupied by individual people in distinct physical locations and circumstances, and can afford a great deal of privacy (either to the occupants, in a hotel, or to those who control the space, as in a prison cell or interrogation room).

Week 4 Reading Response

Ecology of Space


363

In this reading, Brett Bloom describes a bit of the history of how the last major push for government funding of the arts in the US (in the 1970s and early 80s) fostered a system of “alternative” art spaces that could exist outside of the commercial gallery world and, consequently, open the art world up to new creative forms, ideas, and types of artists (especially artists from various marginalized groups) unwelcome to the art market. The dismantling of public arts funding (as a proxy for the general trend of privatization), means that artists now must create their own radical art spaces, but that there is much potential for these types of projects (some of which Bloom describes).

In general, I find Bloom’s assessment of the situation spot-on, and find the types of spaces he’s discussing very interesting. Still, to me, the piece raises more questions than it answers, and I’m left feeling intrigued but also kind of unsatisfied. Bloom is vague on exactly what makes the spaces he describes radical, other than their lack of commerce. His stated avoidance of a specific ideology (as opposed to explicitly anarchist spaces, for example) makes sense, but does little to argue for why these spaces are successful (or sufficiently radical) and others aren’t.

The biggest unresolved question for me is this: is Bloom interested in using radical space to revolutionize the art world, or is he interested in using art to revolutionize social practice? At the beginning, the article leans more heavily on the former—the commercial gallery/museum system is offered as the institution against which alternative art spaces should be defined. The alternative spaces of the 70s and early 80s, we are told, broadened the art world, ushering in new people and new creative possibilities. Presumably, a wave of radical new spaces would do the same now.

I agree, and for me personally, that’s what makes the Internet (and computer tech) such an interesting subject. Even with its deep flaws, Internet culture has certainly given attention to many otherwise marginalized voices, artists and not. At the same time, the availability of relatively inexpensive computer equipment and the philosophies of open source development offer a strong case for digital technology as a mode for anti-capitalist artistic and cultural production.

These are all still very recent developments, so Bloom can be forgiven for not addressing online communities at all seven years ago. However, beyond his critiques of commercial high art, Bloom doesn’t want to offer much of what alternative, radical art looks like, anyway. It’s unclear what makes these spaces specifically “art spaces”. He explains that art is produced and exhibited in these spaces, but no specific art is described or illustrated. Instead “art” hangs as a sort of loose philosophy over the entire endeavor, a vague justification for why these spaces should be used in unconventional ways. When Bloom gets into specifics, it’s not about art (or the economics of image making or anything), but about much more materialist concerns—recycling and gardening and redistribution of excess clothing and community meetings and food.

I find myself worrying about the mechanics of these spaces. How do you get enough people to come, without drowning out the local community? How do you ensure that there’s enough food and space to share? How do you make yourself accessible, while still attending to basic security? How do you set up a space in an inexpensive building (presumably in a poor or at least working-class area) while staying aware of the role that artists often play in gentrification. Indeed, as a recent transplant to New York City, do I have any business setting up “indigenous” spaces anyway?

But, ultimately, these questions seem to be the exact issues that the people in charge of the spaces in this reading are working through. While the Experimental Station’s Dan Peterman hesitates at whether his organization of the building is itself an artistic practice, Bloom seems more certain, describing “a really exploded notion of how art and highly refined aesthetic sensibility could contribute to a larger, more interesting, constantly unfolding daily situation.” But then, why make art objects at all? By describing the many “more interesting” activities artists could coordinate (activities which, to be honest, do generally sound interesting and beneficial to society), it seems to me that Bloom is walking up to, but never embracing, a dangerous proposition: that the most radical thing an artist can do is not make art.

Week 3 Reading Response

Ecology of Space


5

“[T]o transform rather than describe”, Jane Rendell’s description of one of the goals of critical theory seems to nicely sum up much of the work and discussion in this week’s reading. Picking up where Debord left off last week, Rendell argues that the primary point of public art should be to create work that inspires people to question and change the world around them. I found her focus on public art (to the exclusion of art inside the gallery) a bit curious. By her omission, Rendell suggests that gallery art (insulated by the status quo of the art institution) is perhaps immune from the type of pressures that can turn a public artwork into an interesting spatial intervention. Can gallery art perform this crisis-making function, or is it necessarily limited? And given the high stakes that Rendell’s laid out, is making such work irresponsible?

Trevor Paglen does a nice job of tying the previous threads of Marx and Lefebvre into the context of contemporary geography. I’ll confess that I didn’t even know that geography was an active academic discipline (for all the reasons that Paglen outlines) until I started reading some David Harvey a couple of years ago. Still, I appreciated Paglen’s concise explanation of geography as a discipline whose two basic tenets—materialism and human spatial production—can be used to examine any field of human activity. By the end, he’s applied this method of critique recursively, arriving at many of the same conclusions that Rendell draws from the “self-reflective modes of thought” she discusses. Paglan is clearly dissatisfied with art that describes, rather than transforms—in his Art21 interview, he identifies this tendency as the major weakness he found with graduate art education. Quoting Benjamin, he discusses how it is impossible to create critical artwork “outside” of politics. Experimental Geography, then, is his approach towards an art practice that actively engages with politics by the deliberate creation of political space. It would have been nice if he went on to offer more specific strategies for applying these ideas, but he lays a solid intellectual foundation for reading his work and thinking about new work.

And besides, the Interventionists catalogue provides myriad interesting examples of artists engaging in all manner of active spatial practice. Although, I’ll admit that I struggle somewhat with how to unpack this sort of work. Reverend Billy makes sense as a stylized, but still pretty conventional protest, and the Yes Men have a clear goal of embarrassing the large institutions that take them seriously. But many of the projects seem caught between existing as sincere socio-political gestures, while being too abstract or esoteric to be viable templates for larger social action. Is Krzysztof Wodiczko’s Homeless Vehicle a call to action? A prototype? A design fiction?

To me, the work that seemed the most well resolved was either the work that had small stakes (such as Alex Villar’s videos where he disobeys the spatial logic of walls, fences, and railings) or appropriately audacious methods (such as Yomango’s shoplifting as a lifestyle practice). The work also makes me think of Institutional Critique, particularly Michael Asher’s work, though I worry that his work of modifying art gallery buildings falls prey to some of the pitfalls discussed above. I think his work does take seriously its own production of space, but strictly within an art context, perhaps too insulated from the way that context connects to the wider world. Perhaps that’s why I like it so much: when you’re exposing the historical logic of museum exhibition layouts, there are important ramifications, but it’s mostly a fun conceptual exercise. When you’re trying to tackle homelessness, or commercial exploitation, or surveillance, it seems like you should be taking things much more seriously.

But perhaps my desire to insist upon evaluating the effectiveness of these works as political gambits is misplaced. The focus on “tactics” and “experiments” suggests that these projects won’t necessarily work, and that that’s okay, but that each is an important step pushing toward a broader goal.

Week 2 Reading Response

Ecology of Space


3

I had a bit of understanding of the Situationist International before this, but these two readings helped clear things up a bit. As a practice, I think dérive holds a lot of potential for interesting spatial exploration.

Debord is clearly developing his ideas in the context of the urban environment. He contrasts the urban dérive with the Surrealists’ project of wandering in the open countryside in 1923. Dismissing that project, he says, “Wandering in open country is naturally depressing, and the interventions of chance are poorer there than anywhere else.” Buy what about the rural space is “natural depressing”? I think there are two aspect of the city that appeal to Debord: its information density and its human-made nature.

On the one hand, I think that the “interventions of chance are poorer” in the countryside not because one has fewer options (in general, you have greater freedom of movement in the countryside), but because the choices one can make have very little effect on one’s environment. In the city, each block might be very different from the next, while in the countryside, the landscape can stretch on unchanged for miles. In terms of the total number of specific, memorable images and sounds that one might experience, the city offers a much higher density of this type of information than that countryside. This is partially physical (i.e. skyscrapers and buildings occupy space that in nature might be occupied by air), but I think it’s also due to the fact that we don’t generally interpret the details of the natural environment, such as differences in plants, rocks, and water formations, as meaningful in the way we draw meaning from human creations.

To this point, the fact that the city is a human construction seems crucial to the Situationist agenda. As argued in Critique of Urban Geography, Debord’s objective is for people to understand the psychological forces in the city that determine our habits. With an understanding of how the city functions, we can then intervene, with the ultimate goal of revolutionary societal change. In this context, it’s helpful to know that the city is already an accumulation of decisions made by people with specific intensions. By understanding the nature of the city as it exists now, perhaps we’ll be empowered to make our own decisions and reshape the city.

In this way, it’s clear to see how the Debord is drawing on the work of Marx. Marx (by way of Joseph Stalin’s reading) is similarly interested in how the practical foundations of a society, especially the production and circulation of goods, determines the lives of those living within it. Capitalism often seems such a thorny, abstract quantity, but at the roots of Marx are very basic questions: as a human, how can I work to get food, shelter, etc? The focus on dialectic materialism grounds this. Also, given that questions of capitalism/socialism are often discussed in terms of ideology and abstraction, it’s striking how much Stalin is working to lay out a practical historical argument for socialism as the next form of societal organization. Given the failure of the Soviet state, his historical formulation seems optimistic and maybe oversimplifying. Even so, it’s difficult to argue that the issues and conflicts Marx raises are at all irrelevant in our contemporary society.

Meanwhile, I’ve had a harder time getting a handle on Henri Lefebvre’s Production of Space, although I think it will be important to revisit as I continue working on virtual computer-generated space. Lefebvre sets up distinctions between space as it is perceived in our day-to-day existence and space as it is conceived in abstract terms. Does this hold true with the move towards virtual reality? Lefebvre writes a length about the move toward abstract space in capitalist society. What about spaces that take as their foundations geometric ideas and abstract, symbolic representations? Does our representation and experience of these spaces collapse into a point?

Assignment 2: Speech

Reading and Writing Electronic Text


225

I used the speaking dictionary from Carnegie Mellon University’s computational speech group. The dictionary is a super long text document with pronunciations for basically any word you can think of, laid out like so:

...
POET'S 1 P OW1 AH0 T S
POETS 1 P OW1 AH0 T S
POFAHL 1 P AA1 F AA0 L
POFF 1 P AO1 F
...

My rough goal is to cluster words that sound similar (words that rhyme, alliterative words, homophones, etc) into abstract, tongue-twistery text. As it turns out, Python has a good way to calculate the similarity of two lists (in this case, lists of phonemes) using difflib, a module for computing differences in sequences of text.

Working on the text of the recent New York Times article on the FCC and Net Neutrality, I came up with code to generate pairs consisting of a random word from the article, plus the word that was phonetically closest to it.

HOUSEHOLD, SO-CALLED
DIETZ, ITS
CAN, AN
TECHNICAL, CRITICAL
RULES, RULE
BRANDED, BRAND
SPIGOT, BIGGEST
AWAY, WAY
IS, IMPOSE
BEGINNING, BUILDING

To generate my poem, I wrote some code to take pairs of words from the article and then link them with similar adjacent words.

proclaiming in been the a right wrote the
thinking is impose tolls tools to
turn away way were work with waiting until efficiently into
technology activist activists have had branded brand of a right
support he hear dire i think thing that at the
days the a right wrote the
east coast-west
gatekeeper power our future few days dazed brian urban dictionary
window proclaiming proclaimed we week 102
posted a the policy technology activist activists have had branded

Here’s the code:

import sys
import random
import difflib

# Function to find the closest phonetic word
def closest_word(search_word):
    closest = ['', 0]

    for word in words:
        if not word == search_word:
            matcher = difflib.SequenceMatcher(None, phoneticDictionary[search_word], phoneticDictionary[word])
            similarity = matcher.ratio()

            if similarity > closest[1]:
                closest = [word, similarity]

    return closest


# Import and populate the speaking dictionary
cmu = open('cmudict', 'r')

phoneticDictionary = {}

for line in cmu:
    tokens = line.split()

    if tokens[1] == "1":
        key = tokens[0]
        phonemes = []

        # Some syllables have a number indicating stress (e.g. "EH1").
        # We're not using it, so strip that character off.
        for phoneme in tokens[2:]:
            if not phoneme.isalpha():
                phoneme = phoneme[:-1]

            phonemes.append(phoneme)

        phoneticDictionary[key] = phonemes

cmu.close()

wordpairs = {}
lastword = ''

words = set()

for line in sys.stdin:
    line = line.strip()
    line = line.upper()

    if lastword != '':
        wordpairs[lastword] = line

    lastword = line

    if line in phoneticDictionary:
        words.add(line)

# Cycle through lines, starting each with a random word
for i in range(0, 10):
    randomWord = random.choice(list(words))
    lineWords = []

    for j in range(0, 5):
        lineWords.append(randomWord)
        lineWords.append(wordpairs[randomWord])

        if wordpairs[randomWord] in words:
            randomWord = closest_word(wordpairs[randomWord])[0]

            if randomWord in lineWords:
                break
        else:
            break

    print " ".join(lineWords).lower()

Statement of Inquiry

Ecology of Space


229

Broadly speaking, I’m interested in exploring issues surrounding 3d “virtual” space. In terms of personal artistic development, I’ve become increasingly interested in computer generated 3d graphics as a medium for artistic experimentation. To begin with, here’s a brief recap of what I know:

3d virtual space (like all computer software), consists of two components. First, there is data, in this case mathematical descriptions of objects in three-dimensional Cartesian space (usually complex polyhedra with triangular faces, though there are other models). In addition, other objects (such as lights, virtual cameras, physical forces, etc) are represented abstractly and plotted in the same coordinate space. On top of that data are procedures—rules for manipulating these 3d objects and depicting them in a flat image. The traditional set of procedures grew out of mathematics (particularly Euclidean geometry and linear algebra) and were informed both by careful study of real-world optics as well as aesthetic judgements about computer image making.

Historically, the field of 3d computer graphics research emerged primarily as a means to simulate the pictorial space of cinema and photography through a computationally-rigorous application of artistic techniques such as linear perspective. In this context (currently manifest in the ubiquity of CGI in cinema and television), the issues raised by computer-generated 3d space are essentially the same issues of pictorial space already raised by cinema and photography. However, with the real-time processing capabilities of computers, such virtual spaces can now be explored interactively by people and their digital avatars. Rather than merely representational, such spaces now strive to be “immersive”, inviting a new form of spatial analysis.

With virtual 3d spaces, the aim is often some degree of realism. Even when representing worlds of fantasy or abstraction, computer-generated spaces often take for granted much of the organizational logic of “real-world” space. Such spaces attempt to be simulacra of our own space with superficial differences (or without, in the case of projects such as Google Earth). As humanity seems poised to spend increasingly large amounts of time in virtual spaces, a reading of existing spatial practice seems like an important tool for testing the assumptions and affordances of 3d virtual space in the computer.

Some questions come to mind: Given their mathematic foundations, are virtual 3d spaces the ultimate rationalist space (with the corresponding rationalist ideology and politics), or does their artificial nature allow for different geometries? Can virtual spaces be discussed outside of the context of their visual representation? Given that the narrow slice of what the user sees is the only space explicitly rendered, can the remaining environment be said to exist? What is the economic condition of virtual “property”, especially in a video game such as Minecraft where industrial resources are (effectively) infinite? When all information about the virtual world is densely stored in a centralized database, are notions of privacy and surveillance meaningful? What are the politics of a space controlled (and designed) by a single programmer, or a corporate entity? What does occupation or dissent look like in these spaces?

I hope to combine the theory we’ll be discussing in this class with a close “reading” of some of the technological processes underlying 3d graphics engines. The end result will be some sort of virtual space in the web browser (using WebGL) that explores some of these issues. In rough terms, I’m imagining some sort of didactic game—that is, what would an essay about 3-dimensional space look like as an explorable 3-dimensional space.

Assignment 1: Experimental text justification

Reading and Writing Electronic Text


4

While experimenting Python’s string manipulation, I noticed that Python has tools for padding strings to certain lengths in order to center, left or right-justify text rendered in a monospaced font. The fourth popular text layout—text justified with both margins is notably not supported, as that traditionally involves variable-sized spaces.

While considering this problem, it occurred to me that padding strings with extra spaces between words would look clunky, but then I hit upon a solution—vowels. Whereas consonants provide much of the definition of words, vowels can be repeated or omitted with a minimal impact on readability. The following Python program justifies a column of text to the width of the first line, simply by duplicating or removing vowels (omitting capital vowels, as they often occur at the beginning of words, and it looks more awkward to chop them off).

# Justify text by adding and subtracting vowels
import sys

vowels = ['a', 'e', 'i', 'o', 'u']
lineLength = 0

for line in sys.stdin:
    line = line.strip()

    if lineLength == 0:
        lineLength = len(line)
    else:
        index = 0

        while (len(line) < lineLength) or (len(line) > lineLength):
            if line[index] in vowels:
                if len(line) < lineLength:
                    line = line[:index] + line[index] + line[index:]
                    index = (index + 2) % len(line)
                elif len(line) > lineLength:
                    line = line[:index] + line[index + 1:]
            else:
                index = (index + 1) % len(line)

    print line

As an example, take this one of Shakespeare’s sonnets.

Let me confess that we two must be twain,
Although our undivided loves are one:
So shall those blots that do with me remain,
Without thy help, by me be borne alone.
In our two loves there is but one respect,
Though in our lives a separable spite,
Which though it alter not love's sole effect,
Yet doth it steal sweet hours from love's delight.
I may not evermore acknowledge thee,
Lest my bewailed guilt should do thee shame,
Nor thou with public kindness honour me,
Unless thou take that honour from thy name:
  But do not so, I love thee in such sort,
  As thou being mine, mine is thy good report.

When run through this Python script, each line is transformed, resulting in the following, nicely-justified text:

Let me confess that we two must be twain,
Althoouugh oouur undivided loves are one:
S shll thse blots that do with me remain,
Wiithoout thy help, by me be borne alone.
In ur two loves there is but one respect,
Thoouugh iin our lives a separable spite,
Whch thgh t alter not love's sole effect,
Yt dth t stl swt hrs from love's delight.
I maay noot eeveermoore acknowledge thee,
Lst my bwiled guilt should do thee shame,
Noor thou with public kindness honour me,
Unlss thu take that honour from thy name:
Buut do not so, I love thee in such sort,
As th bing mine, mine is thy good report.

Beautiful.

Final Project: Play Space

Physical Computing


6

David Gochfeld and my final project was “Play Space”, a musically-activated space that used a Kinect to track the motion of people and transform that motion into musical output. We envisioned the project as a sort of collaborative instrument, one which would be usable even by people who just wander into the space. Along those lines, I think the project was generally successful—it quickly engaged people, but we also found that we, at least, enjoyed teasing out more sophisticated control of our musical output.

System Diagram

As far as materials were concerned, the project was quite simple. We used the Kinect camera for our tracking and a Processing sketch with the OpenNI library for interpreting the data. From there, we sent our musical output (in MIDI form) to a copy of Logic Pro X, which handled the musical instrument synthesis and final audio output. Besides the Kinect, our only hardware requirements were a decently fast computer, speakers, and the necessary cabling.

We set ourselves the challenge of creating a space that was both immediately engaging but also interesting and rich as a musical instrument. The biggest challenge was communicating to our users what was going on using only audio cues, and indeed many of our first user tests were confusing for people. Our simple instructions (a taped rectangle around the active area and the injunction to “PLAY”), seemed to help a bit—the people who moved in the most interesting ways (not coincidentally, the people with dance/movement backgrounds) created more interesting sounds and more quickly were able to feel out the possibilities for the interface. One remaining challenge is how to encourage more people to move and explore in this way.

We didn’t have very clear ideas of the exact mappings we wanted between movement and music, so we experimented in many directions through trial and error. Because of this working process, I feel like there are so many ways the ideas here could be adapted for various uses. I’ve begun to conceive of this as less of a discrete project, and more as a solid foundation for future research.

Final Project: Musical Mappings

Physical Computing

As part of David’s and my final project, we experimented with various methods for mapping position and movement of people in a space to musical output. As we explored, we were fundamentally trying to solve this list of (sometimes contradictory) challenges:

With this in mind, we partially implemented a variety of different mapping philosophies, most of which didn’t even survive our own personal testing. Not all worked for what we needed, but I think all have potential as musical interfaces. Here are approaches we tried, with a few notes:

Option 1: Position Based Pitch, No User Tracking

This is the simplest method. If there is something interesting (however your algorithm defines that) in a specific part of the screen, then play the pitch associated for that part of the screen. This can be done with just the shapes of users, or it can be done with the centers of blobs. However, this is just based on checking the on/off state of each cell every frame, so it doesn’t have a persistent memory of where people are or how they move over time.

Drawbacks: Without tracking users, we’re incredibly limited in what we can do. In particular, we have very little information over time, so we don’t have much we can do with interesting types of note attacks or decays.

Option 2: Position Based Pitch, User Tracking

Adding a sense of different users to the previous technique. This way, you can choose to only attack notes when the user moves, or enters a new cell, or something. Also, you can assign different users different instruments or voices, which we discovered was crucial for user understanding. In the end, this was the approach we took, with multiple adjustments.

Drawbacks: Because the mapping of pitches is based on screen, rather than physical, space, it’s sometimes difficult to tell when one is between pitch areas. Also, by creating a large enough pitch grid to be legible, it forces the user to move around quite a bit for a small amount of output control. We addressed this by allowing certain gestures that could adjust the pitch relative to the root pitch of the area. Also, by limiting the system to a small number of pitches, the music can be more harmonic, but it also limits the potential range, adding to the audible confusion of too many people playing notes within a constrained pitch space.

Option 3: Direction-Based Pitch

I briefly experimented with a system where movement in one direction would cause a series of ascending notes and movement in an opposing direction would cause descending notes, and vice versa. This allowed us to break up the rigid pitch grid, and the fact that the directions didn’t have to directly oppose (just mostly oppose) meant that the user could move in more interesting patterns, while running up and down the scale.

Drawbacks: Because pitch was movement-based, there was less control over when specific notes played, which confused the mapping. Also, it was difficult to play larger intervals, or specific sequences of notes, given the limitation of moving up and down relative to the user’s previously played note.

Option 4: Human Sequencer, Non-Position Based

With this system, each participant is assigned a note in a sequence. The pitch of that note could change based on position, and the length of the note could vary based on the amount of space the person occupies, but the fundamental sequence would be that Person 1’s note plays, Person 2’s note plays, and so on until it loops back to Person 1 again. By spacing out people’s contributions over time (rather than on different pitches or voices), more people can contribute to a more complex melody, without the music becoming to chaotic.

Drawbacks: The initial state (where one person enters and gets a loop of their own note, which they can then modify) is quite discoverable, I think, but also really boring. Without having three or more people, there’s little that can be done with this system. Also, by adding and removing notes to the sequence (and varying the notes’ durations), the system has a pattern, but very irregular rhythms.

Option 5: Human Sequencer, Position Based

As a variant on the previous idea, we also tried out a much more literal sequencer: we mapped the space on the floor to time and looped across, playing notes for each person the system encountered with time offsets relative to their position. To help communicate this, we added a basic beat that would play when users were in the space.

Drawbacks: This, in particular, suffered from the lack of visual feedback. It was clear that something was happening, but since the virtual “playhead” was immaterial, it was hard to tell when one’s note was about to be played. In addition, having to wait until the loop reaches the user means that exploratory gestures don’t necessarily have prompt feedback. This has potential, but it requires a user that’s more experienced/knowledgable than our intended users.

Final Project: Blob Detection

Physical Computing

For my final project, I’ve begun working in collaboration with David Gochfeld. After abandoning my initial idea, I came up with a new direction that was much more music-composition-based, much at the same time that David was moving in a similar direction, so it made sense to combine forces. David’s got a much better write-up of the project idea, so to put it briefly: we wanted to use the Kinect to track participants’ movement through a room and then feed that movement into a music generation engine. Instead, I’m going to focus on a few of the technical details he didn’t cover, in particular, how we tracked participants.

The problem: We’re working with the Kinect’s depth camera, but we’re mounting it overhead, so most of the Kinect’s built-in person identification/tracking tools are useless to us. (Related problem: I have a somewhat stubborn interest in implementing things from scratch.)

The solution: write our own light-weight blob detection algorithm, purely based on depth data. It didn’t need the sophistication of something like OpenCV, but it did need to be fairly adaptable as we wanted to quantify different aspects of the data. Looking around, I found this page to be a particularly helpful reference, and much of my work draws from the concepts as described there.

Simple Blob Detection

Blob1

First, apply a basic depth threshold so that everything above a given line (in this case, a few inches above the floor) is ignored. With a top-down view of an empty space, this groups people pretty cleanly.

Blob2

Blob3

Now, iterate through all the pixels until you find a pixel that is above the threshold, but that hasn’t been assigned a blob index yet. Set this pixel as belonging to blob number 1 (in this case, red), and then perform a recursive flood fill in all directions to find the other pixels in this blob. Continue where you left off, checking each pixel, but now ignoring any pixels that are already in blob number 1.

Blob4

Blob5

When we encounter another unassigned blob pixel, assign that one to blob number 2. You’ll need to flood fill in all directions to catch things like that long bit that extends off to the left.

Blob6

Continue this process, and soon you’ll have all the blobs filled in. One caveat: If you actually implement this recursively (i.e. with a setBlobIndex function that calls a bunch of setBlobIndex functions on its neighbors), then you may get stack overflow exceptions when enough people enter the frame (also the resulting Processing error message is inscrutable). To get around this, we stored a list of cells that we wanted to fill next, and then filled them in order.

In the video below, we’ve implemented exactly that. The color of the blobs is mapped to their relative size (the biggest one will always be red and a blob of size 0 would be purple-red), and the white circle is the highest point (determined right after the blobs have been assigned). Note that the system currently doesn’t have a persistent sense of who people are or how they move. That will be fixed next.

Tracking Individual People

For everything we were doing, we realized that having a persistent sense of who people were was crucial. This involved creating a class for participants that could keep track of changes in velocity, assigned musical instrument, and so on, and keeping a list of people in the scene. First, we calculate the centers of mass for all of our blobs identified at the end of the last step, by averaging their pixel x and y values. Then, we get rid of all blobs that are too small, and then look at all our blobs, see if there’s an existing person whose last center of mass was closest, and then update that person with the new blob data. If no person matches, then create a new person with the blob, and if people are left over, then delete them. From here, we were able to quickly measure things such as velocity, position, bounding box dimensions, and so on. In addition, building our own person tracking system allowed us to do more simpler person tracking, and also exceed the Kinect’s five-person limit for skeleton tracking.

The code is on Github, if you want to peruse.

Glove Runner

Physical Computing

For my Physical Computing midterm project, I worked with Lirong Liu on a glove-based game controller.

Our initial discussions revolved around a variety of gestural control schemes. In particular, we discussed various ways that hand gestures in space could be tracked, using gloves, computer vision, and other approaches. Some topics of discussion were pointing/moving in 3-d space, and various gestures for instrument control. After much discussion, we settled on a controller for a Mario-style side-scrolling game, where the player would make their character run by physically “running” their index and middle fingers along a table top. I think this gesture is attractive for a number of reasons. It has a very clear correspondence with the action performed on screen, and although the controller gives little physical feedback, the assumption that users would run in place on a table top helps ground their actions. Also, it seemed like a lot of fun.

Glove Mockup

From there, I began working with a physical prototype. To begin with, I took a pair of flex sensors (shown at left with a plug we added for better interfacing) and attached them to my fingers using strips of elastic. From this prototype, it was clear that the best sensor response came when the tip of the flex sensor was firmly attached to the fingertip and the rest of the flex sensor could slide forward and back along the finger as it bends.

Test Processing Output

Reading this sensor data into Processing, I was able to quickly map the movement to a pair of “legs” and get a sense of the motion in the context of a running character. For a standing character, we found that just two changing bend values (one for each sensor) could produce some very sophisticated, lifelike motion. Meanwhile, as I worked on the physical prototype, Lirong set up a basic game engine in JavaFX with a scrolling world and three different obstacle types.

At this point, we both worked on the software for a while, with Lirong setting up a system for converting various finger movements to discrete events (for steps, jumps, etc) and me working on various issues related to graphics and animation. In the end, we wound up using our sensor input with two different levels of abstraction: the high level (the specific running and jumping events) controls the actual game logic, while the low level (the actual bend values of the sensors) controls the character’s animation.

After that, I sewed up the two pairs of gloves shown in the video above, allowing the flex sensors to slide back and forth along the fingers. As we worked on the glove design, we tested with various users to identify potential sizing issues. From there, we built a simple, self-contained system for doing basic user control, and wired everything up.

Code can be found here: https://github.com/mindofmatthew/pcomp_midterm

A few challenges we faced:

A few things I think we did well:

Physical Computing Project Plan

Physical Computing

Prototype Music Composer

Last week, I put together a simple paper prototype of my musical composition machine and tried it out with my classmates to see if the interaction was understandable. In my original sketch of the design, I thought that the device would have both a musical keyboard and a text keyboard. That seemed excessive, so for this iteration, I assumed that the user would use a separate MIDI-based music input of their own choosing, and then enter text with a built-in QWERTY keyboard. Even this proved problematic, however—for many users, the small size of the keyboard seemed uncomfortable to use. It was suggested that my device could instead interface with a USB keyboard. In principle, that suggestion makes sense to me, but the idea of farming out so much of the interaction to pre-existing interfaces makes me uncomfortable with the project.

In general, the direction of my user testing tended towards complexity. Because the output media—punched paper tape— is so linear, I decided that the interface should allow for recording a finite number of musical bars before writing out those bars to paper and recording the next set. In general, testers seemed uncomfortable with this mode of working, and wanted more sophisticated score editing tools. As the interface is already too complex, I decided that my efforts would be better focused on building a tool that could convert MIDI scores to my punched paper format and letting users compose music using any of the already available programs for composing MIDI scores. A good, solid solution, but now out of scope for the class.

So, what about physical computing?

Project Plan

For my final, I think I want to shift in a related direction, which is interfaces for sequencing music. Most sequencers and arpeggiators take musical input and then transform those notes around into complex patterns. For my project, I’m interested in a system that begins with a random sequence of notes and allows the user to filter that random sequence into something cohesive. The knobs on the left will control variables for the interpretation of a randomly-generated sequence, controlling aspects such as playback speed, root tone and the distribution of high vs low notes. The buttons on the bottom can be used to optionally filter the sequence to specific notes on the chromatic scale, and the knobs on the right can be used to control the rules for how the sequence is constructed. The output of the device is a standard MIDI signal, which can then be recorded or synthesized.

I’ve worked with MIDI and the Arduino before, so I foresee the greatest challenges with this project being the layout and selection of different controls. I intend to expose a lot of different choices to the user, so laying everything out cleanly is imperative. In addition, these controls may exceed the capacity of the ATMega, so I’m interested in exploring strategies for multiplexing or alternating between inputs.

InterfaceSketch

System Diagram

SystemDiagram

Initial Bill of Materials

Physical Computing Final Project Concepts

Physical Computing

Right now, I’m generally interested in interfaces for musical controllers or sequencers. Are there specific movements that could lend themselves to interesting tonal changes? What about some sort of individual components that can be physically manipulated to control some sort of musical structure?

Concept1

For example, for my midterm project in Automata, I’m building a music box that transforms data punched in paper tape into voice. For that project, I’ll probably be die-cutting/laser-cutting my punched paper strips, but I’m interested in the challenge of how to build a device for punching these strips. Specifically, we have decent interfaces for inputting text and musical notation, but how would a hybrid work (in order to both perform text and lyrics)? Is it possible without a lot of tedious labor for the operator?

Automata Midterm Proposal: Singing Music Box

BookScanCenter_1

Concept

My midterm would consist of a small singing music box. It will read songs encoded as punched binary data on a strip of paper (probably standard receipt paper), and then will sing those songs as it draws the paper in a synthesized voice.

Composition

The user feeds a strip of punched paper into the back of the music box. The paper is pulled into the box’s drive wheels, and pulled forward for a few lines. At that point, the music box begins singing in a synthesized voice. The paper stops momentarily, until the music box reaches the end of a syllable, at which point the paper is drawn in for a few more lines, keeping time with the music.

Context

Music boxes, piano rolls, and similar automatic musical technologies share some deep philosophical roots with modern digital computing (roots that have become obscured as computers have moved away from punched cards and tape for memory, towards optical and magnetic media. This exposes those roots, while also reimagining a traditional interface for the music box (a wooden box that moves through punched media) towards an application it was never intended for (reproduction of the human voice).

BOM

BookScanCenter_3

Signal chain

The music box will take DC input to power the microprocessor, data reader, and feed motor. The Arduino will store the minimum amount of information necessary to run the SpeakJet chip. Most of the program data will come from the paper feed. Each line (or each collection of lines) will encode a syllable, a pitch, and an amount of time to wait. The Arduino will interpret these bytes, further translate them into an appropriate signal for the SpeakJet, and output audio. The paper feed will be designed to only read in the necessary paper for the current note, in order to better follow the pace of the song. For now, I imagine that pauses in the song will be fixed, but for future extensibility, the music box could also support a MIDI metronome signal in order to synchronize with other devices or music boxes.

Color

Visual Language

Color Visualizer

For my color composition, I decided to play around with Two.js, a javascript graphics library. The visualization is a sort of spiky line graph that moves up and down, and as the lines move up and down, the height of the segments above the middle of the page determines the hue offset, saturation, or brightness.

My color composition is located at this link. Use the left and right arrow keys to change between hue, saturation, and brightness modes, and any other keys to change the base hue.

Code

Instead of drawing every frame (like Processing), Two.js allows you to create vector shapes and then modify the properties of those shapes over time. So, for this, I started by creating two sets of points: one set that ran along the midline of the screen at random distances, and a second set running along the bottom of the window, directly below the first set.

// Points is a 2d array: Row 0 holds all the points along the bottom
// of the screen. Row 1 holds the points that move up and down above
// the mid-line.
points = new Array();
points[0] = new Array();
points[1] = new Array();

// From left to right, pick random values between 0 and the width of
// the animated figure. Create two points at this horizontal position.
for(var i = 0; i <= width - 5; i += Math.random() * 10 + 5) {
	// These points are relative to the center of the screen
	points[0].push(new Two.Vector(i - width / 2, two.height / 2));
	points[1].push(new Two.Vector(i - width / 2, 0));
}

// Add the final two points, so the figure is the exact width of
// the variable 'width'.
points[0].push(new Two.Vector(width / 2, two.height / 2));
points[1].push(new Two.Vector(width / 2, 0));

Then, I created a Two.js object that manages all of the graphics, a container object, “group”, that centers the content

// Create a new Two.js object to hold all the artwork.
two = new Two({fullscreen: true}).appendTo(document.getElementById('home'));

// This group is a container for the art. Move it to the center of
// the screen.
group = new Two.Group();
two.add(group);
group.translation.set(two.width / 2, two.height / 2);

// Now, loop through each pair of vertical points in our points
// array. (That is, points[0][x] and points[1][x])
for(var i = 1; i < points[0].length; ++i) {
	// Make a polygon with this vertical pair of points and the
	// previous vertical pair of points.
	var poly = new Two.Polygon([points[0][i-1], points[1][i-1],
				    points[1][i], points[0][i]]);

	// Now, add this polygon to the container group
	group.add(poly);
}

From there, I keep track of a variable ‘mode’ that is set to 0, 1, or 2, for hue, saturation, and lightness. On arrow key presses, I switched out the mode, and updated the basic hue.

var mode;
var hue;

//This is triggered whenever the user presses a key
function switchMode (event) {
	// First, if left or right is pressed, adjust mode up or down
	if(event != null && event.keyCode == 39) {
		mode = (mode + 1) % 3;
	} else if(event != null && event.keyCode == 37) {
		mode = (mode + 2) % 3;
	}

	// Now, pick the default background color, based on the color mode
	if(mode == 0) {
		// For hue mode, pick a random hue
		hue = Math.random() * 280;
		backgroundColor = hue + ', 60%, 50%';
		var label = "hue";
	} else if(mode == 1) {
		// For saturation mode, medium gray
		hue = Math.random() * 360;
		backgroundColor = '0, 0%, 50%';
		label = "saturation";
	} else {
		// For brightness mode, black
		hue = Math.random() * 360;
		backgroundColor = '0, 0%, 0%';
		label = "brightness";
	}

	// Now, set the background to the background color
	document.getElementById('home').style.background = 'hsl(' + backgroundColor + ')';
}

With a random hue set, the only thing remaining was to update the position and color of the animated graphics. This code is called once per frame:

// Wobble all the points in the array points[1] up and down.
for(var i = 0; i < points[1].length; ++i) {
	// First, use a sine wave to move points up and down
	points[1][i].y = (Math.sin(frameCount * 0.05 + points[1][i].x * 80) *
			 (maxLength - minLength) / 2 + (maxLength + minLength) / 2) * -1;

	// Now, scale the movement of those points based on their distance from
	// the horizontal center, giving us nice high peaks in the middle, tapering
	// off to the sides.
	points[1][i].y *= Math.abs(Math.abs(points[1][i].x / (width / 2)) - 1);
}

// Because we've modified the points, their associated shapes have already updated.

// All that we need to do now is modify the colors for each polygon.
for(var polyName in group.children) {
	// Iterate through each polygon in the group.
	var poly = group.children[polyName];

	// The polygon has two heights: the height of its left side and its
	// right side. Calculate the average height.
	var avgLength = ((-poly.vertices['1'].y - poly.vertices['2'].y) / 2) / maxLength;

	// Now, use that average height to compute either hue offset, saturation,
	// or lightness, depending on the color mode.
	if(mode == 0) {
		var color = 'hsl(' + (hue + avgLength * 80) + ', 60%, 50%)';
	} else if(mode == 1) {
		var color = 'hsl(' + hue + ', ' + avgLength * 100 + '%, 50%)';
	} else {
		var color = 'hsl(' + hue + ', 60%, ' + avgLength * 100 + '%)';
	}

	// Fill and stroke the polygon with this color.
	poly.fill = color;
	poly.stroke = color;
}

Logos

Visual Language


24825

This week in visual design, we’ve been discussing logos. To begin, I’ve been poking around at the history of the DC Comics logo.

Variants collected by Comics Alliance.
Variants collected by Comics Alliance.

As a popular comic book publisher, DC has always had a clear visual identity, drawing on the stars and swooshes of their superhero titles. The most iconic is probably the 1976 “DC Bullet” logo, designed by Milton Glaser. In 2005, DC rebranded, replacing the logo with a much more dynamic swoosh that I feel is a bit over the top.

Two years ago, they rebranded again, revealing a new logo designed by Landor. When a single black and white version of the logo leaked, it met widespread criticism—the logo was too boring, the gradient looked bad, it lacked the excitement of the comics it came from.

DC Logo Variations Today

DC Logo on Comic Covers

Once DC announced the rest of the brand, I think these fears were unfounded. There are still problems (the gradient is pretty obnoxious), but the logo’s simplicity (a geometric “D” folding back like a page to reveal a “C”) makes it a great canvas for all sorts of experimentation.

In particular, the logo looks fantastic on the cover of comics. Tucked in the corner, with the logo’s left edge flush with the comic’s spine, the page metaphor becomes even more obvious. In the exact same corner of every DC title, the logo can subtly echo the color scheme of the page while standing apart—providing a simple, understated constant in an otherwise loud, changeable composition.

The second part of the assignment was to design a new logo for ITP, something that could represent the program both to people familiar and unfamiliar with the program. According to the creative brief, the logo should “reflect the rigor/seriousness of the program but also the creative and experimental nature of the program”.

Sketches

After sketching out some ideas, I settled on the idea of “connection”. To me, the philosophy of ITP is all about making unconventional connections—between skills and viewpoints, between different disciplines, and between different technologies. I decided that the logo would be a series of juxtapositions—chance combinations of different icons representing aspects of ITP. I settled on the “t” as a plus sign that anchored the brand across logo variations.

Logo Type Experiments
A few type variations. I tried to not use Century Gothic (as this is my third week in a row to use it), but in the end, I couldn’t resist.

After figuring out the typography, I began hunting down icons from The Noun Project. Based on the weight and position of the “i” and the “p”, I established boundaries to ensure that all of the icons I chose would subtly reflect the proportions of these letters.

Quadrant of Logos

I don’t intend these variations to stand on their own. Rather, I see them being presented alongside the logo (see above), or in an more interactive context (hover over the logo to the left). At smaller scale, I feel like the wordmark could also stand on its own.


All the icons came from the Noun Project, which licenses their icons for free under Creative Commons, with attribution.

Computer by Daniel Shannon. Microchip by Arthur Shlain. Puppet by Juan Pablo Bravo. Drill Press by jon trillana. Tree by Ferran Brown. Yarn by Marie Coons. Cell Phone by Joris Hoogendoorn. Handset Phone by Juan Pablo Bravo. Television by Alexandria Eddings. Radio Tower by iconoci. Bacteria by Maurizio Fusillo.

Business Cards

Visual Language


11504

Business Card

For this week’s visual language assignment, we were asked to design some sort of calling card or business card. My design is above. I like the visual contrast between the television static imagery and the clean, geometric black and white text. The composition is based off of a simple half-inch grid, which defines the edges of the two white text boxes. The text is set in lower case Century Gothic, one of the fonts I chose last week.

Animated Business Card

The card is double-sided, with identical text on each side, but different static images. My original plan was to attach a thread inside each card, so it could be spun like a thaumatrope, giving the illusion of moving static. On the left is an animated simulation of what this would (theoretically) look like. Unfortunately, in practice, I couldn’t get it spinning fast enough to make a convincing illusion. In the end, I left the cards double-sided, without adding the spinning mechanism.

Morse Code

Automata


19037

For this week’s exercise with solenoids, relays, and other electromagnetic switches, I’ve been working on using a relay to control an 125V AC current load from the wall with the Arduino. For now, I was mostly just making sure I understood how the relay circuit worked, so I just focused on blinking a light.

This program takes the light-blinking hardware and allows the user to type text through the serial monitor, which is then blinked in Morse code. It’s kind of like an Aldis lamp, in spirit, if not in actual functionality.

Here’s the code:

// Length of the shortest unit (a dot, pronounced "dit") in millis
int ditLength = 100;

// Output pin
int lamp = 10;

// Pull in one character at a time
char buffer[1] = {' '};

// Table of all valid morse code values
int codeTable[36][5] = {{1,3,0,0,0},  // A
                        {3,1,1,1,0},  // B
                        {3,1,3,1,0},  // C
                        {3,1,1,0,0},  // D
                        {1,0,0,0,0},  // E
                        {1,1,3,1,0},  // F
                        {3,3,1,0,0},  // G
                        {1,1,1,1,0},  // H
                        {1,1,0,0,0},  // I
                        {1,3,3,3,0},  // J
                        {3,1,3,0,0},  // K
                        {1,3,1,1,0},  // L
                        {3,3,0,0,0},  // M
                        {3,1,0,0,0},  // N
                        {3,3,3,0,0},  // O
                        {1,3,3,1,0},  // P
                        {3,3,1,3,0},  // Q
                        {1,3,1,0,0},  // R
                        {1,1,1,0,0},  // S
                        {3,0,0,0,0},  // T
                        {1,1,3,0,0},  // U
                        {1,1,1,3,0},  // V
                        {1,3,3,0,0},  // W
                        {3,1,1,3,0},  // X
                        {3,1,3,3,0},  // Y
                        {3,3,1,1,0},  // Z
                        {3,3,3,3,3},  // 0
                        {1,3,3,3,3},  // 1
                        {1,1,3,3,3},  // 2
                        {1,1,1,3,3},  // 3
                        {1,1,1,1,3},  // 4
                        {1,1,1,1,1},  // 5
                        {3,1,1,1,1},  // 6
                        {3,3,1,1,1},  // 7
                        {3,3,3,1,1},  // 8
                        {3,3,3,3,1}}; // 9

// Basic setup stuff
void setup() {
  Serial.begin(9600);
  
  pinMode(lamp, OUTPUT);  
}

void loop() {
  // Check if there's incoming data
  if(Serial.available()) {
    // Read in one byte
    Serial.readBytesUntil('/', buffer, 1);
    
    int currentCharacter = buffer[0];
    int currentIndex = -1;
    
    // Check valid character ranges and find the corresponding index
    // in the morse code table.
    if((currentCharacter >= 'a') && (currentCharacter <= 'z')) {
      currentIndex = currentCharacter - 'a';
    }
    
    if((currentCharacter >= 'A') && (currentCharacter <= 'Z')) {
      currentIndex = currentCharacter - 'A';
    }
    
    if((currentCharacter >= '0') && (currentCharacter <= '9')) {
      currentIndex = currentCharacter - '0' + 26;
    }
    
    if(currentIndex != -1) {
      // Serial feedback stuff
      Serial.print(buffer[0]);
      Serial.print("  ");
      
      for(int i = 0; i < 5; ++i) {
        // Loop through until we hit a '0' in the letter sequence
        if(codeTable[currentIndex][i] != 0) {
          // First, print a dot or dash for serial feedback
          if(codeTable[currentIndex][i] == 3) {
            Serial.print("-");
          } else {
            Serial.print(".");
          }
          
          // Now, flash the light on and off 
          digitalWrite(lamp, HIGH);
          delay(codeTable[currentIndex][i] * ditLength);
          digitalWrite(lamp, LOW);

          // Wait one dit between flashes
          delay(ditLength);
        } else {
          // We finished the letter early
          break;
        }
      }
      
      // Wait two more dits between characters
      Serial.print("\n");
      delay(ditLength * 2);
      
    } else if(currentCharacter == ' ') {
      // If we've got a space, wait four more dits (7 total) between words
      Serial.println(" ");
      delay(ditLength * 4);
    }
  } else {
    // For serial feedback: if we just finished printing characters, add a
    // blank line between this and the next thing the user wants to write out
    if(buffer[0] != ' ') {
      Serial.println(" ");
      buffer[0] = ' ';
    }
  }
}

Typography

Visual Language


16061

Abril Fatface

Abril Fatface, a modern serif with very high stroke contrast. I like the flourish on the “y” and the way the double “t” joins. When it comes to serif type, I have a deep love for slab serifs. They have some of the respectability of serif typefaces, but with a more modern appearance, I think.

Charis

If I needed something not as loud, I might use a face like Charis SIL. It’s still a slab (or very close to one) and has very clean shapes, but it’s the sort of typeface that would work well for setting a large block of text.

Chunk

At the other end is something like Chunk, a heavy slab serif that’s great for titles and headings (even at pretty small sizes).

Century Gothic

Century Gothic, a lovely geometric sans-serif. It’s upper-case letters are hit or miss (in my opinion), so I set this in all lower-case. I like geometric typefaces a lot—this one sets up a nice contrast between the round shapes (like the “a” and “m”) and the straight lines of the “t” and “h”.

League Gothic

Another good sans-serif: League Gothic. It’s tall and thin, but not in a way that looks off. I particularly like the way that the top of the “a” curves around in what appears to be a perfect half-circle.

Avenir

Finally, another of my favorite geometrics is Avenir. It’s not as stark as something like Futura or Century Gothic, but it still retains enough of those straight lines and perfect circles to be interesting.

Here’s the whole set. In general, I want my typography to be visually distinctive without being too decorative or over the top. I think this set of typefaces strikes that balance well.

Names

In addition, here are a couple type treatments designed to express certain concepts: blinds, dissonance, and angle.

Expressive Type 2

Blog 2: Observation

Physical Computing


19262

In order to observe interactive technology use in the wild, I staked out a pair of document scanning stations in the NYU library. These stations, with a scanner attached to a specialized computer kiosk, allow for scanning books, chapters, articles, notes, or any of the other paper information one finds in the library. I assumed that the users (as a sample of University students and faculty) would not be experts, but would be more computer literate than the general public. I assumed the system would be used mostly for books and multi-page scans, and that it would support various digital file transfer systems (given that it wasn’t a photocopy system).

Having used many frustrating scanners in the past, I didn’t have high expectations for usability, so I was pleased that most of the people I observed could operate the mechanism with relative ease. One tap on the touch screen, and you’re prompted to pick a delivery mechanism (email, Google drive, etc). Another tap and you’re faced with a page of scan settings. Here users diverged, between those who immediately accepted the defaults and those who scrutinized their options at length. The deliberators were often those who were scanning large documents and wanted to perfect the settings at the outset (a tricky task without visual indication of the effect of a given setting). Once settings were approved, then the user pressed one more button and the machine scanned the page, automatically cropped and rotated it, and displayed it for the user to edit further. On one extreme, I saw a user approach and successfully scan a single-page document in less than 90 seconds.

At this point, I noticed one major problem with the workflow. With a scanned page open, it was unclear for many people whether the “scan” button in the corner (which they’d initially pressed) would now replace the current scan with a new scan, or add a new scan as a second page. Assuming the former, I saw multiple people instead push the “Next” button, only to backtrack when they were confronted with an interface for emailing their scanned document. In fact, the workflow for multiple pages was quite tidy—keep hitting scan on the same screen and it keeps adding pages. Once they settled into this rhythm, users generally took no more than 10 seconds a page, but the interface did little to suggest that this flow was possible.

This process constitutes a very clear interaction by Crawford’s definition: the user instructs the computer to scan a page, the computer processes and outputs an image of a scanned page, the user evaluates this image and responds to the computer accordingly. On the large scale, I felt like all the users understood the interaction. However, on the small scale, certain details caused problems. In particular, I think the touchscreen interface is lacking. It worked pretty well early on in the process, when one must choose one large button from four or six. Inevitably, though, the user wishes to actually save the scanned file, and this means typing an email address, or password, or file name on a vertically-mounted touchscreen keyboard. The users I observed were typing maybe ten words per minute, in one of the most inefficient parts of the whole process. Often, the button wouldn’t even register the tap. Whatever visual feedback was supposed to work here had failed, causing the user to wait (in hopes that the computer was stalled) before trying to press the button again. Without the physical affordances of buttons, these virtual buttons gave little indication of whether they’d been pressed, let alone what angle, force, etc., was required to effectively activate them.

Automata Prototype: Owl 1

Automata


45634

Prototype 1

This week in Automata, we’ve started working with stepper motors. As part of this, I’ve been experimenting with an ultrasonic range finder mounted on a motor to control rotation. After initial tests with a rough mockup (above), I built a small, wooden character to house my mechanism. Because of the shape of the range finder and the head-turning rotation I was trying to create, I immediately thought of an owl as a design motif.

The basic algorithm is to stand still until something passes through the owl’s field of “vision” at a close enough range. The owl then waits for the object to move away, at which point it sweeps left and right for increasing amounts of time, trying to find another object (or the same object) within the close enough range. If it goes too long without finding anything, it will give up and wait for something else to pass by.

So far, the head moves slowly and sometimes triggers randomly. I think I need to fine-tune the mechanism so that the head doesn’t tilt and shake around so much. In addition, my current design limits the full span of the head turn. Perhaps with an improved design, the owl can make a more complete rotation.

Owl Automata Test 1 from Matthew Kaney on Vimeo.

Signage Found in New York

Visual Language


21469

For my second week of Visual Language, we were assigned to look around New York for examples of good and bad signage.

Fire/Police Call Box
The engraved text on the button reads “Push”. That unfortunately didn’t show up in the photograph.

First, and example of some really well-done signage. This call box is probably 30-40 years old, but the instructional design very much holds up. The use of large text and iconography is clear, and the instructions are concise. Additionally, there is an excellent hierarchy of information. The complete instructions are printed out at the bottom for a casual viewer who has time to leisurely read. However, someone who is in a hurry can figure out what to do first without reading anything, at which point they receive a reminder of the next step. This philosophy—making sure that viewers get the information they need when they need it and not before—is key to good signage.

Below, were a few examples of signage I found that were poorly-designed, misplaced, unclear, or some combination of the above. As it turned out, all of my examples involved arrows. I think arrows are so brilliantly simple, but so easy to completely mess up when one’s not paying attention. And when that happens it (apparently) bothers me.

ATM

For example, the ATM is not down there. The arrow immediately misdirects attention and should have been physically cut off the sign. To be fair, I think this reflects poorly not on the store but on the ATM company, who for some reason decided that this sign would be best hung directly above the ATM. In most cases (such as this one), I can’t imagine that placement would be ideal. And, besides, a bright red “ATM Machine” sign without an arrow would convey the same message hung above the ATM, and would ultimately be more flexible.

Door Opens Outward

More strange arrow choices. To start with, which door? The one the arrow’s pointing to? That’s obviously not the case. It must be the other one, then. However, that door’s opening is protected by the wall between the buildings, so it’s unlikely to catch anyone by surprise. Indeed, it’s impossible to both read this sign and be in the path of the door. And again, the arrow: does the door somehow open past the arrow? And why is a straight arrow used to illustrate a curved path?

No Smoking

And finally, it seems like a terrible idea to use the word “this” and an arrow in such close proximity if they don’t refer to the same thing. It’s a simple sign, but the strange layout of content makes it much more confusing than necessary. (And, given that it’s probably a mass-produced sign, here in the doorway of an NYU building, there’s no reason it wasn’t better designed.)

No Smoking Revised

Here is the same sign with my revisions. Bold type with plenty of negative space pulls the eye toward the main message. The secondary message about the air intake is placed at the top of the sign (which makes sense spatially), but in a less visible manner. In addition, I removed the “above”, because the arrow communicates that unambiguously.

Concept: Paper Automata

Automata

Paper Automaton Illustration

As a starting point in my Automata class, I’m exploring the concept of two-dimensional paper automata. I’m drawing inspiration from the types of articulated paper puppets often used for shadow puppetry or animation. Because they lie essentially flat, I’m interested in the mechanical challenges posed by this form. I’m also interested in cut paper (either laser cut or die cut) as an especially accessible form of digital fabrication, and am curious to see how sophisticated of mechanisms can be produced in 2d paper with these technologies.

I’m envisioning a series of animated figures who could perform as part of a larger narrative. The illustration above describes a modular system for these figures, where the figures could be designed as self-contained paper constructions with an exposed gear. This paper construction could then be slotted in to a machine box, where an internal motor and gear would connect with the paper gear and drive the motion.

As an initial experiment with the material (and as a means to get comfortable with common mechanisms, I prototyped two mechanisms in card stock. In order to create joints, I considered various approaches, but ultimately settled on a circle of paper cut into a star shape, with the points of those stars folded through matching holes in other components and then glued down. I started with a series of crossed levers (motion No. 144 in 507 Mechanical Motions) in order to get the joints working.

Scissor Levers

Scissor Motion

Overall, the prototype worked quite well. The joints were strong, but allowed full motion with minimal friction. When kept flat (either holding by hand or by resting on a tabletop), the paper had plenty of strength for the mechanism to work. However, if I wasn’t careful, the paper had a tendency to bend, which completely wrecked the motion. This was clearly pushing the size limits for the card stock I was using. For better structure, I’ll probably try a combination of stiffer paper and smaller individual components in the future.

The crank mechanism (motion No. 92), worked as well, if not better. After some trimming, the motion was smooth, and the main structural piece was the only piece that tried bending. With wheels, cranks and levers working, the next major challenge will be gears. Gears are fundamentally three-dimensional in their operation, so the question stands whether the design can be modified into shapes that consistently interleave while still lying flat.

Crank

Circular to Linear Motion

Graphic Design Analysis

Visual Language


19106

Areas of My Expertise

For my first assignment in Visual Design, I’m going to discuss the cover of John Hodgman’s The Areas of My Expertise. This book is loosely based upon the structure of an almanac, but filled with absurd information, made-up trivia, and other such nonsense. This book launched Hodgman’s career of film and tv appearances (particularly on The Daily Show and as the PC in the “Get A Mac” ad campaign), and eventually led to two followup volumes.

At the time, though, Hodgman was mostly an unknown writer and humorist, and the design of this book does an excellent job of setting the tone for what to expect.

Image from The Old Farmer's Almanac
Image from The Old Farmer’s Almanac

The design draws inspiration from the Farmer’s Almanacs from about two centuries ago—with lots of text, complete sentences, and variations in font size–while pushing this aesthetic to its absurd limit. In addition to the main information, every bit of available space is crammed with text: Lists, notes, descriptions, and even (in small print) the full text of the book’s introduction. At the root, the design is a humorous illustration of the tension between information and confusion. The various text and figures that crowd the book’s cover attempt to inform, but the end result is a chaotic mess. This conflict between sense and nonsense reflects the book’s absurd, deadpan humor, as well as its overabundance of content—charts, footnotes, captions, asides, digressions, appendices and the like. Whether you find this approach to the cover intriguing or off-putting says a lot about whether you’ll enjoy the book.

Basic Grid Structure
Grid Structure (Click for Animation)

The composition of the page is ultimately pretty basic. The cover is subdivided, first horizontally, and then vertically, and then horizontally, and so on. Like the cover itself, this structure is somewhat arbitrary, with heights depending on how many lines of text will fit in a space. However, even for something so intentionally crowded, the design makes sure that you can understand the important information. First, the color scheme is very simple: a bright blue and orange, as well as white text. The colors contribute to the cover’s visual loudness, while dividing the content between important (orange) and not (blue).

Color Scheme

The text, meanwhile, is mostly set in medium weight Futura, with different sizes indicating different levels of importance. Most dramatically, the author’s name is more than twice as large as the next-largest text, allowing it to stand out from the sea of letters. Most importantly, the title receives a unique type treatment. It’s set in a different, condensed typeface, is even larger, and has a solid black drop shadow (the only black on the cover). Because of this font change and use of black, the title immediately commands the viewer’s focus.

Type Detail

Blog 1: What is physical interaction?

Physical Computing


18297

In The Art of Interactive Design, Chris Crawford defines interaction as the process through which two actors take turns listening, thinking, and speaking, like in a conversation. After defining his terms, he explains himself with some nice examples: computers are interactive, books and movies aren’t; dancing with another person is interactive, dancing to music is not; live performances can be interactive, but are hardly ever interestingly so. This definition is pretty cut and dried, and it makes a lot of sense, especially given Crawford’s bias toward computer interfaces. If one is imagining information conveyed through sight or sound (most computer output, to be sure), it’s easy to think of discrete moments of communication back and forth with processing time in between. We imagine a horror movie viewer shouting at the unlistening screen and appreciate our interactive media all the more.

Unfortunately, once we start considering touch, everything goes murky. I can shout at a movie screen to no effect, but if I touch it, things happen. The screen will move, or resist, or transform its shape. This process of action and response sounds kind of like our interaction cycle—are all physical objects interactive?

Crawford has no patience with this line of reasoning. No, such actions don’t count. In his definition, the actors must be “purposeful”—a deceptively philosophical concept used here as shorthand for “a computer.” He dismisses the refrigerator light switch as a trivially simple interactive mechanism, before promptly changing the subject. Left unsaid: is there a machine sufficiently complex to count as interactive? A piano? A typewriter? A mechanical calculator? Do any of these machines “think” enough to really interact? In this light, Bret Victor’s “Brief Rant on the Future of Interaction Design” seems relevant. Victor is a champion of physical objects. He admires the way we manipulate them, the subtle information their weights, temperatures, textures and so on impart. Physical objects can be acted upon, and they can communicate back, but Victor never actually uses the i-word when describing hammers or jar lids.

My interpretation is that both writers are driving at the same continuum from opposite ends. On the one end are simple physical objects, which I would call reactive, and on the other are full interactive systems. And what separates the two? Nothing specific, I’d argue, especially not Crawford’s requirement that an interactive agent think or act purposely.

So, what defines interaction, then?

Most everything is reactive, meaning that it behaves (broadly speaking) like an interactive object. A hammer may not interact, strictly speaking, but it will react in specific ways, which is almost as interesting. However, if we must propose an arbitrary distinction, I believe it is this: interactivity is not determined by how elaborate an actor’s thinking ability is, but rather by how much novelty the actor injects into the interaction. We don’t think of a rock as interactive because when we move it, it moves as expected, no more or no less. The simplest interactive system may therefore be a handful of dice. Roll the dice, and the result is unexpected, forcing us to reevaluate and keeping us engaged. A system can be sophisticated or not, predictable or not, but as long as each new round adds new information, we (the humans) don’t get bored, and the system has successfully interacted.

At the same time, we shouldn’t discount merely reactive objects. As Bret Victor points out, there’s a wealth of information to be found in the ways that physical controls react to human touch. Ideally, the physical interactions we design should blend both ends of the spectrum. Small, reactive details make things more comfortable and understandable, and a broad interactive design keeps the process interesting. The “pictures under glass” that Victor criticizes are great case studies of a design that’s interactive, but insufficiently reactive.

Of course, a designed object can also be reactive instead of interactive. The computer mouse is often cited as a pinnacle of interaction, but I contend that it is less interactive (by my explanation of Crawford’s definition) than purely reactive. The mouse rarely gives the user new information. Rather, it does what it’s told, accurately, but boringly. If the person and the mouse are having a conversation, that conversation is pretty one-sided.

In light of this idea, I thought back to an electronic artwork I read about years ago, Rafael Lozano-Hemmer’s Pulse Room. This piece (where a user’s pulse controls the flickering of a lightbulb in a large array) still attracts me for the reasons it did then—it’s impressive, yet elegant, and so simple to use. However, I now wonder whether it should really be considered interactive. Crawford would probably say it was, but ultimately, it responds in exactly one way to any type of input. It’s a good response, to be sure, but when designing interactions, I feel we could stretch further. How do we respond in interesting ways that keep our users engaged? That, to me, is the fundamental question of interaction design.