sounds of pictures // képek hangjai

31: SoundRaw / SoundDraw

2012-01-31T00:18:00.000-08:00

Sound draw / SoundRaw again....

In this post I introduce the Sound draw app for Mac/Windows. It's a standalone or Max-patch version of the installation presented before, with some add-ons. I tried to apply all of my experiences gained in this month while building the examples above. This is a very first version, but please take a look and send me feed-back!

Here's a screenshot:

And here are the files: http://kbalazs.periszkopradio.hu/sounddraw/ with documentation, etc.

Hereby also i publish the patch for Max5 used in several examples - the harmonograph-style audification tool. The working title of it is "SoundRaw", and look for the screenshot before deciding to download it or no:

The patch can be downloaded here. enjoy it!

This is the last post for a while in the blog. I would like to thank all the readers and contributors to worked with me on this "impossible mission" - maybe we found something interesting for the future reader/listener. I wish everybody good noisy sonifications against conventional parameter mappings...

(blog)lecturi salutem,
Kovács Balázs, Santa Barbara, 01/31/2012.

30: flow

2012-01-30T01:06:00.000-08:00

Waterbottle-cap ventillators follow bikers' flow

Flow is a relatively easy task to track - in this example i decided to go "out of the box" and test it in real-world situation. You'll see, how my custom-built waterbottle-cap and -pen ventillators can rotate according to the huge mass of flowing bikers on the UCSB campus. I'm also a biker and like how they flow and like the rotating of these objects.

How it works?

In the example I used a camera for forward visual input to the computer. The processing were done in Jitter, the controlling task with Arduino.

The setup:

The patch: (can be downloaded here)

And the Arduino code here. I use Adafruit library and motor shield, anyway Standuino is also a perfect solution (just for the sake of the advertisement;)

kind invitation for tomorrow, joint with presenting the last entry of this blog:

29: movement to waveform

2012-01-28T21:37:00.000-08:00

SoundLines

Christopher Jette: Sound

Katharine Hawthorne Movement

Special edition for this sunday is a contribution by Christopher Jette, a composer and PhD candidate at UCSB-MAT. The video above is a documentation of his work SoundLines as performed in august 2011 in Minneapolis. The description is as it follows:

"SoundLines uses two individual lines of the live video of a dancer as the basis for sound. A greyscale version of the live video matrix is read in two horizontal positions. These lines are translated into a waveform that is the basis for the structured live electronic sound improvisation. The basic waveform changes when the dancer moves. The electronic music performer has a fixed score defining the arch of the piece. Multiple iterations of the basic waveform are layered and blended in order to reinforce and juxtapose the movement with sound."

Also, this piece will be performed Friday February 24, 2012 at CCRMA, Stanford CA.

Documentation

Scanning

Interface

Interface - unlocked

Recommended links:

Szigetvári Andrea: Lengőajtó/Swinging Doors - similar approach with color tracking
Eyesweb gesture/movement analysis toolkit

28: shapes

2012-01-28T01:23:00.000-08:00

In this example i try to persuade Sound Draw to play different sounds when moving over different shapes. In this very raw experiment the system makes a linear audification for each coherent blob.

How it works?

The process is similar to the Sound Draw, with the only difference, that while the system recognizes the separated blobs, it forwards the data for an audification (similar to the #2: linear scanning audification).

The patch and application will be published soon, after presenting it on tuesday 1/31 at UCSB-MAT:

-invitation:

27: running

2012-01-26T23:04:00.000-08:00

I watch the children running all the day, this work is a hommage for their hard job. I let the system follow the main (closer) and the smaller (further) movements, and threat them differently to drive two separated granular-like synthesis and a panning according to the guys' position. I added a stop-start mechanism as well.

How it works?

In this case i apply the same methods as the facebook movie: simply extract the frame difference, and get the defference between the movement of the background (whitout this, when for ex. somebody runs before a chair, and he runs away, this could be regarded also a movement of the chair), and calculate the position of the running kid. Then i just apply a granular-like modification (with munger~) and a panning to the sound recorded before. The system could be developed to detect the direction of the running, and also to be more exact while making difference between movement in the fore- and background.

The MaxMSP/Jitter patch is here.

About the video

The video is recorded in our flat in Goleta, CA, 12/10/2011.

26: draw a keyboard

2012-01-26T10:25:00.000-08:00

Visual keyboard

(download this video here)

To develop the previously introduced Sound Draw i imagined a paper based keyboard and controller system. Which is unusual (or we could call it simply abnormal) in the age of touchscreen-based keyboards, but why not? The example above present a spelling keyboard, which can be used with blue "buttons". Below there's a tonal version of it as well.

(download this video here)

How it works?

The system extracts the blue channel, and watch for changes on it. If a change occurs, it finds the greatest blob, and returns the coordinates of it, resulting the ground for the pitch mapping.

The patch for MaxMSP/Jitter is here (sorry for being messy).

25: adaptive mechanism

2012-01-25T01:26:00.000-08:00

Sometimes the installation achieves such a level of interaction, that the human "interruption" is not needed anymore... this is when the system is self-adaptive, which means that it changes his parameters according to his surrounding. This video above presents a very basic, but efficient method to use self-tuning in sonification. The brightness of the video is modified according to one pixel's actual value on the bottom center: if this pixel is white (cloud covers), nothin happens, if it's darker (no cloud), the change the values of the entire picture in order to let this only one white again... (not a democratic attitude, sure)

The role of sound in the example is a negative one: if we hear pitch changes (and we will), then to self-tuning process is not correct, but is in effect, and in average we hear a constant frequency.

How it works?

We have to set-up a feedback, which has two independent output: one goes to the screen, one goes to the manipulation process. The latter contains the pixel color analysis, and the calculation which returns the value of the target brightness.

Patch for Max4 here.

Todo:

The system could be developed for let a user select a target color instead of the white.

About the video:
The video is recorded in the window of UCSB-MAT / Pluriphonic Lab, november 30, 2011.

24: background substraction 2.

2012-01-24T11:33:00.000-08:00

In this example You see birds moving around a tree. Their movement is mapped to the resulting sound's pitch and duration, the movement speed/object mass is to velocity.

How it works?

The system extracts the background while constantly comparing the running average of the movie and the actual frame (see this doc on background substraction). Then the frame differences will be counted and thresholded. In this point, cv.jit.label helps us to find the largest moving object, and let us possible to us this coordinate for the mapping process.

Download MaxMSP patch here

About the video
The video is recorded beneath a tree in the Family Student Housing, Goleta, 01/18/2012.

23: two-level tracking

2012-01-23T01:28:00.000-08:00

Face-gesture tracking

In the example above, I present an interactive-like system, which tracks the eye and mouth, while utilizing their movement to control the sound playback. This video has a speciality as well: during the record, I didn't heard the sound what i influenced with my gestures.

How it works?

In this case, we have to constuct a two-level tracking system. The first one looks for the region (eye, mouth) and stabilizes the changes caused by moving the entire face. The second, while strongly built on the first level, performs a simple analysis gathering the informations about direction and expansion of the move.

here's a screenshot of the patch working:

the patch is undocumented yet, if needed, i can send it.
The video is recorded 10/28/2011 in the Pluriphonic Lab of UCSB-MAT.

22: av-feedback

2012-01-22T01:27:00.000-08:00

Jeremy Keenan: Light Loop (2011)

This sunday's special is the first non-digital sonification presented in the frames of this blog (it's necessary to mention, that a significant history of sonification and audification was done before the digital era!). Light loop, by London based sound artist Jeremy Keenan present an audio-visual feedback loop - as he wrote on the project's site:

"A room full of 16 guitar amplifiers, 16 guitar pickups modified with light dependent resistors, a DMX controlled moving head lamp, 4 amplitude sensors and an Arduino microcontroller with DMX control shield.
The light from the DMX lamp shines on the light dependent resistors, causing feedback at the amplifiers. The feedback is picked up by the amplitude sensors, which control the movement of the DMX lamp. The changing light from the DMX lamp causes new patterns of audio feedback, which trigger subsequent movement of the DMX lamp..."

How it works?

Here are some photos about the setup, provided by Jeremy:

The room

Guitar amp + sensor

Sensor + cable

Guitar amplifier

DMX setup

21: movement analysis

2012-01-20T23:24:00.000-08:00

Facemovie

this video can be downloaded here.

Facebook is a central community medium nowadays. I recorded a series of screenshots in july 2011 a couple of days at there focusing to the friends on chat box, and made a video from them, using the changes controlling the resulting sound.

The work is made in august 2011.

How it works?
The system records every minute a picture file named with the timestamp. Then reads back and keeps the differences between the frames. The differences are then divided to color channels and controls the pitch and density of the sounds.

The recorder patch for Max5 is here,
The player patch for Max4 (and up) is here.http://kbalazs.periszkopradio.hu/hanganyagok/sop/20-facemovie-patch2.mxb

20: wavelet decomposition

2012-01-19T18:32:00.000-08:00

Dara O Shayda: Fall Berries (2011)

Today's featured work is by Dara O Shayda, Toronto based artist, with her work titled Fall Berries.
Description: "The main idea behind this new composition technique is to use images for generation of sounds. There is nothing new

about this and there are quite a few such software tools on the market e.g. MetaSynth’s Image Synth. However the novel idea is to use the Wavelet Theory to analyze the image and generate other interim images to use for variety of additional sounds: http://lossofgenerality.com/wavelets/ (use for the tutorials for Wavelets)

For an example the author chose the image above (credit: http://files.capturethisphotography.com/labels/Great%20Swamp.html)."

The sonified version:

Fall Berries by Dara O Shayda

Recommended links:

19: foreground separation

2012-01-19T01:23:00.000-08:00

Depth analysis on a 2 dimensional image seems to me a "black" magic. How we could attempt to perform it when human beings need also at least 2 linked eyes to it... I'm relatively new to computer vision, and therefore i just have found few signs of the solution. The first was a paper about low-pass filtering some channels on the YUV color space. I followed the receipt, but nothing happened, however i found an another solution. And two more with further thinking it. One is by extracting and closing the edges (these are the sharp lines, which must be equal with some focused area), and mask the whole picture; the other is to blur the entire image and substract it from the original.

Just show You for the sake of comparison:

After this success (maybe my job would be better done as computervisioner than sound artist;) I tried to make it sound. The result is not so interesting, just a demonstration, how the system steps on the head of the flowers.

The MaxMSP/Jitter patch used can be downloaded here, the video above is available here.

About the picture

The photo is taken in Solvang, California. This is a tourist center for those who're interested in danish-dutch culture (the city is founded by danish people), and can find a public restroom there which is reconstructed by Arnold Schwarzenegger.

18: background / foreground separation

2012-01-18T11:39:00.000-08:00

On 18th january 2012. Wikipedia blacked out his english pages due to the piracy laws going to introduced in the US. In this video example the system separates the blacked out page to background and foreground, and threates them differently in order to make sounds from the pixel contents. The sonification process affects the picture as well, some parts will be deleted, some will be underlying - like in the reality.

17: sound draw

2012-01-17T00:16:00.000-08:00

This is a manual sounddrawer. Doesn't need any sensors, but a camera, paper, colored pens. Doesn't need sensor glove or reactable as well. The sound output needs to be tuned, i try to develop it in the forthcoming days.

How it works?
The computer vision system looks for red and blue/green colored "blobs" (groupd of coherent parts) realtime, segments them, while the sonification system adds the mapping by the x/y positions, mass, etc., and steps through these objects depending on their distance. As You see, it has bugs (the drawing hand becomes also tracked, etc), but the idea hopefully can be figured out.

Sounddraw as installed at UCSB-MAT, 01/17/2012.

Recommended links:

Draw a Sound by the vOICe - fft based approach w/ java
Drawdio - electronic pen by MIT
IanniX - graphical sequencer (thanks for info to CDM!)

16: destructive audification

2012-01-16T01:00:00.000-08:00

I sonify away ("elszonifikálom") the flag of Jobbik

Sorry, this is a political statement influenced by the flag-burning action by the far-right party Jobbik performed in Hungary last saturday. I just imagined how could I destructively sonify their flag - this video above is the result. I'm not a politician, just a citizen of this beautiful country called Hungary, but now i feel that we, citizens can perform changes what the politicians can't / don't want do.

How it works?

I apply the technique described as destructive audification: the image changes realtime as the sonification happens, making possible to achieve the silence / blank image. The user can set the reading/scanning speed (frequencies of two sinewaves), and the overwrite color.

The patch is inspired by the harmonograph frequency-visualization instrument, and works as a standalone frequency-drawing program as well.

a screenshot:

and here You can download the patch for Max5/6- as always, i have to apologize for the mess inside it.

15: Voice of Sisyphus

2012-01-14T22:43:00.000-08:00

download the video here.

Special edition for this sunday is a work of UCSB-MAT masters student Ryan McGee, presenting bidirectional image sonification and image manipulation, described as follows:

"Voice of Sisyphus is a multimedia installation by George Legrady with software development by Ryan McGee and Joshua Dickinson. The novel image sonification software scans and translates regions of the image into audio waveforms. The work was displayed at the Edward Cella Gallery in Los Angeles from November 5th, 2011 to February 4th, 2012."

The method of his works present several new approach regarding the latest 14 entries of this blog:

the software looks for fft, low-pass and high-pass filters of the image portion selected, in conjunction with
strong connection of image-preprocessing and audio synthesis
user-selectable region, base frequency, interactive tools.

Here You can see a screenshot of the back-end software:

About the image
The image, titled "Hungarian ball" is taken by George Legrady in Montreal in the 70s.

14: color tracking

2012-01-14T01:25:00.000-08:00

My favourite hobby is catching and tracking a colored pixel, following it and let it go away. In this video i try to track the turquoise-colored antenne of the butterfly - without success but resulting an interesting animation and interaction between the movement and the implied sound. The tracking is working well on red and blue colors, and in this phase i still tried to extend the spectra of possibilities.

13: sonic stepper

2012-01-12T20:46:00.000-08:00

This example presents two different virtual steppers between dots drawn on a paper. While stepping, the distance and direction between dots influences positively or negatively the sound output (which is anyway two very basic midi instrument).

(download the video in .mov format here)

About the picture

The picture is created one of my children with glass-paint technique - on paper.

12: seismograph

2012-01-11T21:59:00.000-08:00

Seismograph sonification demo

Seismograph data generally can be a good source for interpretation and research while being sonified and compressed in time. I just made a small application which can do it 4 different ways, allowing the listener to analyze seismograph data very quickly. In this example You'll see/listen to 11 days of earth-movement in a few seconds, while the peaks of seismic movements can be analyzed, possibly expected.

Sonified version:

The video presents 4 different way to make a sound analysis of seimograph data images, based on parameter mapping sonification.

Download the original video here.

How it works?

Unlike other seismograph sonification methods, this example presents parameter mapping instead of audification. It makes possible to fine-control the sound output. The above example present 4 different output:

- 1/ the density of the curves is influencing a frequency of a sine wave;

- 2/ the running average of the density is counted and is manipulating a sine wave frequency on the right channel - the left channel is similar to the first variation;

- 3/ the difference of 2 and 1 is counted and the mapping method is the same;

- 4/ the density modulates the pitch of a preloaded soundfile on the left channel, the right is the unchanged version.

Download the application (MaxMSP/Jitter patch) with some example pictures here.

About the picture

The picture - and the data used in the example - are coming from the seismograph data of recent helicorder display archive at http://pasadena.wr.usgs.gov/webicorder/. I've choosen the data of Santa

Barbara, and therefore i remember personally all of the small seismic movements.

11: histophon

2012-01-10T22:47:00.000-08:00

Histograms are the visual equivalent of sonograms: extracts the color intensity levels according to saturation. I tried to use them for the today's work in order to find sonic realization of the picture above. The starting point was to excluding the pink background, and then let the system to jump between the flowers' red parts, while constantly playing their notes, and changing their hue. While changing the color content, the sound will also changing.
This work is inspired by the visual treemap (what i met yesterday on George Legrady's class) - a 2-dimensional nested rectangle presenting informations. I just tried to adopt this method of visualization to the sonification, and this is an alpha-version result.

Sonified version:

(download the video)

About the picture:
The photo is taken on the Los Corneros street, Goleta CA, 10/15/2011, during travelling to garage sales nearby.

10: spider-web

2012-01-09T22:00:00.000-08:00

Imagine a hyperactive spider who weaves a web accross every sharp edges! We try it also virtually, and it's really not so hard than in the nature. After the last edge detection-based example, here i'm going to present a work which utilizes the edge for creating a sonic-visual image using their shape. The "spider" simulated by this example goes around the leaf and his (unusually loud) sound is determined by his movement speed and direction.

Sonified version:

Download only the video or the sound.

How it works?

After the last edge detection-based example, i figured out a faster edge following method (I didn't tried there once again, but i hope it could be more efficient on those work as well). The method stores the white pixels' (= edge point) coordinates in a list and recalls them randomly or linearly - therefore we can avoid the slow search-for-edgepoint process. Then i let the system found a center point, and go thru these points, drawing line segments to form a web. During this it calculates the segments' direction and length, the density of the points, and apply them all to the fm-like sound synthesis.

The sound applies a very basic spatialization too: the lines beginning from the left and right are panned to the according channels.

I include a screenshot of the patch, and very sorry, but the patch is a mess, so i can't attach it, but i'm ready to post if interested, just drop me a mail!

About the picture
The picture is taken 11/16/2011 in Goleta, on a street what we call Lemon street, because there is a forest of rich orange and lemon trees:)

09: edge-follow

2012-01-09T07:11:00.000-08:00

re-AHA-izer

Let the picture talk! that's the motto of the second section in this month. During these examples we'll explore some possibilities to extract coherences in images, and build a sonification method around them - instead of apply a pre-existing method to the picture and let the listener understand it (or not) as done in the previous examples. We'll mainly apply the tools of computer vision to sonification, keeping the narrow way signed in the motto.

The first one in this block is devoted to the metanarrative music video of the '80s one-track band A-ha called "Take on me". In this video we see cross-sections of reality (of the reader), virtuality (of the cartoon) and embedded reality (of the reality inside the cartoon), marked with pencil sketches or realistic frames. In this manner, re-Aha-izer extracts the edges of the image, then finds these lines and plays them, while erasing them in the same time.

How it works?

The system is looking for edges (with the cv.jit.canny object), then for the pixels of the edges, and attempts to follow their lines as You see in the following screenshot:

During the search+erase process we hear sounds generated by the X/Y positions of those pixels. The process is very slow, as You see on the video above. A small excerpt from the original version can be listen'd here, and the MaxMSP/Jitter patch can be downloaded here.

About the picture
The original photo is taken in the "Royal plaza inn" at Indio, where we stayed on 12/14/2011.

08: reversible audification

2012-01-07T15:38:00.000-08:00

The "sunday special" for today is a video demonstration recorded in november 2011. It presents a process which sonifies waves with 2 dimensional scanning audification, and then converts the sound back to video. The documentation presents You ratio changes of the scanning speed on the X and Y axis. The question of the experiment was the following: can the sonification process be so "clear" which allows for us to reverse it? if yes, then we're on the right way with scanning audification, if not, then we have to change our ways.

However the example works, the conclusions for me tend to say: no. The vision of "clear audification" maps the modalities pretty well as You see above - but we never can't find the optimal ratio between the recognizable sound and the recognizable video, rather we have to choose among them, or should rely on both of them. The ways now are to find new approaches by forgetting or rethinking the old ones.

How it is working?
The example above has nothing new added to the previous examples. The patch utilizes the image synth methods back and forth. The patch can be downloaded below:

patch for MaxMSP/Jitter here • if needed, there's the video used in the example

07: spectra-scanning

2012-01-06T22:09:00.000-08:00

Scanning seems to be immortal - otherwise we found the possibilities to move away from this technique. This example sonifies the image faster than the color synth, and slower-deeper than image synth, while provides direct mapping between the colors (RGB channel) and the spectra (FFT synthesis).

Sonified version:

Sounds of Pictures #7 - Scanning into the spectra by Xrc

How is it working?

The system scans the whole image top-bottom / left-to-right. The rows are splitted to red and green channel and processed serarately as follows: more power in color means more power in a frequency band raising from left-to-right. This is a very basic sonification technique, one should develop it to be more accurate in spatialization and/or in grounding connections between pixel values and frequency bands. This is the topic of following examples later.

The MaxMSP/Jitter patch used for the example (needs this and this as well!)

About the picture:

The picture is taken in Pismo Beach, 10/8/2011 at the motel where we stayed. It was a very interesting (or could be call "authentic") place.

06: image synth

2012-01-05T23:37:00.000-08:00

Picture 6/1.

Image synth (which is not equal with synthesizing images) could be a radical version of data bending or the color synth: it goes toward the pixel-level, forgotting the content of the whole picture. However image synth use the whole image as only one wave period, which is so fast, and therefore it's impossible to recognize the details - You can just feel the changes in comparison. Therefore i made two different versions - first from the picture above, then from the picture below.


Picture 6/2.

Sonified versions
of the picture 6/1:

of the picture 6/2:

How they work?

Besides that i'm afraid they don't work at all, the working method is fundamental for the works presented in this blog in the future posts. The keyword is the jit.peek~ object in MaxMSP/Jitter, which makes possible to threat an image matrix as a sample. After that i just let the system use the hue and saturation channels of both pictures and make a separated left and right channel, setting subjective values of the scanning rate (which are in the above example the currency rates of hungarian forint and several other devisas) for the playback.

The MaxMSP/Jitter patch used for the examples above.

Picture info

Picture 6/1 presents fresh-baken "cocoa-snails" and taken on 1/4/2012.
Picture 6/2 presents Channel Islands from the beach of Isla Vista, and was taken on 12/25/2011.

05: color synth

2012-01-04T21:51:00.000-08:00

Move the Bee!

Color synth is probably the most frequently used type of sonification-based works. The general concept is to extract color informations from pixels or regions of the image, and then link these data with pitch, timbre, velocity etc. informations. This audio file is a documentation of such a project, where the extracted data simulates a bee, searching for flowers. If the flower is found, then the movement slows down, otherwise it's faster.

Sonified version:

Sounds of Pictures #5 - Move the Bee! by Xrc

How it is working?

The system simulates a bee, who is searching for orange-colored areas. If it's successful, remains there for a while, otherwise move around. Meanwhile the "movement" is linked with sound, which is - like a color synth - defined by the actual color informations.

I used the HSL (hue-saturation-luminance) version of the picture above. It allows the system more freedom to search and follow a color. The MaxMSP patch used can be seen and download below.

Download the MaxMSP/Jitter patch

About the picture
The picture is taken on 11/13/2011 at the Santa Barbara Mission.

Recommended links
Color synth applications:

04: image filter

2012-01-04T15:38:00.000-08:00

Waves and wavefronts are common in sound and nature. In this work i made a very basic spectral version of wave-signs in the sand, rendering lower parts to higher frequencies, scanning bottom-up.

Sonified version:

Sounds of Pictures #4 - Image filter by Xrc

How is it working?
The process is similar to the scanning methods presented before, but in this case the system converts the rows of the image to the spectra of the sound. This is a general technique used in such audio spectra editor applications like Metasynth, Spear, Audiosculpt, etc. In this case i change the filter mask from white noise to recorded audio, (surprisingly) sounds of waves recorded at the mesa of Isla Vista.
The scanning goes back and forth on the picture.

Picture info
The photo is taken 12/25/2011 on the beach of Isla Vista.

Recommended links

9m14s over Vietnam (by Atau Tanaka + Sensorband)
vOICe - navigation system for blind people based on camera input/image filter
PAD28 - other (probably better) solution with oscbank~
Image sonification@25floor.it - research description with examples

03: image as score

2012-01-03T15:50:00.000-08:00

Playing from a score could be regarded as the oldest image sonification method. I use the same strategy by scanning left to right, then jump to the next row, while the heads of the notes are the white dots. In this work i gave the freedom to the notes to be pitched as the dots are big or small, and the tempo of scanning is also determined by the content of the actual "score" values.

Sonified version:

Sounds of Pictures #3 - Image as Score by Xrc

How is it working?
The system scans the image in 7 rows after each other. The rows are splitted to 5 smaller rows, and analyzed separatedly. After thresholding the original picture I use the computer vision library by Jean-Marc Pelletier to extract data of the cells about size and position of the dots. After then i link these data to audio parameters pitch, velocity, and tempo of the playback. The result is a system which consists of small drops, similarly to the image above.

The thresholded version of the image above

The MaxMSP patch used for the work

Picture info

picture taken 10/09/2011 at Pismo Beach, CA

Recommended links

Bruegel variations, project by Sándor Vály

02: Audio QR Code

2012-01-02T20:08:00.000-08:00

Audio QR Code

With the spread of 2 dimensional visual codes like QR code we could imagine the audio version of them. There are some realizations for embedding audio codes in the spectral domain (for ex. this), i just tried a sonification connected to the QR code standard with the goal that make a sonic representation of those images like above, and let them possible to store the same information, which means that is it possible to get the image code back from the sound. As result, the first version can perform this, the second is just on the way to it.

Sonified version 1: impulses

Sounds of Pictures #2a: Audio QR Code - impulse by Xrc

How it works? This is a basic linear scanning sonification, where the black pixels are 1, the whites are 0 in the audio signal - therefore the sound file represents impulses. The playback is relatively slow, because the need of decoding which works even flawless. I attach the MaxMSP patch which can demonstrate it.

Sonified version 2: sine waves

Sounds of Pictures #2a - Audio QR Code - sine by Xrc

How it works? A linear scanning sonification again, but multi-layered: it plays 7 rows at once and represents on 7 different sinewave frequency. It could be developed to fft based approaches, but the keyword which i look'd for is the periodicity: let the user make possible to choose any sound as carrier for the sonification, while the recognizing system watch for the repetitions and extracts the information. It's just a vision for now.

Recommended links:

01: Data bending

2012-01-01T16:52:00.000-08:00

Scanning synthesis with data bending

listen to the sonified version:

description:

Image title: Bank of America building, Los Angeles (12/31/2011)
Sonification method: data bending, vertical scanning

howto:

The audio is made by simply loading the picture file in an audio editor. The picture must be saved as RAW (headerless) format in Photoshop or other image editor; the file must be opened also as RAW (for example Audacity can do such thing), possibly with the similar bit depth that the image has (8). You can play with audio manipulations on image file or vice cersa; and also with multichannel (eg. RGB) pictures to multichannel (3-channel) audio transfer.

recommended links: