A Sound In Image Experiment

There are some usefull apis since Flash Player 10, I have never been playing with. Yesterday, I realized one experiment with Sound + SampleDataEvent and created quick demo for converting music into a bitmap and vice versa.

There is a Sound.extract() method in ActionScript allowing you to grab raw sound data from a sound object. It provides you ByteArray of 32-bit floating-point values, which can be converted to a Number using ByteArray.readFloat(). The resulting values are -1 < value < 1...

I tried to add and multiply the Number to get unsigned integer and it appeared that only first 16 bytes are changing while the rest 16 remains unused (0xffff). I did not get any further to investigate if it is related to the type conversion or its a limitation of Sound api. While for BitmapData you provide 32 values (unsigned integers) and the converted sound value use only 16, next 16 bites can store additional sound value. That perfectly fits for left + right channel. BitmapData handles 32-bit ARGB values. In my case, left channel will occupy first AR, right channel GB values, so please do not expect the result to be Mona Lisa portrait, but rather it is a chaotic (noise like) color and alpha result. Following image contains 5 seconds of Vivaldi:

If you are able match image width with music rythm, some more interesting result may appear. And here is the demo player:

Sound In Image – wonderfl build flash online

There is a lot of commented code in the demo, feel free to uncomment for generating images from your mp3s.

Some quick math about image size vs. music length:

  • in sound, the audio data is always exposed as 44100 Hz stereo
  • so there is 88200 32-bit floating-point values per second (left + right channel)
  • two values can be stored in a pixel, that results in 44100 pixels per second
  • that means you can store almost 18 seconds in 1024×768 ARGB image

It would be nice to see how can image compression, watermark etc. affect the resulting sound/size. Or how would some image filters affect the resulting sound… A lot of experiments ahead :-)

1 comment so far

  1. Flo - derhess.de November 6, 2011 15:34

    Nice experiment! A student of our university create a sound / music peace out of skyline picture from New York. He tranfsormed the bitmap data via Fast Fourier Transformation to a spectrogram. This data produced the sound and he created the pictures backwards with the spectogram data. So it is almost your way vice versa. Some information about his performance: http://www.jyd.at/skylines/

Leave a comment

Please be polite and on topic. Your e-mail will never be published.