← Return

An Experimental Streaming Platform

Live Streaming Platform As An Instrument, Experience And Pastime ·

Archival Links

Code and Media Archive, includes more history focused on the code.

NIME 2020: Live Streaming Platform As An Instrument, Experience And Pastime

NEXT Festival 2020: Live Streaming Platform As An Instrument, Experience And Pastime

Vasulka Kitchen 2021: Taming of the Stream: Performance with(in) protocol

Web Audio Conference 2021: Live Streaming Platform As An Instrument, Experience And Pastime

The History

In my early, naive years of learning software development, I became enthusiastic about how the nodes in a network would function together and produce novel phenomena.1 I had a vision of streams of audio, in place of websites, in some way explorable and interlinked. Just as you might "surf" through pages and their interconnecting links2, you would close you eyes a listen, moving spatially through a soundscape, listening to voices and music like disembodied speakers as you moved past them.

Just like early visions of hypertext, which were hopeful and utopian3, there are unjustified and implausible design issues with this auditory web project. However, instead of spending my time exploring plausibility, I worked towards this vision, building as I learned.

2015

After only a few months working in my first full-time software developer job, I designed and (attempted to) build a collaborative writing tool. Not in the multi-user editor sense. Instead, the idea would be that you would write some text, perhaps a paragraph or two, and connect it to a prior text. Like a labyrinth, endless passages where a story could go. A kind of centralized hypertext centered around prose.4 I retired this project as it felt too under-developed.

2016

I started to shift my focus towards 3D graphics as I was studying it at the time. I now had enough knowledge to explore WebGL and built a prototype of the of this sonic cyberspace. The idea was a kind of brutalist style labyrinth of rooms. Each room containing a stream of spatial audio, eminating from some object.5 I would show experimental musicians and presented it at creative coding meetups, to gauge what kind of interest there was.

2017

My first attempt at building something collaborative involved a small audio player playing a simple 10 second audio loop. Participants would record a second of audio, which would upload to a server and be inserted at random in the loop. I shared it on social media into niche groups of people who enjoyed experimental music.6

2018

I was developing my ability to write software and use it as a form of creativity. As I wanted to spend time abroad, I felt like Berlin would be a good place to potentially explore both skills. I relocated, but shifted my focus to graphics for this period. I briefly explored collaboration tooling further for another creative coding meetup, where I combined a web app for participants, a go server, and a C++ renderer, to demo a collaborative drawing system.

2019

Around the later half of 2019, I explored the use of Icecast and multi-channel audio interfaces. I was certain this would be the best approach to building a working system of the vision. Eager to share my design, I submitted my project as a workshop for NIME, and it was accepted soon after.

I was primarily a web developer, so the software and existing code to learn from was unfamiliar territory, and testing audio was much harder than testing a user interface or some API.7 By the time I had something working, I realised it would only work on a device like a Raspberry Pi - so anything collaborative would depend on every participant owning and operating one. If I was going to demo it, I would need to supply a Raspberry Pi to all the participants.

I turned to using the web platform as an alternative. Icecast did stream to browsers, but getting the audio out seemed to be much more complicated. I learned about how modern streaming techniques don't just send endless streams of data. Instead, they split and encode segments of the audio.8

2020

The main issue I had was how audio should be chunked.9 The problem was compounded by the design of the system. You get random sequences of segments from any particular part of any given audio stream.10 When it plays, the flow from one sequence to another was meant to sound seamless. Not only did a set of sequential segments need to be chunked seamlessly, they also needed to blend smoothly into completely different audio.11

Once I figured out how to make it work and had the system running, COVID-19 had spread, and global lock down had begun.12 I presented my project remotely, which was received warmly, but I didn't quite get to the point of having a group of people all making music simultaneously. A few days later, I was invited to present the same workshop for the online version of the Slovakian "NEXT Festival" later in the year.

I presented the workshop at NEXT, which was fun, but again - not much participation. At the end of the workshop, I was approached by a curator from Vasulka Kitchen. They wanted to work with me to seek an arts grant in collaboration with other artists. I was excited for the opportunity, as I knew of the Vasulkas, and was interested in how the grant process worked.13

2021

I had moved to south-west Germany and it was coming into spring, so I wanted to try using the tool for producing music out in nature. This was probably the most enjoyable period of use of this platform. I would take a few speakers, my phone, laptop, and ride my bicycle into the woods.14

This was to be the most productive year with the platform. I completely rebuilt the platform and gave it a completely different user interface. I managed to do a few jam sessions with various people using it. VK and I got the grant and part of it was a few demos and a collaborative performance. I applied to ZKM but unfortunately was rejected. I had workshop with web audio conference, and wrote a paper for it.

Winding Down

I received a lot of feedback during this time for how I could improve it or different approaches I could take. After all the effort, I felt as though my enjoyment for working on it was beginning to wear. The COVID lock down restrictions were being lifted and I wanted to focus on other things.

I think this project suffered from enthusiam for too many competing visions over what "sonic cyberspace" was meant to be. It converged towards something that was more of a performance tool, which never really was used in a way that fully realised its potential.15 As a kind of auditory journey tool, any kind of addition of a visual would make it irrecoverably a VR virtual world-like system, and any reduction of it to a single musician would make it interactive sound art.

Learning

As a summary, what I was able to learn through the pursuit of this vision was:

  1. It is only with those nodes can you get these phenomena: feeds, chat, data aggregation. But I think it's hard to notice the relationship, the aggregate hides the pieces until they're gone. ↩
  2. Some affordance provided like a hyperlink underline but in haptic or auditory form, allowing you to teleport into a new auditory space. ↩
  3. Not to say Project Xanadu was fated to never be realized, but it is considered vaporware, just like this project. ↩
  4. I had never written at novel length before, and challenged myself to actually try writing long prose. Writing and starting post-grad took my focus away from this project. ↩
  5. I got as far as building a kind of single room version of this, but the complexity of browser performance and learning how to build streams was a new stage that would take much longer. ↩
  6. The result was definitely interesting. It wasn't quite what I had in mind, however, but it simply working was a helpful nudge towards building the more advanced ideas. ↩
  7. It was very easy for it to make very loud and annoying sound when something is slightly wrong. Handling buffers and latency make testing anything in real time very difficult - sometimes you would wait many seconds before sound would become audible, so testing this repeatedly make the feedback loop very tiresome. ↩
  8. The receiver then decodes each segment and appends the audio data to a buffer. By using WASM, I could encode segments of recorded audio and send them to the server, and the same segments would be distributed for playback. ↩
  9. I chose not to read any papers and learn about how all this segmenting concept worked, and didn't use any existing tool to do so, instead I wanted to figure it out myself. ↩
  10. The choice to use randomness rather than just streaming audio as it was recorded, on reflection, really moved the project away from the original vision. I embraced a kind of philosophy of trying to introduce playful spontaneity while preventing any contributor from being able to dominate the combined sound. This would effectively make it no longer "spaces of audio streams" and more like an experimental performance tool. ↩
  11. Cut it wrong you can get weird pulsing sounds or clicks. ↩
  12. Suddenly, my plans to visit the UK and present at NIME were no longer possible, but streaming technology was now very relevant to artists. ↩
  13. At this stage, I felt the user interface and codebase needed to be improved in order to make it more capable as a performance tool. The grant would be an excuse to continue working on developing it. ↩
  14. I had left my job at Super.AI when I moved. This project played a part in getting my next job at Skribble - I was asked to go through my code and explain it to some engineers on the team. ↩
  15. Like most naive, ambitious projects, it was the journey that mattered yada yada. ↩

That's all • Scroll to the top