← Return

Android Multi Device Demo

A project as summary of learning for Android ·

Nostalgia

This project really brought me back to my University years, studying Industrial Design. The downside to a lot of commercial work is you start with building a brief, and if there's no sense that you'll make money (typically millions of dollars), it gets nixed before anything tangible is produced.1

A System of Experiments

There were a lot of little things I wanted to explore and learn over the prior months, and it culminated in the design of system to really demonstrate how each part could work together as a whole. The center of my focus was around what I could do with my Android phone2, and whether it was plausible to do these things in Rust.3

What I learned and intended to combine was:

Back in Germany, I worked on a physical computing side-project, exploring the Web Serial API.7 Although working with electronics is fun, it's a discipline in itself and post-COVID supply chain issues, the general difficulty of the hobby, and relative danger of electricity made me contemplate better approaches to the rapid development of tangible user interfaces for software. I had exposure to OpenCV and AR before, especially at University, and had an idea that paper could be a potential medium.8

The Mechanism

By turning a knob, this would turn some paper fudicial markers. This would then be reflected into a mirror, which the phone camera would pick up. The phone could adjust some internal state, which would be shared with another phone. The other phone would be in use as a DSP system9. The two phones would not have manual pairing - the mDNS system allows them to automatically find and connect over a shared network.

Honestly, the result was pretty janky and I've focused elsewhere since working on this project. Ultimately, it was a messy-but-complete way to bring together the separate projects I had explored over the Winter.

  1. The fun with stuff at University or hobby projects is that this isn't really a problem, but you bare the cost and time you invest. ↩
  2. Specifically, the power of the processor: Snapdragon 8 Gen 1. ↩
  3. I had experience with using the NDK before and built a few Android apps, but for whatever reason, it always felt way harder and less satisfying than I hoped. ↩
  4. Mostly the use of ArUco markers and dot detection, which provided the most consistent detection. ↩
  5. The low-latency Android Audio SDK. ↩
  6. A mistake I've made in the past was not prioritising this for dealing with reptitive but complex steps such as orchestration. ↩
  7. I hadn't grasped how a peripheral could communicate with a computer before, and found it really fascinating the way it all works. To this day I run the Awesome list for Web Serial API projects. ↩
  8. Paper is often used for prototyping all kinds of design projects, and it's already well explored as part of TUI systems in the Dynamicland project. ↩
  9. This separates the compute and location for vision detection and DSP, both quite heavy processing tasks. The DSP was relatively heavy as the plan was to use it with a variety of effects for my bass guitar. ↩