## Nostalgia

This project really brought me back to my University years, studying Industrial
Design. The downside to a lot of commercial work is you start with building a
brief, and if there's no sense that you'll make money (typically millions of
dollars), it gets nixed before anything tangible is produced.[^1]

## A System of Experiments

There were a lot of little things I wanted to explore and learn over the prior
months, and it culminated in the design of system to really demonstrate how each
part could work together as a whole. The center of my focus was around what I
could do with my Android phone[^2], and whether it was plausible to do these
things in Rust.[^3]

What I learned and intended to combine was:

- mDNS-SD broadcast and discovery
- OpenCV.js[^5]
- WebSocket server/client architecture
- Rust via NDK
- Oboe[^4] and DSP
- OpenGL ES 3.0
- Paper-based TUI
- Dev tooling / automation[^6]

Back in Germany, I worked on a physical computing side-project, exploring the
Web Serial API.[^7] Although working with electronics is fun, it's a discipline
in itself and post-COVID supply chain issues, the general difficulty of the
hobby, and relative danger of electricity made me contemplate better approaches
to the rapid development of tangible user interfaces for software. I had
exposure to OpenCV and AR before, especially at University, and had an idea that
paper could be a potential medium.[^8]

## The Mechanism

By turning a knob, this would turn some paper fudicial markers. This would then
be reflected into a mirror, which the phone camera would pick up. The phone
could adjust some internal state, which would be shared with another phone. The
other phone would be in use as a DSP system[^9]. The two phones would not have
manual pairing - the mDNS system allows them to automatically find and connect
over a shared network.

Honestly,
[the result was pretty janky](https://codeberg.org/bytetrie/android-multi-device-demo)
and I've focused elsewhere since working on this project. Ultimately, it was a
messy-but-complete way to bring together the separate projects I had explored
over the Winter.

[^1]: The fun with stuff at University or hobby projects is that this isn't
    really a problem, but you bare the cost and time you invest.

[^2]: Specifically, the power of the processor: Snapdragon 8 Gen 1.

[^3]: I had experience with using the NDK before and built a few Android apps,
    but for whatever reason, it always felt way harder and less satisfying than
    I hoped.

[^4]: The low-latency Android Audio SDK.

[^5]: Mostly the use of ArUco markers and dot detection, which provided the most
    consistent detection.

[^6]: A mistake I've made in the past was not prioritising this for dealing with
    reptitive but complex steps such as orchestration.

[^7]: I hadn't grasped how a peripheral could communicate with a computer
    before, and found it really fascinating the way it all works. To this day I
    run the
    [Awesome
    list for Web Serial API projects](https://github.com/bytetrie/awesome-web-serial).

[^8]: Paper is often used for prototyping all kinds of design projects, and it's
    already well explored as part of
    [TUI systems in the Dynamicland project](https://dynamicland.org/).

[^9]: This separates the compute and location for vision detection and DSP, both
    quite heavy processing tasks. The DSP was relatively heavy as the plan was
    to use it with a variety of effects for my bass guitar.