On the 25th of May, I gave a lightning talk about lo-fi AR at EnthustiastiCon, a conference that celebrates programming. To use the organizer’s words, the conference is about ‘the strange, the wonderful, and the clever solutions to unusual problems’. This blog post aims to sum up the talk for those who missed it. Additionally, I’ll add some links to resources I found useful when starting with lo-fi tech.
To make sure everyone in the audience understands the tech behind AR, I started my talk with short definitions that I then challenged.
Augmented Reality, often abbreviated as AR, alters a user’s perception of reality. Compared to AR, Virtual Reality replaces one’s perception with a digital environment. Current understandings and examples of both forms are mostly visual.
On the left, you can see a typical headset called HTC Vive. Its functionality relies on able-bodied human, binocular vision. Inside the headset, one finds two lenses with which a human user looks at two images that show the same content seen from slightly different angles. The images are displayed on an internal screen. The human brain turns these 2D images into a 3D world. The headset on the right works similarly. The difference is that it is way cheaper. Instead of a built-in screen, one puts a smartphone into it which then serves as a screen. That comes with certain limitations but also possibilities!
What happens, however, if binocular human vision isn’t the limit for our tech? Different animals perceive their surroundings in different ways. Other mammals like cats and rabbits, but also most birds and fish, have partly binocular and partly monocular vision. Some birds can additionally see UV light. Insects have compound eyes, and spiders have up to eight eyes. Bats don’t rely much on vision and use echolocation to detect prey. Sharks have electrical perception.
With my collaborator Jasper Meiners I have developed experiments to open augmented reality up to multi-species perceptions. We started our AR experiments with research about animal perception. In the experiment that I presented at EnthusiastiCon, we have asked ourselves what happens when eyes become detached from the head and how human testers react to multi-eyed vision.
Jumping spiders of the family Phidippus Regius can see sharp in the range of 30 cm. In the picture, you can see my jumping spider Cyan. With her eight eyes she is an expert when it comes to multi-eye perception.
To simulate a multi-eyed vision, my first idea was to use multiple webcams and the game engine Unity. At that point, Jasper and I worked with the headset of the HTC Vive which I showed you in the beginning. The problems began when we tried to handle the input of multiple cameras via one USB card on a VR PC running Windows. When it finally kind of worked by using one USB port per camera, we attached the cameras to my wrists and knees and arranged the video streams in my field of vision. This was exciting, but the cables of the headset limited my movements and I couldn’t freely explore my new set of detached eyes.
We wondered how to read the video streams from the cameras without being tied to a heavy computer. Our idea was to use one of the lightest computers we can afford, the Raspberry Pi. Since people build home surveillance systems with Raspberry Pis, it is relatively easy to find examples for fetching camera output. When you use a Raspberry Pi camera board module, there’s great documentation online on how to stream, modify and analyze videos with Python and OpenCV. For my purposes of just streaming videos, I chose a tool called Motion. Motion can stream videos and also trigger scripts depending on changes in the video footage.
I used a router to locally stream the videos to a website. With only HTML and CSS, Jasper arranged the videos and used blend mode for overlapping effects. On the slide you can see both how Motion looks on a Raspberry Pi and how the video streams are arranged via HTML.
To attach Raspberry Pis and webcams to the human body, we built a lo-fi AR suit. The Raspberry Pis and power banks are sewn into the back of the suit. Two webcams can be used on the upper half of the tester’s body and the other two on the lower half.
This is a recording of a performance we did with the lo-fi AR suit at a media arts festival in Kassel. I am performing with the suit. People in the exhibition see the same imagery on a monitor that I am seeing in my headset. They liked watching themselves being seen by my webcams. I can only use the suit for half an hour and afterwards feel like I swam for two hours. It’s apparently hard work for the brain to process four videos that are connected to one’s body.
Coming back to the initial research questions, this experiment indicates that a different configuration of visual inputs does translate into a different sense of space and leads to adapted movements. If you want to actively use your knees as visual inputs, moving the body to the ground feels quite reasonable.
So, why would you get started with lo-fi AR? AR isn’t finally defined as a specific technology with specific tools yet. There’s lots of room for experimentation. Lo-fi means not only to figure things out with inexpensive technology but also to see the value in imperfections, glitches and the experimental. Another reason to get started is that developing lo-fi AR applications is a lot of fun! Or because you like pixelated flowers!
Throughout my work with AR, I have used a range of approaches. If you want to get started, here’s what you can do:
I hope you feel encouraged to start your own AR experiments. Why, for example, not build lo-fi AR that only relies on touch? Thanks again to EnthusiastiCon for the opportunity to present on lo-fi AR. I enjoyed the conference very much and can recommend y’all to go there next year if you can.