I am currently doing a group project for the Human Computer Interaction course at the Media Technology MSc Programme, where we’re exploring the possibilities of interaction with paper and computers. The key part of the project is to figure out a way to get non-technical users to develop their own (game) controllers that can be build solely using paper. The controller has to control a game or application on the computer.
The project is still in a very early and experimental/explorational phase, but here is a demonstration video of one of the prototypes. In this prototype, it was possible to use AR markers to build a virtual dashboard where the users could create and control certain input devices (knobs, sliders, switches) by moving AR markers around.
My starting point would be the same: Have a camera monitor the phone’s screen, and a computer to control the phone’s keypad.
The difference I wanted to make was sending the key-presses using an RS232 or Bluetooth serial connection. This would eliminate the need for external hardware pressing the buttons, or the need for altering the cellphone by hot-wiring the buttons. This was a very simple task, all I needed was the AT Commands Online Reference. The phone I used (Sony Ericsson T610) supports the AT+CKPD command (Control Keypad), thus giving me the power to emulate the entire keypad over a serial connection.
The second difference would be on the graphical user interface. At first I thought it would be best to control the phone using the PC’s keyboard, but whilst developing I found out it would be better to be able to click the phone’s actual buttons using your mouse. I rewrote the key-press part, so when a mouse-click on the video-stream is registered, it translates the mouseX and mouseY to a certain button on the phone, and sends the corresponding key-press using the Bluetooth connection.
The end result: A live video-stream of the phone on your computer and being able to use your mouse to click the phone’s buttons within this stream. A demonstration:
Note: The original cellphone button pusher also is an amazing piece of art. I get why the over-enginereerd method is chosen, and it is quite astonishing how it all worked out. This article is purely to demonstrate how it can done in a simpler way.