Projection wigdets for instrumented environments

Within traditional Desktop GUIs widgets represent interactive components, that can be used to navigate through information structures or to manipulate information. A widget is a function object and thus modularly reusable. It also possesses a recognizing effect. If a user has handled a widget before and learned its function mode, they can use it again. Augmented Reality widgets, based on conventional 2D widgets, were sketched and integrated into the instrumented environment by means of light projection, in order to enable complex interactions.

Visual design and function

click to enlarge

The user should be able to recognize the projected widgets and to operate them by hand. When designing the visual appearance, one has to take the properties of light projection into account. It is not possible to darken a surface in the same way that one can lighten a surface only by sheding light onto it. The seamless visual integration into the environment is an important aspect too. The figure on the left illustrates the design of a checkbox, one in a standard black-to-white contrast and one inverted for projection. 14 shades of grey are used in the visual design including black and white. This is a benifit regarding people with color asthenopia and colors are reserved for encoding additional information, e.g. highlighting an important action.

click to enlargeIf 2D Buttons are enabled and ready to use, they usualy appear three-dimensional upraised. When the user activates a button, it appears to fall inward and thus imitates the physical behaviour of real-world buttons. This gives the user an intuitive feedback, in response to their action. A virtual light source situated to the top left of the widgets creates a shadow and thus achieves the visual effect of pushing. The standard background color of 2D GUIs is a medium shade of grey to enable objects to be darker or brighter in contrast. The button is similar to a cave, because a change of the relative position of the light source is unlikely and would be inconsistent with other widgets.

click to enlargeclick to enlargeThis visual phenomenon is less effective regarding light projection, since light can only be added and thus virtual objects can only appear to be brighter than the surrounding surface. The design considers this fact, as only the middle part of the button falls inward when beeing activated. The remaining edge strengthens the desired visual effect of pushing and enables a seamless integration in the surrounding area. If the button is not enabled and is currently not intented for use, it appears darker than enabled widgets.

The functional behaviour of the AR widgets corresponds to their 2D pendants. The developed widgets are as follows:


The core of the hardware implementation is the steerable Projector-Camera-Unit Fluid Beam. The digital camera delivers high resolution images for calibrating the Regions of interest (ROI), i.e. the sensitive areas of the widgets. A continous video stream is analyzed and in the case of a significant change within an ROI, the corresponding widget is notified. The widget adjusts its state and notifies all registered event handlers. The widgets are placed within a container, similar to Java Swing widgets within a JFrame.

Demo application

click to enlargeThe demo application AR Jukebox is controlled by the developed widgets. The user activates the application by touching the powers witch (1). They are then able to open the Combobox to view the list with the items (2) and to search within this list (3). After selecting a song, the user can start the playback by touching the "Play" button (4). The button changes its function to "Stop". Using the slider, the user can adjust the volume (5). While the song is playing, the user selects another song. By touching the "Stop" button, the user interrupts the playback and starts the new song by touching the button again (6). When finished the user may switch the application into stand-by mode by touching the power switch (7). A running playback is interrupted and the widgets are disabled. At any time the application can be enabled again (8).