(Image source: The American Resolution)

A couple weeks ago I was at dinner with some friends. At some point, talking about recent trips, people wanted to see pictures. As iPhones ware passed around (always a little worried that swiping to the next photo might reveal something not intended to be shared), we realized that we could exchange photos with Apple’s new AirDrop feature. While this was pretty cool, the technology wasn’t seamless, and didn’t work for everyone. So the conversation took a detour into settings and configurations screens. The joy of sharing devolved into an IT lesson where some people got confused and frustrated. We eventually put our phones away and switched topics.

The problem

We know we use technology constantly our daily lives, both individually and socially. But the event also illustrated how far we still have to go to invent something that’s both powerful and effortless.

This is especially true with our mobile devices. We rely on them as memory aids and as ways to enhance conversations by introducing new content. Talking about a movie we look up the director to see what else they did; we take a note of a book that a friend recommends; we check the reviews to find a restaurant when we’re in an unfamiliar neighborhood. The devices give us access to virtually infinite content; but as we use them, our attention to our surroundings, and the people we’re with, diminishes. Conversations take a pause. We get distracted.

With human factors and design research we know to design interfaces that are based on user needs and take into account the contexts in which they’re used. For example, a running app has larger buttons so it’s easier to hit when running; and a navigation app reduces information clutter to prevent driver distraction.

But these are instances of interfaces designed for specialized contexts. It’s when we use general-purpose apps in unintended environments that we get into trouble. And it ranges from real danger, such as when using your phone while driving, to minor distraction, such as looking up some information while at dinner.

For the “danger” category, there’s lots of work being done trying to find individual design solutions that are safer. But what about the “distraction” category? Is there a better way to weave general purpose connectivity into our lives? I’m not claiming to have any solutions, but bear with me as I do a bit of a ramble (and link in some previous posts)…

Device solutions

One approach is through augmentation. There’s a lot of talk of how augmented reality can aid manufacturing, and there are plenty of one-off advertising toys. But I can’t think of many examples of general purpose apps that use augmentation. Perhaps Yelp’s Monacle or Nokia’s City Lens? But these examples are still fairly specialized (they help you explore a city) and don’t especially facilitate social interaction.

Google Glass takes augmentation in the opposite direction, skipping social entirely, and focusing on just the wearer. While there are some interesting design opportunities, the technology feels like a backwards step — it’s inward looking, increases isolation, and discourages sharing. Taking this to the extreme, Domestic Robocop (and Augmented City) projects an absurd possibility of this future.

Domestic Robocop

The introduction of the iPad was a interesting advance for social interfaces for it freed us from the desktop. Because it was portable, it could be used almost anywhere. And, because it had a larger display, multiple people could look at it simultaneously.

There have been a couple specialized apps that treat the iPad like a multi-user device. Cotracks, for example, lets multiple people make music together. And Scrabble connects the iPad to each player’s iPhone for multi-user and multi-device play.

Scrabble

But, we’re not yet all carrying around iPads and placing them out in the middle of the table. And while it’s larger and more social than a mobile phone, it doesn’t scale well for more than a couple of people.

Environmental solutions

What if we stepped away from the idea of individual & personal devices? What about larger-scaled technology — stuff that’s built into our environment?

Well, technology that’s built into the environment tends to date quickly and be expensive to update. And there are plenty of challenges to designing for sharable technology which would also know your preferences and link to your networks. But there are some interaction examples that are noteworthy and from which maybe we can learn.

Microsoft’s Surface Table (now called PixelSense) is a touch-sensitive display, usually mounted as a table , around which people can gather, and that recognizes objects placed on it. It’s a nice, and relatively seamless, way for the display to connect to the real world. It aims to support natural user interface (NUI) design principles so that the interface effectively becomes invisible to users.

But the Surface Table never really took off. It was expensive, had too specific a form factor, and there were few applications that really took advantage of the new design principles that the technology offered. Samsung now offers a larger, more flexible and powerful version.

Microsoft Surface Table

Other real-world examples of technology built into environments tends to be in support of fairly constrained use-cases — not the general purpose functionality hinted at with Surface. For example, high-end video conferencing systems such as Cisco, Halo, and Oblong, build technology into meeting spaces. (Media:scape is in somewhat more natural environments.)  They let you bring your content and provide ways to structure and work with it. While they’re built around meeting taxonomies, they might be expanded and generalized for other ways in which we share.

Halo conference room

Media:scape installation

For non-business examples, I think back to more specialized things like the dining tables at Inamo and Barneys. Inamo’s is quite narrow in its use, really just being a way to order food. The Barneys table also gives you food and fashion content, appropriate to when you’re in their store shopping. But both are still solitary interfaces, without means to share with your fellow diners.

Barneys genes@CO-OP table

Future technologies

Looking at future technology, stuff still in the concept and research phases, we see the emergence of some pretty powerful ideas that could support more general-purpose activities.

“Projection-lights,” such as Berg Lamps and LuminAR (among others), look at how we might be able to build projection and gesture recognition into our everyday world. It nice thinking about how we could get away from screens and have information displayed seamlessly anywhere we may need it. Berg’s explorations of how projection exists as a design medium, and the general rules for “smart light”, are especially interesting. This, combined with all of our devices being connected as part of an internet of things, seems full of exciting possibilities.

LuminAR projection bulbs

Science fiction interfaces, as well as vision films, portray future scenarios and how new technologies could be incorporated into our everyday world. They actually are closest to portraying a future where information seamlessly is part of the everyday context. But, as Scott Smith pointed out, these films often “show our lives as simplified and passive, with technologies that magically anticipate our needs.” They show solutions to specific problems — not overall new interface approaches.

Corning’s A Day Made of Glass

Back to where I started

I think my last post, about food and dining, was partly to blame for all this talk. I wanted a restaurant where I could share with my fellow diners, and also get content that would enhance my dining experience. But now I want something more generalized. I want new ways for us to get and share information. Ways that are transparent and seamless. Ways that don’t distract or cause us to loose the human connections we have with others.

This post isn’t about solutions… technology, and our expectations of it, change too fast. Instead it’s about challenges and opportunities. This post also isn’t intended to cover all the research that’s happening in this field — as there’s lots, with a great and deep history. Instead, hopefully it’s also about how the future can be much more than what we currently have. And it’s a reminder that we need to continue designing new interaction paradigms.

Related posts:

1. Are Phones Leading OS Innovation?
2. Star Trek: PADDs
3. Productivity Future Vision

Take a look at Receipt Racer — a microproject by Joshua Noble and undef.

Done in just one day, as part of the CreativeApplications.Net OFFF2011 Workshop, the game is a fascinating mix of interactivity and print. Using a thermal receipt printer, a racing track is printed. The racing car is projected onto the printout with a micro projector, and a game controller lets the player steer.

The creators note that using 50 meters of paper for one race run isn’t very efficient… “Ecologically it’s pretty much a disaster, just like any real race car.”

(Link via Interactive Stuff.)

Related posts:

1. Handheld Projectors
2. LuminAR
3. Life Moves Fast

SixthSense: Newspaper showing live video news

There’s something a little amazing that happens every time you set up a video projector. All the cables get connected, you turn the projector on, and during the process of placing it, the image may show up on an unexpected surface. It’s cool, so you move the projector around to see what it looks like in other places. For example, at home, we often watch movies outside on a screen, but have tried projecting onto the leaves of a tree, or the stairs.

Projection mapping is one application of this sort of fun. But the projector is fixed – so the applications are more about changing how the surface is perceived. Handheld projectors give more control to the user, encouraging interactivity. And as these small pico-projectors get more common, this sort of magic will be just an everyday thing. So it’s interesting to see the research taking place to figure-out applications of the technology.

Motion Beam is a project from CMU and Disney Research that looks at how handheld projectors can be used to interact with, and control, characters. The team has developed a couple interaction principles, and applied them to character and racing games. They also have a cute demo of how a projected character can interact with physical items in the environment — in this example, bouncing when it moves over a trampoline.

SixthSense is a super-cool project done by Pranav Mistry while at the MIT Media Lab’s Fluid Interfaces Group. The project looks at how a gestural user interface, combined with camera and projector, can sense the environment around the user, and project relevant supplemental information. There are definitely aspects of augmented reality here — but projecting the additional information, rather than using a display, gives it a really interesting immediacy.

The Pasta&Vinegar blog recently posted about a report on how people may want to use such technology. They noted some social responses. One wasn’t so surprising — that people were reluctant to project personal information such as txt messages. But the fact that other observers had little interest in reading the projections of others was an interesting contrast. (Coincidentally, Russell Davies recently experimented with something along these lines, as he projected a twitter feed alongside his television at home.)

Motion Beam: Character Game

Motion Beam: Game sequences

Motion Beam: Interaction Principles

SixthSense: Using palm for dialing a phone number

SixthSense: Boarding pass showing the live flight status

Related posts:

1. LuminAR
2. Caprica Paper
3. Dragging Video Objects

Way back in 2006, at the Wired NextFest, Saturn sponsored a “green” exhibit, highlighting their hybrid technology. It was a 4,000 square-foot installation featuring a life-sized, interactive holographic people, CAD projections onto vehicles, and a interactive/reactive wall with user-generated content.

I first learned of the project when I was judging the Communication Arts 2007 Interactive Annual and said it was beautiful and poetic. It may not have had deep interaction, but that was ok for an exhibition space as crowded as this — as the audience has a limited attention span.

I particularly liked the way information was projection mapped onto the vehicles, giving them a kind-of transparency, or augmented reality quality. A very engaging way to communicate under-the-hood concepts.

The reactive grass wall was big, green, and lushly fluid. It responded to people’s movements and displayed content from a connected kiosk. The wall was developed by the Barbarian Group, and their in-depth write-up of the project has a surprisingly detailed discussion of the Processing code that was created for it.

Goodby, Silverstein and Partners lead the project. Obscura Digital did the cool displays.

In-development view of programmed grass.

Related posts:

1. Lovedisk
2. Talk to Me @ MoMA
3. Cooking Dinner and Project Oasis

This is interesting… Intel Labs’ Oasis Project uses your kitchen countertop to recognize the foods that you place on it, and projects a variety of interactive content.

It’s still early research, and so the sorts of interactions possible, as well as the content that’s projected, are pretty basic. But the video suggests a variety of ways people could be assisted in a kitchen of the future.

Actually, when I first watched the demo, the first thing that popped into my mind was David Small’s Food for Thought — where he used a laser sign cutter to write messages onto food. It’s not directly related to Oasis, but how we associate food with technology, as a cultural and experiential issue, is the connecting theme.

In thinking more about where Oasis-like kitchen-based R&D could go, a great inspiration should be this beautiful video, Cooking Dinner Vol. I by William Hereford. It shows a notably hands-on, refreshingly tactile, approach to food and cooking. It’s so different from the antiseptic way, almost germ-phobic and lawsuit-eager, that food is usually treated. I’d love to see how a system like Oasis could be developed to encourage such holistic changes to how we think about, and engage with, food.

[Project Oasis link via Interactive Multimedia Technology; Cooking Dinner link via SwipeLife and KN.]

Food for Thought, David Small (1999)

Intel, Project Oasis, interface detail

Cooking Dinner, William Hereford

Related posts:

1. Dreamy Visions
2. Interative Gastronomy
3. Oblong and Before

Palm: Interacting with the projection

Take a look at this series of commercials from Palm, part of their “Life moves fast. Don’t miss a thing” campaign. They show a kind-of augmented reality, holographic-style mobile UI.

Unlike the Chase Blueprint ads, which showed some pretty innovative ideas for holographic interfaces, these essentially project what’s happening on the screen. The point isn’t to present a new interface — instead it’s about communicating how you can use a mobile interface while you’re on-the-go. That, and not feel like you’re staring at a screen, isolated from the world around you.

In reality such an interface might be a bit intense as it followed your vision as you turned your head. But it could be interesting to think about how the interface might change when you stopped moving and wanted to focus on a detailed task. In that case the information density of the display could transform to be more like an iPad.

Microsoft has a similar ad for their Windows Phone 7 series — although their users are staring at the phone, not at the projection. Perhaps more honest, but not as engaging.

Windows: Interacting with the screen

Related posts:

1. Windows Phone 7 Series
2. Manual Deskterity
3. Handheld Projectors

It looks like we’re one step closer to the Luminous Room. Take a look at this demo of LiminAR. It’s a project by Natan Linder, a student of Pattie Maes in her Fluid Interfaces Group at the MIT Media Lab.

LiminAR is made up of two components, the Bulb and the Lamp:

The LuminAR Bulb combines a Pico-projector, camera, and wireless computer in a compact form factor. This self-contained system enables users with just-in-time projected information and a gestural user interface, and it can be screwed into standard light fixtures everywhere. The LuminAR Lamp is an articulated robotic arm, designed to interface with the LuminAR Bulb. Both LuminAR form factors dynamically augment their environments with media and information, while seamlessly connecting with laptops, mobile phones, and other electronic devices. LuminAR transforms surfaces and objects into interactive spaces that blend digital media and information with the physical space.

The demo is a little rough (it could use a bit of design polish) but it’s a great idea. The combination of smart positioning, along with recognizing gestures to reposition the display, is really cool.

(Link via Engadget.)

Related posts:

1. Oblong and Before
2. Handheld Projectors
3. Crazy UIs

I just discovered Glowing Pathfinder Bugs and think it’s super-charming. Created by Squidsoup, and originally shown in 2008, the piece was commissioned by Folly Gallery for Portable Pixel Playground.

Glowing Pathfinder Bugs, an interactive art project primarily aimed at children, uses projection to visualize virtual bugs on a real sandpit. The bugs are aware of their surroundings and respond to its form in their vicinity. By altering the topography of the sand, participants affect the bugs’ environment in real time, facilitating direct communication between them and computer-generated creatures.

The projected digital bugs, combined with the sand pit, gives users a really tactile, and hands-on, experience. I love how, in the video, one kid yells “kill it!” at one of the bugs — but most everyone else seems absorbed and enchanted in the experience.

A couple technical details on the project are available here.

Update: Also check out Mud Tub, by Tom Gerhardt. It’s cool, and similar — if slightly messier!

Related posts:

1. Digital Natives
2. Handheld Projectors
3. Cooking Dinner and Project Oasis