Breaking Development: Beyond Mobile

by October 22, 2013

In his presentation at Breaking Development in Nashville TN, Josh Clark looked toward the future of devices and interaction design. Here's my notes from his talk on Beyond Mobile: Where No Geek Has Gone Before.

  • Mobile devices are today's most personal computer. They give us superpowers through the sensors within them: GPS, camera, microphone, gyroscope, etc.
  • People assume mobile is the companion experience but it is actually the primary experience. Mobile can do more than traditional PCs.
  • When we think about sensors, we think about understanding our environment. But we should be focused on interpreting our environment or even changing it.
  • Games and toys are often more forward-looking because they are free to play.
  • Skinvaders is a game that overlays controls on top of people's faces using the front-facing sensors.
  • Ikea allows you to scan pages of their catalog and position furniture in your house to see what it might look like in your apps.
  • Layar was an augmented reality browser that pivoted into using print layouts as input. Just point your camera at a magazine and get digital information about it.
  • Minimize input to maximize output. Make software design as smart as possible to minimize work on the part of users.
  • Table Drum uses the microphone on a mobile device to turn sounds in the real world into sounds in the app.
  • Don't force people to interact with machines. Design for sensors instead of the screen.
  • Sometimes the best touch interface is no touch at all. Too much time on screens can be dangerous, we're distracted.
  • Speech and 3D gestures are input types that are coming of age just like touch did a few years ago.
  • All the ways we communicate with each other as humans are starting to be recognized by machines. What do we do with those inputs in an interface.
  • HP's Envy has a touch screen, leap motion controller, keyboard, and trackpad. All these inputs can be used at any time in the applications.
  • Anything that is trying to replace the mouse as a pointer is going to loose. Instead think about how controllers like Leap Motion can use three dimensions.
  • The Leap motion controller can take input from physical objects and turn them into digital interfaces.

Designing for Sensors

  • Services like Square put custom sensors in places like registers and smartphones to collect payment information.
  • As we add sensors everywhere, we can collect information from anything. The cost of internet connections and sensors is dropping dramatically.
  • Sensors turn anything into a controller.
  • We've been spending the last few years making the digital world more physical. Today, more of the physical world is becoming digital. There's an interesting nexus here.
  • People who are really serious about software should make their own hardware -Alan Kay. More companies are becoming these kinds of organizations.
  • How do we make peace with all our devices when everything is connected?
  • Remote control interfaces allow smart devices (like our phones) to control dumber devices like our TVs. The phone acts as a controller.
  • Sensors, mirroring, and remote controls are our current ecosystem of devices. Migrating interfaces are next.
  • Migrating interfaces allow us to move between devices and continue our tasks seamlessly. Example: take a call on your phone using car audio, then easily switch to phone handset.
  • Displays can connect to devices that carry operating systems and apps for you.
  • We can design the spaces between our devices. There's magic there.
  • Gesture and speech create magic together. The issue is imagination not technical capabilities.
  • Sifteo cubes are aware of each other and can transfer information and actions between each other.
  • Passive interfaces just do their work without needing any interaction from us.
  • We tend to think of technology as always getting smarter, bigger, more powerful. But we'll also have a lot of dumb devices around us.
  • How do you design interfaces to reach all these devices? You need an API -that's your application. Its creative control in an uncertain world.
  • Sensors, mirroring, remote control, passive and migrating interfaces create a cloud of social devices that are there when we need them.
  • We need to think beyond single interfaces. They're just temporary containers for our services. Presentations deprecate.


  • Push the use of sensors to minimize input and maximize output.
  • Think social: how can we move things between the devices in our lives?
  • Build an ecosystem for your content.
  • We're all cloud developers now. Our content and behaviors need to converge in the cloud.
  • Your application is the API. Think beyond a single presentation layer.
  • New input methods are coming fast and furious, embrace them.
  • The future is already here. We've got decades of material to work with right now.