With the Apple iPad launched and scores of other tablets and e-readers hitting the market, I think it’s important to step back and look at the larger trends. We’re in the middle of a major shift towards ubiquitous computing, cloud based personal storage, and tangible interaction. It’s a shift away from the generic computation typified by the “personal computer,” which never really achieved the individuality or specificity implied by the term “personal.” In short, we’re experiencing the emergence of The New Ecology of Things, where a network of heterogeneous, smart objects and spaces create opportunities for a more personal and meaningful landscape. This is what I’d like to explore:

• Where we’ve been and how the personal computer has made us soulless
• Where we’re about to be #1 with the emergence of digital slabs
• Where we’re about to be #2 with a new form of design that’s a hybrid of software and product
• Where we may be going and the future of the designer in an era of bespoke objects

The Past – The Personal Computer Has Made Us Soulless

There are many signs that all is not well with our day-to-day work life. John Hockenberry’s review of Michael Wolf’s The Transparent City contemplates the crushing homogeneity and conformity of modern work revealed through Wolf’s photographs of life seen through Chicago’s skyscrapers. Describing one photo, Hockenberry sees “12 random floors of eggshell white, computer screens on brown desks, and wall-hung bookshelves.” Sound familiar?

The article goes on to discuss how the environment for “knowledge work” is unlike factories where the space is specifically suited to the activity of making things. The knowledge working context has devolved to the point where “offices have become stacks of boxes for people who get paid to think out of them.” But I believe this is not only a problem of architecture and environmental design. Our daily activity has been squeezed into the narrow channel of interaction with the personal computer and its attendant posture, furniture, and detachment from the needs of the person. The digital tools we use have played a large role in creating this disembodied, deadening uniformity.

Similarly, Matthew B. Crawford has been driven out of the office and into his motorcycle repair shop as described in “Shop Class as Soulcraft: An Inquiry Into the Value of Work.” In the book, Crawford discusses how knowledge work has become vague and disconnected from the concrete, meaningful outcomes of manual labor. But again, perhaps it’s not only the type of work, but the manner in which the work is accomplished. The disconnection from the physical isn’t limited to paper-pushing knowledge workers, but architects, graphic designers, recording engineers and others whom we think of as having “satisfying” jobs where things are made. As creative workers, we’ve seen our day-to-day work compressed from a productive, bodily engaged studio environment down to the almost motionless “mouse-crouch” that plugs us into the virtual. Seduced by the power of the personal computer, we have morphed from active, engaged, social, interactive people to sedentary, soulless slugs perched in front of our glowing screens.

The personal computer has created a homogenous, static and context free environment for work and play that removes activity from the meaningful and productive character of acting and thinking in the embodied, physical environment. This needs to change.

The Emergent – Slabs: A Step Towards Re-Engagement

The iPhone, Android platform, Monome (its clones), Siftables, and now the iPad represent a new form of computing device that I call the slab. Slabs are hand-held, generic platforms with a range of sizes and capabilities (e.g, touch screen, GPS, accelerometer, gyro, WiFi, speaker, mic, etc.) that, in effect, turn into something new with each different application they run.

Slabs are different from personal computers. First, because they have a smaller, simpler form factor and a direct, touch based interaction. Second, because slabs do one thing at a time, in that the device effectively becomes the app once it’s launched, and the separation between software, hardware and interaction dissolves. When you switch apps on a slab, you get a whole new device that engages you as a unified, tangible object. This works because the app is the device (and all the discussions about slabs “multitasking” have new meaning in this context). Third, slabs will eventually be cheap enough that one can use multiple devices at a time, at a location and context appropriate to the physical space and task at hand.

Finally we can begin to disconnect from soulless trap of the (im)personal computer. By using slabs, we customize and re-engage with our environment and feel the consequences of our activity. Instead of being tied to a “workstation,” we can move around the work/play space, utilizing slabs and their spatial relations to one another. We use specialized tools and work practices as slabs morph to the needs of each activity (rather than the other way around). We touch things again. Instead of immersing ourselves in the virtual, we re-engage with people and things in the world. We work more easily with others, sharing and collaborating in physical space, where “here, look” and “take this and work on it” are literal statements that once again become the norm. We customize our tools and ultimately I hope, make or acquire our own specialized tools (see the bespoke objects discussion below).

In addition to their embodied and embedded character, slabs excel at leveraging the affordances of computation and networks. With configuration and data always backed up in the cloud, slabs can be easily reincarnated if scrambled, broken, or lost – and with this, interaction, meaning, and ideas become more important than the object. The cloud also enables different slabs to work across space and time while still being in the here and now. Our digital work is no longer tied to a single workstation, but can manifest in different forms, on different devices, with activity-specific functionality.

In short, there’s the potential to get the best of both worlds – the material and social character of the physical, along with the flexibility, power, and ubiquity of the computational. We gain a heterogeneous collection of devices, specifically suited to the activities at hand. For example, a graphic designer might have a large, work table slab for standing and working on layouts. Or a lap sized slab to sit with and edit an image in a concentrated mood. A narrow slab might sit on a table to keep track of a to-do list. And of course, the designer would use a few 8 1/2″ x 11″ slabs at a client meeting to pass around the table for discussion.

The Emergent – Sofducts: A Challenge for Designers

For the designer of these systems, the slab presents an interesting set of challenges. In particular, there’s a hybrid character to apps. Apps are software, yet as described above, the app becomes something more like a physical product once launched on a slab. This merging of app + slab leads to something I call the software/product or sofduct.

On one hand, the sofduct is distributed like software with almost no cost of goods. On the other hand, the user perceives the sofduct like a product because the interaction feels similar to that of  a manufactured object. You hold it, press buttons, shake it, etc. For example, GPS navigation systems used to be sold as a traditional product, in a box, physically shipped, with a warranty card and customer service phone number. Now, the sofduct version gives you the exact same functionality but is downloaded and runs on a slab as a piece of software. To the user, the end result looks and feels just like the traditional physical product. The sofduct is very disruptive in this way.

For one, whole business models are being destroyed by the sofduct. You can now buy the MotionX-GPS Drive app for $2.99, and get turn-by-turn navigation for $2.99 a month or $25/year (not to mention Google’s free turn-by-turn GPS on Android). In some cases, in-app purchasing of add-ons and features creates a modular “product” model, where the sofduct is actually a range of product possibilities that can be selected and customized by the user. Can traditional GPS units and other physical products survive this kind of competition?

But I’m especially interested in how sofducts disrupt the role of designers. Unlike software for a computer, a sofduct has to meet the expectations for a traditional product. The high-finish aesthetics, ergonomics, and conceptual integrity of physical product design will be assumed by users. Likewise, simplicity and clarity of interaction are critical. The perception of “product-ness” will also influence user expectations for reliability and customer service – we want products to simply work. Because of this, visual and interaction screen designers need to adopt the above considerations and aesthetics of product designers as they develop sofducts. Or better yet, collaborate with a product designer as a member of the design team.

On the other side, product designers entering the sofduct realm need to understand the traditions and expectations for software. Users want constant and rapid, usually free upgrades. Product design tries to get it perfect before launch, since there’s no turning back after you send the device to manufacturing. But with a sofduct, it may be better to put out a really good, but simpler version on the market quickly, and use a software model for product planning where upgrades are rolled out on a strategic schedule. Plus, customization and the integration of media are different from the fixed character of physical products, requiring a deep understanding of interaction, typography, and visual design that requires the experience of software and screen designers.

Sofducts are are new category for design, merging the focus, situated character, and physicality of an object with the malleability, customization, and media richness of software. This requires an integration of disciplines, including software development practices with product design, screen design with haptics, interactive/interaction design with materials sensibility, media production with physical interactions. Further, new business and design opportunities emerge, and require a complete rethinking of design and implementation for this new category.

The Future – Bespoke Objects

As slabs and sofducts create an emerging design landscape today, designers need to prepare for further disruptions and repositioning of their skills. Soon, trends in hardware and software will open up the possibility for low-cost, custom-built systems for individuals and specific applications. In the same way that one can have a bespoke suit tailored to a perfect fit and style, it may soon become possible to have a bespoke object with the hardware, software and design features tailored to the perfect fit and style for you and your intended use.

By this, I don’t mean the custom manufacturing typified by NIKEiD and others in recent years (though that will likely happen as well). What I do mean something literally like the local tailor, working out of a shop around the corner. The production of bespoke objects on the local level is becoming possible because of rapid advances in desktop 3D printing, system-on-a-board components, open-source software and hardware, and the DIY culture growing around these trends.

With cheap, off-the-shelf computational components and the ability to print 3D parts, the digital tailor will soon be able to hang their sign out and make individual or short-run custom objects full of ubicomp goodness. People will want these because a generic, mass-produced slab won’t always be suited to their particular circumstance or activity. Moreover, having a custom designed ensemble of complementary, networked objects, specifically crafted to your way of working will be the hallmark of the enthusiast and expert alike. We’ll want to assemble our own unique ecologies of things, from tiny watch sized objects, through tablets, to big activated interactive walls.

Assuming the bespoke object becomes a reality, what does this mean for the designer and design firms? Will it put designers out of business? I think that in the same way sofducts are disrupting design practices and business models, bespoke objects will create major disruptions for designers. If even a portion of product design and manufacturing moves to a decentralized, local model, many individual designers and design companies will have to adjust.

I see a few possibilities. First, those digital tailors are designers. Or at least the most successful ones will be. Just because some of the parts, software and 3D models will be off-the-shelf, bespoke objects will also have custom aspects and are systems that must be integrated for a specific person or task. That’s the job of a designer. Would it be so bad if designers are small business owners around the corner, selling locally in person and internationally online? Second, the off-the-shelf interactions, interfaces, systems-on-a-board, 3D models, etc. – i.e. the ecosystem around the bespoke object – all need design, and I’d expect a market to develop for small and large organizations to design and produce the necessary (virtual and physical) components that enable the digital tailor to operate.

The New Ecology of Things

Nearly 20 years ago, Mark Weiser published his seminal paper on ubiquitous computing in Scientific American, “The Computer for the 21st Century” (scan, reprint in PDF – you really should read it!). This remarkably prescient work predicts much of what’s becoming a reality today (and perhaps Apple’s iPad name is a tip-of-the-hat to Weiser’s taxonomy of tabs, pads, and boards). More recently, Microsoft showed their vision of 2019 video. It’s a great visualization of the potential fluidity of interaction, but the homogeneity and sense of virtuality (a window into something) does not capture the more tangible, gritty, idiosyncratic, embodied, embedded character I hope for in The New Ecology of Things.

Both visions emphasize the screen as the dominant system. Certainly the screen’s power to change dynamically; show combinations of text, image, animation, and video; and to support interaction through touch is remarkable. Yet I think one of the most interesting challenges for designers is to look beyond the screen slab, and imagine how other computationally enhanced objects with texture, kinetic motion, haptic feedback, sound, and light can be integrated into this new ecology of things. Let’s make sure we leverage the joy and power of all the human senses and abilities in our designs.

All projects are working interactive demos that use the Make Controller in combination with our NET Lab Toolkit (Pen & Book didn’t use the Make).

Net Desk - Chris Becker & Ryan D’orazi 
Focuses on the affordances of paper comps of magazine and other printed material. Each piece of paper is tagged, and acts as a vector into all of its versions, related documents, and relationships with the people and files that created the content of the page. By swiping the paper over different areas of the desk, the user instantly sees these different aspects of the page on a large screen. Reflects the activities of real-world magazine production. Paper, custom built table, Barcode reader, Wii controller, 50″ display, light sensors.

Whisper Stones - Kylan Coats
Plays with the idea mythical character of the stone by allowing people to lean down, pick up a one of several stones, and anonymously leave or hear and audio message. Small and large stones, $10 radio shack solid state recorders, switches.

Pen and Book - Chris Lauritzen & Hunter Sebresos 
Explores and embraces the affordances of pen and paper by imagining simple, non-precious, mass produced and cheap digital pens and e-books. Enables the user to write on standard paper, yet instantly email the new document. Also, mark up a book and access the notes with any reader, anywhere. Paper, modified ball-point pen, Wacom tablets, projector, Flash, AppleScript.

Memory Apparatus – Hyun Yang
Inverts the “problem” of ubiquitous surveillance video, and demonstrates how people can access the histories of their own lives through the near future of continuous video capture of everything, everywhere. By rotating the two axis of the Astrolabe like object, one can dial in moments and locations of your life, seeing yourself from the perspective of a surveillance camera. Knob, Wii controller, video.

Postgeheimnis (The mail project) – Christiane Holzheid & Julia Tsao
A series of speculations (some mocked up in video scenarios, others prototypes) that explores ideas for applying some of the beneficial affordances of traditional mail and packages to electronic forms of mail. For example, in one of the working prototypes, an authorized friend is able to leave a video message for you and assign it to an object (e.g. a lamp) in your home. The lamp blinks when a message is waiting, and you simply touch the lamp see the video message played back in place of the light cast by the lamp. Micro projector, speaker, pressure sensors, two computers, Flash media server.

Shop Conscious – Nicole Chan & Haelim Paek
A system that enables the shopper to rate and find more information as they shop. They can create different rating profiles for themselves, weighting sustainability, product quality, and social network criteria. It includes a portable and dressing room media component. iPhone with LiveView app streaming flash from laptop, proximity sensors, light sensor.

Brainstorming Egg – Haemi Yoon
This project imagines a quarter sized projector that can be stuck to any surface (like your pant-leg) and turns it into a brainstorming area. Using a egg shaped object as the interface, the user can gesture to create a topic, manipulate media (e.g. scrub video), and make connections between ideas. Accelerometer, XBee wireless module, Wii controller in combination with Jonny Lee’s whiteboard software.

NetCreator – Austin Lee & Yoo Kyoung Noh
By imagining a future startup company, this project uses a series of video simulations and working prototypes to explore how everyday objects can mediate communications and create new meanings in their surroundings. For example, the working prototype demonstrated how a user could read information on their coffee cup, transfer that article to another person’s cup, send secret messages, or seamlessly transition from the short version on the cup to a full version displayed on an e-paper table. iPod Touch with LiveView app streaming flash from laptop, pressure sensor, projector, knob.

Update: Behind the scenes of the making of the video

Microsoft Office Lab’s Vision 2019 video

MDP Alumni Sebastian Bettencourt’s Beyond The Fold Newspaper Project

Here are links to several of my students’ past projects:

• 2004 – wall – scott nazarian, nikolai cornell
• 2005,2006 – mirror – in search of identity, infiniti interactive - nikolai cornell, phil van allen, others
• 2006 – interactive table – acura oracle – nikolai cornell, jonathan jarvis, phil van allen, others
• 2006 – handheld augmented reality – telepath – matt mcbride
• 2007 – e-paper newspaper – beyond the fold - sebastian bettencourt

In my current New Ecology of Things class, and in my upcoming research, we are working on an in-depth exploration of how people can work, play, and be entertained using digital affordances, yet moving away from the “mouse crouch” of sitting at a computer. One difference I have from the Microsoft vision is this: I believe each different activity can have its own set of affordances rather than the seeming standard ones envisioned in the Microsoft video. For example, writing an essay should be quite different from say, designing a website. I call this approach “anti-homogenous.” What we don’t need is a new ubiquitous interface and set of gestures. What we do need is gaining back some of the benefits of work and play spaces that are suited to the activity at hand.

This line of thinking was inspired by an article in Metropolis Magazine by John Hokenberry on an excellent new photography book by Michael Wolf called The Transparent City. The photos are studies of high-rises in Chicago, and show the uniformity of our current workplaces. An excerpt from the article:

The steel mill can’t be confused with the meatpacking plant or the typewriter-assembly factory. But here one has no idea of what goods or (more likely) services emerge from these cubicles and boardrooms.

Microsoft is proposing a new way of working, but in many ways, it is reproducing the idea of one-size-fits-all that’s embodied by operating systems like Windows. I’m not saying that Microsoft got it all wrong, they did a very good job up to a point. But perhaps we need to think beyond the homogenous approach. Instead, people need to be able to create their own work/play spaces that afford the kind of activities they are doing at the moment.  What would that heterogenous approach look like?

View Video (95meg)

I’m very interested in how tangible objects can be used in interesting ways to interact with information on screens. This video collects together a series of experiments on the use of a range of object prototypes. In making these, I imagined a screen in front of me (in some cases a standard size screen, in other cases a wall sized screen), and manipulated the various objects as if I was controlling and interacting with content on the screen. It was more an experiment in the affordance of the objects in relation to screens than thinking of specific applications.

This project was aided by the help of my graduate students Jonathan Jarvis, and Parker Kuncl.

American Honda and George P. Johnson have donated one of their Oracle Multi-touch Tables to the Media Design Program. We now have it permanently in our graduate studio where it is available for faculty and students to develop new applications. In particular, we’re interested in exploring how large sets of text and image content can be explored in a collaborative way with multiple users.

The Acura Oracle project was initiated by MDP alumni Nikolai Cornell at George P. Johnson for the Acura stand at the international auto shows (including Detroit, Chicago, New York, and LA). I participated as a consultant in the initial creative brainstorming and technical definition. Bringing in Moto Development to create the multi-touch technology and optics, I coordinated Moto with the creative team, advised on interaction issues, and developed test projects to explore different interaction possibilities.

After the first year deployment at the auto shows, American Honda and GPJ lent the Oracle to the MDP for our summer term. I taught my Interactive Objects and Spaces class with the Oracle as one of our experimental design platforms. Several student and faculty projects came out of this, and IOS student Jonathan Jarvis then went on to intern at GPJ, implementing a version of his class project which became the Oracle application for the 2007/2008 auto show season. 

One of the Oracles has now been permanently donated to the MDP.

See also a page put up by Bill Buxton, one of the original people to implement multi-touch systems back in the ’80s.