Information is a Material

Most of this talk comes from a chapter in my upcoming book, called “Smart Things”, it’s on ubiquitous computing user experience design. It’s available for pre-order now and will ship sometime between the end of August and the middle of September.

Peak MHz

I call this phenomenon Peak MHz.

Unlike oil, we’re not literally running out of CPU clock cycles, but we are seeing a reevaluation of how we understand the value that computers provide, and this has resulted in a shift in the strategy of microprocessor makers. What happened in 2004 was, broadly speaking, that chip manufacturers saw that we were running out of uses for big, energy-hungry, hot processors, and they shifted the game. Since 2004 the competition has shifted from raw CPU to making smaller, cooler, cheaper chips that can do as much work as older chips, while using fewer resources.

Here’s a slide from a talk Paul Otellini, the CEO of Intel, gave last year. Notice that instead of talking about numbers going up, processor manufacturing has become all about pushing numbers down. Instead of competing on doing more with more, they are now competing on doing more with less.

One of the main effects of this shift is that in addition to pushing the price and energy consumption of the latest CPUs down, it also pushes the price of all previous processing technologies down along with it. For example, at the beginning of the Internet era we had the 486 as the state of the art and it cost $1500 in today’s dollars. It’s the processor that the Web was built for and with. Today, you can buy that same amount of processing power for 50 cents, and it uses only a fraction of the energy. That is the same 3 orders of magnitude drop as the increase in speed to 2004. This is not unrelated, because both are the product of the same underlying technological changes.

Here’s another Otellini slide. It’s essentially saying “look, we’re making the same thing smaller and cheaper every year.”

When a technology falls in price this much, it opens up enormous possibilities and creates fundamental changes in society. Steam engines similarly lowered the price of harnessing energy by orders of magnitude…and the Industrial Revolution was born as people found all kinds of new uses for readily-available mechanical energy.

You can see similarly transformative effects if you look at what happened when the price of extracting aluminum dropped by two orders of magnitude in the late 19the century, or when electric motors became significantly cheaper and smaller in the 1920s. When something becomes cheap, it quickly joins the toolkit of things we create our world with. It becomes a design material. Sometimes for better and other times for worse.

Because cheap processors have drastically lowered the cost of taking information in, evaluating it, manipulating it, rearranging it, and acting on it, information is very quickly becoming a material to design with. It is no longer unthinkable to have an everyday object use a small embedded processor take a small piece of information—say the temperature, or the orientation of a device, or your meeting schedule—and autonomously act on it.

This capability of everyday objects to make autonomous decisions and act using arbitrary information is as deep an infrastructural change in our world as electrification, steam power, and mechanical printing. Maybe it’s as big of a deal as bricks. Seriously, it’s a huge change in how the world works, and we’re just at the beginning of it.

Design Properties of Information

Automatically sense the world

Autonomously act on the world

Remember

Repeat exactly

Image: UPI: http://www.upi.com/enl-win/9b95da78f449e1a5dc28a05efc4d55a4/

But it also means that device behavior can be replicated exactly. We’ve become acclimated to it, but--stepping back--the idea of near-exact replication in a world full of randomness and uncertainty is a pretty amazing thing, and is a core part of what makes working with information as a material so powerful.

Image: N-Trophy, 2000-2003, Kelly Heaton, Feldman Gallery: http://www.feldmangallery.com/pages/exhsolo/exhhea03.html

Create complex behavior

Image found on: http://www.vansairforce.com/community/showthread.php?t=51435

Compare that to a traditional gyroscopic autopilot where every single component is unique, it does very little, and to change its behavior you have to completely reengineer it.

Isn't this stating the obvious?

Now that we’ve broken information as a material down to components, what does this mean for the design of devices?

Object-oriented hardware

What you see here are mostly all prototypes, but these devices are the atoms of information as a material, the blocks from which other information devices will be made.

Most images from Jacob Nielsen’s PhD, “User¬Configurable Modular Robotics” (9.2MB PDF)
Also LittleBits from Bdeir, Hoefs, et al.
Tinkerkit from Tinker.it
Bug Labs

BlinkM

OK, end of sales pitch.

Smart Things

The Internet of Things

Information as Decoration

Shelf by Jean-Louis Frechin
"Enteractive" floor by Electroland
Buddy Beads by Ruth Kikin-Gil>

Intelligent Environments

From Herman-Miller’s “Always Building” (5.4MB PDF)
TU Delft’s Interactive Environment class
Usman Haque’s Sky Ear

All Design is a Negotiation with Materials

Thank you.

[July 10, 2010 update: added links to most of the references]

This chart demonstrates that we hit the era of what I'm calling Peak MHz in about 2004. That's the point when processor speed effectively peaked as chip manufacturers began competing along other dimensions. Those other dimensions--energy efficiency, size and cost--are driving ubiquitous computing, as their chips become more efficient, smaller and cheaper, thus making them increasingly easier to include into everyday objects.
For those who grew up during the 1990-2004 era, this can be quite confusing, since CPU speed was how the value of computing devices was commonly measured. Now that is shifting to how that power is applied. In other words, it's gone from being a discussion of raw power, to how that power is applied (for a similar phenomenon, see the superbike top speed competition among motorcycle manufacturers, which ended with the 2000 Suzuki Hayabusa agreement).

Information is a material

• What it means to treat information as a material.
• The properties of information as a design material.
• The design possibilities created by information as a material.
• How information as a material enables The Internet of Things, object oriented hardware, smart materials, ubiquitous computing, and intelligent environments.

Citations to references can be found here.

Chapter 6: Information shadows

Part 6: WineM, an example of design with information shadow

Figure 6-10. ThingM's WineM smart wine rack (Photo by Tod E. Kurt)

In 2007, my company, ThingM, used these ideas to design a smart wine rack (Figure 6-10). It was created to demonstrate one endpoint of a service based on wine information shadows. Every bottle of wine in it has an RFID tag (Figure 6-4), with an RFID reader in every cell of the rack. The rack, in turn, connects to an online information shadow service. This service aggregates wine information from the Internet and all of the racks that are connected to it. [Footnote: As this was a proof of concept, we implemented only bare-bones functionality and worked with a wine data aggregator (Inertia Beverage) to verify that we could get appropriate information in a production environment.]

When someone associates an RFID with a specific wine bottle, the service would connect it to all of the other wines of the same type. Many wine management services can already analyze a wine collection and recommend wines resembling those in the collection. Our service went one step further: every bottle could serve as a subscription to a data feed from the winery and to a social network of enthusiasts with similar interests. A winery could have a sale for existing owners, or recent drinkers, of its wines. Then, every rack that contained a bottle of that wine would get a message, and light up the wine in a specific color, or send a text message that said "your 2004 Domaine Roger Perrin Chateauneuf-du-Pape has mail!"

WineM is an experiment to understand how the potential of information shadows can be expressed in a good user experience. It is designed to keep the user experience focused on the experience of choosing and drinking wine. It minimizes the presence of a full-purpose computer while still providing the full power of Internet information exchange.

Figure 6-11. WineM control panel (prototyped on a Nokia 770 tablet)

For example, the interface is a faceted classification browser (Figure 6-12). Every click adds another constraint to the search set and lights up the appropriate bottles in the rack. Thus, it's possible to organize the wine not just by year or grape, but also by current market price or number of bottles in stock (or all of the above).

Hallmarks took the difficult process of identifying the manufacturer of a given piece of flatware by its style and instead made it a matter of matching a small stamp to pictures in a catalogue. Similarly, this kind of information exploration would have been very difficult with a traditional wine rack. However, once the bottles had their information shadows stitched to them using RFIDs and a simple database connected to the massive amounts of wine information online, it was relatively easy.

Next month: Chapter 8, Service Avatars

Citations to references can be found here.

Chapter 6: Information shadows

Part 5: Design with Information Shadows

A systematic approach to user experience design can reduce the possibility vertigo of multiplying two such nearly infinite sets. Despite the speed and novelty of changing technologies, people's underlying needs and desires change slowly. What has changed is that a new powerful tool is now available to address those needs.

The use of information shadows is still in its infancy, but several interesting design properties of information shadows have emerged:

• They simplify the design of certain kinds of devices.
• They allow designers to treat dedicated devices like physical embodiments of Web services and create mashups.
• They allow mass customization of experiences without mass customization of objects.
• They allow devices to be self-disclosing for disposal and recycling.
• They blur the line between devices and services.
• They create novel, pleasurable, entertaining experiences.

Information shadows simplify devices

Physical/Network mashups

Ubicomp mashups attempt to move computation off the desktop and integrate it with the artifacts of everyday life. They extend beyond the Web and combine the functionality of both software and hardware components. (Hartmann et al, 2008)

Fried and Torrone's 2009 Tweet-a-Watt project (Figure 6-9) is one such mashup. It posts electricity use to Twitter, using the same API that's normally carries people's Twitter posts about their own activities. But Twitter can easily broadcast information about devices, making hour-by-hour updates about energy use accessible to humans and readable by software.

Ubicomp device user experience designers can hook up information about objects to data sources on the Internet using the same APIs and protocols used by web site mashups. [Footnote: Bleecker (2005) takes it further, and defines the term blogject to describe devices that act like people on the Internet. They can blog, they can post to Twitter, they can reply to human conversation. For Bleecker, "blogjects become first-class a-list producers of conversations in the same way that human bloggers do—by starting, maintaining and being critical attractors in conversations around topics that have relevance and meaning to others who have a stake in that discussion."] These physical/network mashups build on existing web design methods and provide a familiar set of web concepts to describe how physical objects and online information can interact.

Mass customization

Conversely, merging information shadows with rapid manufacturing techniques such as 3D printing allows for the instantiation of data in a physical, purchased object. Materialise, a Belgian 3D printing firm, sells a line of intricate designer lamps for the high-end furniture market (under the .MGX brand). Each lamp is individually printed. When first introduced, every lamp came with a disk containing a CAD file describing how to recreate that lamp. It's the lamp's DNA and part of its information shadow. Since Materialise keeps copies of the files, the lamps are, in effect, immortal: if a lamp is broken, they can print another one. Each lamp can be unique, or replicated as often as a buyer wants.

What happens to "mass production" when an object's physical form is based on a unique digital file? In a sense, we can now go back to a pre-Industrial Revolution era of unique objects. But now the uniqueness stems not from the imperfections and unpredictability of hand craft processes, but from a manufactured object’s relationship with its informational shadow.

Smart disposal and recycling

For example, information about the materials from which the object is made can be mashed up with a database of municipal recycling rules to generate instructions for how to locally recycle the object. San Francisco—where I live—has an advanced recycling program that automatically distinguishes between many materials. However, I still don't know if I can put a steel car part, Styrofoam or shoes with a "recycle" logo on the sole in my plastic recycling bin. The rules of what is acceptable, and how to prepare it, change regularly. An information shadow mashup linked to each object could clarify that question instantly, directing me to take my esoteric recyclable to a specific location or to treat it in a specific way.

Similarly, complex items, such as consumer electronics or robotic toys, are difficult to recycle because they require too much disassembly and contain unknown materials. For the municipality, these objects' information shadows could contain disassembly instructions and complete materials lists. With more information, city systems could know what do to with old toys besides sending them to the dump.

Information shadows enable new kinds of services

By giving objects unique identifiers, shadows allow those objects to become the subjects of services that track them and interact with them. Everyday objects can become subscription services. [Footnote: Again Sterling got here first. In Sterling (2001) he describes a furniture subscription system that creates one-off customized furniture on demand.]

In the days before the breakup of AT&T, Americans didn't own their own telephones. They leased them from the phone company. Although "Ma Bell" limited the range of phone choices, the phone company was required to repair broken equipment. The company could arrange update the whole system systematically and thoroughly, whenever it wanted. Though not ideal, the system had its benefits. It was also nearly impossible to replicate without the resources of an enormous company like AT&T. Information shadows could facilitate similar—but not so resource-intensive—for many other kinds of products and consumers.

For example, a shoe company could sell sustainable shoes by subscription. The shoe easily disassembles, yet is sturdy and comfortable. Buying the shoe means buying into a subscription for that shoe. As one part wears out, or as fashions change, the shoe can be disassembled, and mailed to a central warehouse, which mails back a replacement part. The shoe's information shadow says exactly which replacement it requires.

More directly, unique item-level identification allows for services that determine authenticity and trace provenance. In some parts of Africa, 30% of pharmaceuticals are counterfeit. mPedigree is using unique identifiers, printed under a scratch-off material, to identify authentic drugs (Schenker, 2008). Sending the number by text message to a trusted central location checks the authenticity—and then the expiration date—of the pharmaceutical. If the identification number is valid and the medicine's shelf-life has not expired, the system sends back another text message with a simple affirmation. Similarly, a purchaser can use the information shadow of a grocery item to trace its progress back to the farm where it was made and verify whether their farming practices are sustainable and humane. Similarly, an expensive designer handbag can be quickly authenticated.

The service possibilities of information shadows are enormous.

Entertainment

That is, of course, just the tip of the iceberg. Mediamatic, a technology design organization, challenged a group of designers at the PICNIC 08 conference in Amsterdam to create social games using RFID tags that every conference participant was given (Mediamatic, 2008). After a week of hacking, the 30 people in the workshop had created ten functioning games (Table 6-1).

Table 6-1. Games developed usink ikTag RFID tags in a week of hacking by groups at the PICNIC 08 conference. The text is from a flyer printed by Mediamatic, the organizer of the hacking week.

• ikRun. Run from the conference to the PICNIC Club and record your fastest time and finishing photo. Scan your ikTag at the start next to the E-Art dome, scan again to finish and win!
• Friend Drink Station. Free drinks for new friends! Mediamatic offers a free drink and a new friendship in the network. Just swipe your ikTag and push the button.
• Department. Use the ikTag to see what the Department of Information Security & Privacy knows about you. The DISP is buying privacy and selling security.
• ikCam. Swipe your ikTag to add your portrait to your profile. Or gather up to 20 friends with ikTags and make shapshots!
• Breathalyzer. Use the ikTag, blow into the straw and test your alcohol intake. The outcome will be published on your profile. Compare your drinking skills with others at PICNIC.
• ikWin! Use your ikTag to challenge someone in a battle for Google ranking. Two scissor lifts will go up, the more hits, the higher you go.
• Mobile Massage Couch. Sit down on the two seater with a new friend, use your ikTag to get a free massage. You can win bonus time as a gift from the crowd.
• DuckRace. 2 players start their race cars with their ikTag. The race track is based on your profile and network. The audience will influence your race car with their ikTag.
• Breedrs. Drop your ikTag in the Breedrs Pond and see it evolve into a creature, with DNA based on your profile. Is this love or war?
• Vbird. Contact the Vbird with your ikTag, help it fly, meet new friends and find the interactive film in your profile.

Tomorrow: Chapter 3, Part 6: WineM

Citations to references can be found here.

Chapter 6: Information shadows

Part 4: The Internet of Things

The Center's mission is to create an "Internet of Things" that will: merge the centuries old "network of atoms" (the production, distribution, sale, use & disposal of products) with the "network of bits" (the Internet).

- Kevin Ashton, executive director of the MIT Auto-ID Center, 1999

In 1999, MIT Auto-Id Center's vision was that non-electronic things should also have digital identities. The Center's main focused was automatic identification [Footnote: Leading to a number of visionary proposals for how such identification would work (Brock, 2001-1, 2001-2, etc.).], rather than the digital social life such identification creates for things. Information shadows, however, enable more symmetrical experiences than just identifying non-digital objects, as economically and socially powerful as that is. The Internet of Things has by now taken on a broader meaning. It describes the collection of all objects with information shadows, whether those objects' relationship to the Internet is asymmetrical (as in the case of identification and tracking) or symmetrical.

Figure 6-8. FedEx SenseAware (Courtesy FedEx)

The possibilities created by feedback between an object and its information shadow are immense. A shipped object can conceal its actual destination, only revealing the next step in its path to a shipper and dynamically adjusting its route if diverted. It can validate its authenticity and refuse to function if the person holding it is unauthorized, with the list stored in its information shadow. A FedEx's SenseAware smart tag (Figure 6-8), for example, reports not just the location of its package, but also the environmental conditions of its transfer. As it travels, it adds its location, temperature, pressure, humidity, and whether (and when) its box has been opened to its information shadow. In a more speculative scenario, a SenseAware-equipped box could theoretically reroute itself if storage conditions were likely to have caused its contents to spoil. [Footnote: FedEx does not currently offer this service. This scenario is speculative only.]

Despite its roots in shipping, an Internet of Things could use information shadows in a wide variety of ways. A motorcycle could have a strain gauge built into its chain, and store both the status of the chain, and the specifications for a replacement, in its information shadow. When the chain was stretched to the point where it needed replacement (an important part of motorcycle maintenance that can lead to very dangerous conditions if not performed), the motorcycle could log that state in its information shadow. This, in turn, could trigger an alert on the motorcycle owners' phone and on the motorcycle's dashboard. The repair shop could interrogate the motorcycle for the kind of chain last used and the chain performance characteristics indicated based on sensor data taken since the last replacement.

Tomorrow: Chapter 3, Part 5: Design with Information Shadows

Citations to references can be found here.

Chapter 6: Information shadows

Part 3: Point-at things

Systems of information shadows are created by promoting standard methods for associating a physical object and information about it. Experience designer Tom Coates coined the term point-at things while compiling ways to organize BBC's program directory. [Footnote: See (British Broadcasting Corporation, 2000) for the information architecture standard the BBC produced as part of the process described in this section.] In that project, he realized that "once you [uniquely identify] a programme episode then something really significant happens—you can give it a name, make it addressable, you can—for the first time point at it. Better still, you can move from pointing at something to glueing handles onto it. And once you have such a handle, then you can pick up the programme and throw it around and stick labels on it and join it together with other programmes." (Coates, 2004) [The original post.]

[Footnote: A similar term to point-at things is Bruce Sterling's spime, a neologism that merges space and time, because those are two key data points in an object's identity. As Sterling puts it, "every object worthy of human or machine consideration generates a small history. These histories are not dusty archives locked away on ink and paper. They are informational resources, manipulable in real time. […] The key to the spime is identity. A Spime must therefore be a thing with a name. No name, no spime." (Sterling, 2005)]

His realization, which translates directly to organizing information shadows, is that when a unique identifier is attached to an object, it becomes possible to collect the metadata about that object into a single information shadow. That unique identifier is the leverage point with which to access and manipulate the whole information shadow in relation to similar shadows.

For example, when the online retailer Amazon branched out from selling books to selling other kinds of products, they needed a way to identify every item they sold. Extending the International Standard Book Number (ISBN) they were already using, they created the Amazon Standard Identification Number (ASIN) to uniquely identify every product they sell. This allowed all of the items they sell to become, in Coates' term, point-at things. It became possible to precisely identify exactly which product was being linked, discussed, etc. That precision allowed for a wide degree of flexibility and power—which other services could use to build upon Amazon’s inventory systems. As of late 2009, ProgrammableWeb.com lists more than 300 services that use Amazon's ASIN system to create additional services. [Footnote: Using Amazon's eCommerce API.] Pickii, for example, uses ASINs, Amazon's list of categories, and user ratings for individual products to create a top-10 best-reviewed list for virtually every Amazon product category.

Matt Biddulph, one of Coates' colleagues at the BBC, and Ulla-Maaria Engeström (née Mutanen) took this idea further. While useful, Amazon's ASIN was limited to products Amazon sold. What about the "many small producers especially in developing countries do not have access to create unique identifiers, and therefore, their products are not equally visible or recommendable online" (Biddulph and Mutanen, 2006)?

Figure 6-6. Unique item on Thinglink, showing Thinglink ID in the lower right-hand corner of the photo

In response to this observation Mutanen created Thinglink, "a service where anybody can register unique identifiers for objects that they want to identify as unique." The service generated a unique alphanumeric identifier (Figure 6-6), and served as the metadata clearinghouse to "aggregate online discussion around particular objects, track the history and transformation of objects, [and] socialize around particular objects." (Biddulph and Mutanen, 2006) In other words, Thinglinks were designed to create point-at-things out of any object by using user-generated unique identifiers to connect single items to their information shadows. [Footnote: As of the end of 2009, the service is still running at thinglink.com, but has shifted its focus from products created by small producers to high design objects.]

When data network access seems almost everywhere, the movement of information about objects within groups of people then intersects with their physical movements. Friends' recommendations, for example, can affect buying choices and preferences, while purchasing an object creates a data event that feeds up through the store's inventory, to the distributor's, all the way back to the manufacturer. Along the way, humans and automated systems make decisions that affect what happens to the object, whether the object is bought, sold, gifted, destroyed, or perhaps refurbished, extending the object’s social life.

Figure 6-7. Nokia 3220 with the NFC shell, the first mass markt mobile phone with a built-in RFID reader (Courtesy Nokia)

As optical identifiers, such as 2D barcodes, become more popular and devices such as the RFID reader on the Nokia 3220 (Figure 6-7) become inexpensive, this symmetry means that nearly everything can be uniquely identified on the cheap. [Footnote: At the time of writing, late 2009, there are more than 60 barcode and 2D barcode scanners in Apple's iPhone App Store.] Coates initially used point-at things to refer to digital objects, but it may soon be common to literally "point at things" to access their information shadows.

Tomorrow: Chapter 3, Part 4: The Internet of Things

Citations to references can be found here.

Chapter 6: Information shadows

Part 2: Information shadows

Then Wendy saw the shadow on the floor, looking so draggled, and she was frightfully sorry for Peter. "How awful!" she said, but she could not help smiling when she saw that he had been trying to stick it on with soap. How exactly like a boy! Fortunately she knew at once what to do. "It must be sewn on," she said, just a little patronizingly.

- Peter and Wendy by J. M. Barrie (1911)

Figure 6-2. Amazon listing for Tickle Me Elmo, showing manufacturer metadata

An Amazon product listing (Figure 6-2) shows a lot of information, but there's much more information about the product than just the official specs. An enormous quantity of user-generated content exists on the Internet tied to nearly every product. Virtually everything made or grown has been reviewed, discussed, photographed, mocked, praised, prodded, measured, disassembled, and hacked. Until the Internet, little of this social life was available; now there's a flood.

The digitally accessible information about an object can be called its information shadow. [Footnote: The use of "shadow" to describe a relationship between physical objects and digital information goes back to Westin's 1967 description of "data shadows." My use is closest to Greenfield's 2006 use of it: "the significance of technologies like RFID and 2D barcoding is that they offer a low-impact way to 'import' physical objects into the datasphere, to endow them with an informational shadow."] Nearly all industrially created objects have rich information shadows, even if those shadows are invisible to their owners and users.

Wine bottles, for example, have very rich information shadows. Along with the traditional bottle-level data (such as when it was bottled, what grapes were used, who bottled it, etc.) wines have a huge social life generated by thousands of websites, blogs, rating services, and books. Wine enthusiasts probably spend as much time discussing wine as they do drinking it, and they've created a lot of content with easily accessible identifiers, namely the vineyard and vintage. [Footnote: Bruce Sterling describes the kind of social relationship we can have with wine in "Shaping Things," his 2005 book-length essay on ubiquitous computing and design.]

Everyday objects have been separated for a long time from their information shadows, as Peter Pan was from his actual shadow. The complexity of finding, organizing and accessing this information divided the world of objects and the world of information shadows. Even if accessible through a computer, information shadows were unavailable when they could provide the most value: in choosing between different products to buy, or in figuring out how to use a new tool. For example, barcodes aren’t human readable. So for a long time, only those with barcode readers and access to specialized databases could use them. [Footnote: This of course is no longer the case, as barcode reading cameraphone applications have proliferated in recent years, but that's a relatively recent phenomenon relative to the three decade-long history of bar codes in common usage.] Which basically only included retailers and their employees. Even then the kinds of data available to retailers were limited. Standard Universal Product Code (UPC) barcodes identify classes of products, not individual things. Moreover, there is no universal database of barcodes. So a retailer's information about a given object is limited to data it has bought or generated on its own. Typically, like the early Borders Books system, retailers only keep enough information around to price items and order more.

A few industries have systematically generated and employed more extensive information shadows. In particular, the manufacturing and shipping industries have been tracking and identifying individual objects for years. In those industries every mile traveled by every piece of inventory directly affects revenue. So manufacturing and shipping systems provide fine-grained information shadows to identify and locate objects. For example, Wal-Mart and the American military—two organizations that ship a huge variety of things to a dizzying number of locations—have enthusiastically adopted identification technology (Myerson, 2006) to increase efficiency.

For consumers, ubiquitous computing attaches the information shadow to the object, like Wendy does to Peter Pan's shadow. It does this using three key technologies:

1. Inexpensive machine-readable item-level identification technologies (see Sidebar) uniquely mark every object.
2. Wireless networking makes the information shadow of objects accessible to devices in more places.
3. Networked information aggregation services create a standard way of accessing information shadows that are produced simultaneously in many places at once.

Sidebar: Item-level Identification Technologies

• Barcodes. These are most commonly seen representing the Universal Product Code that's on most retail packaging. They are used to uniquely identify items in corporate inventory systems and in certain niche markets, such as wine storage.
• 

  Figure 6-3. QR Code that reads "Ubiquitous Computing User Experience Design" when decoded
  2D Barcodes. These come in many styles, from the Datamatrix code found on postal service package labels, to exotic fiducial markers used in motion sensing applications. QR Code markers (Figure 6-3), designed for easy reading using mobile device cameras, are increasingly popular.

• RFID. Radio Frequency IDs (Figure 6-4) come in many shapes and sizes—from subway access cards, to paper and metal stickers, to ceramic cylinders that sit in the stomachs of cows. Small radios inside them broadcast a unique identifying number. Because radio waves can transmit through solid materials, RFIDs can embedded inside other objects (shipping palettes, clothing tags and animals). Passive RFIDs, the most common kind, gather energy from radio broadcasts received from RFID readers and reflect that energy back to the readers in the form of a wireless stream of data communicating their unique identifier. They do not need their own source of power to function, but consequently work only within relatively short distances. Active RFIDs use their own power sources to transmit data farther, but are more expensive and often require batteries.
  

  Figure 6-4. An RFID attached to a wine bottle (photo Copyright 2007 by Tod E. Kurt, used by permission)
  

• Smart cards. SIM cards are small processors packaged inside plastic cards. They transmit a unique identifier when in direct contact with a reader. Because they contain an active processor (typically powered through contact with a reader), they are capable of much more complex interaction. Thus, they can encrypt information, require authentication before transmitting their identifier, poll sensors, etc. Smart "buttons" that resemble coin cell batteries are a similar technology. These buttons also communicate through contact with a reader that also acts as a power source.
• Magnetic stripes are familiar from credit cards and hotel entry cards. They are an established technology, and are easily created or changed, but are neither as compact, convenient or secure as some of the others on this list.
• IPv6. For devices that are actively communicating through a network, IPv6 is an extension to the familiar Internet Protocol standard designed to uniquely identify trillions of devices. It was formally defined in the mid-1990s, and while uncommon in current Internet devices, it will likely become an identification standard for smart devices communicating over a network.

Tomorrow: Chapter 3, Part 3: Point-at Things

Citations to references can be found here.

Chapter 6: Information Shadows

Part 1

Every master goldsmith shall have a mark by himself.
- King Edward III, 1363, (Chaffers and Markham, 1905)

Figure 6-1. Silver hallmark indicates that Aaron Hadfield of Sheffield, England made this knife in 1834 (Courtesy Leopard Antiques, Cape Town)

However, King Edward’s system of hallmarks (Figure 6-1) came to be much more valuable. As one of the earliest instances of catalogued metadata associated with manufactured goods, the system of hallmarks enabled a much different relationship between people and their possessions. Hallmarks re-associate objects with their origins. They systematically, consistently, connect two people (the owner and the maker) through an object. This silver knife (Figure 6-1) is not anonymous: if we can decode the hallmark, we know who made it, when, where, and of what material. In encoding information into objects, hallmarks link those objects to data.

That linkage has continued to prove valuable and important in unexpected ways. Identifying individual objects in the world builds a bridge between any given object and the available information about that object. It allows for physical, everyday objects to act in the world of symbolic data, and vice versa.

Before ubiquitous computing, only expensive things such as precious metals, currency, or large machines were individually identified with any regularity. [Footnote: Such as vehicle identification numbers, building addresses and serial numbers. Brock (2001) has a good list of earlier identification schemes, their format, and the motivation for their creation.] Only the cost of the item—or the social cost if the item went untracked (as with firearms)—justified the labor and monetary expense of labeling individual items and maintaining the metadata. Ubicomp includes a number of item-level tracking and identification technologies (see Sidebar) that dramatically lower the cost of adding machine-identifiable codes to objects and employing devices to read those codes automatically. Other information aggregation services can then build on these technologies to cross the gap between what a thing is, and the meanings people give it.

An early success in item-level identification

Borders Books started as a university town bookstore in Ann Arbor, Michigan. It was not significantly different from other bookstores in the town, except for a small innovation. Louis Borders, who founded the company with his brother Tom in the 1970s, had studied computer science. Thanks to this exposure to the power and mechanics of information processing, he realized that putting a unique computer punch card in every book in his bookstore would automatically identify the book when a computer read the card (Raff, 2000). The punch card system allowed Borders Books to take a near-instantaneous inventory of their store to identify which books had sold and when. Other bookstores conducted laborious annual inventories, at best. But Louis and Tom Borders could accomplish the same thing quickly and nearly at any time they wanted.

Here is how it worked:

• Every book had a punch card inserted into it, like a bookmark.
• Cashiers removed the punch cards and set them aside while ringing up a purchase.
• Periodically, all the collected punch cards went to a computing center, which generated a report on the sold books
• Using the report, company managers could track books sold at which stores and when they sold.
• This allowed store managers and book buyers to identify buying patterns and anticipate shortages.

This seemingly small change gave Borders a significant advantage over its competitors, because they could keep a much wider variety of books, keep the popular ones perpetually in stock, and cater to local buying tastes and trends. They didn't have to shut down and count all their books like other stores, and they didn't have to rely on their hunches to figure out which books sold well in which stores. They soon outgrew their original Ann Arbor store and founded a software company—Book Inventory Systems—that sold their system to other bookstores. Today, they're a huge international chain. [Footnote: Which, it should be noted, is ironically struggling to compete with online book retailers whose advantages are created with many of the same technologies Borders pioneered.]

Item-level identification gave Borders a competitive advantage because it linked the physical object—a book—to important information about it. Some of that important information was permanent, such as the name of the book and its author. Other important information was contextually generated when the book sold—such as the store location and date of purchase. Equally important is that Borders Books was able to integrate the information tracking smoothly into the shopping experience. Most Borders customers may have never known how the punch card in every book, casually removed and put away by the cashier, was the lynchpin of their bookstore's success.

Tomorrow: Chapter 3, Part 2: Information Shadows

Problem: the US Postal Service is in financial trouble

• Casual letters have largely been replaced by email
• And package delivery has to compete with FedEx, UPS, DHL

Proposal: the Postal Service should become the geographic DNS….

This, more or less, is exactly what the USPS already does, but it's still tied to the sender writing the actual physical address on the letter. However, as any recipient of a slightly mis-addressed letter that still arrived knows, the service is actually pretty good at figuring out where the letter is going. The USPS is already resolving ambiguous address data into physical locations.
It's been doing it for years:

(Image Public Domain from Wikimedia)
Why not make name-to-location resolution the primary role of the postal service?
For example, rather than having your address be "Your Name, 1234 Oak Street, Town, State, Zip Code" you could pay to have it be "Your Name, Town, USA." Microsoft could pay to have their address just be "Microsoft, USA." It works for "Santa Claus," why can't the USPS charge MS to make it work for them?

On the back end, the postal service could provide a number of routing services using the infrastructure they already have. The "Microsoft" letter could go either to a regional office or to a central location, depending on what Microsoft wanted to pay for.

…and charge for it.

Let's look at the numbers.

According to the 2002 US Census, there are approximately 6 million small businesses in the US that have employees (there are 18 million other firms with no employees). For the purposes of extreme simplification, assuming that the smallest businesses aren't interested at all (which is probably not all true). That leaves almost 2.5 million businesses with 5 or more employees.

Given this single figure, here are three scenarios:

• Worst case: 1% of firms are interested in paying $100 a year to have this service. That equates to annual revenue of $2,500,000. Not bad, and could probably pay for itself, perhaps charging more than $100 for services for the largest firms (such as routing Microsoft's mail appropriately).
• Median case: let's assume that 30% of the firms buy in. That's $75MM per year.
• If 60% take it up, then that's $150MM per year.

My point is that they could just copy the way that DNS works from a service design perspective to create a highly lucrative line of business that sits on top of, and uses the infrastructure of, the business they already have. A business with an enormous increasing ongoing revenue stream.

And then become the primary provider of geographic Point of Interest resolution

They also have an extensive network of people and machines (the largest civilian vehicle fleet in the world, according to Wikipedia) that could continually add to the database.

Then they could charge for bulk data rental for other services to build on this.
As the importance of location based information grows, the value of a USPS API will only grow, while the value of their primary business (moving stuff around) shrinks.

The USPS is a special organization that has a unique structure (part independent corporation, part government agency) which gives them a significant competitive advantage. They also have enormous resources for converting physical location information into data, and vice versa. They are also losing 4 billion dollars a year (1MB PDF). Moving atoms will only become more expensive as energy becomes more expensive. The USPS needs to start moving bits.

Last month I posted a draft of Chapter 1.

Citations to references can be found here.

Part 4: Sidebar: Too Much Metaphor, Magic Cap

General Magic's Magic Cap was an early operating system for mobile devices. It can be thought of as an early attempt at a user-centered ubiquitous computing experience design. It is also an example of a product that followed its metaphor too far.

Sony Magic Link (Courtesy Sony)

Designed for portable, networked computing, the launch of Magic Cap in 1994 predated the Palm OS by several years. Magic Cap provided similar functionality to the Palm OS, but with one crucial difference: from the start General Magic had designed Magic Cap for networked communications. Both devices that ran Magic Cap, the Sony Magic Link and the Motorola Envoy, were tablet PCs with built-in networking (the Sony had a phone modem and the Motorola had an early wireless modem).

On the back end, Magic Cap's architecture used software "agents" written in a language called Telescript, which could run outside of the device to perform actions on the Internet and deliver results back to the user. [Footnote: Magic Cap mixed its metaphors, too, describing their software having simultaneously as "agents" (Tardo et al., 1996) and a "cloud" (Hendler, 2000) while using neither of those two concepts in the interface in that way (there is a cloud in the Downtown sky, but it is unrelated to the Internet, which is an office building).]

Key members of the Macintosh development team led General Magic. They decided to leverage their work on the Macintosh interface and extend the desktop metaphor to a networked device environment.

Figure 3-6: Magic Cap home screen.

The Magic Cap home screen (Figure 3-6) centers on an image of an office desk. It is a surprisingly literal interpretation of the desktop metaphor after a decade of mass popularity. Perhaps they zoomed out from the two-dimensional “desktop” view to emphasize how a portable, mobile device would move off the desktop and into the three-dimensionality of a real room.

Figure 3-7: Magic Cap hallway.

However, the realistic, spatial implementation of an "office" metaphor constrained them further. Thus, users had to walk down the "hallway" (Figure 3-7) to access functionality located in different rooms.

Figure 3-8: Magic Cap Downtown.

The literal interpretation of the office building metaphor even produced a city street. The town center (Figure 3-8) had an Internet office building and a diner with a Web browser accessible through a movie poster. The interaction resembled a sideways scrolling adventure game more than an operating system for business users (Sony's target audience for the device).

It is difficult to say whether the decision to structure the user experience as an adventure through a software suite primarily caused Magic Cap’s business failure. It is clear, however, that the metaphor, and General Magic's literal interpretation of it, significantly constrained the design. Even in the early days of mobile device usage, the literalism may have hurt comprehension as much as it helped. Many screens required significant "signage" to explain what various iconic images meant. Other icons, such as the "magic lamp" icon located at the bottom of every screen, simply did not make sense within the town center metaphor.

Extending the desktop metaphor to buildings may have seemed like a good idea initially, but it became increasingly baroque in its details. Ultimately it turned using the operating system into a long walk through an unknown city full of confusing signs, which is possibly the least magical experience of all.

Next Month: Chapter 6, Information Shadows

Last month I posted a draft of Chapter 1

Citations to references can be found here.

Part 3: Designing with Metaphors

Metaphors are complicated tools. They inspire us to make new associations and can communicate complex ideas quickly, but they also constrain thought. Connections that may make sense in the metaphor's source concept may not exist in the target. For example, Netscape's Navigator browser provided little guidance on how to reach a destination, like an actual navigator would. If its exploration metaphor is interpreted literally, browsers are more like boats, and search engines (which, coincidentally, little resemble engines from a user's perspective) are the navigators. As documented by Blackwell (2006), the history of metaphors in interaction design has gone through boom times -- the 1980s success of the desktop metaphor -- to times of extreme criticism and failure (the infamous Microsoft Office 97 paperclip). [Footnote: Blackwell (2006) is indispensable for those interested in the history and cultural role of metaphor in human computer interaction.] Despite the criticism, however, metaphors remain powerful and valuable tools. They are one of the most straightforward ways to tap into existing knowledge to create a familiar narrative out of novel functionality.

This chapter provides a set of benchmarks for thinking metaphorically about ubicomp UX design. If the assumptions behind a project recall one of these metaphors, it is possible to ask the following questions about the design metaphor as a way to understand the project's limits and possibilities -- and your own as a designer.

• What is the comparison that this metaphor is making? What class does it say that the design and the metaphor belong to?


• What is the list of tools and activities associated with the source concept? How would those map to the experience being designed?


• What are the visual images the source concept evokes? What are the interaction patterns that it implies? Are there necessarily positive outcomes to those patterns? Negative ones?


• What is the purpose of using the metaphor? What exactly do you need it to accomplish? What associations is it supposed to evoke and what actions will the metaphorical associations make easier?


• What are the boundaries of the metaphor? At what point do the differences between domains become so great that the metaphor hurts more than it helps?

Sidebar: Dan Saffer's Guidelines

The following guidelines from ubicomp designer Dan Saffer (2005) provide a sequence for approaching the design of interaction metaphors and designing with metaphors. It begins with a focus on the culture of use and ends with advice for identifying and discarding metaphors that are no longer valuable.

1. Metaphors are cultural. Different cultures have different conceptual frameworks, especially about abstract ideas like time. Be conscious of differences when picking metaphors that span multiple cultures. Not only are metaphors culturally specific, but they can also be limited to specific audiences within that culture. If you do not have a desktop, the desktop metaphor could be meaningless to you.
2. Metaphors are contextual. Be aware of the context in which the metaphor is used. What can work in one medium or domain may not work elsewhere. Unless you are deliberately juxtaposing for effect, metaphors within a product should fit the context in which they will be used. The subject matter of most projects will likely be rich with its own metaphors. Finding and utilizing them can make powerful connections between the product and its context of use.
3. Fit the metaphor to the functionality, not the other way around. Metaphor should be a tool to help users comprehend unfamiliar content or functionality. So when using a metaphor within a product, start with the new, unfamiliar (to users) material you have and make that the subject of the metaphor, not the referrer. Awkward situations are more likely to happen when functionality is shoehorned into a metaphor. In the best case, metaphors should support concepts, not be supported by concepts or be the concept.
4. Use metaphor to uncover otherwise hidden aspects of the material. While "banking is a game" might be an inappropriate metaphor when used inside a product, it could be a powerful metaphor to use during concept development to show what banking is not. And even perhaps what it is.
5. Discard process metaphors when necessary. Metaphors that have been used in brainstorming or during the design process can grow constrained or simply be inappropriate for users. In some cases, it is better to have no metaphor at all than an inappropriate one.
6. Do not let your metaphor ruin key features. Designers need to realize that all metaphors can obscure as much as they illuminate, and they should choose their metaphors so they do not obscure or distort key features. Microsoft╒s recycling bin in Windows OS is cute, but is less clear than a trash can or a dumpster would be.
7. Choose metaphors that are appropriately scalable. The desktop metaphor has lasted as long as it has because it scales very well. Many varied tasks fit well into its framework; likewise, the folder metaphor. Other metaphoric choices (an envelope instead of a folder, say) may not have scaled so well. (On the other hand, using the metaphor of a workbench instead of a desktop might have supported many activities, not just working with paper.)
8. Let your metaphors degrade and die. Once an appropriate metaphor is understood, it becomes nearly unconscious ("dead"), only to become apparent again with effort. Although this has been criticized, this is a good thing, as intermediate and advanced users should not have to bother with the metaphor and deal more directly with the underlying material. It is only inappropriate metaphors that continue to hinder more experienced users. This is, in fact, a good test for whether or not a metaphor is appropriate. Metaphors can be a double-edged sword.

The rest of this chapter are my own guidelines for metaphor design.

1. Create Specific Metaphors

   To give the widest possible overview, the previous list organizes metaphors into very broad categories. In practice, metaphors for each specific project are much narrower. In deciding to make an enchanted object, for example, designers pick a specific magic item to emulate (say, a magic mirror). Then, the rest of the design process can move from the first principle that "in video conferencing the computer screen is a kind of magic mirror."

   Design decisions flow from this more narrow metaphor, but the broader metaphor provides guidance for how that specific experience fits into a larger set of ideas. If computers are invisible, for example, how do people know where they are? Are they just invisible to people, or also to each other? How does someone tell this specific information processing-enabled invisible experience apart from all the others? How do they know where one invisible computer ends and the others begin?

2. Good metaphors describe function, not appearance

   
   Metaphors should describe deep functional similarities, not superficial resemblance. A well-chosen metaphor maps many of the experiential qualities of one kind of interaction to another. Magic wands in myth, for example, are swung around and pointed at objects to activate actions. The Nintendo Wii captures these qualities well, creating a relatively clear relationship between familiar stories and a new game controller. A game controller that looked like a traditional magic wand (glittery star on top, etc.) but operated like an ink pen would be simply confusing.

3. Metaphors Imply People

   
   

   When we propose that a computer be presented as a metaphorical office or typewriter, one of the things we are really describing is the intended user of this computer, describing him or her as an office worker or typist.

   
   - Blackwell (2006)

   Metaphors establish not only ideas about the meaning of physical affordances and potential ways to use them, but also about the people involved. A metaphor communicates a set of roles for people involved in the device's use. For example, a digital device resembling a stethoscope implies that the user is similar to a medical clinician and that its intended use is diagnostic. If the device is not medical (i.e., rapid transit personnel use it to verify passengers' RFID passes), the mismatch can produce awkward misunderstandings. The transit workers may feel misrepresented by the device, and the passengers may worry about what exactly it is scanning.

   Schank and Abelson (1977) theorized that people reason about the world using expected sequences of actions akin to film scripts. Metaphors embody those scripts by affecting the design of an object╒s form and functionality. By linking the new to the familiar, metaphors communicate how people and devices should act together. Devices are props to carry out those scripts. (Yet another metaphor!) The more explicitly devices rely on metaphor, the more explicitly they reference known characters and how they behave. If the script referenced by the prop does not fit the needs of the people actually using it, or is unfamiliar to them, the prop will be less successful. Thus, the choice of the metaphor should start with the expectations, needs, desires, and actions of the people involved.

4. Use metaphors to inspire, not constrain

   
   

   Almost any metaphor, even an arbitrary one, can trigger new ways of thinking about a product or new solutions to a design problem.

   
   - Saffer (2005)

   In the 1920s, surrealist artists invented a game of map misdirection. Using a map of one city, they would try to navigate another city. They would pick a destination, and simply follow the map. For example, a trip in Berlin using a map of Paris would end at a barber shop in the suburbs instead of the Eiffel Tower. In the process, apart from having some absurdist fun, they would learn more about both Berlin and Paris.

   Similarly, generating new ideas is one of the most powerful results of using metaphor to design user experiences. When mapping one idea to another, the differences between the two domains highlight aspects of both that might not otherwise be noticed. A "butler" device that can understand some spoken comments, for example, probably reaches the limits of its actual understanding very quickly. The designer may well imagine what an actual butler would do in a similar situation. The metaphor raises the question: How do people communicate when they do not understand a phrase, even if they understand each individual word? It also points to a new domain of functionality: more than speech recognition, the system might need to be presented more like a student with limited knowledge of a foreign language than a smoothly polite butler. Similarly, the system may need to imitate how people pause to think in a conversation as a way to explain pauses for processing.

5. Use metaphors to explain, not hide

   
   Metaphors can help explain the functionality of unfamiliar technologies and inspire reflection on how relationships between people and interactive systems unfold. They are, by definition, a distortion of the capabilities and functionality of the technology they are used to explain. If not approached carefully, that distortion can hide vital aspects of the technology from users. The right level of concealment is highly context-dependent, as Saffer's guidelines point out. On one end of the spectrum, toys designed to stimulate the imagination of children can fully embrace the magical metaphor. Misalignments between what children believe is happening and the toys actual functioning are tolerable if they support open-ended, imaginative play. At the other end, airplane cockpit designers must confirm that any metaphorical relationship helps pilots respond correctly to changing conditions and is more cognitively efficient than a non-metaphorical design. Thus, interfaces that simulate hydraulic flaps but control a mechanism that is not actually hydraulic12 need careful evaluation to make sure that they really assist pilots.

Ultimately, metaphors may be most useful during first encounters with a new technology. Declaring that a new technology resembles a familiar one, even if that apparent resemblance is only skin-deep, may lower anxiety and help people transfer existing skills to a new tool. In the transition to a new tool, a few conceptual misunderstandings, or inefficient use, may be a small price to pay for reducing a normal reluctance to try something new. However, as the technology becomes familiar, the metaphor may lose its value. Eventually, the metaphor that seemed so helpful may start creating more problems than it solves. Where this point lies is unique to every product and every group of new users. Recognizing that metaphors have failed can be humbling, but metaphors, like all tools, need regular maintenance.

Tomorrow: Chapter 3, Part 4: Too Much Metaphor, Magic Cap

Last month I posted a draft of Chapter 1

Citations to references can be found here.

Chapter 3: Interaction metaphors

Part 2: A Catalog of Ubicomp UX Metaphors

The following metaphors are ones that I have observed in ubicomp user experience design. This list is certainly not exhaustive and there are many natural overlaps in the concepts involved. I include it here to prompt creative thinking about how people relate to new technologies and how to design for them.

[Footnote: See Barr et al. (2002) for a different taxonomy of user interface metaphors.]

(Image from PICOL)

Organizational Metaphors

1. THE FACTORY The world is a factory.

   Domotics can be defined as the set of elements that, when installed, interconnected and automatically controlled at home, release the user from the routine of intervening in everyday actions and, at the same time, provide optimised control of comfort, energy consumption, security and communications.

   - Bravo et al. (2006)

   Our kitchen supports the automatic generation of web-ready recipe pages, with other possible applications including actual cooking assistance, and communication or education across distances, cultures and generations.

   - Siio et al. (2004)

   The field of "home automation," for example, treats the house as a kind of factory by implying that repetitive activities should be automated to maximize production of leisure. In this metaphor, people own the factory. Their role is to act as overseers, organizing an efficient assembly line of appliances to produce ever more free time. The implication is that automation will automatically make more time for pleasurable activities.

   Most "labor-saving" products fall into this category. Their core assumption is that technology should and will eliminate repetitive actions.

   [Footnote: As documented in Cowan (1983), one of the unintended consequences of such technologies is that they tend to raise expectations, rather than eliminate labor. From an experience design perspective this could be because they do not try to address how that time could be filled, and whether the repetitive work is drudgery in the first place.]

2. PUBLIC UTILITY
   Information processing is a utility, like electricity.

   If computers of the kind I have advocated become the computers of the future, then computing may someday be organized as a public utility just as the telephone system is a public utility.

   John McCarthy (1961), quoted in Garfinkel and Abelson (1999)

   From the earliest days, computing has been characterized as a new kind of electrification. Ubiquitous computing likewise continued to embrace it.6. The metaphor's implication is that information processing will be as accessible, in as many places, as electricity. In this view, the movement of computing out of personal computers is analogous to early twentieth century electrification of ordinary homes and workplaces, and will create applications as pervasive as electric lighting. In this metaphor, the walls and floors of the spaces we inhabit house conduits filled with information processing that enlivens devices as electricity powers lamps.

   Many telecommunications networks implicitly rely on this model of information processing by treating the traffic they carry with a neutral distance. The idea of "network neutrality" in effect demands that the Internet backbone treat packets as the electrical infrastructure treats electrons. Some electrons may heat a lamp filament, while others turn a motor. The electrical infrastructure does not differentiate in supplying them — it raises prices based only on aggregate network demand.

3. BACK TO NATURE
   Ubiquitous computing returns us to a pre-industrial life.

   There is more information available at our fingertips during a walk in the woods than in any computer system, yet people find a walk among trees relaxing and computers frustrating.

   - Weiser (1991)

   This metaphor implies that technology and nature can coexist in a way that de-emphasizes technology to the point that it becomes indistinguishable from nature.

   [Footnote: Without, it should be noted, defining what "nature" is. See Marx's (1964) description about how American notions of pastoral coexistence with Industrial Revolution technology, the "Machine in the Garden," resulted in the disruption of American pastoral life.]

   Projects that use this metaphor emphasize the "organic" nature of interaction (such as Rekimoto, 2008) and the way that computers can free people to leave their offices to sit on the beach.

   [Footnote: It should be noted that one project that literally put computers into the forest. Ambient Wood (Rogers et al., 2004) focused on the educational possibilities of ubicomp technology outside the classroom, rather than attempting to create a pre-industrial landscape.]

4. THE VAPOR
   Computation is a cloud that surrounds us.

   This metaphor compares the permanent availability of information and information processing to a vapor enveloping (and possible penetrating) every person and every object. Like Pigpen’s cloud of dust in the Peanuts comic strip by Charles Schulz, everyone perpetually moves within a cloud of digital information. The clouds extend beyond our reach and do not have a defined shape or boundary. Events that happen in the cloud may be outside of the control of any one person in it.

   Projects such as Urban Atmospheres (Paulos and Jenkins, 2005), Digital Aura (Ferscha et al., 2004) and Personal InfoCloud (Vander Wal, 2005) evoke this idea. The terms "pervasive computing" and "ambient intelligence" derive from this metaphor: in pervading an area, vapors process information ambiently.

5. PARALLEL UNIVERSES
   Technology gives us access to a parallel universe with different laws.

   It [appears] to the user that the virtual and real objects coexist in the same space, similar to the effects achieved in the film Who Framed Roger Rabbit?

   - Azuma (1997)

   

   A vision of "cyberspace" underlies many views of Internet-based information access. In this vision, computers create, reveal, or possibly contain a parallel universe. Implicit in this vision is the notion that while technology may provide access to another universe, the universe exists independently of the technology. For example, the term "physical computing" implies that other kinds of computing are not physical, they are not here, which means that they happen elsewhere, perhaps a parallel universe. In this vision, the device creates a window, and beyond that window lies a radically different world with its own physics and geography. Science fiction gives us the canonical examples of this alternate space, from William Gibson’s original notion of cyberspace to Neal Stephenson’s Street in Snow Crash (Stephenson, 1992).

   Augmented reality applications often rely on this metaphor as well. When the screen of a mobile device superimposes digital images on a live camera feed, it is presented as a window on another world where, like Roger Rabbit's Toon Town, data and humans cohabit. Similarly, Hertzian space (Dunne and Raby, 2001) is presented as a parallel dimension created by devices using the radio frequency spectrum, and the term "physical computing" implies that other kinds of computing are not physical, they are not here, which means that they happen elsewhere, perhaps a parallel universe.

6. THE INESCAPABLE PRISON
   Omnipresent technology imprisons us.

   When every action is recorded for perpetuity, in a seemingly objective manner, and there is a likelihood that the consequences will be realised, then Bentham's Panopticon becomes fully realised.

   - Dodge and Kitchin (2007)

   This dystopian metaphor implies that information technology will imprison its users, empowering governments, organizations, and individuals to create novel forms of tyranny. As people increase their dependence on information technology and share personal information, they decrease their ability to act (and possibly think) freely. If taken too far, or left unchecked, proponents of this metaphor suggest a coercive environment might emerge. For example, Albrecht and McIntyre (2005) argue that governments and corporations will inevitably use RFIDs to uniquely identify and track devices and individuals to force people to behave in certain ways. Others suggest that omnipresent sensing and recording devices will create a situation of sousveillance (Mann et al., 2003): a world in which people are constantly spying on each other and being spied upon.

Interaction Metaphors

1. TERMINALS EVERYWHERE Every surface is a display.

   

   This metaphor takes the laptop as its point of departure and imagines the functionality of laptops embedded into a variety of flat surfaces. General purpose, network-connected displays expand or contract and become embedded in furniture, architecture, or clothing, but their imagined functionality remains basically the same. The metaphor implies that interaction with these displays may differ in some details, but that we will use them for much the same purposes as we do today.

   The Microsoft Surface is an example of a device that follows this interaction metaphor by imagining a coffee table as a data terminal.

2. INVISIBILITY
   Computers are ever present, but transparent.

   It is invisible, everywhere computing that does not live on a personal device of any sort, but is in the woodwork everywhere.

   -Weiser (1994)

   The implication of Weiser's quote, the title of Don Norman's The Invisible Computer (Norman, 1998) and the European "Disappearing Computer" initiative (Streitz and Nixon, 2005) implied that computers can vanish from human perception while continuing somehow to exist. The metaphor establishes a powerful image: everyday life stays exactly the same as today, but is simultaneously altered by invisible computing everywhere.

   How do we interact with these invisible computers? Since Weiser's original statement, scientists, engineers, and designers have struggled to implement invisibility. One model is the car. Though contemporary automobiles contain dozens of microprocessors, most drivers will never recognize many of these tiny computers as such. [Footnote: Cars have included multiple computers in the form of dedicated microprocessors since the early 1980s. Yakal (1983) listed five in the 1984 Lincoln Continental alone.] Instead, they just feel the brakes react to road conditions, turn on satellite radio, and cheerfully ignore the automatic gear adjustment.

3. ANIMISM
   Digital devices become animals or people.

   I don't want to argue with my car about where I want to go.

   - Mark Weiser, quoted in The Plenitude (Gold, 2002)

   Any uniquely detectable physical object may become a Passenger.

   Streitz et al. (1999)

   

   One broad definition of animism is the belief that objects have will, intelligence, and memory, and that they interact with and affect our lives in a deliberate, intelligent, and somehow conscious way (Kuniavsky, 2003). Animist metaphors for devices illustrate that we might interact with them as we interact with friends, pets, or pests.

   Virtually every project that emphasizes how devices react to people is in a sense animist. The Nabaztag is a rabbit. Roomba robotic vacuum cleaners were designed to emulate insect behavior (Kurt, 2006). And one widely cited scientific project is described as an "aware home" and "living laboratory" (Kidd et al., 1999).

4. PROSTHETICS
   Ubicomp extends our bodies.

   To me, the primary motivation behind the information appliance is clear: simplicity. Design the tool to fit the task so well that the tool becomes a part of the task, feeling like a natural extension of the person.

   - Norman (1998)

   The computer is a more sophisticated extension of the central nervous system than ordinary electric relays and circuits.

   - McLuhan (1968)

   
   Figure 3-4: Motorola WT4070/90 wearable terminal.
   (Copyright 2008 Motorola)

   Technology attached to human bodies that is not biological acts as a kind of prosthetic (Figure 3-4). In this view, ubiquitous computing replaces human biological functions with silicon ones, or gives people a kind of super power by amplifying their natural senses. The comic book superhero Batman is an archetypal prosthetic technology user. Special crime-fighting technologies enhance Batman's natural abilities, but they do not remove the need to engage with the world directly. Batman fights crime in person, with his hands, instead of sending a remote-controlled robot to do his fighting for him.

   

   From the notorious Lovegety dating device (CNN, 1998) to the adidas_1 running shoe, many wearable computing products want to extend natural senses and physical capabilities. With industrialized countries facing a wave of aging population, cognitive abilities are also up for augmentation as well. The Forget-me-not (Lamming and Flynn, 1994) is one of the earliest computer-based attempts to aid memory.

5. ENCHANTED OBJECTS
   Electronic devices behave like enchanted objects.

   
   Figure 3-5: Ambient Orb by Ambient Devices.
   (Courtesy Ambient Devices)

   Some devices add computational functionality to well-known — if often fictional — objects. By referencing the name, form, or behavior of magical objects in myths and fairy tales, their makers ask users to ignore the technical details and focus on the resulting experience.

   One waves around a Nintendo Wiimote, for example, much like a magic wand, while the Ambient Orb (Figure 3-5) resembles a fortune teller's crystal ball. Accenture's Magic Medicine Cabinet (Wan, 1999) "takes advantage of the situated nature of the medicine cabinet but extends it from a passive storage space into an interactive appliance." Enchantment — the addition of a magical spell to an existing object — implies that the object is mostly like its mundane counterpart, but with key behavioral differences created by information processing technology.

Mixed Metaphors

Sun Microsystems Unveils Open Cloud Platform.

Since explicit metaphor design is not a typical part of experience design, it is not surprising that many, possibly even most, ubicomp projects mix metaphors freely. For example, Starner et al. (1997) described an augmented reality system that acts as a "butler/confidante" that enhances memory by overlaying the results on a video image of the current surroundings projected through a worn video display. Parsing the description carefully, it seems that the system is a human-like prosthetic that presents a parallel universe of personal memory data. Mixed metaphors might not make systems any less functional or valuable in practice, but looking for inconsistencies and contradictions in metaphors does identify potential design issues to resolve. Does the system represent itself as a single butler or perhaps as multiple butler–agents? Is the butler's reasoning expected to appear human? Does the butler disappear when the heads-up display is removed, or could you call him (it?) on the phone?

Tomorrow: Chapter 3, Part 3: Designing with metaphors

Last month I posted a draft of Chapter 1

Citations to references can be found here.

[Note: don't let the theoretical nature of Chapter 1 and this chapter give you the idea that the book is all theory. It's designed to be a practical tool for working designers, but I feel that a theoretical grounding is important to put the practical examples and advice in context.]

Chapter 3: Interaction metaphors

Part 1

No simple analogical model is sufficient to completely explain the operation of a computer system. Computer systems are too different from familiar, everyday non-computational systems.

First of all, we made the screen layout resemble a desktop; displaying pictures of objects you'll have no trouble recognizing. File folders. Clipboards. Even a trash can.

Mapping one category of ideas to another is the basis of linguistic metaphor, which The MIT Encyclopedia of the Cognitive Sciences (Wilson and Kiel, 1999) defined as a "class inclusion assertion." In this definition, a metaphorical comparison implies that two things belong to the same class of objects, even if the two things do not seem to share any directly comparable characteristics.
[Footnote: The MIT Encyclopedia of Cognitive Science (Wilson and Keil, 1999) gave the example of "my lawyer is a shark" as a linguistic metaphor. Metaphors are also not the only way to compare two objects. Marcus (1998) listed a number of non-metaphoric concepts that compare two sets of ideas. His list includes models, analogies, similes, metonymy, and synecdoche.]
Hence lawyers are sharks and love is a flower. Once established, a metaphorical relationship allows reasoning about an unfamiliar concept using what we know about the more familiar one. The details will not match exactly, and terrible analogies regularly surface (how does calling the Internet "a series of tubes" [Footnote: As per former US Senator Ted Steven's infamous June 28, 2006, description.] help us understand e-mail?), but metaphorical comparison can be a basis for reasoning when little is known. Going further, Lakoff and Johnson (1999) argued that cognition is entirely metaphorical. For them, abstract ideas can only be comprehended in terms of metaphors to concrete concepts, even when the metaphorical match is imperfect. For example, they write time is often characterized in terms of money. People talk about how much time something will take, or whether an activity will save time even though it is not possible to stockpile time or to trade it for goods or services as we might with money.

Metaphors have long been part of how we think about design. In 1923, the architect Le Corbusier famously wrote that "a house is a machine for living in." It was a statement crafted to evoke the architectural capabilities of the new materials and engineering techniques of the twentieth century, even as it implied that building dwellers are more like cogs in the machine than operators of it. In the 1970s and early 1980s, another famous metaphorical comparison of life inside buildings to the operation of machines became a key factor in popularizing personal computing.

Before the late 1970s and early 1980s, computers were largely controlled by typing words into command-line and "glass teletype" interfaces. Xerox PARC's office desk metaphor, popularized by Apple's Macintosh computer, made many capabilities of the computer system (such as storing files in hierarchical directories) accessible through representations of familiar office objects (such as folders). This mapping proved very successful. Even though the onscreen objects did not behave exactly like their real-world counterparts -- no one keeps a stack of glass windows on the top of their office desk, for example -- evocations of familiar objects were more helpful than a command-line-based operating system in acquainting a larger audience with the capabilities of their personal computers. A machine, it seems, could be a house for living in. [Footnote: Thanks to Liz Goodman for this observation.]

The position of ubiquitous computing resembles that of computing before the 1970s: we have a palette of new technologies without metaphors to communicate their power or operation. If Lakoff and Johnson are correct, and reasoning is largely metaphorical, then people encountering these unfamiliar technologies will always rely on metaphors to interpret them. From a design perspective, then, it makes sense to discuss the kinds of associations new technologies could or should prompt. And it also makes sense to identify the metaphors designers consciously or unthinkingly employ. Just as it may be impossible to reason without metaphors, it also may be impossible to design without them. Design requires using tools skillfully, and metaphors are the tools of thought.

[Note: I am deeply indebted to John Lawler, whose "Metaphors we compute by" (1987) class at the University of Michigan's Residential College was fundamental to my thinking about how people interact with computers. I also thank Bruce Sterling for his review of an early draft of this chapter, and to Dan Saffer for giving me permission to reprint his guidelines for design with metaphor.]

Tomorrow: Chapter 3, Part 2: A Catalog of Ubicomp UX Metaphors

In the interview I talk about the book (of course) and ThingM's upcoming products. I also took the opportunity to think about how I've noticed the trend of services that provide data streams, rather than just units of data:

I think that there’s a really interesting trend in opening up data sources. Pachube works as a free data stream brokerage that sits on top of TCP/IP and HTTP to provide a kind of semantic resource location technology for small net-enabled devices that has been missing. This kind of data openness is being matched by things such as the US Governement’s open data initiative at data.gov.

The trend I see here is a combination of openly sharing data sources and streams and creating business models around making technology layers that make those data streams meaningful and valuable. Both Pachube and data.gov are a kind of search engine for data streams, rather than documents, which I think is a very powerful concept.

This is definitely related to the discussions around syndication that have been going on for years (since the launch of RSS), to micro-content, and to various services that add structured semantic information to Web-accessible data. However, I think what we're seeing now goes beyond those largely abstract discussions to create a more pragmatic understanding of what it means to create meaningful sources of data, rather than just meaningful units of data.

It means, as my last sentence implies, that there are enough data sources--whether it's sensor data automatically collected, organized and tagged by Pachube or the human-created sources of data presented by data.gov--that we can start having search services for such data. The conversation becomes again about "wrangling" information shadows, as I discussed in my NASIG keynote two years ago.

In that discussion I talked about how journal subscriptions--which are a kind of knowledge white hole, wellsprings of specific kinds of information--represent a model for how information shadows can be organized and managed in the future. Well, it looks like we may be closer to that, and that the wrangling may be a combination of automated tagging and human curation.

Does this mean that Google will soon be automatically cataloging data streams? I'd be surprised if they're not already.