Tools for Designing Smart, Collaborative Things

With the ability to quickly create working AI prototypes that intelligently move, see, hear and speak, designers can go beyond the cliches of AI, and bring collaborative systems into people’s lives that are more humane, more personal, maybe even more inspiring.

Research Fellowship in Delft, Netherlands

In Fall of 2017, I was in residence at TU Delft on a research fellowship to develop concepts and prototypes of a tool for designing smart, collaborative things. Funded by Design United, I worked with Professor Elisa Giaccardi (department of Industrial Design Engineering) to start developing the Delft AI Toolkit visual programming environment for the interaction design of AI.

A Visual Design Tool + ML & AR for the IxD of AI

This project builds on my prior research (NETLab Toolkit and Animistic Design), and will ultimately allow designers to easily and iteratively shape the perceived personality, character, and behavior of an AI product. 

The tool provides a drag-and-drop, visual programming environment to build the interaction and behavior of an AI system. In addition to building the logic of the application, the tool allows the designer to incrementally prototype the AI, form, and behavior. First by simulating the AI, before committing to the training and implementation of an actual machine learning algorithm. Second, by simulating the physical form and behavior using augmented reality, before committing to actual hardware.

Simulate, then implement AI

To support the simulation of the AI, the design tool will have a “personality control panel” to shape the system’s behavior and ml models driving its behavior. It will also have a visual marionette system so the designer can “wizard-of-oz” AI behaviors in real time while observing users. The tool will then allow the implementation of these approaches to AI in a functional machine learning system, such as a small robot with and onboard machine learning engine, camera, microphone, and motors.

Simulate, then implement hardware

I’ll also be exploring the potentials of using augmented reality as part of the designer’s process. Creating working, physical prototypes is critical to the design process, but it can be time consuming and expensive.

The tool will ultimately support augmented reality simulations of the AI project, so a designer can quickly test and update the form/behavior of an embodied experience without the complexities of electronics and mechanics. The designer or user will “see” and interact with the AI system as an object in real space through AR. Then the tool can deploy the same simulated virtual behaviors into a physical prototype.

Stay Up to Date

Read more about my ideas and this project in Design United’s interview. For the latest version, see the releases page and documentation.