Interaction

Lumi Mirror

Senior Capstone

10 Weeks to Build a Real World Software Project

As part of the Stanford computer science major, all seniors have to spend 10 weeks building a significantly sized software project for their senior capstone. We would be evaluated on our ability to work cohesively as a team (peer reviews), individual code contributions (github review), and the project’s level of execution and completion (project fair).

Our team of 4 built the software and assembled the hardware behind Lumi, a gesture-controlled smart mirror that plays music, takes photos, and displays glanceable information like the time, the weather, and your twitter feed.

Project

IoT, Interaction, Web Development

Role

UI/UX Design, Visual Design, Front End Development

Timeline

March – June 2016

User Pain Points

What Problems Might a Smart Mirror Solve?

We began by taking note of our daily rituals to identify potential pain points. Our most frequent interactions with mirrors were during our morning routines of getting dressed and brushing our teeth. During these moments, we’d often be fiddling with our phones, switching between a handful of apps to check the weather, our day’s schedule, and our social media feeds. Our screens would inevitably get covered in a mixture of water and toothpaste as we brushed our teeth or we’d awkwardly perform a one-armed balancing act while checking email as we got dressed. Observing friends, we noticed many of our female peers switching between a mirror and a laptop screen to view makeup tutorials, leaving their keyboards covered in eyeshadow as they frustratedly clicked pause and rewind.

Brainstorming, we came up with smart mirror features that might be able to alleviate these pain points or provide value to the user: A calendar view to see the day’s activities at a glance, a radio player to hear the news, a countdown timer for brushing your teeth, a camera to record a daily photo journal, a Twitter feed to keep up with friends, a video player to see makeup tutorials…

We created a cohesive list of features we wanted to implement:

  1. Clock
  2. Countdown Timer
  3. Weather View
  4. Twitter Feed
  5. Email Feed
  6. Stock Market Feed
  7. Daily Photo Journal
  8. NPR Radio Player
  9. Spotify Music Player
  10. YouTube Video Player

Touch vs. Gesture

Choosing an Interaction Method

Once we had the list of features we wanted to build, we needed to figure out how users would interact with the mirror. Early on, we agreed that our augmentations should not impinge on the seamless experience of using a regular mirror.

From our research, we found that the most common method of interacting with smart mirrors was through a touch interface. There were pros and cons to this design.

Touch Interface Pros:

  1. Thanks to Apple, touch is already a commonly used interaction method — most users are educated with how it works and UI/UX patterns are established.
  2. With the currently available technology, touch arguably provides the most accurate and reliable method of interacting with a smart device.

Touch Interface Cons:

  1. Touch requires the user to be within arms distance of the device, greatly limiting a smart mirror’s ability to be used at a distance as a full-body mirror in addition to a close-up mirror.
  2. Touch causes the user to leave fingerprints and potential other debris on the reflective glass, impinging on the mirrors primary function as a reflective surface.
  3. The hardware required to support responsive and accurate touch interactions (not to mention multi-touch) for a device as large as a mirror is expensive.

While touch would certainly be the most reliable way to interact with a smart mirror, given our priorities and use cases, touch wasn’t appropriate as it greatly hinders the mirror’s ability to serve as a normal mirror. Further, while touch is a great input mechanism for primary devices like smart phones and tablets where a high level of granularity and accuracy is required, a smart mirror isn’t meant to be a primary device, making touch input overkill.

Weighing the pros and cons of a gesture controlled interface with our mirror’s use cases, it was clear that gesture control was the best direction to go in:

Gesture Control Pros:

  1. Gesture control enables the smart mirror to be interacted with at a much wider range of distances from the mirror.
  2. Gesture control leaves the reflective surface clear of fingerprints and other debris, allowing the smart mirror to perform its primary function just as well as a regular mirror.
  3. Sensors for gesture control (e.g. Kinect) are widely available and affordable.

Gesture Control Cons:

  1. Gesture control is not a commonly used interaction method — some users may not be familiar with how it works and UI/UX patterns are not as well established.
  2. The current state of gesture control technology is far from perfect. Input is not as granular nor accurate as touch.

While gesture input would be less accurate and granular than touch, this level of sophistication was appropriate for the smaller type of role we thought a smart mirror should play in a user’s life.

Building Lumi

Software Engineers Build Hardware

To build the mirror display, we purchased a 34” 2K monitor and affixed a two-way reflective piece of acrylic to the front (purchased from TAP Plastics). The two-way acrylic reflects light on the front but allows light to pass through the back. This allows the smart mirror to work as a fully functioning, reflective mirror when the monitor is asleep.

Lumi Mirror Construction One
Lumi Mirror Construction Two
Lumi Mirror Reflection One
Lumi Mirror Reflection Two

When content is displayed on the monitor, it passes through the two-way acrylic and appears on the surface of the mirror.

Lumi Mirror UI Plus Reflection

To capture gesture input, we used an Xbox 360 Kinect sensor that we placed at the bottom of the mirror and tilted upwards for optimal gesture capture. To run the display and process gesture input, we had a computer running behind the monitor. Lastly, we mounted speakers behind the display for audio output.

Lumi Kinnect

Primary Design Challenge

Our primary design challenge was creating a set of interactions for controlling the mirror that would be intuitive to the user, intentional to the Kinect, and powerful enough to navigate a UI. Because we weren’t familiar yet with the technical limitations of the Kinect, our process became highly iterative: describe a gesture, sketch a resulting UI, prototype the gesture on the Kinect, repeat.

The set of functions our interaction system needed to be able to handle included:

  1. Adding/removing widgets
  2. Initializing widgets (i.e. permissions, onboarding, etc.)
  3. Customizing widget locations
  4. Launching widgets (e.g. playing music, stopping music)

Cursor Method

Grid Method

Inspired by PCs, our first designs employed a cursor controlled by the user’s arm to navigate the mirror’s interface. The user simply raises her arm to chest height to “grab” the cursor. The cursor would then follow her hand as she moved it around. This turned out to be a poor design for several reasons. Firstly, tracking the hand and mapping it to a precise x, y cursor coordinate was inaccurate and unreliable. Secondly, capturing and then processing a gesture takes a non-trivial amount of time, resulting in a significant delay between the movement of the hand and the movement of the cursor on the mirror.

In our next iterations, we focused on designing a system that would mask this sluggishness by requiring less granular gestures. Instead of precises x, y coordinates, we divided the screen into a grid of areas that could be selected as the user moved her hand to different areas on the screen.

While this improved consistency and accuracy, especially in the middle of the screen, near the edges of the screen, this system often failed to capture the gesture, making it too frustrating to use in a daily routine.

Ultimately, we realized that we were trying to make the mirror do too much. Day to day, the user wouldn’t need to set the widgets’ permissions, customize the widgets’ locations or add/remove widgets – these were all system settings that the user would likely only need to set once. We moved all of these tasks into a separate web app that the user could set on her laptop or mobile phone and then forget about. The mirror would only need to handle launching widgets.

Our final solution was a clever radial menu interaction that appears in the center of the mirror when the user raises her arm out to chest height. From there, the user moves her hand in any one direction to highlight an option. Once an option is highlighted, she simply holds for two seconds and the option is selected.

This system worked extremely well as it could only be activated from a highly specific, intentional movement and was very forgiving in that it required little granularity and focus in movement to accurately select a desired option. Further, it greatly improved perceived responsiveness and felt snappy compared with previous iterations.

However, a tradeoff with this design was the number of apps it could handle. We found that it worked best with 5 apps, any more than that and the Kinect has a harder time differentiating options.

Lumi Mirror UI Plus Reflection
Lumi Mirror UI Plus Reflection

UI & Visual Design

Along with our interaction system, our corresponding UI went through several iterations as well. In early iterations, we envisioned having live app cards, similar to Microsoft Windows Phone live tiles. These would allow the user to glean useful summary information at a glance. Because we wanted to maximize the amount of reflective mirror surface, we hid these app cards behind a menu button on the bottom left. Hovering over this button revealed the app cards that users could scroll through by swiping their hand. As we learned more about the limitations of the Kinect’s gesture recognition capabilities, we realized this was actually a poor design.

Mail Iterations
FB Iterations
ESPN Iterations
Lumi Mirror Iteration 1 Passive
Lumi Mirror Iteration 1 Approach

In these early iterations, we didn’t have a good grasp on appropriate use cases for a smart mirror. We viewed it just like a larger tablet with the ability to launch media rich applications like Facebook and ESPN. However, as we talked to more users and explored the technical limitations of our hardware, we realized that while we could implement those applications, they weren’t appropriate for a smart mirror and would lead to a very poor user experience as they required a very accurate and granular method of input. Further, hiding the app cards behind a menu was simply a bad user experience as it required too many steps to access information that should already be available at a glance.

Taking these learnings, we adjusted our UI to have the widgets already displayed along the unused edges of the mirror. We went through a variety of iterations and visual treatments to find designs that were glanceable and felt lightweight visually.

Time Final Version Time Iterations
Stock Ticker Final Version Stock Ticker Iterations
Mail Feed Final Version Mail Feed Iterations
NPR Radio Final Version NPR Radio Iterations
Calendar Final Version Calendar Iterations
Timer Final Version Timer Iterations

We knew iconography was going to be extremely useful in keeping our user interface lightweight yet intuitive. To maintain visual cohesiveness, we designed a custom set of icons.

Widget Icons
Weather Icons
Sunny Closeup
Partly Cloudy Closeup
Final Mockup

The Software

PyKinect, Node.js, and a Python Web Server

We built the mirror's software using Node.js since the majority of our widgets (e.g. Spotify music player, weather, stock, email etc) provided web APIs. The monitor simply displays a full sized web browser that loads a webpage connected to a local Node server running on our laptop. The UI elements were built and animated with HTML, CSS, and Javascript. In addition to the mirror GUI, we built an additional web service that enables users to customize the location of widgets on the mirror display through a drag and drop interface as well as manage permissions for the widgets (e.g. Twitter, Spotify, email, etc).

For gesture recognition, we utilized the PyKinect framework and built a separate Python server to process and handle the input and feed it to the Node.js server.

Weather Widget
Twitter Widget
Lumi

Results

Project Fair

At the project fair, Lumi’s gesture recognition and radial menu design performed remarkably well with a variety of user heights and under a range of lighting conditions. Feedback from users and faculty were overwhelmingly positive. By popular vote, Lumi won "Best Senior Capstone Project".

Project Fair