As part of the Stanford computer science major, all seniors have to spend 10 weeks building a significantly sized software project for their senior capstone. We would be evaluated on our ability to work cohesively as a team (peer reviews), individual code contributions (github review), and the project’s level of execution and completion (project fair).
Our team of 4 built the software and assembled the hardware behind Lumi, a gesture-controlled smart mirror that plays music, takes photos, and displays glanceable information like the time, the weather, and your twitter feed.
IoT, Interaction, Web Development
UI/UX Design, Visual Design, Front End Development
March – June 2016
We began by taking note of our daily rituals to identify potential pain points. Our most frequent interactions with mirrors were during our morning routines of getting dressed and brushing our teeth. During these moments, we’d often be fiddling with our phones, switching between a handful of apps to check the weather, our day’s schedule, and our social media feeds. Our screens would inevitably get covered in a mixture of water and toothpaste as we brushed our teeth or we’d awkwardly perform a one-armed balancing act while checking email as we got dressed. Observing friends, we noticed many of our female peers switching between a mirror and a laptop screen to view makeup tutorials, leaving their keyboards covered in eyeshadow as they frustratedly clicked pause and rewind.
Brainstorming, we came up with smart mirror features that might be able to alleviate these pain points or provide value to the user: A calendar view to see the day’s activities at a glance, a radio player to hear the news, a countdown timer for brushing your teeth, a camera to record a daily photo journal, a Twitter feed to keep up with friends, a video player to see makeup tutorials…
We created a cohesive list of features we wanted to implement:
Once we had the list of features we wanted to build, we needed to figure out how users would interact with the mirror. Early on, we agreed that our augmentations should not impinge on the seamless experience of using a regular mirror.
From our research, we found that the most common method of interacting with smart mirrors was through a touch interface. There were pros and cons to this design.
Touch Interface Pros:
Touch Interface Cons:
While touch would certainly be the most reliable way to interact with a smart mirror, given our priorities and use cases, touch wasn’t appropriate as it greatly hinders the mirror’s ability to serve as a normal mirror. Further, while touch is a great input mechanism for primary devices like smart phones and tablets where a high level of granularity and accuracy is required, a smart mirror isn’t meant to be a primary device, making touch input overkill.
Weighing the pros and cons of a gesture controlled interface with our mirror’s use cases, it was clear that gesture control was the best direction to go in:
Gesture Control Pros:
Gesture Control Cons:
While gesture input would be less accurate and granular than touch, this level of sophistication was appropriate for the smaller type of role we thought a smart mirror should play in a user’s life.
To build the mirror display, we purchased a 34” 2K monitor and affixed a two-way reflective piece of acrylic to the front (purchased from TAP Plastics). The two-way acrylic reflects light on the front but allows light to pass through the back. This allows the smart mirror to work as a fully functioning, reflective mirror when the monitor is asleep.
When content is displayed on the monitor, it passes through the two-way acrylic and appears on the surface of the mirror.
To capture gesture input, we used an Xbox 360 Kinect sensor that we placed at the bottom of the mirror and tilted upwards for optimal gesture capture. To run the display and process gesture input, we had a computer running behind the monitor. Lastly, we mounted speakers behind the display for audio output.
Our primary design challenge was creating a set of interactions for controlling the mirror that would be intuitive to the user, intentional to the Kinect, and powerful enough to navigate a UI. Because we weren’t familiar yet with the technical limitations of the Kinect, our process became highly iterative: describe a gesture, sketch a resulting UI, prototype the gesture on the Kinect, repeat.
The set of functions our interaction system needed to be able to handle included:
Inspired by PCs, our first designs employed a cursor controlled by the user’s arm to navigate the mirror’s interface. The user simply raises her arm to chest height to “grab” the cursor. The cursor would then follow her hand as she moved it around. This turned out to be a poor design for several reasons. Firstly, tracking the hand and mapping it to a precise x, y cursor coordinate was inaccurate and unreliable. Secondly, capturing and then processing a gesture takes a non-trivial amount of time, resulting in a significant delay between the movement of the hand and the movement of the cursor on the mirror.
In our next iterations, we focused on designing a system that would mask this sluggishness by requiring less granular gestures. Instead of precises x, y coordinates, we divided the screen into a grid of areas that could be selected as the user moved her hand to different areas on the screen.
While this improved consistency and accuracy, especially in the middle of the screen, near the edges of the screen, this system often failed to capture the gesture, making it too frustrating to use in a daily routine.
Ultimately, we realized that we were trying to make the mirror do too much. Day to day, the user wouldn’t need to set the widgets’ permissions, customize the widgets’ locations or add/remove widgets – these were all system settings that the user would likely only need to set once. We moved all of these tasks into a separate web app that the user could set on her laptop or mobile phone and then forget about. The mirror would only need to handle launching widgets.
Our final solution was a clever radial menu interaction that appears in the center of the mirror when the user raises her arm out to chest height. From there, the user moves her hand in any one direction to highlight an option. Once an option is highlighted, she simply holds for two seconds and the option is selected.
This system worked extremely well as it could only be activated from a highly specific, intentional movement and was very forgiving in that it required little granularity and focus in movement to accurately select a desired option. Further, it greatly improved perceived responsiveness and felt snappy compared with previous iterations.
However, a tradeoff with this design was the number of apps it could handle. We found that it worked best with 5 apps, any more than that and the Kinect has a harder time differentiating options.
Along with our interaction system, our corresponding UI went through several iterations as well. In early iterations, we envisioned having live app cards, similar to Microsoft Windows Phone live tiles. These would allow the user to glean useful summary information at a glance. Because we wanted to maximize the amount of reflective mirror surface, we hid these app cards behind a menu button on the bottom left. Hovering over this button revealed the app cards that users could scroll through by swiping their hand. As we learned more about the limitations of the Kinect’s gesture recognition capabilities, we realized this was actually a poor design.
In these early iterations, we didn’t have a good grasp on appropriate use cases for a smart mirror. We viewed it just like a larger tablet with the ability to launch media rich applications like Facebook and ESPN. However, as we talked to more users and explored the technical limitations of our hardware, we realized that while we could implement those applications, they weren’t appropriate for a smart mirror and would lead to a very poor user experience as they required a very accurate and granular method of input. Further, hiding the app cards behind a menu was simply a bad user experience as it required too many steps to access information that should already be available at a glance.
Taking these learnings, we adjusted our UI to have the widgets already displayed along the unused edges of the mirror. We went through a variety of iterations and visual treatments to find designs that were glanceable and felt lightweight visually.
We knew iconography was going to be extremely useful in keeping our user interface lightweight yet intuitive. To maintain visual cohesiveness, we designed a custom set of icons.
For gesture recognition, we utilized the PyKinect framework and built a separate Python server to process and handle the input and feed it to the Node.js server.
At the project fair, Lumi’s gesture recognition and radial menu design performed remarkably well with a variety of user heights and under a range of lighting conditions. Feedback from users and faculty were overwhelmingly positive. By popular vote, Lumi won "Best Senior Capstone Project".
Built in collaboration with Dan Guo, Brian Yang, and Sloane Sturzenegger (from left to right).