“Dance With Your Hands” With Tap Tap Revolution!
Meet our team! Dan Kershner, Katherine Lordi, Samantha Nolle, and Jordan Sechler
Introducing… Tap Tap Revolution! A Spin-off of Dance Dance Revolution
Thinking back to fun childhood memories spent at the arcade playing games and winning tickets in exchange for prizes, Dance Dance Revolution (DDR) was always one of the largest games in the entire arcade that required the most physical activity.
DDR is fun to watch (and play!) because it requires rhythm, coordination, dancing ability, and willingness to potentially look silly in front of strangers. Keeping bodily expression and fun as the primary design goal for this design sprint, our team decided to recreate an online version of the beloved DDR game called “Flash Flash Revolution” that uses the basic arrow keys. In order to make the game expressive using a 3D UI, we enabled users to “dance with their hands” through the use of Python code and visual detection of the user interface using a webcam. In turn, “Tap Tap Revolution”, our rendition of the game, was born.
Our Design Process
Choosing the Game… Trial and Fun!
Before settling on Tap Tap Revolution for our game, we brainstormed a handful of fun and popular web games that we liked from our childhoods, and brainstormed ideas for the 3D UI by trying gestures ourselves through “Wizard of Oz” style testing. In Wizard of Oz testing, developers define a set of actions for game input for the participant, and the participant uses those actions to interact with the game. However, meanwhile, one developer acts as the controller.
While brainstorming, we were careful to focus on games that were doable in the context of the tools we had available to us, namely the Leap Motion Controllers and laptop webcams. We wanted to choose a game that not only worked easily with these forms of technology, but that would also be intuitive and easy to learn for first time users.
Most importantly, we wanted a game that was fun and engaging (emphasizing fun and expression). We imagined that “engaging” could come in many forms, such as nostalgia, movement, colorful or exciting visuals, a fast pace, and/or use of music or auditory elements. Inspired by “Making Things Engaging”, we sought important elements of engaging products including beauty of interaction, enjoyment of the experience, rich actions, and doing rather than thinking. From there, we sought to settle on a game that incorporated these aspects of an engaging experience, doability using the tools available to us, and intuitiveness for users.
Idea #1 Slither.io
The first web game we considered was Slither.io, a game in which a user controls the continuous motion of a colorful snake to “eat” orbs and not get absorbed by other snakes. We felt that this game would integrate easily with the capabilities of the Leap Motion in the sense that the snake may be moved to the left or right or with a boost of speed. We envisioned a user moving their flattened palm to the left or the right to control direction, and pinching or unpinching their fingers to control speed. We imagined that these motions would need to be explained to a user, but since there were only three variations to remember, the learning process would be simple. Additionally, we felt that the element of nostalgia for past users, as well as colorful visuals and movements (including variation of speed), would all contribute to an engaging experience for users.
After “Wizard of Oz”-ing the Slither.io idea for Leap Motion, we decided to think about games involving more interactive hand gestures to control characters.
Idea #2 Frogger
The second web game we brainstormed was Frogger, a game in which the user controls the forward motion of a frog across a game board with moving tiles and obstacles. We felt that the objective of this game could be integrated with the capabilities of the leap motion through the clenching and unclenching of a fist to control forward motion of the frog. We imagined that like Slither.io, these motions would also need to be explained to a user, but as they were few in number, they would not be overwhelming for a user to remember. Furthermore, we felt that Frogger also contributed to a feeling of nostalgia for users, as well as the excitement of both moving game pieces and the movement of the frog.
After “Wizard of Oz”-ing the idea for Frogger using Leap Motion, one reoccurring issue encountered was the quickness required to move the frog. Therefore, test users were a little overwhelmed with the speed of the game. However, we noticed that users quickly picked up the mapping of frog motions to hand motions with little instruction.
Chosen Idea: Flash Flash Revolution
Our team thought of DDR (Dance Dance Revolution) because we found a version of it online called Flash Flash Revolution (FFR). We liked this game for a number of reasons. In addition to being familiar and nostalgia provoking like Frogger, the element of music and musical coordination made FFR fun and engaging to play. By trying the hand gestures ourselves through Wizard of Oz testing and creating life size arrow keys on paper that corresponded to the arrow keys on the keyboard, we found that not only did we have to move our hands to match the screen, but also to match the rhythm of the music. This game combined the engagement of hand gestures with the excitement of music through coordination of hands and song. Additionally, unlike the other games we considered, two hands are needed to play rather than just one!
Andrew Ko’s chapter on hands and discussion about Hand Tracking had great points that set the FFR idea apart from the others. Hand tracking systems work well with tasks in 3D that require pointing and spatial manipulation (such as dancing and hitting arrow keys) because “they treat the hand as a continuous input device and estimate the hand’s real-time position and orientation.” In our hand tracking system, the webcam and color recognition techniques would help track when a key is pressed in the FFR game. Even though hand tracking systems introduce gulfs of evaluation and gulfs of execution, according to Ko, the severity of these gulf may be lower. In addition to the fun of the FFR game, this was another motivation in choosing it for our final product.
Formative Testing: “Wizard-Of-Oz”ing Tap Tap Revolution
By means of “Wizard of Oz” testing of our Tap Tap Revolution idea with other students, we were able to see if the tapping motions that we planned for the game were indeed fun, expressive, and intuitive for users for the design of the 3D UI.
Following the completion of some wizard of oz testing for Tap Tap Revolution, we saw enjoyment and ease of use by users in the following ways:
- Users enjoyed having a precise location to “tap” physically.
- Users became more and more comfortable the more they “played.”
- Users found the mapping of hands keys to arrow keys and music intuitive and amusing, especially when many have tried DDR or another alike version of the game before.
Building Tap Tap Revolution: Python and Vision Detection via Webcam
We began coding a prototype in p5.js to see if game implementation was even possible. It worked by using the webcam to identify four specific pixels on the screen and examine the color of each. When the color of one of these pixels changed, which would happen when a user covered it with their hand, it would register that change and then create input to Flash Flash Revolution (FFR). However, we quickly discovered that p5.js had no easy way to create input for a browser game. This caused us to switch to python, which was actually very easy due to a few libraries we imported. It worked much the same as the original prototype, and could now send key presses to the game. This allowed us to map each of the four pixels to an arrow key, so that when the user covered a pixel, the proper arrow key that it represented would be pressed and given as input to FFR.
This seemed to work, but we were getting lots of false positives because the color of a single pixel was very unreliable. To fix this, we took the average color of a circle with a radius of 64 pixels, rather than the color of a single pixel. This gave us much more reliable results and virtually no false positives. To improve user experience, we placed white sticky notes at the location of each of these circles, so all they had to do was cover one of these sticky notes with their hand to simulate pressing an arrow key.
User Testing Post-Development
Once we had a working game, we had several people test it and give us feedback. Some people who tested the game for us had trouble with the controls because they would let their hands hover over the sticky notes, which caused false positives. However, once we explained how it worked with the webcam they made sure to keep their hands back until they wanted to trigger an input. Once they figured out how it works and got the hang of it, most users really enjoyed the game. They told us it was fun to play a game familiar to them from their childhood with totally new controls.
Some users told us that while it was fun and a cool new way to play the game, it was not a perfect input method because it was more difficult than the traditional way of just using the arrow keys on the keyboard. For many of the harder songs available to play, it was near impossible to keep up with our new control method. However, since the point of this project was to create a fun new input method to the game, rather than one that made it easier to play, we decided to stick with our implementation because the users were having fun on the easier songs. It may not be the ideal input method for playing this game well, but it definitely seems to be a fun one.
Demo Day & User Feedback
The demo day went very well, and most people who tried our game thoroughly enjoyed it and had fun with it! The feedback that we received showed us that most people loved the game and controls we implemented. They said it was fun to play, very intuitive to control, matched their motions well, and was simple overall. The 3D user input also better connected to the real version of the Dance Dance Revolution game that evokes nostalgia among users.
People also gave many suggestions for improvement, revolving around the controls and the game. Most people wanted the controls to be more precise and reliable. It wasn’t too easy to quickly cover the right spot on the sticky notes to activate an input, which caused the game to be more difficult than it would be with normal controls. Some people also wanted more controls and interactivity, such as a pause button or the ability to choose other songs. Currently the only way to do these things is to use the actual computer with the real controls, not our 3D input. Implementing a few more controls and tuning the current inputs to be more precise could definitely improve our game for future iterations.
Conclusion: Designing for Fun/Expression
Through designing for fun and expression in this design sprint, our team implemented a 3D UI and designed an interface that prioritizes bodily expression and fun. “Tap Tap Revolution” creates a fun, new way to play the beloved game Dance Dance Revolution that allows people to “dance with their hands” and follow the rhythm of the music and hit the arrows to earn points. The intuitive and expressive hand motions along with a variety of song options and difficulty levels also brings out players’ competitive spirit. Our team’s goal was to create a game that provides a fun user experience. Through brainstorming, wizard of oz testing, and user testing after implementation, we met our goal and identified ideas for future iterations of “Tap Tap Revolution.”
Find the source code here!
