Over the past two years, there has been a tremendous amount of focus on what we see in the VR world through products like the Oculus Rift, or how we move and interact with the Virtuix Omni treadmill or the upgraded Sixsense STEM System. But where is the feeling? The haptics? That nudge that tells you you aren't dreaming?
Today, MTBS is proud to bring you our interview with Dr. William Provancher, Founder of Tactical Haptics and Associate Professor at the University of Utah. William will talk about the benefits of haptics, big challenges that need to be overcome, and where things are headed with his new device!
MTBS: Welcome to MTBS! Before we talk about your latest invention, tell us about your work at the University of Utah. What do you teach, and what skills do students walk away with?
WP: For eight years, I've been teaching Mechanical Engineering Senior Capstone Design, Electro-mechanical Design (Mechatronics), and Haptics (The study of touch sensing and feedback).
I've always been interested in creating and building new things and grew up in a house where my dad had a machine shop and wood shop in the basement. This guided my path through school and led me to work as a structures and mechanisms designer at Lockheed Martin Missiles and Space for five years, as well as my current interests in creating novel haptic feedback devices.
So I bring that creative element to all of the above courses, which are all project-based classes – in addition to teaching the fundamentals of their associated topics. This means that whether students are learning about design methodology, electronics, programming, human neurophysiology, perception, device design, or controls, students learn and apply this knowledge to a project of their own. They are great courses that enable students to be productive in their future research or work.
MTBS: Let's talk a little bit about haptics. What are haptics and why are they important?
WP: Haptics are the study of human touch sensing and feedback. It's the nexus of neurophysiology, perception/psychology, mechanical engineering, electrical engineering, computer science and a number of other sub-disciplines. And the key is that in order to be a successful haptics researcher or designer, one needs to know at least a little about many of these.
In modern technology, two of the reasons people are interested in haptics include: 1) that it provides a greater level of immersion in gaming and VR, and 2) it can be a possible means to bypass the glut of visual and auditory information we receive. It's like having a special, separate channel of communication for communicating with the brain! This second point is the point of active research by psychologists.
MTBS: There have been some excellent haptics devices on the market before. For consumers, the Novint Falcon is my personal favorite, though it never really took off. Are there physical challenges with making a good haptics device? Are there roadblocks that get in the way of completing the experience?
WP: The greatest challenge with designing a haptic device for consumers, like the Novint Falcon, is that the device must have a relatively high fidelity design in order to present forces without also having other unintended artifacts, such as vibrations or clicking or noise from the device transmission. And to do this well costs money. Unfortunately, currently it doesn't appear that most consumers place a high enough value on the added utility provided by these more expensive haptic interfaces for them to be commercially viable in many cases. And in the field of gaming, where the Novint Falcon was targeted, there is another challenge which is consumers are less likely to buy a new haptic device if there isn't enough content or games already designed for it. Conversely, developers are reluctant to develop new games for devices that don't have a huge install base. So hence there is a chicken and egg scenario that is challenging.
MTBS: You're taking a big leap of going from university professor to entrepreneur. You're obviously convinced that you are on to something. Have you always been interested in VR, or is this a recent development? What sparked your interest, and how did this come about?
WP: I've generally been interested in VR, but with a specific focus on creating new haptic devices. More recently I participated in a US National Science Foundation (NSF) program called I-Corps, which is meant for helping Principal Investigators (PIs) of prior NSF research understand the commercial potential of their research. A big part of the I-Corps program is the "customer discovery" process, where we discovered that the people we were talking to were very into the "experiences" we could create in virtual environments with our haptic devices.
Around this same time, along came the Oculus Rift, and it just made sense that our technology would make the perfect marriage with the Rift and the Hydra (now the Sixense STEM System).
MTBS: How does the Tactical Haptics controller work? What sensations is it best at translating?
WP: Reactive Grip™ touch feedback can be used to convey motion and force information using tactile feedback integrated into the handle of a device. The device utilizes sliding contactor plates in the handle. Translational motions and forces can be portrayed along the length of the handle by moving the plates in unison in the corresponding direction; whereas moving opposing slider plates in opposite directions creates the feeling of the device's handle wrenching within the user's grasp. Because of its ability to recreate the skin sensations of actually holding the in-game object, this type of feedback creates an engaging experience that brings gaming interactions to life. For example, you can feel the resistance of striking an opponent with a sword, the kick of a gun, the stretching of an elastic object like a slingshot or bow, or shifting weight as you swing a flail in the air. We have named this feedback technology "Reactive Grip" Tactile Feedback, since the device responds or reacts to the user's motions and actions.
This technology is compatible with motion-based controller interfaces, such as the Nintendo Wii, Sony Move, or Microsoft Kinect, and could also make a formidable pairing with a head mounted display for unprecedented levels of immersion and realism in a virtual reality gaming environment. The current game controller prototype uses a Razer Hydra as its motion tracking system, but our tactile feedback technology is not tracker dependent.
MTBS: You've been showcasing different prototypes as you go from conference to conference. How have people been reacting to your invention? What improvements have you been making along the way?
WP: People are typically quite shocked at how engaging and realistic our touch feedback is, as can be seen in the above reactions from GDC2013.
The main improvements we've made since the device that we showed at GDC2013 (above left) include: significantly reduced size and weight, handle ergonomics, and ergonomically integrated trigger, thumb-stick, and buttons. The improved device is shown on the right.
MTBS: I know this is apples to oranges; the Novint Falcon is a product that has a push/pull sensation and it has the advantage of being stabilized on a table. Yours is using the handle to express haptics feedback. Why did you choose to have it as a baseless controller? What purpose were you trying to fill that a table-based controller wouldn't do as well?
WP: My technology produces force-torque-like sensations with lower cost and device complexity than force feedback devices, and without the small workspace limitations that are inherent to grounded force feedback devices.
Hence, my technology may, when mass produced, come in at a price point that consumers may be willing to pay, while overcoming the issue of the small workspace of devices like the Falcon.
MTBS: You haven't launched a Kickstarter yet. How have you been funding this effort? Who has been responsible for creating the prototypes and software support?
WP: This technology has come out of research conducted in my university lab that has been supported by NSF, including the NSF I-Corps grant I mentioned above.
MTBS: While the concept of the hardware seems simple enough, how would you describe the software? How do you go about translating a virtual world to something the dynamic handle will understand? Is there a basic algorithm involved?
WP: Our software works in combination with the physics engine that is part of Unity. We use the physics engine from Unity to calculate the forces and torques that would be felt at the "virtual hand". We then translate the forces into proportional motion of the sliding plates in the handle of the device along the length of the handle. And we move the sliding plates in the handle in opposite directions along the length of the handle to represent the computed torques from Unity. When both forces and torque are present these sliding plate motions are added together and superimposed.
See starting at 0:16 for an explanation of applied forces and torques with our controller:
MTBS: Video games seem to be the lowest hanging VR fruit right now. Is Tactical Haptics best suited for video games, or do you see a brighter future in the more professional markets. Why?
WP: Yes, our tech is suited to both VR and mainstream gaming, as we can do both well and can be inexpensive enough to be viable in mainstream video games.
In addition, there are also great higher-end applications, such as to enhance the touch feedback of a surgeon using a minimally invasive or robotic surgical system, or could be a more compelling interface for upper limb rehabilitation. We also see a good match with VR interfaces such as zSpace or AR interfaces such as the CastAR system.
MTBS: If we lived in a perfect world, what games would you like to see retrofitted with your device? How would the experience be different than what we have now?
WP: Skyrim and Left4Dead would be at the top of the list. We currently have great demos for portraying interaction with melee (sword, flail) and projectile (currently slingshot, but a bow would be an easy, less sophisticate addition) weapons.
You could actually feel simulated resistance when you engage your opponent's sword or hit their armor. Or you could feel the shifting weight of the flail as you swing it over your head and its impact when you hit your opponent. Or you could feel the changing resistance as you pull back the pouch of your slingshot and have this resistance shift side to side as you adjust your aim.
MTBS: Tactical Haptics struck a relationship with Sixense and their recently Kickstarted STEM System. What do they bring to the table, and why was this advantageous for you and the controller? What feature does it add to the experience?
WP: We've already been using the Razer Hydra, a Sixense licensed technology, and the STEM represents the next leap forward for their magnetic tracking technology. We currently have issues with the Hydra's tracking accuracy and limited workspace that the STEM greatly improves upon.
For those interested, we will still plan to support the Hydra as a clip-on option. We will also try to work with people that want to use other trackers.
MTBS: Vireio Perception are free open source VR drivers developed by the MTBS community that let you take existing games and play them in VR even though they weren't designed for that purpose. There are commercial options too like DDD's TriDef Ignition and VorpX.
In the case of adding a new interface, a modern example of this is the HalfLife2 mod (not the native VR support version) that added Razer Hydra interaction to the game after it was already released.
Do you think games and software will require native VR support to work with your device, or will it be possible to extrapolate an equivalent through a driver solution? Why?
WP: The short answer is probably no, unless as part of a game mod, you also had access to physics engine data or this info was already being generated by the game for some reason and was available for our haptic device drivers to use.
For the case of just taking an existing game and plugging in drivers and my device, in order to get our feedback to work, you would need a game that had already generated and made information about force/torque available to our drivers. Think of this like you would having a house wired to be "cable ready".
The games would need to be "force/torque ready" or have access to the physics engine to regenerate force/torque information on the fly, which may be possible on open games such as HalfLife or Skyrim, but as many of the interactions in these games are just gesture or button based, some work is likely necessary to get these games to provide force/torque commands to our haptic game controllers.
Also, the reason I didn't say "haptic ready" instead of "force/torque ready" is because most people think of haptics as vibration feedback and the data that drives vibration feedback does not contain the necessary force/torque information, just frequency and amplitude at best; usually just magnitude and timing -- on/off.
It would be awesome if developers would at least buy into the notion that "better haptic feedback is coming" and started piping the interaction in their games to generate and make data available to make their games "force/torque ready". This would help us get past the chicken and egg issue and allow an install base of haptic devices to grow and add value to customers who want and own haptic devices, since their haptic devices will be able to instantly play this new content on all the games that support their new device.
Unfortunately, I'm not sure that the big game developers will be motivated to make this extra haptic information available for new devices unless they see associated increases in profits resulting from consumer preference for "force/torque ready" games. This probably comes down to finding a game developer that is interested in differentiating themselves by featuring new haptic controllers to interact with their games and forging some standards for other developers to follow.
MTBS: What are your thoughts when people compare your work to say a Nintendo Wii remote or similar gaming device? Are there misconceptions about pricing and what people get for the money?
WP: It's hard to compare my tech to a Wii, and we'll ignore that we plan to use the STEM System for tracking rather than the inertial, plus optical sensing for pointing direction.
The haptic feedback provided by the Wii is a simple vibration, and is hard to compare to our tech which can provide torque-like sensations.
However, it is hard to compete on price to the single motor spinning an off-center mass that creates the vibrations in the Wii-mote. This spinning mass adds approximately $1 to the cost of the Wii-mote. It's also difficult to compare the price of our tech, which we'll initially target making approximately 1,000 of with the cost of a wii-mote, which is made in quantities of hundreds of thousands to millions. The cost of their injection molds becomes inconsequential, not to mention that Nintendo can afford to sell their controllers at a loss if they chose since they could make up for these losses in additional licensed products and games for their system by having consumers buy their low-cost consoles and controllers.
If or when our technology makes it to mainstream video games, the cost of our controllers will certainly come down too, but it can never be quite as cheap as a single spinning motor and mass ($1).
Only to compound the issue is that it is difficult to convey how good our haptic feedback is in words or even videos. One needs to feel our technology to truly experience and understand it. Hence, it can be difficult to have a consumer understand our added value and why they might be willing to pay the additional cost. This is why we wanted to get our devices out into the world so that there are more possibilities for people to discover what our tech can do for VR and video games, and Kickstarter is the best way I could think of to accomplish this as quickly as possible.
MTBS: You've obviously got a prototype already. Why are you doing a Kickstarter? What will the money be used for?
WP: Kickstarter will allow a few things:
1) Help raise awareness and publicity for our technology because Kickstarter is a very visible crowdfunding site.
2) As we are currently just getting off the ground as a company, Kickstarter allows us to get developer kits out to people and establish a developer community and do this in a timely manner to be part of the first generation of devices as consumer VR begins to coalesce.
3) It will also allow us to make additional refinements to the device to make it more functional and take cost out of the design without sacrificing performance. We have started testing alternate actuators and will need to integrate the new STEM tracker in addition to some other design changes that the Kickstarter will drive.
4) This can also demonstrate to potential investors that there is demand for our technology as we attempt to raise capital.
5) Kickstarter provides a company money up front, which can help a starting company, since it can be hard to obtain a loan to buy tooling and components if you don't get paid until stuff ships.
Great stuff! There is more to this interview, and we'll share it closer to Kickstarter time! Thanks for joining us, William.