The need for VR related libraries and uniform driver APIs

[MSat]

There's little doubt about the importance of standardized APIs when is comes to graphics, sound, physics, etc., so with the variety of hardware that could be targeted for VR use in the near future, hardware manufacturers, software developers, and end users would also stand to benefit from an abstraction layer dedicated to this equipment, if not absolutely necessary for widespread adoption.

I want this thread to be a place software and hardware developers could bounce ideas around, to try to get a better grasp of what's needed, and the best way to pursue it in hopes that something useful and agreed upon can come out of it.

I'll start off by saying that since the user state in a VR environment has the possibility to be a lot more dynamic in the man/machine interface than with any other type, user awareness is probably the most fundamental and critical aspect of VR. Without sufficient input based upon a human model, the output will be limited (if not in capability, then in accuracy). So an API that can return various details about the user's state is likely at the forefront of this undertaking. So starting with this core aspect and working it out, I think the rest will slowly fall into place.

Lets make this happen!

[cybereality]

I think it would be cool to have a generic force-feedback layer, which you can pass various properties to and have hardware react to that.

In the most basic sense it could have a vibration meter (ranging from 0.0 to 1.0) which could rumble a users controller (sticking with what is currently available).

More advanced feedback could be 3D, with relative (x, y, z) coordinates to offset by (for example useful for kick-back on a gun). This could also be a constant force, in the case of an object having weight. This is the kind of stuff that might work on a Novint Falcon.

Then there could be temperature feedback (for example, 34 degrees) which could potentially interface with central air or the thermostat in the room.

There could be wind feedback, given in a normal angle (x, y, z) and force (0.0 - 1.0).

I could probably keep going, but I think you guys get the idea.

[geekmaster]

... Then there could be temperature feedback (for example, 34 degrees) which could potentially interface with central air or the thermostat in the room.
...

You could PWM an infrared heater overhead to simulate the hot noonday sun, and turn it off when you step into the virtual shade. The PWM signal could control a solid-state relay, for fast switching times.

Or you could use electric heating in a vest to simulate sun on your back or chest. Perhaps a hot plate under your feet to simulate walking on hot desert sand.

With thermoelectric Peltier devices, you can cause heating or cooling depending on direction of current flow.

A jumpsuit lined with flex tubing could have hot or cold water pumped through it to make the temperature match the VR environment, and you could feed it directly with hot and cold tap water, and drain it back into a sink. Of course that would require flexible tubing that tethers you, unless you carry recirculating hot and cold water in a backpack.

Whether or not practical, these ideas are still "food for thought".

In my experience, it is easier to simulate weightlessness by laying on your back and projecting on the ceiling (or inside a Rift), but that no longer works well now that I have accumualted significant floaters in my eyes. When laying on my back they shift from the bottoms of my eyeballs to settle on the retinas, which is very distracting when watching video above you. Enjoy not having that problem while you are still young.

Although I really would like to try things like the Wizdish, my knees cannot handle much of that these days, so it would not be money well spent for me. Youth is wasted on the young...

[MSat]

It's exactly because of the potentially huge variety of input/output VR equipment that I started this thread! The equipment is only a fraction of the challenge. If every company making something like a temperature feedback device had their own API, then VR would be in a state where sound cards and gamepads were during the bad ol' days of DOS.

A VR-centric API needs to consider (and be open to) various classes of hardware without the need for a developer to support any specific device. For instance, something like the Hydra could be interfaced with by a generic 6DOF joystick API. Any other 6DOF joystick could also be interfaced with using that same API. It eases the burden on the developers, and also assures hardware manufacturers that their equipment will be supported in any game using the API as long as they provide an API compliant driver.

[virror]

Great thread, this is something i have been thinking about as well, some kind of software platform were anyone can make a cool hardware solution to "plug in" to the system.
Something to start with would probably be what the different "base systems" that needs to be supported, like body tracking data, external I/O and so on, then start to refine and split those up into smaller parts and finally start looking at the API implementation details.

[PatimPatam]

Maybe I'm confused.. but is this not exactly the spirit of FreePIE?

http://andersmalmgren.github.com/FreePIE

http://www.mtbs3d.com/phpBB/viewforum.php?f=139

[virror]

But freePIE cant really interact directly to a game right? As an example if i make a VR game in Unity with support for bone movement by tracking and some cool stuff like turning on a fan or something similar, correct me if im wrong here. I think what the OP is after is more of a unified set of APIs for direct integration into game engines?

[MSat]

As virror said, the idea is a VR-centric API that software developers would use to interface with a variety of abstracted input and output hardware. Whereas FreePIE enables a variety of input devices to work in games where they're not directly supported.

Taking virror's fan example, a software developer wanting to turn on the API compatible fan to the medium setting would make a call that's something like:

OpenVR.output.fan(medium)

The specific fan brand and model in this case is irrelevant, and therefore any manufacturer that makes a compatible fan will work with the application. Just like OpenGL/Direct3D abstracts the specific video card in the user's computer.

[virror]

Count me in if you decide to take this somewhere : )

[PatimPatam]

Ok I see.. and i agree, that would be great!

I guess after developing the main platform you would need to persuade a few big names from both sides (HW manufacturers and game companies) to start using it, and get the thing rolling.. Some would be quite difficult to convince at the beginning, but I can see Oculus and ID Software for instance getting behind something like this.

Maybe I could help a little bit as well, making some of the first OpenVR compatible HW for positional tracking

[virror]

Something like this would be awesome to get out, right now its a big problem cause a lot of ppl here on the forums are developing different kinds of cool hardware but if anyone besides themselfs are going to be able to use it they would need custom game support and thats not likely going to happen. But if we could make a standard API that would benefit both hardware designers and game coders.

When/how/were do we start? : )

Edit: Palmer, would Oculus consider to support something like this? Or even take an active part of some parts of the development (like headtracking)?

[MSat]

Unfortunately, my programming skills are limited to a couple obscure languages. I never even wrapped my head around OOP.

I just wanted to get the ball rolling, and see if we can at least generate an outline, and then slowly fill in the blanks. I know there's quite a few programmers here whose input would be invaluable for this undertaking. Before we're able to get any substantial parties involved, we'll at least need something to show. If Oculus wants to get in early in the game, that would be awesome, as they're pretty much the sole face of VR at the moment, so they could have some major sway. It would also show that they want to further VR as a whole, and not just their company.

For initial partners, it might be good to align with FreePIE, FreeTrack, and of course, posiTTron

I'll start working on an outline over the next couple days and see what everyone thinks. Hopefully we'll be able to slowly hammer something out.

[virror]

Dont worry about the coding, that will be a later thing.
I think its good to get early input from ppl in the business (like Oculus) so we get it right from the start and don't have to rework everything later.
I will try to think of some good stuff to include as well.

[virror]

Now that I'm home from work i will make a longer post : p

First things first, what layer should we put the driver on, that needs to be decided first i think. Should we cover the whole part from the port on the computer to the engine or just act as a layer in between, or maybe some other configuration. This kind of stuff is not my area so someone else is welcome to come with input here, plenty of ppl that are great at doing that kind of stuff on this forum it seems : )

Thoughts on different "base systems" that should be supported:
Body tracking
Problem here is that there are different ways of doing tracking, one is to measure the angles of the bones (IMU) and another is to measure the position in 3d space (Razor) so i guess we need to make sure it can use all variations. Another thing here is to make it friendly for the user to get the skeleton data because different games uses different parts of the body, some might only use the arms as an example and on the other hand some equipment might only give arms while others give the full body.

Body feedback
Some kind of API for restricting the body and to give the feeling of touch.

Movement tracking
For things like treadmills and that kind of stuff, basically just a value on how fast you move and direction?

External I/O interface
For writing/reading to/from external sources like sensors, fans and stuff.
I think it has to be a bit more generic than then "setFan(0.4);" because we don't want to update the driver all the time someone comes up with another crazy invention : )

Probably also need a way to detect what devices are attached that supports OpenVR so the software can make some automatic setups that might be needed.

Just some random rants : p

[MSat]

Alright, so here's a basic premise. Not particularly detailed, but it's a start.

flowchart.jpg

Starting with the 'Installed compliant hardware list' database, it's essentially a description of the installed hardware with some other details like the name and path of the device used by the OS. I'm thinking the database entries should be made during driver installation. The application can read the list so that it knows what compliant hardware is available.


The 'Hardware configuration utility' is an optional piece of software the user can run to specify what hardware is to be used in what way. For instance, via the utility the user could specify that a PS Move controller is to be used for head tracking, and a Razer Hydra for hand tracking. This utility then builds a 'Global hardware configuration' database which any application can load so that the user doesn't have to specify it for every single different app. That said, the app wouldn't have to use this database, as it could load the 'Installed compliant hardware list' instead, but then the program would have to manage the configurations on its own - It would probably be better on the developers and end-users if the application used the 'global hardware config' database instead.


With the specifics of the hardware known, and the desired configuration, the application can then either communicate directly with the hardware drivers, and provide its own processing and management algorithms, or it could leverage VR-oriented software libraries (for things like sensor fusion, skeletal modelling, etc) and talk to drivers indirectly.

Not really part of it, but I wanted to include in the chart anyway is the API interface middleware. This could be something like FreePIE - a program that accepts API compliant calls made to hardware that doesn't officially support them.


That's all I have for now. If there's any interest, I'll continue working it out. As always, help/suggestions is wanted.

[virror]

Looking good so far : )
I just wish i could more of that driver stuff : p

[geekmaster]

Alright, so here's a basic premise. Not particularly detailed, but it's a start.

[img]http://www.mtbs3d.com/phpBB/download/file.php?id=2043&t=1[/img]

Shouldn't there be a dashed line connecting the "VR Libraries" box with the "API compliant interface middleware" (HAL) box?

[MSat]

Shouldn't there be a dashed line connecting the "VR Libraries" box with the "API compliant interface middleware" (HAL) box?

Yes, there should. I thought it was obvious enough to not require criss-crossing lines all over the place though.

[geekmaster]

Shouldn't there be a dashed line connecting the "VR Libraries" box with the "API compliant interface middleware" (HAL) box?

Yes, there should. I thought it was obvious enough to not require criss-crossing lines all over the place though.

Because the thread title is about the HAL, I should think that the HAL box is the one thing that should NOT be missing a connection.
tion.

Such assumptions require a certain degree of common experience, which cannot be counted on in a public forum such as this.

Better to simplify the diagram by dropping connections to things NOT related to the HAL, such as one of the redundant compliant drivers...

IMHO.

Of course, this is a great start. A little rearranging could clarify the structure a bit, and make room for the missing connection without crossovers.

[Fredz]

The StreamInput initiative from the Khronos Group could be an interesting part of a standard virtual reality API in the future. It's still a work in progress and it seems quite low-level for now, but it could at least help abstract the different input devices, and more complex interaction models could be build upon this basis.

See : http://www.khronos.org/streaminput/

[MSat]

Shouldn't there be a dashed line connecting the "VR Libraries" box with the "API compliant interface middleware" (HAL) box?

Yes, there should. I thought it was obvious enough to not require criss-crossing lines all over the place though.

Because the thread title is about the HAL, I should think that the HAL box is the one thing that should NOT be missing a connection.
tion.

Such assumptions require a certain degree of common experience, which cannot be counted on in a public forum such as this.

Better to simplify the diagram by dropping connections to things NOT related to the HAL, such as one of the redundant compliant drivers...

IMHO.

Of course, this is a great start. A little rearranging could clarify the structure a bit, and make room for the missing connection without crossovers.

I changed the title of the thread to something I believe is now more correct/appropriate to the goals of the project, as the interface middleware should not really be considered part of it.


@Fredz
Thanks for the heads up. I just quickly glanced over that site, but it looks like it might cover a lot of bases when it comes to input. I'm going to read more about it now and see how it fits in, or how it could be leveraged to at least solve part of the puzzle.

[Owen]

I still think its too early to think about standards. We don't yet know what input methods are going to catch on, or even what will exist in a year, or how that data will be integrated into games. The need really hasn't presented itself yet.

[virror]

I don't really agree there, when we have a its already to late : p
Its not that hard to predict a bunch of different types of inputs because it should not matter how the hardware works, just what kind of input it is e.g. skeleton tracking.

[STRZ]

Important is that the game developers allow the usage of a bunch of input devices simultaneously, and calibration and the assigning/mixing of those.

[MSat]

I still think its too early to think about standards. We don't yet know what input methods are going to catch on, or even what will exist in a year, or how that data will be integrated into games. The need really hasn't presented itself yet.

Of course, we can't really foresee the future of technology, but those should be addressed as they come up. There are a bunch of systems that do exist right now, and those are what we should be targeting. Of course, the API should be future-looking enough so that it is still applicable to systems that don't currently exist, but it doesn't have to be set in stone. It's no different than OpenGL/D3D which have also grown with time. Otherwise, there wouldn't have been a need for them to go beyond V1.0


I'm curious if StreamInput will take off, or what its status even is. Apparently it was being proposed back in 2011 and a spec was supposed to be hammered out within a year From what I can tell though, it seems to be pretty much in line with what I'm talking about - just need to add some VR/AR-specific libraries developers can use, and a hardware configuration utility + database to make everyone's lives easier. If Khronos can do the heavy lifting by promoting an API spec among hardware manufacturers, the developer and user oriented aspects of this would be a much more reasonable undertaking.

[virror]

Best would of course be to support an already ongoing project, i guess we have to find out whats going on there and then if they are alive try to get our stuff in there, if not i guess we could make a spec ourself and then try to convince ppl its good : )
Getting hardware and software devs on the train is essential or its just going to be a pile of files on a drive somewhere : p

Edit: Seems like they want to do the same thing but after going through the site they seem kind of dead, we will see.
Edit2: Sent a mail asking about the status : )

[virror]

Got an answer:

Hi Victor,

Thank you for your inquiry about StreamInput.

StreamInput is in active development and we are nearing the completion of the first draft specification. The focus of the 1.0 development has been enabling fusion of motion, vision and touch sensors for mobile devices with particular attention to power efficiency.

Several companies, most recently Tobii (leaders in eye-tracking), have joined Khronos specifically for StreamInput participation. We also continue to have the support of depth camera vendors such as PrimeSense and SoftKinetic – both of which have contributed their proprietary APIs as contributions to StreamInput.

The unanimous consensus of the participating members is that the draft specification has taken a very positive step in creating a framework that the membership believes solves the most pressing industry friction points. Due to the frenetic pace of the industry – we do need more participation cycles to push the spec to completion and ratification in a timely way.

I would be happy to discuss this with you if you have any questions, or you wish to explore this further.

Best regards,
Neil

[MSat]

Interesting. Hopefully the final spec will be available soon.

[Mart]

Just posting to show my support for a single API for VR I/O.

I'd also like to say that I disagree that it's "too early to think about standards". We've seen countless battles for standardization in this industry over the years - many that span over a decade.

It's all about getting as many developers behind a single standard before it becomes mainstream. If this is left until after the Rift/VR takes off, it will be much harder to accomplish.

[Mystify]

But how do you standardize something when it has not been determined what the standardization needs to address?

[MSat]

But how do you standardize something when it has not been determined what the standardization needs to address?

That's part of the standardization process

The most important aspect to address initially IMO is input hardware. Common devices like joysticks and gamepads for the most part have been standardized via the USB HID class. An IMU, the Hydra, or practically any other 3/6 DOF device could be made as a sort of generic HID device, for which a common driver could be used. Devices like an IMU could provide raw data from the on-board sensors, and an application can either use public sensor fusion libraries, or its own algorithms.

While optical systems could in theory be handled similarly, it would probably be better for hardware manufacturers to provide drivers with standardized functions for returning data such as absolute position of tracked nodes, skeletal models, point cloud data, etc.

I think that covers all the bases in terms of input hardware.. Am I missing anything?

[virror]

I think you covered the basics, and the system could later be expanded with ease as new technology starts to evolve like forcefeedback/tactile systems and so on.
As Msat said, its a lot harder to make a standard when there is already multiple products out there that uses their own API.

[Mystify]

But early codification of a standard limits your ability to innovate in ways not supported by the standard. And I think we are still at the point where innovation is key.

[MSat]

But early codification of a standard limits your ability to innovate in ways not supported by the standard. And I think we are still at the point where innovation is key.

As I had mentioned before, no software can be totally future proof by anticipating every new piece of hardware that might come out. Was DirectX or OpenGL pointless because there have been (and will be) numerous revisions since they were first released? There is a lot of hardware here now from which software could benefit from by implementing a standardized API. Even for future hardware, a well thought out framework can still be forward-looking enough to be applicable to currently unknown systems. That said, positional input devices have to be based on reality, so data can ultimately be reduced to absolute or relative 6DOF point/node data max. Ultimately, the underlying method by which that data is obtained is irrelevant. For example, lets say that a new 6DOF controller is invented that derives absolute position by harnessing the power of pink unicorns - to a game, absolute position is absolute position regardless of the hardware used. What's more important is that the data be available in a specified way. That way game developers can support Joe's Razer Hydra, and Jane's pink unicorn controller without having to write specific support for either.

[Mystify]

But early codification of a standard limits your ability to innovate in ways not supported by the standard. And I think we are still at the point where innovation is key.

As I had mentioned before, no software can be totally future proof by anticipating every new piece of hardware that might come out. Was DirectX or OpenGL pointless because there have been (and will be) numerous revisions since they were first released? There is a lot of hardware here now from which software could benefit from by implementing a standardized API. Even for future hardware, a well thought out framework can still be forward-looking enough to be applicable to currently unknown systems. That said, positional input devices have to be based on reality, so data can ultimately be reduced to absolute or relative 6DOF point/node data max. Ultimately, the underlying method by which that data is obtained is irrelevant. For example, lets say that a new 6DOF controller is invented that derives absolute position by harnessing the power of pink unicorns - to a game, absolute position is absolute position regardless of the hardware used. What's more important is that the data be available in a specified way. That way game developers can support Joe's Razer Hydra, and Jane's pink unicorn controller without having to write specific support for either.

And when you are dealing with something as basic as a 3d point and orientation, sure. I feel like that is the least of the concerns for a VR API, and something that is relatively well established by this point. Still, what if you have a gun controller which, in an attempt to get more precision on the aiming, deals with 2 points to establish the aiming, and having to translate this to the API's format would result in a loss of precision. However, what about things like haptic feedback? They are accomplishing such a large variety of things that I don't see how you can standardize it yet. Or motion capture. Somethings work with skeletons, others point clouds, new solutions may do something else.

[MSat]

Yes, 6DOF is well established, but AFAIK, every device with such capabilities has its own driver and API, which is my whole point. I'm not sure how you figure that a standardized API would somehow prevent devices from being precise - it really comes down to the hardware. You could make all the data variables 512 bits long in the API if you really wanted to - that would ensure that accuracy was never diminished due to artificial software limitations (yeah I know 512 bits is silly, it was just an extreme example to point out why this is a non-issue). Skeletal modelling is fairly well established and fits in the framework regardless of the underlying hardware used. Same with point cloud. I think you're missing the point which is to abstract the hardware from the supplied data.

You mention haptic feedback, but what about it? Classes will need to be created (and revised, refined, amended) for the various types of devices that do exist, or may exist. It's silly to say that standards shouldn't be created for the hardware currently available because we don't know what devices there will be in the future. Really, setting some standards now is important to get developers to even create *any* hardware. If software developers don't have an easy way to support a variety of similar hardware, hardware manufacturers are at great risk to not have their equipment supported at all. If there's 3 different companies making similar rumble seats, why shouldn't developers be able to use a single API to control all 3?

If this was a non or insurmountable issue, then the creation of the DirectStream standard wouldn't be underway.

[Mystify]

I guess my objection is trying to group all of these things together. An API for motion control alone may make sense. An api for point cloud data makes sense. But just because both could be useful for VR doesn't mean those APIs need to have anything to do with each other. I mean, a keyboard and mouse are standard components for many things, does that mean a mouse API and a keyboard API need to be related?

[MSat]

I'm not talking about lumping a disparate amount of devices into a sort of singular object, but rather having the same API for all the devices that fall under a particular class. So for instance, all devices that generate point cloud data would fall under a point cloud class. 3DOF devices would fall under a 3DOF class, and so on. If you want to utilize a device from a certain class, you can by making standard API calls regardless of who the device is made by. This is what mice and keyboards do - they fall under their respective class, so generally any mouse or keyboard that is plugged into the computer works with a generic driver - applications don't care whether Microsoft or Logitech made it, as the API is all the same.

[Mystify]

I'm not talking about lumping a disparate amount of devices into a sort of singular object, but rather having the same API for all the devices that fall under a particular class. So for instance, all devices that generate point cloud data would fall under a point cloud class. 3DOF devices would fall under a 3DOF class, and so on. If you want to utilize a device from a certain class, you can by making standard API calls regardless of who the device is made by. This is what mice and keyboards do - they fall under their respective class, so generally any mouse or keyboard that is plugged into the computer works with a generic driver - applications don't care whether Microsoft or Logitech made it, as the API is all the same.

I'm not even sure what I'm trying to argue anymore. I think I misunderstood what you were trying to accomplish. Carry on.

[Direlight]

All we can do is support things like Oculus Rift and Leap Motion. If they get popular enough, OS developers will have to add them as official HIDs. End goal should be plug and play functionality.