Sound behavior with speakers and headphones

[LordJuanlo]

Not sure if this issue has been discussed already, but I have been thinking about it for the last days. Let's imagine our character is in a room, heading and looking north. Then a door opens at southwest. If we are looking north, we should hear the door at the southwest. What does it mean with speakers or headphones?

- Speakers: sound comes from our rear left speaker
- Headphones: sound comes from our left headphone and uses HRTF algorithms to make it sound in our back

So if we are looking at the same place we are facing, the game audio engine always make the sound come from the same position, no matter if speakers of headphones are used.

Now imagine our character is still heading north, but we turn our head and look south west, then the door opens. What should happen now?

- Speakers: our character is facing north, so the sound must still come from our rear left speaker. We have turned our head left and back, physically looking at the rear left speaker, so we will hear it sounding in front of us, because we are looking at the rear left speaker.
- Headphones: the sound should NOT come from left headphone, it should be centered, because our speakers have moved together with our head.

Now we aren't looking at the same position that we are facing, so the game audio engine must behave differently depending on our audio system.

CONCLUSION: If we are using speakers (stereo or surround, it does not matter), the sound position should be relative to our character's torso. But if we are using headphones, the sound should be relative to our character's head. Of course, all of this should happen when aiming is decoupled from view, as it should be with every Rift-ready game.

[EdZ]

For the near term, it might be best to just go with a blanket "headphones only" label. Sound will already be relative to the viewport orientation, so no further coding should be needed.

External fixed sound sources have an additional problem above 'just' torso-tracking (orientation and position): real-space and game-space may not stay in fixed alignment to each other. Brantlew's Redirected Walking tests are an example of this, but not the only way this can happen. You'd need to keep track of both the measured torso orientation & position, but also accumulate the absolute offset of the environment from the virtual position. Fix the real and virtual environments to each other and this isn't an issue, but adds what may be excessive constraints on the movement space (no redirected walking, no movement amplification, etc).

[cybereality]

If you are using headphones, and the game view moves in relation to a head-tracker (for example using mouse-emulation), then everything should work fine right now.

[LordJuanlo]

Depending on my mood (and the game), sometimes I prefer to play with my 5.1 speakers, and other times I prefer to play with my earphones. I think anyone planning to develop games/apps with Oculus Rift support in mind should take those things into consideration.

[cybereality]

Well I think most gamers would want headphones, unless you are developing your own game and can control the position of sound.

[bobthedog007]

I agree. I never play games with headphones.

Given the choice I would prefer to use my home theater system.
I can get a nice kick in the chest from the sub-woofer, something that is lacking in headphones.

If doom 3 doesn't support torso based sound I guess I will need to start looking for a decent set of headphones.

[2EyeGuy]

It's a head mounted display. So it's not unreasonable to assume people will use head mounted sound with it. It's strange that people would put a big thing over their eyes, but would balk at wearing headphones.

Wanting the bass sounds to shake your body is a legitimate reason to want external speakers, but often people have crazy ideas about 5.1 surround sound being better than head-tracked headphones, even though we only have 2 ears.

But I did think about that same issue when I was trying to design my own all-encompassing VR SDK. I thought it might be possible to place speakers where-ever you wanted around the room, and have the game render to all those speakers correctly as you moved around the room (I know how to do that with display screens). But I couldn't work out how to do that properly for sound.

It would be good to have an option for head-tracked headphones or normal fixed speakers though. At least, if we can get absolute yaw values.

Anyway, John Carmack talked about the head-tracked headphone sound in DooM 3, so we expect that's going to be present. I don't know about the fixed speaker support, but I guess it won't be present, unless we make a driver for it.

[brantlew]

I agree. I never play games with headphones.

Given the choice I would prefer to use my home theater system.
I can get a nice kick in the chest from the sub-woofer, something that is lacking in headphones.

If doom 3 doesn't support torso based sound I guess I will need to start looking for a decent set of headphones.

If seated, you can use something like a ButtKicker to get a similar bass effect using headphones.

[cybereality]

I'm still itching to try out the Aura Interactor. I know its probably crap, but been eyeing it for years.

[youtube]http://www.youtube.com/watch?v=0NSHASDuD_A[/youtube]

[2EyeGuy]

If you want to use an Aura Interacta, I'd recommend generating your own special "audio" specifically for it, using DirectSound to direct different sounds to different channels.

[LordJuanlo]

It's a head mounted display. So it's not unreasonable to assume people will use head mounted sound with it. It's strange that people would put a big thing over their eyes, but would balk at wearing headphones.

It may be strange, but I want to do it, and it shouldn't be THAT hard to implement. For me, a real 5.1 speaker system renders sound more accurately than headphones. I mean: to me, it's much easier to tell if a sound comes from my back using my speakers than headphones. The HRTF algorithms aren't that good (yet).

So... all I want is THE FREEDOM to choose my sound system. Quoting my first post: If we are using speakers (stereo or surround, it does not matter), the sound position should be relative to our character's torso. But if we are using headphones, the sound should be relative to our character's head.

[jaybug]

If we are using speakers (stereo or surround, it does not matter), the sound position should be relative to our character's torso. But if we are using headphones, the sound should be relative to our character's head.

Why the torso? Shouldn't it be relative to your head, regardless? If you turn your head left, then the speakers would have to reflect that by putting the front channel on the left, since that's the way you are facing.

[bobthedog007]

Thinking about it more, this might be tricky to implement.

The easiest case will be when you are seated facing forward (relative to your speakers), for example a driving/flight sim where your torso/car/cockpit is always assumed to be fixed.

What happens though if you are playing an FPS where you are standing on the spot but can still turn your entire body, and not just your head, to face any direction. Now your torso is facing the back speaker. How will the game know that your center speaker is no longer the forward position?

What about games like Doom 3 where the torso and head are fixed together and not separately controlled?

It seems that this would only work when the game knows what direction your head is facing relative to your speakers.

[MrGreen]

If we are using speakers (stereo or surround, it does not matter), the sound position should be relative to our character's torso. But if we are using headphones, the sound should be relative to our character's head.

Why the torso? Shouldn't it be relative to your head, regardless? If you turn your head left, then the speakers would have to reflect that by putting the front channel on the left, since that's the way you are facing.

Headphones move with your head, speakers don't.

[jaybug]

Yeah, but the point is that the speakers should reflect where your head is turned, not torso.

If you're looking to the left, it doesn't make sense that the "front" sound is still coming out of the front speakers, does it?

[kalabalik]

Yeah, but the point is that the speakers should reflect where your head is turned, not torso.

If you're looking to the left, it doesn't make sense that the "front" sound is still coming out of the front speakers, does it?

If you hear a sound from your left and turn the view towards the sound so you look at the source, the speakers will center that sound to your centre channel; thus effectively putting that sound in your right ear as you look to your left in relation to your speaker setup. The speakers and sound functions is relative to your monitor and centre channel. That's not a problem in headphones as they follow your head.

[jaybug]

Either I'm going crazy, or you are essentially saying the same thing that I'm saying.

I get what the problem/challenge is with speakers vs. headphones. I'm just questioning why the OP wants to target the position of the torso, rather than the head.

Edit: I just realized that OP might simply be talking about when you move your body, which is the problem of positional tracking. Still, if you're gonna track anything it should still be where your head is facing.

[2EyeGuy]

It's nothing to do with your torso, unless you're wearing cybereality's speaker-vest on your torso.

The VR game has to remember the mapping between the real world and the virtual world, because that changes when you use the joystick or mouse to turn, or when the game turns you automatically (eg. standing on a rotating platform in the game). It doesn't change when you turn your head though. For Head-Mounted Displays and head-mounted speakers (headphones), your head-tracker angle is added on top of the real-to-virtual mapping to render the sound and graphics. But for speakers that are at fixed locations in the real world, you just use the real-to-virtual mapping and ignore the head-tracking, when rendering sound.

That's all pretty straightforward if you think about it.

[urban]

Thinking about this, I hope they can put a headphone jack on the unit, and linked in through the hdmi, that way I can use the built in audio on my nvidia card

Would probably be using some nice comfy, quality earbud headphones, hopefully plugged into the rift

[2EyeGuy]

I don't think they're using HDMI, except as an optional HDMI adapter you can use. I think it actually runs on DVI.

[urban]

I don't think they're using HDMI, except as an optional HDMI adapter you can use. I think it actually runs on DVI.

Dvi still transports audio so should be ok still if it is dvi on the pc end

I'm using a dvi to hdmi cable at the moment on my pc, plugged into to a HDTV, and I get pc audio coming out of the tv speakers

If not for the dev kit, I'd like to see a headphone jack on the consumer unit, maybe even a built in microphone

[EdZ]

Dvi still transports audio

DVI does not transport audio: it's video only. A nonstandard connector on a video card may push HDMI out of a DVI connector, but this is not a DVI signal.

[MrGreen]

I don't think they're using HDMI, except as an optional HDMI adapter you can use. I think it actually runs on DVI.

Dvi still transports audio so should be ok still if it is dvi on the pc end

I'm using a dvi to hdmi cable at the moment on my pc, plugged into to a HDTV, and I get pc audio coming out of the tv speakers

[urban]

Dvi still transports audio

DVI does not transport audio: it's video only. A nonstandard connector on a video card may push HDMI out of a DVI connector, but this is not a DVI signal.

Sorry I just got confused by the game and movie SOUNDS coming out of my HDTV whilst connected to my pc via a standard DVI to HDMI cable

[jaybug]

The VR game has to remember the mapping between the real world and the virtual world, because that changes when you use the joystick or mouse to turn, or when the game turns you automatically (eg. standing on a rotating platform in the game). It doesn't change when you turn your head though. For Head-Mounted Displays and head-mounted speakers (headphones), your head-tracker angle is added on top of the real-to-virtual mapping to render the sound and graphics. But for speakers that are at fixed locations in the real world, you just use the real-to-virtual mapping and ignore the head-tracking, when rendering sound.

That's all pretty straightforward if you think about it.

So straightforward it's giving me an aneurism. :p

I just understood everything as I was writing this reply. I knew i was bound to have a "i'm an idiot" epiphany eventually.

[LordJuanlo]

So straightforward it's giving me an aneurism. :p

I just understood everything as I was writing this reply. I knew i was bound to have a "i'm an idiot" epiphany eventually.

Glad you got it mate. English it's not my native language so I may have not expressed it as clearly as I wanted.

[jaybug]

Nah, in hindsight you explained it fine, I was just having an mental breakdown

I couldn't "get" the idea that sound would be "disconnected" from the screen.
For what it's worth, I would definitely also like to see speakers supported in this way too.

[EdZ]

Sorry I just got confused by the game and movie SOUNDS coming out of my HDTV whilst connected to my pc via a standard DVI to HDMI cable

It's easy to be confused, as DVI and HDMI are bandied about a lot, and not necessarily correctly.
In your case, the physical connector on the GPU and one end of the cable is DVI, but the actual connection standard and link protocol in use is HDMI.
In the case of the Oculus' link between the HMD and the panel driver board, the physical connectors and cable are HDMI, but the actual interface used is (almost certainly) LVDS. The link between the driver board and the PC is DVI, both physically and electrically. At no point is the HDMI interface used, as that would require a licensing fee.

[hughJ]

Worth remembering that however good a properly set up speaker system might be (ie. perfectly balanced and evenly distributed around you), and not tied to your camera view, it's still not going to provide any support for modelling the roll and pitch of the head without you installing speakers in your floor and ceiling.

Out of curiosity, do any sound engines in modern games do proper ray/wave casting to model reverb, occlusion, etc?

[urban]

Sorry I just got confused by the game and movie SOUNDS coming out of my HDTV whilst connected to my pc via a standard DVI to HDMI cable

It's easy to be confused, as DVI and HDMI are bandied about a lot, and not necessarily correctly.
In your case, the physical connector on the GPU and one end of the cable is DVI, but the actual connection standard and link protocol in use is HDMI.
In the case of the Oculus' link between the HMD and the panel driver board, the physical connectors and cable are HDMI, but the actual interface used is (almost certainly) LVDS. The link between the driver board and the PC is DVI, both physically and electrically. At no point is the HDMI interface used, as that would require a licensing fee.

thanks for that, I understand what's going on now