It is currently Sun Sep 22, 2019 7:05 pm



Reply to topic  [ 30 posts ] 
 Sound behavior with speakers and headphones 
Author Message
Binocular Vision CONFIRMED!
User avatar

Joined: Thu Jul 26, 2012 12:55 am
Posts: 224
Location: Spain
Reply with quote
Not sure if this issue has been discussed already, but I have been thinking about it for the last days. Let's imagine our character is in a room, heading and looking north. Then a door opens at southwest. If we are looking north, we should hear the door at the southwest. What does it mean with speakers or headphones?

- Speakers: sound comes from our rear left speaker
- Headphones: sound comes from our left headphone and uses HRTF algorithms to make it sound in our back

So if we are looking at the same place we are facing, the game audio engine always make the sound come from the same position, no matter if speakers of headphones are used.

Now imagine our character is still heading north, but we turn our head and look south west, then the door opens. What should happen now?

- Speakers: our character is facing north, so the sound must still come from our rear left speaker. We have turned our head left and back, physically looking at the rear left speaker, so we will hear it sounding in front of us, because we are looking at the rear left speaker.
- Headphones: the sound should NOT come from left headphone, it should be centered, because our speakers have moved together with our head.

Now we aren't looking at the same position that we are facing, so the game audio engine must behave differently depending on our audio system.

CONCLUSION: If we are using speakers (stereo or surround, it does not matter), the sound position should be relative to our character's torso. But if we are using headphones, the sound should be relative to our character's head. Of course, all of this should happen when aiming is decoupled from view, as it should be with every Rift-ready game.

_________________
Image
Comunidad española de RV / Spanish VR Community


Thu Sep 06, 2012 2:41 am
Profile WWW
Sharp Eyed Eagle!

Joined: Sat Dec 22, 2007 3:38 am
Posts: 425
Reply with quote
For the near term, it might be best to just go with a blanket "headphones only" label. Sound will already be relative to the viewport orientation, so no further coding should be needed.

External fixed sound sources have an additional problem above 'just' torso-tracking (orientation and position): real-space and game-space may not stay in fixed alignment to each other. Brantlew's Redirected Walking tests are an example of this, but not the only way this can happen. You'd need to keep track of both the measured torso orientation & position, but also accumulate the absolute offset of the environment from the virtual position. Fix the real and virtual environments to each other and this isn't an issue, but adds what may be excessive constraints on the movement space (no redirected walking, no movement amplification, etc).


Thu Sep 06, 2012 3:32 pm
Profile
3D Angel Eyes (Moderator)
User avatar

Joined: Sat Apr 12, 2008 8:18 pm
Posts: 11394
Reply with quote
If you are using headphones, and the game view moves in relation to a head-tracker (for example using mouse-emulation), then everything should work fine right now.

_________________
check my blog - cybereality.com


Thu Sep 06, 2012 8:57 pm
Profile
Binocular Vision CONFIRMED!
User avatar

Joined: Thu Jul 26, 2012 12:55 am
Posts: 224
Location: Spain
Reply with quote
Depending on my mood (and the game), sometimes I prefer to play with my 5.1 speakers, and other times I prefer to play with my earphones. I think anyone planning to develop games/apps with Oculus Rift support in mind should take those things into consideration.

_________________
Image
Comunidad española de RV / Spanish VR Community


Fri Sep 07, 2012 1:39 am
Profile WWW
3D Angel Eyes (Moderator)
User avatar

Joined: Sat Apr 12, 2008 8:18 pm
Posts: 11394
Reply with quote
Well I think most gamers would want headphones, unless you are developing your own game and can control the position of sound.

_________________
check my blog - cybereality.com


Fri Sep 07, 2012 6:43 pm
Profile
One Eyed Hopeful

Joined: Sat Jun 09, 2012 11:39 am
Posts: 10
Reply with quote
I agree. I never play games with headphones.

Given the choice I would prefer to use my home theater system.
I can get a nice kick in the chest from the sub-woofer, something that is lacking in headphones.

If doom 3 doesn't support torso based sound I guess I will need to start looking for a decent set of headphones. :P


Wed Sep 26, 2012 11:36 am
Profile
Certif-Eyable!

Joined: Tue Sep 18, 2012 10:32 pm
Posts: 1139
Reply with quote
It's a head mounted display. So it's not unreasonable to assume people will use head mounted sound with it. It's strange that people would put a big thing over their eyes, but would balk at wearing headphones.

Wanting the bass sounds to shake your body is a legitimate reason to want external speakers, but often people have crazy ideas about 5.1 surround sound being better than head-tracked headphones, even though we only have 2 ears.

But I did think about that same issue when I was trying to design my own all-encompassing VR SDK. I thought it might be possible to place speakers where-ever you wanted around the room, and have the game render to all those speakers correctly as you moved around the room (I know how to do that with display screens). But I couldn't work out how to do that properly for sound.

It would be good to have an option for head-tracked headphones or normal fixed speakers though. At least, if we can get absolute yaw values.

Anyway, John Carmack talked about the head-tracked headphone sound in DooM 3, so we expect that's going to be present. I don't know about the fixed speaker support, but I guess it won't be present, unless we make a driver for it.


Wed Sep 26, 2012 12:44 pm
Profile
Petrif-Eyed
User avatar

Joined: Sat Sep 17, 2011 9:23 pm
Posts: 2220
Location: Menlo Park, CA
Reply with quote
bobthedog007 wrote:
I agree. I never play games with headphones.

Given the choice I would prefer to use my home theater system.
I can get a nice kick in the chest from the sub-woofer, something that is lacking in headphones.

If doom 3 doesn't support torso based sound I guess I will need to start looking for a decent set of headphones. :P


If seated, you can use something like a ButtKicker to get a similar bass effect using headphones.


Wed Sep 26, 2012 12:55 pm
Profile
3D Angel Eyes (Moderator)
User avatar

Joined: Sat Apr 12, 2008 8:18 pm
Posts: 11394
Reply with quote
I'm still itching to try out the Aura Interactor. I know its probably crap, but been eyeing it for years.


_________________
check my blog - cybereality.com


Wed Sep 26, 2012 6:03 pm
Profile
Certif-Eyable!

Joined: Tue Sep 18, 2012 10:32 pm
Posts: 1139
Reply with quote
If you want to use an Aura Interacta, I'd recommend generating your own special "audio" specifically for it, using DirectSound to direct different sounds to different channels.


Wed Sep 26, 2012 11:17 pm
Profile
Binocular Vision CONFIRMED!
User avatar

Joined: Thu Jul 26, 2012 12:55 am
Posts: 224
Location: Spain
Reply with quote
2EyeGuy wrote:
It's a head mounted display. So it's not unreasonable to assume people will use head mounted sound with it. It's strange that people would put a big thing over their eyes, but would balk at wearing headphones.


It may be strange, but I want to do it, and it shouldn't be THAT hard to implement. For me, a real 5.1 speaker system renders sound more accurately than headphones. I mean: to me, it's much easier to tell if a sound comes from my back using my speakers than headphones. The HRTF algorithms aren't that good (yet).

So... all I want is THE FREEDOM to choose my sound system. Quoting my first post: If we are using speakers (stereo or surround, it does not matter), the sound position should be relative to our character's torso. But if we are using headphones, the sound should be relative to our character's head.

_________________
Image
Comunidad española de RV / Spanish VR Community


Thu Sep 27, 2012 4:41 am
Profile WWW
Binocular Vision CONFIRMED!
User avatar

Joined: Sun Sep 09, 2012 4:55 pm
Posts: 302
Reply with quote
Quote:
If we are using speakers (stereo or surround, it does not matter), the sound position should be relative to our character's torso. But if we are using headphones, the sound should be relative to our character's head.


Why the torso? Shouldn't it be relative to your head, regardless? If you turn your head left, then the speakers would have to reflect that by putting the front channel on the left, since that's the way you are facing.


Thu Sep 27, 2012 4:54 am
Profile
One Eyed Hopeful

Joined: Sat Jun 09, 2012 11:39 am
Posts: 10
Reply with quote
Thinking about it more, this might be tricky to implement.

The easiest case will be when you are seated facing forward (relative to your speakers), for example a driving/flight sim where your torso/car/cockpit is always assumed to be fixed.

What happens though if you are playing an FPS where you are standing on the spot but can still turn your entire body, and not just your head, to face any direction. Now your torso is facing the back speaker. How will the game know that your center speaker is no longer the forward position?

What about games like Doom 3 where the torso and head are fixed together and not separately controlled?

It seems that this would only work when the game knows what direction your head is facing relative to your speakers.


Thu Sep 27, 2012 5:00 am
Profile
Diamond Eyed Freakazoid!
User avatar

Joined: Mon Sep 03, 2012 1:36 pm
Posts: 741
Location: QC, Canada
Reply with quote
jaybug wrote:
Quote:
If we are using speakers (stereo or surround, it does not matter), the sound position should be relative to our character's torso. But if we are using headphones, the sound should be relative to our character's head.


Why the torso? Shouldn't it be relative to your head, regardless? If you turn your head left, then the speakers would have to reflect that by putting the front channel on the left, since that's the way you are facing.

Headphones move with your head, speakers don't.


Thu Sep 27, 2012 5:47 am
Profile
Binocular Vision CONFIRMED!
User avatar

Joined: Sun Sep 09, 2012 4:55 pm
Posts: 302
Reply with quote
Yeah, but the point is that the speakers should reflect where your head is turned, not torso.

If you're looking to the left, it doesn't make sense that the "front" sound is still coming out of the front speakers, does it?


Thu Sep 27, 2012 7:36 am
Profile
Two Eyed Hopeful

Joined: Thu Aug 23, 2012 5:53 am
Posts: 80
Reply with quote
jaybug wrote:
Yeah, but the point is that the speakers should reflect where your head is turned, not torso.

If you're looking to the left, it doesn't make sense that the "front" sound is still coming out of the front speakers, does it?


If you hear a sound from your left and turn the view towards the sound so you look at the source, the speakers will center that sound to your centre channel; thus effectively putting that sound in your right ear as you look to your left in relation to your speaker setup. The speakers and sound functions is relative to your monitor and centre channel. That's not a problem in headphones as they follow your head.


Thu Sep 27, 2012 8:07 am
Profile
Binocular Vision CONFIRMED!
User avatar

Joined: Sun Sep 09, 2012 4:55 pm
Posts: 302
Reply with quote
Either I'm going crazy, or you are essentially saying the same thing that I'm saying.

I get what the problem/challenge is with speakers vs. headphones. I'm just questioning why the OP wants to target the position of the torso, rather than the head.

Edit: I just realized that OP might simply be talking about when you move your body, which is the problem of positional tracking. Still, if you're gonna track anything it should still be where your head is facing.


Thu Sep 27, 2012 8:34 am
Profile
Certif-Eyable!

Joined: Tue Sep 18, 2012 10:32 pm
Posts: 1139
Reply with quote
It's nothing to do with your torso, unless you're wearing cybereality's speaker-vest on your torso.

The VR game has to remember the mapping between the real world and the virtual world, because that changes when you use the joystick or mouse to turn, or when the game turns you automatically (eg. standing on a rotating platform in the game). It doesn't change when you turn your head though. For Head-Mounted Displays and head-mounted speakers (headphones), your head-tracker angle is added on top of the real-to-virtual mapping to render the sound and graphics. But for speakers that are at fixed locations in the real world, you just use the real-to-virtual mapping and ignore the head-tracking, when rendering sound.

That's all pretty straightforward if you think about it.


Thu Sep 27, 2012 8:41 am
Profile
Two Eyed Hopeful

Joined: Thu Aug 23, 2012 8:31 am
Posts: 50
Reply with quote
Thinking about this, I hope they can put a headphone jack on the unit, and linked in through the hdmi, that way I can use the built in audio on my nvidia card

Would probably be using some nice comfy, quality earbud headphones, hopefully plugged into the rift


Thu Sep 27, 2012 9:13 am
Profile
Certif-Eyable!

Joined: Tue Sep 18, 2012 10:32 pm
Posts: 1139
Reply with quote
I don't think they're using HDMI, except as an optional HDMI adapter you can use. I think it actually runs on DVI.


Thu Sep 27, 2012 9:18 am
Profile
Two Eyed Hopeful

Joined: Thu Aug 23, 2012 8:31 am
Posts: 50
Reply with quote
2EyeGuy wrote:
I don't think they're using HDMI, except as an optional HDMI adapter you can use. I think it actually runs on DVI.


Dvi still transports audio so should be ok still if it is dvi on the pc end

I'm using a dvi to hdmi cable at the moment on my pc, plugged into to a HDTV, and I get pc audio coming out of the tv speakers

If not for the dev kit, I'd like to see a headphone jack on the consumer unit, maybe even a built in microphone


Thu Sep 27, 2012 9:45 am
Profile
Sharp Eyed Eagle!

Joined: Sat Dec 22, 2007 3:38 am
Posts: 425
Reply with quote
urban wrote:
Dvi still transports audio
DVI does not transport audio: it's video only. A nonstandard connector on a video card may push HDMI out of a DVI connector, but this is not a DVI signal.


Thu Sep 27, 2012 9:56 am
Profile
Diamond Eyed Freakazoid!
User avatar

Joined: Mon Sep 03, 2012 1:36 pm
Posts: 741
Location: QC, Canada
Reply with quote
urban wrote:
2EyeGuy wrote:
I don't think they're using HDMI, except as an optional HDMI adapter you can use. I think it actually runs on DVI.


Dvi still transports audio so should be ok still if it is dvi on the pc end

I'm using a dvi to hdmi cable at the moment on my pc, plugged into to a HDTV, and I get pc audio coming out of the tv speakers


:?


Thu Sep 27, 2012 9:58 am
Profile
Two Eyed Hopeful

Joined: Thu Aug 23, 2012 8:31 am
Posts: 50
Reply with quote
EdZ wrote:
urban wrote:
Dvi still transports audio
DVI does not transport audio: it's video only. A nonstandard connector on a video card may push HDMI out of a DVI connector, but this is not a DVI signal.


Sorry I just got confused by the game and movie SOUNDS coming out of my HDTV whilst connected to my pc via a standard DVI to HDMI cable


Thu Sep 27, 2012 10:04 am
Profile
Binocular Vision CONFIRMED!
User avatar

Joined: Sun Sep 09, 2012 4:55 pm
Posts: 302
Reply with quote
2EyeGuy wrote:
The VR game has to remember the mapping between the real world and the virtual world, because that changes when you use the joystick or mouse to turn, or when the game turns you automatically (eg. standing on a rotating platform in the game). It doesn't change when you turn your head though. For Head-Mounted Displays and head-mounted speakers (headphones), your head-tracker angle is added on top of the real-to-virtual mapping to render the sound and graphics. But for speakers that are at fixed locations in the real world, you just use the real-to-virtual mapping and ignore the head-tracking, when rendering sound.

That's all pretty straightforward if you think about it.


So straightforward it's giving me an aneurism. :p

I just understood everything as I was writing this reply. I knew i was bound to have a "i'm an idiot" epiphany eventually. ;)


Thu Sep 27, 2012 10:50 am
Profile
Binocular Vision CONFIRMED!
User avatar

Joined: Thu Jul 26, 2012 12:55 am
Posts: 224
Location: Spain
Reply with quote
jaybug wrote:
So straightforward it's giving me an aneurism. :p

I just understood everything as I was writing this reply. I knew i was bound to have a "i'm an idiot" epiphany eventually. ;)


Glad you got it mate. English it's not my native language so I may have not expressed it as clearly as I wanted.

_________________
Image
Comunidad española de RV / Spanish VR Community


Thu Sep 27, 2012 1:54 pm
Profile WWW
Binocular Vision CONFIRMED!
User avatar

Joined: Sun Sep 09, 2012 4:55 pm
Posts: 302
Reply with quote
Nah, in hindsight you explained it fine, I was just having an mental breakdown ;)

I couldn't "get" the idea that sound would be "disconnected" from the screen.
For what it's worth, I would definitely also like to see speakers supported in this way too.


Thu Sep 27, 2012 2:13 pm
Profile
Sharp Eyed Eagle!

Joined: Sat Dec 22, 2007 3:38 am
Posts: 425
Reply with quote
urban wrote:
Sorry I just got confused by the game and movie SOUNDS coming out of my HDTV whilst connected to my pc via a standard DVI to HDMI cable
It's easy to be confused, as DVI and HDMI are bandied about a lot, and not necessarily correctly.
In your case, the physical connector on the GPU and one end of the cable is DVI, but the actual connection standard and link protocol in use is HDMI.
In the case of the Oculus' link between the HMD and the panel driver board, the physical connectors and cable are HDMI, but the actual interface used is (almost certainly) LVDS. The link between the driver board and the PC is DVI, both physically and electrically. At no point is the HDMI interface used, as that would require a licensing fee.


Wed Oct 03, 2012 4:39 pm
Profile
Two Eyed Hopeful

Joined: Sat Aug 11, 2012 12:56 am
Posts: 52
Reply with quote
Worth remembering that however good a properly set up speaker system might be (ie. perfectly balanced and evenly distributed around you), and not tied to your camera view, it's still not going to provide any support for modelling the roll and pitch of the head without you installing speakers in your floor and ceiling.

Out of curiosity, do any sound engines in modern games do proper ray/wave casting to model reverb, occlusion, etc?


Wed Oct 03, 2012 5:12 pm
Profile
Two Eyed Hopeful

Joined: Thu Aug 23, 2012 8:31 am
Posts: 50
Reply with quote
EdZ wrote:
urban wrote:
Sorry I just got confused by the game and movie SOUNDS coming out of my HDTV whilst connected to my pc via a standard DVI to HDMI cable
It's easy to be confused, as DVI and HDMI are bandied about a lot, and not necessarily correctly.
In your case, the physical connector on the GPU and one end of the cable is DVI, but the actual connection standard and link protocol in use is HDMI.
In the case of the Oculus' link between the HMD and the panel driver board, the physical connectors and cable are HDMI, but the actual interface used is (almost certainly) LVDS. The link between the driver board and the PC is DVI, both physically and electrically. At no point is the HDMI interface used, as that would require a licensing fee.


thanks for that, I understand what's going on now


Thu Oct 04, 2012 1:11 am
Profile
Display posts from previous:  Sort by  
Reply to topic   [ 30 posts ] 

Who is online

Users browsing this forum: Tbot and 4 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Jump to:  
Powered by phpBB® Forum Software © phpBB Group
Designed by STSoftware.