OpenGL to Direct3D to 3D Vision: yes, it works great!
Posted: Mon Feb 04, 2013 2:14 pm
UPDATE: The source code for this solution is available here: https://github.com/tliron/opengl-3d-vision-bridge
Hello friends,
This is going to be a long, long post. Like a few other people, I got the combination to work, but there were many pitfalls along the way, lots of trial and error, and mountains of frustation. I want to share my experience so that others would suffer less. :/ I'm posting to MTBS beause it seems like the best minds are here! I hope to get feedback and corrections, as I'm sure I got some details wrong.
The bottom line is that I can develop my game in pure OpenGL, ensuring cross-platform compatibility, while also allowing for 3D Vision on supported Windows boxes. It really truly works: I run my game in Ubuntu, Mac OS X and Windows, and I'm working on mobile (Android/iOS) versions. On Windows I can put on the 3D Vision glasses and enjoy full stereoscopy.
This post is aimed at programmers: there is no magic switch by which you can enable this for any OpenGL game, sorry. I will explain a bit about that.
OK! Let's get to it. Lessons learned:
1. You know how a lot of Direct3D games support 3D Vision even though they weren't designed for it? This is called 3D Vision Automatic, and it only works for pure Direct3D, not OpenGL. It's really clever, actually: it seems that the driver captures all drawing operations between the BeginScene and EndScene calls, and renders each using slightly different projection matrices for each eye. There's a heuristic for it in the driver, but games can have "profiles" stored in the driver registry tweaking the heuristic for parts of the scene. Of course, many games that are not designed with 3D Vision in mind do all kinds of tricks with rendering (especially for shadows), and that's the main reason they look terrible in 3D Vision Automatic. So, why not Automatic for OpenGL? I can only guess: OpenGL has many, many different ways to "render scenes" due to its long history and many different specs. It seems that it would be much harder to implement 3D Vision Automatic for OpenGL, and there would be much more room for breakage. So, NVIDIA just decided not to even try. [Correction: apparently Automatic for OpenGL was supported until 2008.]
2. But... you don't need to use 3D Vision Automatic. There is a very poorly documented method for putting in the left- and right-eye images in yourself. This involves doing a StretchRect to your device's back buffer surface, whereby the origin surface includes two side-by-side images and a special, magical NVIDIA signature in the last row. Even though the origin is double the width of the destination, and also has an extra row, the driver recognizes the magical signature and turns on 3D mode. Where is the documentation for this sorcery?! You can kinda get an overview of it in this presentation, from page 37.
3. And where is the API mentioned in the presentation? Ha! The only official distribution I found of it is very obscure. It's in the NVIDIA SDK, but not in the usual places. It's actually hidden inside the "StereoIssues" code sample. (Here's a direct link to it.) And even in the code example, it's hidden in the /Direct3D/Source/StereoIssues directory. Finally after a long journey you will find the "nvstereo.h" file mentioned in the presentation. There you will find the definition of the NVSTEREOIMAGEHEADER, including the crucial NVSTEREO_IMAGE_SIGNATURE, which you need to send in your StretchRect.
4. The StereoIssues sample is also the only place I found "nvapi.h" and "nvapi.lib". The header file is surprisingly well-documented for NVIDIA! Unfortunately, it's not overly useful, at least not initially. It will let you hook into the 3D Vision driver's user interface, so that you can detect when users press the assigned keys for turning stereo on/off, changing depth and convergence, etc. But since we're not using 3D Vision Automatic, it's up to us to implement what actually happens when the user presses these keys. Unless you really want to mimic the "3D Vision Automatic experience", you might as well implement these features (and/or others!) in your own game settings. For example, maybe setting convergence is not good enough for you, and you want to let users tweak the field-of-view. We'll get to that later. Also note that most of "nvapi.h" only works in Windows Vista+, and the 3D Vision stuff is only a small part of it. Anyway, the bottom line is that you do not need "nvapi.h" at all to make any of this work. In fact, it might even confuse you. For example, a NvAPI_Stereo_Activate call will not simply make your application work.
5. Let me elaborate that point: because you're not using 3D Vision Automatic, it's going to be up to your application to generate the images for each eye. You'll have to generate each scene twice, from the point of view of each eye. Since I'm using the OpenGL interop, this meant that I actually created two frame buffer objects. I switch between them, rendering the scene to each, before finally "flushing" both eyes to the final Direct3D surface and then the back buffer. Rendering for each eye sounds hard, but actually it's great, because you get complete control over what each eye sees. You don't lose any performance over this compared to 3D Vision Automatic: after all, it's exactly what 3D Vision Automatic itself does behind the scenes. Actually, you can make things work faster than Automatic because you can optimize: some parts of the scene might not have to be rendered twice, and can be more simply copied to each eye surface. Futhermore, there are a lot of things that can be handled in stereoscopy without using any 3D API. For example, making 2D overlays appear distant or near can be done with simple separation. You can even handle things like shadows in 2D using simple techniques. Doing stereoscopy well is hard: it depends a lot on the field-of-view of your game, and exactly what kinds of scenes you are displaying. I suggest you read up on some of the optical theory of this to get the best user experience.
6. That final, magical StretchRect is indeed magical. The NVSTEREO_IMAGE_SIGNATURE actually seems to trigger an entirely different path in the driver than the usual StretchRect API. Once discovered, it enables 3D mode in your driver and nonitor, and displays the two side-by-side images in proper stereoscopy. Yay!
7. How weird is this magical StretchRect? Weird. For example, it seems to ignore the source and destination rects, so you don't have to set them: it always uses the entire source buffer and the entire back buffer. On that note, it seems that the dwWidth, dwHeight, and dwBPP fields in the NVSTEREOIMAGEHEADER are also ignored. Only the dwFlags field seems to be used.
8. This magic is incredibly annoying: there is no coherent error message from the StretchRect and no way to know why the driver is failing to enable stereo. Indeed, sometimes stereo won't work at all, and you will just see the side-by-side images (as well as that bottom row with the signature!). Makre sure that you test the return codes for every single API call until you get to the final StretchRect, to be sure that everything is OK along the way. Another bizarre error that happened to me was that I did get stereoscopy to be enabled, but only the left eye was showing! Again, no error messages, no nothing. I had to do a lot of trial and error to find out what works. Miserable.
9. Fullscreen or windowed? By default, the magical StretchRect will only work in fullscreen mode: it will just be ignored in windowed mode, even if you force 3D mode to be "always on" in the NVIDIA settings. But, as you know, there are a few select applications that do seem to work fine in windowed mode. For example, the browser plugins for Firefox/IE, and the 3D Vision movie player. How come it works for them and not for you? Astoundedly, allowing for windowed 3D mode seems to be triggered by the name of your executable. Yes, as far as I can tell these names are hardcoded in the driver. I have not found entries in the registry. Wow. So, if you want 3D working for your app in windowed mode (especially useful for testing during development) then just rename it to "googleearth.exe". Yes, it's that simple and that stupid. Just, wow. I would really like to get more information on this bizarre issue.
10. Another annoyance: in your call to CreateDevice you can set a flag in D3DPRESENT_PARAMETERS to enable fullscreen mode. I personally have not been able to make fullscreen work properly, perhaps due to my specific environment (I use SDL to create the window), though I found that if my window is fullscreen I can use windowed CreateDevice and 3D Vision still works. Weird, right? But, be that as it may, the CreateDevice can fail with an error code that is not documented in the Windows API. I imagine that this is a particularity in the NVIDIA Direct3D implementation. Anyway, very annoying. How am I supposed to know what an unknown error code means?
11. So far I only talked about 3D Vision, but what about the OpenGL/Direct3D interop? This adds a lot of potential for things going wrong, which is again made doubly miserable due to the lack of error messages. Even if you don't use OpenGL, I suggest you read on, because it taught me further lessons about 3D Vision oddities.
12. First of all, the API for the OpenGL/Direct3D interop is implemented as a WGL extension named WGL_NV_DX_interop. Read the documentation very carefully and make sure you follow the rules for creating your Direct3D device and OpenGL textures and frame buffer objects. Importantly, things work differently for Windows XP and Windows Vista+. The advent of WDDM means that you must use Direct3D 9Ex for Vista+. How can you know at runtime? You can use GetVersionEx and see if the major version is 6+.
13. The example code included in the WGL_NV_DX_interop documentation is minimal and incomplete. For a more thorough example see this great contribution from Snippets & Driblits. Unfortunately, that example is also not a complete application, and is also not well-documented in the code. But I learned a lot from studying it, and especially was encouraged from their bottom line: this whole thing actually works!
14. When creating your frame buffer, make sure to call wglDXLockObjectsNV before you call glFramebufferTexture2D or other OpenGL frame buffer object APIs. If you don't, glCheckFramebufferStatus will return an unknown error code. (I'm getting really tired of these unknown error codes, aren't you?)
15. What kind of surfaces would work? You'll see in many of the examples for 3D Vision posted online that people use CreateOffscreenPlainSurface for the final surface (by "final surface" I mean the side-by-side surface plus the NVSTEREOIMAGEHEADER that you feed into the magical StretchRect). Unfortunately, the OpenGL/Direct3D interop does not support offscreen plain surfaces, and there are limitations in the Direct3D API that mean that if I don't use offscreen plain surfaces for my sources, I cannot use an offscreen plain surface for the final surface. See the StretchRect API documentation for a table of what kinds of source surfaces can be stretched to what kinds of destinations.
16. OK, so if CreateOffscreenPlainSurface doesn't work, you could just use CreateRenderTarget for all your surfaces, right? Well, kinda. The OpenGL/Direct3D interop works fine with this: I was able to create OpenGL frame buffer objects, and use StretchRect to the final surface. But... 3D Vision did not work properly. I got it to turn on, but only one eye was showing. Great. No error code, no nothing.
17. I have no explanation for this final lesson. Why should 3D Vision care about the source from which you stretch to the final surface? But, here's the fact: when I created my source surfaces using CreateTexture and GetSurfaceLevel, the one-eyed bug disappeared. Somehow CreateRenderTarget and CreateTexture create a different kind of raster, with the former, after a StretchRect, making 3D Vision unhappy. I have no idea what the difference could be, because if I disable 3D Vision and just show the final surface on the screen, using either kind of surface produces an identical result. Anyway, with CreateTexture, both OpenGL/Direct3D and 3D Vision are happy. I'm happy, the players will be happy, everybody's happy.
18. Final issue, and it could potentially be a big one for you: OpenGL and Direct3D have different coordinate systems. (0,0) is bottom-left in OpenGL, top-left in Direct3D. So, all your surfaces will appear upside down. One solution is to ask your players to stand on their head ;). Another solution could be to flip the surface using some kind of blit function, but that would require extra memory for a target surface, and is a waste. What I did instead is put a flag in my code, and make sure I render everything upside down in OpenGL when I know I that I am sending it to Direct3D in the end. Three potential sub-issues to consider: 1) The easy part is creating a projection matrix for flipped mode; 2) Since your surfaces are rendered in OpenGL, they are counter-clockwise from the outside, but that will be reversed with the flip. You don't have to change your vertex order. Well, you can, but I imagine it would be very hard to do in your code. A much easier fix is something like: glCullFace(flip ? GL_FRONT : GL_BACK); 3) Finally, if you are doing any 2D overlays, you will also have to render them upside down. That might be harder depending on how much control you have over your 2D rendering library.
Though I'm happy I got this to work, I'm very unsatisfied. Between no error codes, unknown error codes, poor-to-non-existent documentation, bizarre hacks hardcoded into the driver and bizarre bugs (only one eye showing?!), I feel just damn lucky that I got all this to work. Lucky, but also angry about what NVIDIA put me through. NVIDIA, you should really support these features properly. My very reasonable wish list:
1. It would be nice to get better documentation for and access to the API
2. It woulld be nice to get any documentation for the proprietary error codes
3. Would it kill you to add some kind of way to get an error code out of that failed magical StretchRect?
4. Could you please document how a 3D Vision application can run in windowed mode? The insane .exe rename hack is not a solution.
Good luck, everybody. May the force be with you.
Hello friends,
This is going to be a long, long post. Like a few other people, I got the combination to work, but there were many pitfalls along the way, lots of trial and error, and mountains of frustation. I want to share my experience so that others would suffer less. :/ I'm posting to MTBS beause it seems like the best minds are here! I hope to get feedback and corrections, as I'm sure I got some details wrong.
The bottom line is that I can develop my game in pure OpenGL, ensuring cross-platform compatibility, while also allowing for 3D Vision on supported Windows boxes. It really truly works: I run my game in Ubuntu, Mac OS X and Windows, and I'm working on mobile (Android/iOS) versions. On Windows I can put on the 3D Vision glasses and enjoy full stereoscopy.
This post is aimed at programmers: there is no magic switch by which you can enable this for any OpenGL game, sorry. I will explain a bit about that.
OK! Let's get to it. Lessons learned:
1. You know how a lot of Direct3D games support 3D Vision even though they weren't designed for it? This is called 3D Vision Automatic, and it only works for pure Direct3D, not OpenGL. It's really clever, actually: it seems that the driver captures all drawing operations between the BeginScene and EndScene calls, and renders each using slightly different projection matrices for each eye. There's a heuristic for it in the driver, but games can have "profiles" stored in the driver registry tweaking the heuristic for parts of the scene. Of course, many games that are not designed with 3D Vision in mind do all kinds of tricks with rendering (especially for shadows), and that's the main reason they look terrible in 3D Vision Automatic. So, why not Automatic for OpenGL? I can only guess: OpenGL has many, many different ways to "render scenes" due to its long history and many different specs. It seems that it would be much harder to implement 3D Vision Automatic for OpenGL, and there would be much more room for breakage. So, NVIDIA just decided not to even try. [Correction: apparently Automatic for OpenGL was supported until 2008.]
2. But... you don't need to use 3D Vision Automatic. There is a very poorly documented method for putting in the left- and right-eye images in yourself. This involves doing a StretchRect to your device's back buffer surface, whereby the origin surface includes two side-by-side images and a special, magical NVIDIA signature in the last row. Even though the origin is double the width of the destination, and also has an extra row, the driver recognizes the magical signature and turns on 3D mode. Where is the documentation for this sorcery?! You can kinda get an overview of it in this presentation, from page 37.
3. And where is the API mentioned in the presentation? Ha! The only official distribution I found of it is very obscure. It's in the NVIDIA SDK, but not in the usual places. It's actually hidden inside the "StereoIssues" code sample. (Here's a direct link to it.) And even in the code example, it's hidden in the /Direct3D/Source/StereoIssues directory. Finally after a long journey you will find the "nvstereo.h" file mentioned in the presentation. There you will find the definition of the NVSTEREOIMAGEHEADER, including the crucial NVSTEREO_IMAGE_SIGNATURE, which you need to send in your StretchRect.
4. The StereoIssues sample is also the only place I found "nvapi.h" and "nvapi.lib". The header file is surprisingly well-documented for NVIDIA! Unfortunately, it's not overly useful, at least not initially. It will let you hook into the 3D Vision driver's user interface, so that you can detect when users press the assigned keys for turning stereo on/off, changing depth and convergence, etc. But since we're not using 3D Vision Automatic, it's up to us to implement what actually happens when the user presses these keys. Unless you really want to mimic the "3D Vision Automatic experience", you might as well implement these features (and/or others!) in your own game settings. For example, maybe setting convergence is not good enough for you, and you want to let users tweak the field-of-view. We'll get to that later. Also note that most of "nvapi.h" only works in Windows Vista+, and the 3D Vision stuff is only a small part of it. Anyway, the bottom line is that you do not need "nvapi.h" at all to make any of this work. In fact, it might even confuse you. For example, a NvAPI_Stereo_Activate call will not simply make your application work.
5. Let me elaborate that point: because you're not using 3D Vision Automatic, it's going to be up to your application to generate the images for each eye. You'll have to generate each scene twice, from the point of view of each eye. Since I'm using the OpenGL interop, this meant that I actually created two frame buffer objects. I switch between them, rendering the scene to each, before finally "flushing" both eyes to the final Direct3D surface and then the back buffer. Rendering for each eye sounds hard, but actually it's great, because you get complete control over what each eye sees. You don't lose any performance over this compared to 3D Vision Automatic: after all, it's exactly what 3D Vision Automatic itself does behind the scenes. Actually, you can make things work faster than Automatic because you can optimize: some parts of the scene might not have to be rendered twice, and can be more simply copied to each eye surface. Futhermore, there are a lot of things that can be handled in stereoscopy without using any 3D API. For example, making 2D overlays appear distant or near can be done with simple separation. You can even handle things like shadows in 2D using simple techniques. Doing stereoscopy well is hard: it depends a lot on the field-of-view of your game, and exactly what kinds of scenes you are displaying. I suggest you read up on some of the optical theory of this to get the best user experience.
6. That final, magical StretchRect is indeed magical. The NVSTEREO_IMAGE_SIGNATURE actually seems to trigger an entirely different path in the driver than the usual StretchRect API. Once discovered, it enables 3D mode in your driver and nonitor, and displays the two side-by-side images in proper stereoscopy. Yay!
7. How weird is this magical StretchRect? Weird. For example, it seems to ignore the source and destination rects, so you don't have to set them: it always uses the entire source buffer and the entire back buffer. On that note, it seems that the dwWidth, dwHeight, and dwBPP fields in the NVSTEREOIMAGEHEADER are also ignored. Only the dwFlags field seems to be used.
8. This magic is incredibly annoying: there is no coherent error message from the StretchRect and no way to know why the driver is failing to enable stereo. Indeed, sometimes stereo won't work at all, and you will just see the side-by-side images (as well as that bottom row with the signature!). Makre sure that you test the return codes for every single API call until you get to the final StretchRect, to be sure that everything is OK along the way. Another bizarre error that happened to me was that I did get stereoscopy to be enabled, but only the left eye was showing! Again, no error messages, no nothing. I had to do a lot of trial and error to find out what works. Miserable.
9. Fullscreen or windowed? By default, the magical StretchRect will only work in fullscreen mode: it will just be ignored in windowed mode, even if you force 3D mode to be "always on" in the NVIDIA settings. But, as you know, there are a few select applications that do seem to work fine in windowed mode. For example, the browser plugins for Firefox/IE, and the 3D Vision movie player. How come it works for them and not for you? Astoundedly, allowing for windowed 3D mode seems to be triggered by the name of your executable. Yes, as far as I can tell these names are hardcoded in the driver. I have not found entries in the registry. Wow. So, if you want 3D working for your app in windowed mode (especially useful for testing during development) then just rename it to "googleearth.exe". Yes, it's that simple and that stupid. Just, wow. I would really like to get more information on this bizarre issue.
10. Another annoyance: in your call to CreateDevice you can set a flag in D3DPRESENT_PARAMETERS to enable fullscreen mode. I personally have not been able to make fullscreen work properly, perhaps due to my specific environment (I use SDL to create the window), though I found that if my window is fullscreen I can use windowed CreateDevice and 3D Vision still works. Weird, right? But, be that as it may, the CreateDevice can fail with an error code that is not documented in the Windows API. I imagine that this is a particularity in the NVIDIA Direct3D implementation. Anyway, very annoying. How am I supposed to know what an unknown error code means?
11. So far I only talked about 3D Vision, but what about the OpenGL/Direct3D interop? This adds a lot of potential for things going wrong, which is again made doubly miserable due to the lack of error messages. Even if you don't use OpenGL, I suggest you read on, because it taught me further lessons about 3D Vision oddities.
12. First of all, the API for the OpenGL/Direct3D interop is implemented as a WGL extension named WGL_NV_DX_interop. Read the documentation very carefully and make sure you follow the rules for creating your Direct3D device and OpenGL textures and frame buffer objects. Importantly, things work differently for Windows XP and Windows Vista+. The advent of WDDM means that you must use Direct3D 9Ex for Vista+. How can you know at runtime? You can use GetVersionEx and see if the major version is 6+.
13. The example code included in the WGL_NV_DX_interop documentation is minimal and incomplete. For a more thorough example see this great contribution from Snippets & Driblits. Unfortunately, that example is also not a complete application, and is also not well-documented in the code. But I learned a lot from studying it, and especially was encouraged from their bottom line: this whole thing actually works!
14. When creating your frame buffer, make sure to call wglDXLockObjectsNV before you call glFramebufferTexture2D or other OpenGL frame buffer object APIs. If you don't, glCheckFramebufferStatus will return an unknown error code. (I'm getting really tired of these unknown error codes, aren't you?)
15. What kind of surfaces would work? You'll see in many of the examples for 3D Vision posted online that people use CreateOffscreenPlainSurface for the final surface (by "final surface" I mean the side-by-side surface plus the NVSTEREOIMAGEHEADER that you feed into the magical StretchRect). Unfortunately, the OpenGL/Direct3D interop does not support offscreen plain surfaces, and there are limitations in the Direct3D API that mean that if I don't use offscreen plain surfaces for my sources, I cannot use an offscreen plain surface for the final surface. See the StretchRect API documentation for a table of what kinds of source surfaces can be stretched to what kinds of destinations.
16. OK, so if CreateOffscreenPlainSurface doesn't work, you could just use CreateRenderTarget for all your surfaces, right? Well, kinda. The OpenGL/Direct3D interop works fine with this: I was able to create OpenGL frame buffer objects, and use StretchRect to the final surface. But... 3D Vision did not work properly. I got it to turn on, but only one eye was showing. Great. No error code, no nothing.
17. I have no explanation for this final lesson. Why should 3D Vision care about the source from which you stretch to the final surface? But, here's the fact: when I created my source surfaces using CreateTexture and GetSurfaceLevel, the one-eyed bug disappeared. Somehow CreateRenderTarget and CreateTexture create a different kind of raster, with the former, after a StretchRect, making 3D Vision unhappy. I have no idea what the difference could be, because if I disable 3D Vision and just show the final surface on the screen, using either kind of surface produces an identical result. Anyway, with CreateTexture, both OpenGL/Direct3D and 3D Vision are happy. I'm happy, the players will be happy, everybody's happy.
18. Final issue, and it could potentially be a big one for you: OpenGL and Direct3D have different coordinate systems. (0,0) is bottom-left in OpenGL, top-left in Direct3D. So, all your surfaces will appear upside down. One solution is to ask your players to stand on their head ;). Another solution could be to flip the surface using some kind of blit function, but that would require extra memory for a target surface, and is a waste. What I did instead is put a flag in my code, and make sure I render everything upside down in OpenGL when I know I that I am sending it to Direct3D in the end. Three potential sub-issues to consider: 1) The easy part is creating a projection matrix for flipped mode; 2) Since your surfaces are rendered in OpenGL, they are counter-clockwise from the outside, but that will be reversed with the flip. You don't have to change your vertex order. Well, you can, but I imagine it would be very hard to do in your code. A much easier fix is something like: glCullFace(flip ? GL_FRONT : GL_BACK); 3) Finally, if you are doing any 2D overlays, you will also have to render them upside down. That might be harder depending on how much control you have over your 2D rendering library.
Though I'm happy I got this to work, I'm very unsatisfied. Between no error codes, unknown error codes, poor-to-non-existent documentation, bizarre hacks hardcoded into the driver and bizarre bugs (only one eye showing?!), I feel just damn lucky that I got all this to work. Lucky, but also angry about what NVIDIA put me through. NVIDIA, you should really support these features properly. My very reasonable wish list:
1. It would be nice to get better documentation for and access to the API
2. It woulld be nice to get any documentation for the proprietary error codes
3. Would it kill you to add some kind of way to get an error code out of that failed magical StretchRect?
4. Could you please document how a 3D Vision application can run in windowed mode? The insane .exe rename hack is not a solution.
Good luck, everybody. May the force be with you.