Vireio Perception 1.0 Released!

[Baristan6]

I don't have enough time to finish my Vireio/FreePIE project right now, but here is what I currently have.

The binaries:

Perception 1.4 with SocketTracker for FreePIE.rar

The source code:

Perception 1.4 with SocketTracker source.rar

FreePIE with VireioST Plugin source.rar

It works for me with TrackIR in Skyrim, but I haven't done any other testing. Let me know how it works for you and anything that should be changed.

EDIT
these files are out of date use new ones 13 posts below

[cybereality]

Thanks, Baristan6.

Just a note, we were up to version 1.0.4, not 1.4. If this code is working good we should merge it into the master and make that version 1.0.5. I don't have any trackers on me right now so I can't test this new version. But if other people want to test and post their thoughts I would appreciate it.

[mscoder610]

I think it's still just a one-off testing build for now.
I downloaded it earlier, and tried to adapt it to using the Razer Hydra (through FreePIE) in Left 4 Dead 2.

The roll worked fine, I had issues getting the pitch and yaw working though. (Although really, I could probably just use the Hydra's built in mouse emulation for that part). I'll keep looking at it as I have time.

[2EyeGuy]

I pulled dylanwinn's FreeTrack code and merged in Baristan6's FreePIE code to my repository, and did a pull request.
Other people can pull the combined changes from here: https://github.com/CarlKenner/Perception.git
I haven't really tested the changes, except that they compile and run.

[CyberVillain]

@2EyeGuy I had totally missed that you are Carl Kenner

Do you guys think this should be a Core plugin in FreePIE or should it be shipped with Vireio instead?

edit:
I made Baristan6's plugin syntax a bit more FreePIEish

Code: Select all

using System;
using System.Collections.Generic;
using System.Text;
using FreePIE.Core.Contracts;
using System.Net.Sockets;

namespace VireioSocketTracker
{
    [GlobalType(Type = typeof(VireioSTGlobal))]
    public class VireioST : IPlugin
    {
        private Socket socket;
        private double yaw;
        private double pitch;
        private double roll;
        private bool newData;

        public object CreateGlobal()
        {
            return new VireioSTGlobal(this);
        }

        public Action Start()
        {
            ConnectSocket("localhost", 49015);
            return null;
        }

        public void Stop()
        {
            socket.Close();
        }

        public event EventHandler Started;

        public string FriendlyName
        {
            get { return "A name that will be used from GUI"; }
        }

        public double Yaw
        {
            get { return yaw; }
            set 
            {
                yaw = value;
                newData = true;
            }
        }

        public double Pitch
        {
            get { return pitch; }
            set
            {
                pitch = value;
                newData = true;
            }
        }

        public double Roll
        {
            get { return roll; }
            set
            {
                roll = value;
                newData = true;
            }
        }

        public bool GetProperty(int index, IPluginProperty property)
        {
            return false;
        }

        public bool SetProperties(Dictionary<string, object> properties)
        {
            return false;
        }

        public void DoBeforeNextExecute()
        {
            sendData();
        }

        private void SendData()
        {
            if (!newData) return;

            if (socket.Connected)
            {
                string tmpStr = "<VST>";
                tmpStr += "<yaw>" + yaw.ToString() + "</yaw>";
                tmpStr += "<pitch>" + pitch.ToString() + "</pitch>";
                tmpStr += "<roll>" + roll.ToString() + "</roll>";
                tmpStr += "</VST>";

                byte[] msg = Encoding.ASCII.GetBytes(tmpStr);
                socket.Send(msg);
                newData = false;
            }
            else
            {
                ConnectSocket("localhost", 49015);
            }
        }

        private void ConnectSocket(string server, int port)
        {
            socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
            socket.Connect(server, port);
        }
    }

    [Global(Name = "vireioST")]
    public class VireioSTGlobal
    {
        private readonly VireioST vireioST;

        public VireioSTGlobal(VireioST vireioST)
        {
            this.vireioST = vireioST;
        }

        public double Yaw { set { vireioST.Yaw = value; }}
        public double Pitch { set { vireioST.Pitch = value; } }
        public double Roll { set { vireioST.Roll = value; } }
    }
}

A side note, the code uses TCP, this is overkill if a message gets lots its no use to resend it because it will be too old anyway. I would change the code to use UDP or even a SharedMemory pipeline. Shared memory pipelines are very easy to set up. Check the Freetrack plugin in FreePIE.

[Baristan6]

I think it's still just a one-off testing build for now.
I downloaded it earlier, and tried to adapt it to using the Razer Hydra (through FreePIE) in Left 4 Dead 2.

The roll worked fine, I had issues getting the pitch and yaw working though. (Although really, I could probably just use the Hydra's built in mouse emulation for that part). I'll keep looking at it as I have time.

The scale for yaw/pitch/roll isn't finished in this test version, but yaw/pitch should work. Vireio's roll is achieved through the view matrix, but the yaw/pitch is mouse emulation.

A side note, the code uses TCP, this is overkill if a message gets lots its no use to resend it because it will be too old anyway. I would change the code to use UDP or even a SharedMemory pipeline. Shared memory pipelines are very easy to set up. Check the Freetrack plugin in FreePIE.

Yes switching to UDP was one of the things I planed on adding before submitting it to Vireio and FreePie. I forgot about SharedMemory pipelines. That would simplify things a lot.
I can change the code to SharedMemory this weekend if no one beets me to it.
The plugin isn't dependent on any Vireio code, so I think it would make sense for it to be part of FreePIE.

[cybereality]

OK, 2EyeGuy, I have merged that into master. Thanks Baristan6 and dylanwinn.

@CyberVillain: We can pack the plug-in with Perception if that makes more sense to you.

[CyberVillain]

@Baristan6 Yeah, Shared memory will certainly speed things up

@cybereality Hmm, yeah, currently we only have hardware support in FreePIE Core if we start to support software in the core then there is no end to how much plugins there can be in Core I let you guys decide though. Just let me know if you need any help with the FreePIE side of things

edit: I just added a registry key with the install path to the FreePIE installdir so after next version you can get the FreePIE install folder, so if you decide to put the plugin in Perception then you're installer can check that registry key and copy the plugin to the correct place

[baggyg]

Hi Cyber,

Firstly congrats on the Oculus job. Had a chance to try Skyrim using perception yesterday on LG 3d tv. All looking great and running at 60fps (even on 1080p). I had some luck using some of the nexus 3d vision compatibility project correction meshes for sky / water fix etc to correct a few problems. The 3d vision compatibility project they have there does deal with the shadows so may take a look at some point to see if a similar approach that could be adapted to vireio. However certainly very playable even at this stage. Thanks for the hard work. Cant wait to try in the dev kit!

[brantlew]

A side note, the code uses TCP, this is overkill if a message gets lots its no use to resend it because it will be too old anyway. I would change the code to use UDP or even a SharedMemory pipeline. Shared memory pipelines are very easy to set up. Check the Freetrack plugin in FreePIE.

I already implemented a 6DOF named pipe plugin for FreePIE several months ago, but I never checked it into the core because it's a proprietary solution. Very fast and simple. If you want it to be in the core, I can just check it in or email it to you.

[Dantesinferno]

@brantlew sorry to be a bit off topic but can you check out this forum link http://www.mtbs3d.com/phpBB/viewtopic.p ... &start=165 and could you take it into consideration ?

[bobjwatts]

I managed to Baristan6's Socket Tracker FreePie/Vireio implementation working with the iPhone as a headtracker. Nice to have roll, I have been playing Dear Esther, quite a bit of skew on roll but still cool.

I'm going to give HL2 a try later.

Here's what is working for me, I had to swap pitch/roll because of the physical orientation I am using:

Code: Select all

if  starting:
   iPhone.continuousYawMode = True

yaw = math.degrees(iPhone.yaw)
pitch = math.degrees(iPhone.pitch)
roll = math.degrees(iPhone.roll)

vireioST.setYaw(-yaw*40)
vireioST.setPitch(-roll*40)
vireioST.setRoll(-pitch/20)
vireioST.sendData()

[cybereality]

The skew problem when rolling I think I fixed, but only on the Oculus Rift mode.

[Baristan6]

I removed SocketTracker and created a SharedMemoryTracker.
It should be faster, and uses the yaw, pitch, roll multipliers.
yaw, pitch, and roll are now in degrees. Any SocketTracker scripts will need to be changed.

I used FreePIE's Freetrack plugin as an example for the VireioSMT plugin. Didn't know the best way to add it to the main Vireio project, so I started it's own on GitHub.
Feel free to add it to Vireio, and I can delete the standalone project for the plugin.

Vireio Perception
https://github.com/Baristan6/Perception

Vireo Perception SharedMemory Tracker Plugin for FreePIE
https://github.com/Baristan6/VireioSMT/

Let me know if I missed anything.

EDIT
here are the binaries.

Perception and VireioSMT Mar_4_2013.rar

[mscoder610]

Cool, thanks for the update.
All the code works fine for me - on the FreePIE side everything seems stable (I saw occasional script aborts with the socket version, no issues so far with this one).
And yaw / pitch / roll are all working fine for me with this version, with the Hydra.

[jf031]

This is quite a request (and I'm not going to offer any work of my own, since I've barely started programming [and in that outdated language of C]), but could someone fix the "renders every other frame to one eye" quirk, so it works like the other stereoscopic solutions out there (rendering both eye views at the same time)? The 3D effect disappears upon any even midly-quick movement (due to desync), resulting in strong eye/brain confusion. This seems to be a pretty severe problem; might as well not have stereo if it becomes worse than mono with enough movement. If the Vireio drivers are going to be the go-to for the mildly technically competent devkit owners/non-developers (like me), they could really hurt VR early on with this major problem (people associating VR with this desync issue; I can see it now:"Virtualboy 2.0").

Thanks for the hours and hours of unpaid work to get to the point you did, though, Cybereality! I realize that the stereoscopic 3D is just one small part of all your work on the driver.

[cybereality]

Yeah, of course a simultaneous render is the best, but it was a lot easier to render sequentially. It's not impossible to fix, though, but would take some time.

Basically we would have to proxy a few D3D functions that draw the geometry (for example "DrawPrimitive") and switch the buffer or render target to draw in both left and right views at once. That would work fine, at least for fixed-function pipeline. This may get a little more complicated with shaders, but that would be the basic idea.

[2EyeGuy]

A simultaneous render is really hard, like cybereality said. We need to redraw the entire scene twice. There are only about 7 D3D functions that actually do any drawing. If those functions were all rendering straight to the screen then it would be easy enough to hook them and change the render target then draw it, then change the render target again and draw it again. That would probably hurt performance a lot though, I imagine there's a high cost to changing render targets. We might also be able to do the same thing just with viewports and a single render target that's twice as wide.

If they are rendering to an offscreen surface though, then we need to hook the creation of that offscreen surface and make two of them, or make it twice as big. Although we might not want to do that to all offscreen surfaces.

If we don't want to change render targets all the time, I think there are a few alternatives, but they are very advanced. It might be possible to remember all the Direct3D commands for the scene in a buffer, and then play them back twice. Or it might be possible to use some advanced vertex/pixel shading to render to multiple render targets simultaneously (Direct3D supports multiple render targets at once). Or it might be possible to hook whatever functions the game uses for timing and input, and convince the game that no time has passed and nothing has happened between the stereo frames.

[Fredz]

NVIDIA explained how they did it in their driver (starting at p.50 and p.47 resp.) :
- Stereoscopic 3D Demystified - Nvidia ;
- Implementing Stereoscopic 3D in Your Applications - Nvidia.

It's a bit light on details, but there is still good info there and it could help for a start.

We might also be able to do the same thing just with viewports and a single render target that's twice as wide.

Some time ago I tried to implement a page-flipping stereo driver for Linux. I basically created a virtual screen of twice the width of the current graphics mode, then wrote to the CRTC VGA registers to do hardware page-flipping by alterning the scanout to the left or right part of the screen, all synchronized to the vertical retrace by intercepting the GPU IRQ.

I suspect the NVIDIA driver (at least the older one) may be doing the same thing, since there is a visible graphics mode switch when entering the 3D mode. That makes sense since that basically gives a quad-buffer mode (dual front buffer and dual back buffer).

[MaterialDefender]

It's pretty tough for anything that is shader based. Simply swapping rendertargets back and forth like you would normally do is clearly not enough for an interception driver, since the hooked application doesn't know a bit about your extra targets. Intermediate targets simply don't get used. Kudos to those who have this fully working in a generalized way. I'm close myself, but not quite there.

Alternate frame rendering is so much simpler that slowly I'm even starting to think it's the better solution. At least it does work flawlessly without having to jump through hoops.

But considering the heavy shader and shadow artifacts in many modern games with a geometry based solution, a faster depth buffer based approach in many cases looks better anyway in direct comparison, as far as I'm concerned. Although I'm sure that not everybody will agree on that.

[kazak]

I don't have enough time to finish my Vireio/FreePIE project right now, but here is what I currently have.
The binaries:

Perception 1.4 with SocketTracker for FreePIE.rar

I am very happy, this is the first version that works on my PC.

[2EyeGuy]

At least it does work flawlessly without having to jump through hoops.

jf031 pointed out the obvious flaw, so I wouldn't call any of the solutions flawless yet. Maybe you'll have to stick with your approach of offering multiple methods.

[MaterialDefender]

Yes of course, that definitively is a flaw. Without it there would be no reason to try anything else at all when it comes to geometry based 3D. I should have been more precise, I guess . I meant flawless purely in a technical sense.

In regard to my work nothing has changed anyway, there is a z-buffer based (still my main focus) and a geometry based (AFR, at least for the first release) method, switchable on-the-fly for many games. Both methods are different enough to justify the extra work that this implies.