Opentrack output as FreePIE input

[stupor-mundi]

I was using OpenTrack with PointTracker for IL-2 (great battles) headtracking. I've now got an EDTracker, which I like, but which is 3DOF.
Now I'd like to still use the pointtracker input for Y (in opentrack parlance), and possibly x and z, while using the EDTracker for head yaw and pitch. EDTracker also goes through Opentrack into IL-2.
For both my old and new setup I have Opentrack set to Freetrack output, which IL-2 understands fine.

I found I'm able to run 2 opentrack instances at the same time, 1 reading from pointtracker and the other reading from EDTracker (which presents itself as a joystick).

So I thought I might use FreePIE as a mixer, to merge those two streams, and output as freetrack.
I'm sure some of this could be done by skipping opentrack, but I'd like to leave opentrack in there because I like the ability to tweak the curves visually.

So I thought I could output from Opentrack to vjoy devices, which freePIE would read, but saw a thread explaining that freepie will only write to those, not read.

So I wonder, is there a good way to output from opentrack and input into freepie? (without using up the freetrack protocol)

Or should I turn the whole thing around, get EDTracker input and point tracker input into FreePIE first, output to vjoy, which opentrack will accept as joystick input (i would hope) ?

A problem with the latter approach: Pointtracker exists as an opentrack plugin and a facetracknoir plugin, but I can't find a standalone version. Even if it existed, the issue of what-to-output-to, so that FreePIE can read it, would still persist.

[Jabberwock]

There might be some misunderstanding: FreePIE cannot read internally from vjoy, but it should have no problem reading the emulated devices as a regular controller with joystick[] function. I might be wrong here and cannot test it right now, but I see no reason why that would not work.

[stupor-mundi]

Thanks Jabberwock, I retried and was indeed able to get the input with joystick[]! Awesome!

But I ran into a number of new problems.

One is, that EDTracker (or rather, the edtracker software on windows) *MAY* be affected by accessing this? I had used my, fairly new EDTracker device for a few days only, without problems, and now, after tracking fine for 10,20 seconds, it loses connection ('no Device') for some seconds and then resumes tracking.

The possible confounder here is that in the meantime, in IL-2, constantly got that audio alert sound of a (usb) device unmounting/mounting, didnt know what it was, and had to disable the sound alerts. Now it seems likely that it was this device. And this didn't happen while I was running the FreePIE script which would attempt to access joystick[]. I thought that maybe there had been a surprise windows 10 update but can't find one in the history for the relevant dates.

Another is that, on multple attempts to run OT and read, different indexes in the joystick[] array were apparently used.
I tried joystick[0] and so on. Is it possible to use joystick["aname"] and if yes, how would you find out the names?

But the biggie is that, no matter how many vjoy devices there are (i run 3 currently), OT only tries one, probably "vjoy device #1" in vjoy-monitor speak, so, when I start the first OT instace outputting to vjoy, it works, and when i try the second one, it fails with mostly spurious dialogs, but it's clear what's going on.

OT will also output to 'mouse emulation' but I dont really know what that entails. when I tried it it didn't seem to affect the mouse pointer on screen. Is that maybe an additional virtual 'mouse'?

OT will also output to UDP. Would it be significantly harder to read UDP into FreePIE? If I were to use UDP for both OT instances I would be avoiding any of the vjoy / joystick-index related issues...

[Jabberwock]

If there are the USB reconnections, I would say it is more likely a hardware problem - a faulty cable/plug/port or a cable that is too thin/too long. I do not think a software driver could affect that, even less a software that reads the drivers (as it is separated by several layers from that). But who knows...

Yes, you can get the joystick by the name, it should be put in exactly as it appears in the "Game controllers" window - the names might be different in other parts of Windows.

For Opentrack issues, it seems there is little that can be done. You might hunt down the vjoy selection in code and compile a second version with different assignment... Or add an issue to the github, as I see, they are preparing another release, so maybe they could add the selection.

For what it is worth, FreePIE does have UDP input for reading data from its Android IMU application. However, I do not think it works with the Opentrack output - the protocols are probably different. Even more, Opentrack has support for the FreePIE protocol, but only as input, not as output. Another GitHub issue, maybe? But it seems that their resources are quite limited...

Have you tried outputting freetrack from two OT instances with select axes/values only? What happens if you output XYZ from one instance and yaw/pitch from another?

[stupor-mundi]

It turned out to be the cable.
Until I receive the new long cable, I can't properly use IL-2, but I was able to try out your suggestion, Jabberwock.
This succeeded much further than I had expected, but not till the end.
Opentrack was actually content, the two instances, to output to the freetrack protocol without complaining.
Not sure exactly what that (freetrack) really is, but it doesn't appear to be a singleton of sorts, such as a port that is bound.
However, though, under 'output' , in one OT instance, I 'disabled' X,Y,Z, and, in the other, 'disabled' the rotations, it appears that freetrack works in such a way (or OT's implementation of it) that the 'disabled' parts are still getting written.
So that, when I tried this out in IL-2, the view stuttered, fractions of a second, between the two inputs.

I suppose I'll try 'mouse emulation' for the pointtracker OT and will try to find out where that data ends up at.

[Jabberwock]

I wonder what would happen if you disabled the relevant axes/values in the two instances of OT, send them over UDP and then receive them together with a third instance? I think still all six values are sent, but maybe they are somehow flagged as disabled... That way you could use the third instance to bring it all together and send as freetrack...

I know next to nothing about networking in Python, but from what I understood, writing a Python UDP OT receiver could be doable. If you managed to do that in FreePIE, the rest would be easy. Note though that Python in FreePIE is somewhat limited, so I am not sure it is even possible.

[Jabberwock]

Here is an example of such code:

https://stackoverflow.com/questions/384 ... -real-time

[stupor-mundi]

Thanks Jabberwock,

I've adapted the code from the example to the FreePIE situation, like this:

Code: Select all

import socket
import sys


def connect(port):
    global s
    global connected

    connected = False

    host = '127.0.0.1'

    # Datagram (udp) socket
    try:
        s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
        diagnostics.debug('Socket created')
    except socket.error, msg:
        diagnostics.debug('Failed to create socket. Error Code : ' + str(msg[0]) + ' Message ' + msg[1])
        return

    # Bind socket to local host and port
    try:
        s.bind((host, port))
    except socket.error, msg:
        diagnostics.debug('Bind failed. Error Code : ' + str(msg[0]) + ' Message ' + msg[1])
        return

    diagnostics.debug('Socket bind complete')
    connected = True


def fetch():
    global s

    # receive data from client (data, addr)
    d = s.recvfrom(1024)
    data = d[0]
    addr = d[1]

    if data:
        reply = 'OK...' + data
        s.sendto(reply, addr)

        #unpacked_data = struct.unpack('dddddd', data)
        #x = unpacked_data[0]
        #y = unpacked_data[1]
        #z = unpacked_data[2]
        #return (x, y, z)

        return data


def update():
    global s
    global connected

    if connected:
        #(my_a, my_b, my_c) = fetch()

        #diagnostics.watch(my_a)
        #diagnostics.watch(my_b)
        #diagnostics.watch(my_c)

        data = fetch()
        diagnostics.watch(data)


if starting:
    connect(8080)
    freeTrack.update += update

This runs, and indeed receives data from OT.
I'm guessing, this bit in the example:
struct.unpack('dddddd', data)
... refers to some code of the author which was outside of the example scope.

So for the situation at hand, the question will be how does OT package the data before sending off a UDP datagram.
I assume that question is outside of the FreePIE community and falls within the remit of the OT community, which, sadly, AFAICS, doesn't have a forum of its own.

[Jabberwock]

You can hunt some useful bits from the GitHub issues, like here:
https://github.com/opentrack/opentrack/issues/747


Struct is a regular Python module, but it might not be available in FreePIE. You can copy it from a regular Python installation, but that does not always work.

[stupor-mundi]

Using struct worked fine, just with import struct, without having to add anything to FreePIE.
The data from OT turned out to be:
unpacked_data = struct.unpack('dddddd', data)
diagnostics.debug('fetch5')
x = unpacked_data[0]
y = unpacked_data[1]
z = unpacked_data[2]
yaw = unpacked_data[3]
pitch = unpacked_data[4]
roll = unpacked_data[5]
(possibly a different order of yaw, pitch, roll, didn't get around to verify that)

In this way, reading the UDP data from OT worked just fine.

However, I wanted to carry on and read UDP data on two sockets from two OT instances, and, as I did so, in the process of playing around with the 2 OT instances, starting and stopping and changing the config, more bleeding over of state (at least config state) from one instance to the other became apparent, which caused me to have second thoughts about the doability of the 2 OT instances approach.

I was doing config in one instance and then it turned out I had accidentally changed some state on the other, and so on. Thus I'll have to rethink the approach.

But, for what it's worth, it looks as if, if it weren't for the 2-instance OT issues, it would be a viable approach to receive UDP data on two sockets in FreePIE, pick the desired axis, and write those to FreeTrack.

===

I see there is a new 'portable' mode in OT. I'll play around with that a little, maybe it will remove the issues.

[Jabberwock]

As your motivation for using Opentrack are the filter curves (although that might be reproduced to an extent with FreePIE filters), I think you should still look into the 'turn around' approach, i.e. get the raw data in FreePIE, send that to OT and integrate that there (and send to a game).

As you have managed to get the UDP working in FreePIE (great work, could you post the code? I will certainly use that sometime), maybe you should look into getting the UDP input from FaceTrackNoIR? The protocol is described here:

http://facetracknoir.sourceforge.net/Trackers/UDP.htm

Then send it over as joystick data (or UDP?) to OT...

[stupor-mundi]

The OT 'portable' mode, AFAICS, removed any state bleeding from instance to instance.
So I continued on this approach, which seems close to working now. Once it fully works, I'll consider whether the inverse approach would have any significant advantages.
The problem I have at the moment, is with writing to FreeTrack in the update() function. I always had that
freeTrack.update += update
line in there, which I saw as an indication that writing to freeTrack would pose no problem. But maybe something needs to be done, maybe initiallize it or such?
The code is below. With the freeTrack writing lines commented out, it works fine, but with them active, FreePIE just appears to stop, without error, after the first pass through the update function.
My python is very rusty, and maybe the way I'm attempting to log exceptions is missing out on something.

The socket binding is not very robust. Both OT instances, outputting to UDP on those 2 ports, have to be running before this FreePIE script can be started.

If anything breaks during socket binding attempts, or many other conditions, a backgrounded FreePIE bound to those ports may be left over and has to be killed before the next run can be successfull.

Code: Select all

import socket
import struct


def update():
    global rot_sock
    global xyz_sock
    global connected_rot
    global connected_xyz

    diagnostics.debug( str(rot_sock) )
    diagnostics.debug( str(xyz_sock) )

    if connected_rot:

        try:
            diagnostics.debug('connected_rot')
            (got_a, x_a, y_a, z_a, yaw_a, pitch_a, roll_a) = fetch(rot_sock)
            diagnostics.debug('have read rot')
            diagnostics.watch(got_a)
            diagnostics.watch(x_a)
            diagnostics.watch(y_a)
            diagnostics.watch(z_a)
            diagnostics.watch(yaw_a)
            diagnostics.watch(pitch_a)
            diagnostics.watch(roll_a)

            #freeTrack.yaw = yaw_a
            #freeTrack.pitch = pitch_a
            #freeTrack.roll = roll_a

            diagnostics.debug('done_rot')
        except Exception as ex:
            diagnostics.debug('fetch a blew up')
            diagnostics.debug(ex.message)

    else:
        diagnostics.debug('not connected_rot')

    if connected_xyz:

        try:
            diagnostics.debug('connected_xyz')
            (got_b, x_b, y_b, z_b, yaw_b, pitch_b, roll_b) = fetch(xyz_sock)
            diagnostics.debug('have read xyz')
            diagnostics.watch(got_b)
            diagnostics.watch(x_b)
            diagnostics.watch(y_b)
            diagnostics.watch(z_b)
            diagnostics.watch(yaw_b)
            diagnostics.watch(pitch_b)
            diagnostics.watch(roll_b)

            #freeTrack.x = x_b
            #freeTrack.y = y_b
            #freeTrack.z = z_b

            diagnostics.debug('done_xyz')
        except Exception as ex:
            diagnostics.debug('fetch b blew up')
            diagnostics.debug(ex.message)

    else:
        diagnostics.debug('not connected_xyz')

    diagnostics.debug('update end. not blown up')


def connect(rot_port, xyz_port):
    global rot_sock
    global xyz_sock
    global connected_rot
    global connected_xyz

    connected_rot = False
    connected_xyz = False

    host = '127.0.0.1'

    try:
        rot_sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
        xyz_sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
        diagnostics.debug('Sockets created')
    except socket.error, msg:
        diagnostics.debug('Failed to create socket. Error Code : ' + str(msg[0]) + ' Message ' + msg[1])
        return

    rot_sock.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, 1)
    xyz_sock.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, 1)

    try:
        rot_sock.bind((host, rot_port))
        connected_rot = True
        diagnostics.debug('Rotation socket bound to port ' + str(rot_port) )
    except socket.error, msg:
        diagnostics.debug('Bind failed on rotation socket. Error Code : ' + str(msg[0]) + ' Message ' + msg[1])
        return

    try:
        xyz_sock.bind((host, xyz_port))
        connected_xyz = True
        diagnostics.debug('XYZ socket bound to port ' + str(xyz_port) )
    except socket.error, msg:
        diagnostics.debug('Bind failed on xyz socket. Error Code : ' + str(msg[0]) + ' Message ' + msg[1])
        return
    except Exception as ex:
        diagnostics.debug(ex.message)
        return

    diagnostics.debug('Sockets bind complete')


def fetch(sock):
    d = sock.recvfrom(1024)
    data = d[0]
    addr = d[1]

    if data:
        reply = 'OK...' + data
        sock.sendto(reply, addr)
        unpacked_data = struct.unpack('dddddd', data)
        x = unpacked_data[0]
        y = unpacked_data[1]
        z = unpacked_data[2]
        yaw = unpacked_data[3]
        pitch = unpacked_data[4]
        roll = unpacked_data[5]
        return (True, x, y, z, yaw, pitch, roll)
    else:
        diagnostics.debug('no data')
        return (False, 0, 0, 0, 0, 0, 0)


if starting:
    connect(8080, 8081)
    freeTrack.update += update

[Jabberwock]

freeTrack.update += update means that update() will run only if freeTrack values are updated. If nothing is writing to freeTrack, then it probably gets some random values. But if you start writing to it, then it runs once and stops, because nothing else is changing FT values.

update() is meant to save cycles when your inputs do not change. That is why it should be used with your input device (I am not sure if you can bind it to more than one). Without it the code just gets executed at FreePIE regular frequency (which I never remember), which might decrease performance if the code is heavy.

One thing I forgot to mention: if you have access to Kinect, you might try it for head position - I use it for VR with one of the two FreePIE Kinect plugins.

[Jabberwock]

When I wrote above, I have realized I do not remember whether OT supports Kinect. It turns out they have added the support... four hours ago!

[stupor-mundi]

Clearly I had too vague an idea of what the line
freeTrack.update += update
actually meant, when I saw it in examples.

So, in the case of reading UDP, if there were a generic UDP plugin, it would be appropriate to do:
udp.update += update

But since this is generic networking code, no such thing. I now re-wrote/re-tried it, with invoking both my update() and write_out() functions from the, let's say, free floating code (not sure what that's properly called). I.e. the bottom of the script now looks like this:

Code: Select all

if starting:
    setup()
    connect(8080, 8081)

if True:
    update()
    write_out()

In terms of, reading from the sockets, and writing to freetrack, this approach works, i.e. logs and diagnostics show that values are continuously being picked up and written out.

There is the minor problem that I don't really know, due to my lack of experience with UDP sockets, what happens if you attempt to read from them too often or too rarely. But the values shown in diagnostics certainly look legit.

The bigger problem is that the freetrack data apparently doesn't arrive at IL-2, or maybe not at a frequency or suchlike, that makes it look like decent freetrack data to IL-2.

IL-2 (BoX) doesnt have config for choosing a headtracking protocol. In my experience, when freetrack is provided from somewhere (in my case that has only been OT so far), it just switches to it.

I'm writing to freetrack at the default FreePIE update frequency, like this:

Code: Select all

    freeTrack.yaw = g_yaw
    freeTrack.pitch = g_pitch
    freeTrack.roll = g_roll
    freeTrack.x = g_x
    freeTrack.y = g_y
    freeTrack.z = g_z

I'm writing the the exact data for position and angles that I receive from OT.

The only other thing that the freeTrack object can do, according to the code completion, is
freeTrack.update
which is the update event, to be used when reading from it.

In my desperation I attempted to call it, like so
freeTrack.update()
which only leads to an error:
"bound event is not callable"

At this point, I don't know how to proceed, debugging wise, because I have no idea what freeTrack actually IS. Certainly it's a DLL inside FreePIE.

But in terms of IL-2, reading from it, what is it? OT, when choosing FreeTrack output, calls it FreeTrack 2 enhanced, and gives the option of outputting both FreeTrack and TrackIR at the same time, or just one. I believe I tried the variants and they all worked.

Timing-wise, I'm logging a fair amount of stuff at the moment, would that slow down the refresh loop? Looking at the logging, the whole read/write thing only seems to happen at a few Hz, certainly not at the speeds EDTracker or OT pointtracker are capable of.

To get things to happen at higher Hz, would there be a technique in FreePIE?
Maybe a
magicTimer.update += update
or suchlike?

[Jabberwock]

I have no idea why it would not work... While I have not been using (I think... my script became a bit unmanageable) Freetrack from FreePIE for any simulators, I am using it, routinely for VR with Vridge, so I am quite certain it works correctly. My simplest FreeTrack script is just this:

Code: Select all

if keyboard.getKeyDown(Key.A):
	freeTrack.x = freeTrack.x - 1

if keyboard.getKeyDown(Key.D):
	freeTrack.x = freeTrack.x + 1
	
if keyboard.getKeyDown(Key.W):
	freeTrack.z = freeTrack.z - 1
	
if keyboard.getKeyDown(Key.S):
	freeTrack.z = freeTrack.z + 1
	
if keyboard.getKeyDown(Key.PageDown):
	freeTrack.y = freeTrack.y - 1
if keyboard.getKeyDown(Key.PageUp):
	freeTrack.y = freeTrack.y + 1

	
diagnostics.watch(freeTrack.y)

if starting:
	freeTrack.y = 1800
	freeTrack.x = 0
	freeTrack.z = 0

But note that VRidge only expects positional values, so maybe for IL-2 the angles should be initialized/updated as well.

You are saying that you are using the exact values from OT... but they are not FreeTrack values, are they? Maybe the value ranges need to be adujsted somehow? Try to output Freetrack data in OT, capture them in FreePIE and check whether they are reasonably similar to what you were sending.

FreePIE supports also TrackIR output, have you tried this? As I understand, it is harder now as there is some encryption involved, but not in all titles. If it worked for OT, it should also work with FreePIE.

[stupor-mundi]

Looking at your code Jabberwock, and also some sample code I had seen, I gather the values are absolute, not deltas.
Which is also what I have done. If the ranges for freetrack, for example for angles, are different than -180 to 180, let's say larger, it would be difficult to imagine that IL-2 would reject headtracking information because the values are too modest. OTOH, if the freetrack values were just -1.0 to 1.0 or something like that, I could imagine out-of-bounds values to be rejected. Hmm.
I still think there is the possibility of rejection because of a too-slow update rythm maybe.
Regarding timing, I found a thread, and might experiment a little:
https://github.com/AndersMalmgren/FreeP ... -141589044
I've now also tried writing to trackIR with the same result.

Depending on what freetrack and trackir actually are (maybe they are also just udp packets at a port?), I could imagine that IL-2 would reject those based on the identity of who's writing to them, i.e. accept OpenTrack because it can somehow identify headtracking FROM OpenTrack, and so on...

It would be interesting to hear from anyone who's managed to transmit freetrack or trackIR to IL-2.

I'll also ask this on the IL-2 forum...

[stupor-mundi]

Maybe I found something :
https://github.com/opentrack/opentrack/ ... /fttypes.h

the stuff at the bottom, like that
int32_t GameID;

maybe suggests that the sender sets an ID of who is sending?

Possibly IL-2 will only read from certain senders?

[Jabberwock]

I have tested it myself and what is seen as degrees in OT is sent through freetrack as radians. Centimeters in OT are output in FreeTrack as milimeters.

Update speed should not matter, as FreeTrack libraries seem to retain the values over time (that is why in my script I do not have to update them constantly).

[stupor-mundi]

I tried again, converting the degrees to radians, to see if those too-large values had made the freetrack output unacceptable to IL-2, but nothing changed.
I tend towards thinking there is something fundamental about the freetrack output from FreePIE which IL-2 rejects, maybe the version, maybe the GameID, etc.
So I took a different approach and added a third OT instance. This may seem ridiculous, but since the OT instances are now very neatly separated from each other, this was no work at all.
My FreePIE script now outputs to VJoy instead of freetrack, and the third OT outputs freetrack, which IL-2 accepts.
What's nice is that OT, when reading input from a VJoy, you get to pick which one. I.e. I was able to add another VJoy, write to that one, so as not to interfere with the VJoy devices I have already set up for other purposes. This works now, I still have to work on the curves in OT, to make it usable, but it's basically done.

Now that this is working, I'm left with the task of merging my new script with the one that I had been running, to use a MIDI device (Behringer X Touch) as input for IL-2 (dials, sliders, and buttons).

So I wonder what people typically do in FreePIE in this situation. Create Python modules and import them, or just concatenate, and end up with longer scripts?

[Jabberwock]

It might be part of the anti-competition measures on the poart of NaturalPoint. It seems that Battle of Stalingrad has a setting old_trackir... Maybe it is worth a try?

[stupor-mundi]

Since I have it working now, with 3 OT, I think for now I'm gonna use it like this for a bit.
When/if I find the energy to do it, I'll try that oldtrackir setting.
I'll also post on the IL-2 forum, maybe I can get some info about what (or from where) exactly IL-2 will accept.

Should I post the (for now) final version of the script here? It's not very different from the earlier one I posted.

[stupor-mundi]

Having used this now for a while, a few observations.
In practical terms, the use of the 3 instances of Opentrack isn't a problem. When I get fed up starting them by hand, I'll write a script.
Finally I have the head movement working the way I wanted it, in IL-2, it makes a big difference.
There is too much latency. For some time I thought I'd get used to it, but it's too much. That really has negative effects when zooming in, in IL-2.
With a multi tier system it's tricky to figure out the contribution of the various bits, but, based on this thread
https://github.com/AndersMalmgren/FreePIE/issues/69
I think the execution of the 'free floating' code I have in my script is 64hz, the call rate of the FreePIE main loop.
Or maybe it's a bit slower if the execution time of my script goes on top of that.
I did notice that removing lots of those diagnostics.debug() statements made it a little faster.
So I think it's basically those 64hz which are responsible for the latency.

So I wonder: is there a way (short of changing code in FreePIE and building it myself) to tell FreePIE to execute something more frequently?
Since I only use this with IL-2, where pretty much only 1 cpu core is ever remotely used up (it's gpu limited), I have basically unlimited cpu to throw at this...

[Jabberwock]

Yes, there is:

https://www.mtbs3d.com/phpBB/viewtopic. ... 39&t=18724

[stupor-mundi]

Oooooh!

Awesome, that worked!