fingers and hand tracking technology

[JDuncan]

First you should read my "arm tracking technology" thread to see what basic idea I'm using here.

Basically, the idea is there is one spot on the body that has a sensor and reciever and the sensor is in one spot. This is like lighting up on spot on the ground in a dark room with one flash light and the flash light never moves.

This sensor and receiver is on the neck like a dog collar or oxen collar and is held on the neck so it doesn't shift around when the person moves.
The sensor and receiver are right next to each other so when the person bends down the flash light doesn't move, if you use the flash light in a dark room analogy.

Now you have the arm technology show that from sensors you build up a hierarchy that shows where the bones are. So you have a sensor on the elbow and a sensor on the wrist and these show where the forearm is and the sensor on the elbow shows the humerus. Then the sensors on the elbow and wrist go to the sensors on the neck and the SW see's these sensors and estimates the position of the forearm and humerus bones, then the SW shows the forearm and humerus bones moving when the sensors move.
That is arm tracking technology.

Hand and finger tracking technology is the same idea, you need a static position to reference the dynamic positions. The static positions are the still flash lights in a dark room, the dynamic positions are the moving flash lights in a dark room.

The moving flash lights are the sensors on the arm, the one one the elbow and wrist, the still flash light is the one on the neck.
So the neck is used to give the still flash light but there is;
1 sensors on the wrist = 1 sensor
1 sensor on each knuckle = 5 sensors
1 sensor on each bending part of each finger that is not the knuckle = 9 sensors
= 15 sensors

These 15 sensors go to the receiver on the neck, then the SW sees the sensors on the wrist and hands fingers and decides where the bones are.

Come to think of it, the finger tips could use sensors too couldn't they? Then the length of the bone that has the finger nail on it could be found by the SW, so that's 5 more sensors, one for each finger tip and that would bring the total to 20 sensors per hand.
40 sensors for 2 hands, and 41 sensors total if you include the sensor on the neck.

Then the receiver on the neck gets these sensors which have a unique ID the SW can use to find the location of the bones by seeing how far the sensors are from each other.
e.g. the wrist sensor is the far away from the knuckles sensors, the knuckles sensors are this far away from the middle of the fingers sensors, etc.

Then if you want to get fancy there can be a haptic skin that can give feed back to the hands touch sensation, so in VR the hand touches something the sensors go to the receiver the SW decides the hand touches something and the hand feels that thing.

Unless the hand has some kind of robotics to hold the hand in position when it touches some VR object, like a ghost the hand will pass through that object.

So 41 sensors to get the hands up and running with VR, and 43 sensors if you want the arm included, you would add one sensor to the elbow. Now the entire hand and arm is articulated in VR. And with haptic skin on the glove you can feel the VR world too.

[android78]

I think that the sensors for every part of the fingers that bend is a little redundant. If you have know the size of your hand, and the length of the bones in the finger, you only really need to sense where the back of your hand is, and the location and orientation of the finger tips. This will give you pretty accurate position for every finger bone given the way the hand is able to move.

While all of this is a good mental exercise (working out how many sensors are required to know exact body position), given the cost of sensors, I think it best to look at what is the minimum number of sensors that could get reasonable estimation of the body position.

[JDuncan]

If you guess the hands position and only know for sure a few bones position, then a sensor on each fingertip would be nice, and then one on the back of the wrist, then have some cookie cutter SW figure out the size of the hand from these 6 sensors and then create a fake hand shape, then the finger tips move and animate the fake hand.

Add a sensor to the elbow and neck for 8 sensors and you have full arm and hand tracking in VR. 15 sensors for both arms and hands.

[HeliumPhoenix]

Using the bend measurements instead of trying to 3D track the fingertips has a major benefit: Latency. 3D position sensors are considerably slower in response time than simply reading an ADC output of a voltage. Many of the bend sensors are also quite cheap to implement. So while you use more of them, the cost works out similarly.

Also, positional sensors of any accuracy are still a bit bulky compared to a fingertip. Not sure how easily one could build a glove that those would both (A) stay on the glove during any sudden hand movements, or (B) allow the fingers to move easily without getting in the way in common positions.

It's a set of tradeoffs. With the right options, optical bend sensors (like in the old Lanier gloves) can give great response and latency, and be quite accurate.....but the cost and fabrication issues rise up with that.

[JDuncan]

https://www.youtube.com/watch?v=nObCJFLvqrQ



I thought about how to get the hand controller working without wireless sensors and decided that a gametrak method would work.

If you look at the hand when you hold your fingers out straight and stiff, then bend the fingers, if you bend the knuckles the fingers move too.

So the knuckles is one part of movement and the fingers bones are the other parts.

So if you tracked the hand you need to track the knuckles separately than the fingers bones.

Tying a one end of a string to the fingers bending points, and the other end to the gametrak joystick, you can translate the fingers motion to the gametrak joystick.

The gametrak joysticks are held above the fingers on a angled board, at a 45 degree angle when the fingers are held out flat and stiff. The angled board starts at the knuckles so the movement of the knuckles doesn't move the joysticks the fingers bending points are tied too.

Then the knuckles have a string tied to them, and the other end of the string to a gametrak joystick, the joystick is held over the knuckles and held on a angled board that starts at the wrist and goes over the back of the hand at a 45 degree angle.

Then you input the gametrak joysticks into SW and read it in VR. Then this is inexpensive and has no lag.

Then to find the roll of the hand you first find what part of the hand arm makes the hand roll. If your hand is flat so the fingers are all on the same horizontal plane then you tilt the hand so the thumb is on a different horizontal plane than the pinky this is rolling the hand.

When you roll the hand you move the forearm but not the humerus bone.

So if you had a two rods joined at a angle joint, and the angle joint was where the elbow is, then one of the rods is tied to the humerus bone and the other rod is in one spot above the forearm.
When the forearm twists the rod above the forearm doesn't twist.

Then you can tie one part of the string to the wrist and the other part of the string to the tip of the rod above the forearm. On the tip of the rod above the forearm is a joy stick and the string is tied to this.
So when the wrist turns the joy stick moves.

Now you can track the turn of the wrist, the movement of the knuckles, and the bend of the fingers, all mechanically using proven technology.

The joy sticks then input into SW and this into VR.

And most importantly, the string on the fingers is lightweight and unobtrusive since it's just tied to the fingers using something like a ring.