arm tracking technology

[JDuncan]

In my eye tracking idea, I used the kingpin technology.

First I took the grid, that is the "button" in the kingpin technology.
I first made the eye to form the grids dimensions, then got the eye to look at the grid light up and this was the eyes position relative to that lit grid.

Then I showed the stereoscopic picture to the eye based on what part of the grid the eye was looking.

That was the eye tracking section.
Second was the neck tracking which let me judge the depth the person was looking at, it used the same grid idea but the grid looked different than the previous grid, so that what the grid showed could let me judge the distance the person was looking at.

That's about the skinny on what the eye tracking idea was.

In hand tracking, I need to have two parts;
- the hand motion when the arm is bent and holds the hand as clode as it can to the armpit
This is the same as the eye tracking in the eye tracking idea.
- the arm moving out so the hand is distanced from the armpit is was held close to.
This is the same as the neck tracking in the eye tracking idea.

I found that if you hold your hand by your armpit, and the palm is facing downwards, so the thumb is besides the chin.
That is you wiggle the single finger starting with your thumb and moving to your pinky.
Then the finger that is next to the wiggling finger, and closer to the pinky, will wiggle slightly, like the two are trying to rub together.
And if you wiggle the finger a lot then the finger beside it that wiggles slightly will will more too.

This was the wiggling finger is copied by the finger besides it.

Conversely, if you start with the pinky moving to the thumb, and wiggle the one finger at a time.
That the finger besides the wiggling finger and closest to the thumb will wiggle.

So if your counting the fingers inwards to outwards, thumb to pinky, the copycat finger will be the finger closer to the pinky.
Or if your counting the fingers outwards to inwards, pinky to thumb, the copycat finger is the finger closer to the thumb.
This is because your feeling where your moving too, it's a compelling feeling that you put your hand where your hand is moving.
This is visible when you imagine a object to your right and then try to touch it and you watch how your fingers bend, then try again but the object is to the left not the right - your fingers move differently, it's a touch association in the mind I think.

The grid in this case is the wiggling finger, the eye in the case, of hand tracking is the copycat finger.
So the grid lights up and then the eye follows the grid and this has a value, grid lit up by main wiggling finger = copycat finger position.

If you hold your hand below your elbow and then bend your forearm upwards. The amount you bend your forearm affects the amount you bend the arm attached to the shoulder, the same as the copycat finger, just the same.

The forearm is the main wiggling finger, and the arm bone attached to the shoulder is the copycat bone that copies the forearm.
So if you map the amount the copycat bone moves, it moves because the forearm moved.
So if you can have the forearm be the grid, the copycat bone that is inside the shoulder joint is what follows the copycat bone, as the eye, if you see the parallel between eye tracking and this.

So when the arm is held next to the armpit, the forearm is held still against the ribs, so the arm attached to the shoulder is still.
Then the copycat fingers are made to equal the main wiggling finger.
Then using eyetracking terminology, the forearm moves, and the bone attached to the shoulder is made to equal the forearm, if the forearm is the grid that lights up, and the bone attached to the shoulder is the eye.

Now the grid and tracking of the grid is understood, you just need to figure out how to get the grid to mean something in software, then match the copycat to the grid in software.

This would then let you have hand tracking in virtual reality. This was on theory not actual implementation. For implementation I can't say what can read the bones of the hand and forearm to make the grid, thenwhat can read the copycat bones in the arm and hand. Maybe colors on the arms that then show how the fingers will wiggle?

If the arm moves outwards, then the fingers grid finger and copycat finger is understood in SW.

[android78]

I think I got lost about a quarter of the way through. Is this a rehash of that song 'heads and shoulders, knees and toes'?

[JDuncan]

TBH I got a bit lost when I was explaining it, so the typing was a bit off, and the words rambled on a bit as I refined what I had understood in my psyche verbally.

But today is a new day, and I have a better understanding from my previous attempt and now I will try to detail it again but my precisely.

Bend your hand and arm on your left, so it is close to your left armpit.
The same for the right, so the right hand is close to your armpit.
This is the starting position to get hand measurements for tracking.
________
The fingers section

Now sway your left hand to the left, and as you sway your left hand to the left curl your middle fingers in a tapping motion.
You will see the left ring finger wiggle more than the index finger.

Now sway the left hand to the right, and as you curl your middle finger so it is tapping up and down, and as you do watch your left index fingers wiggle compared to the way it wiggled before.

The left index finger wiggles differently based on which direction you sway your hand.

This is associated with touch in the mind.
That is, if your going to touch something, your fingers move based on the direction of the motion hand.

If you move your hand up, the fingers will curl one way, then if you move you hand down your fingers will curl a different way than when the hand was moving up.

The deductive conclusion to this phenomena is;
the direction the hand moves, will show how the fingers will typically curl.

Not only that, but wiggling one finger gets one other finger to wiggle as well.

And this copy cat wiggling finger is directly beside the wiggling finger.

And if the wiggling finger has one finger on either side of it, the copy cat finger is on the side the hand is moving too.

And if the middle finger moves far up and down, the copycat finger moves far up and down too.

I think the copy cat finger is understanding what the wiggling finger is doing.
For instance, it's like they are talking to each other, and the wiggling finger is talking and the copy cat finger is listening.
__________

The arms section

Now the humerus bone is connected to the forearm bones and shoulder.

If the forearm moved the hand close to the armpit, then holding the humerus bone the same position swing the forearm around freely, up, down, side to side.

Now hold the left arm straight of in front of you and put your right palm below the left arms elbow, then bend your forearm upwards slowly.
Then lower the forearm back down, gently.
Then lift the forearm upwards fast and in a jerking motion.
Chances are, the hand below the elbow moved less when the arm was bent slowly than when the arm moved up fast.

That is the principle that shows, the way the forearm moves, is how the elbow thus humerus relates to the forearm.
Thus the forearm is what is before the humerus in terms of motion.
______________

Conclusion

The arm has the humerus follow the forearm.
The hand has a copy cat finger that's based on the direction of the hands motion.

So, if you can get the motion of the hand known, you can figure out what is the copy cat finger.

And the forearm moves the hand.

Therefore, if the forearm moves, the hand moves, the copycat finger moves.
And how the forearm moves is how the humerus moves.

So if the forearm and hand and fingers move one way, the humerus follows this logically.

e.g. the forearm pushes outwards, and the hand extends the length of the forearm and the fingers point straight on the hand, the humerus does what?
The humerus moves the forearm outwards is what.

the hierarchy of arm motion is thus;
forearm
hand moves
finger moves
copycat finger moves
humerus moves
= arm and thus hand motion.

[JDuncan]

Based on what I said before, for hand tracking you must track the forearm first, the hand motion second, the humerus third, in that order.

Now the hierarchy is the humerus and hand are both children to the forearm parent.

The hand is connected to the forearm, and follows the forearms initialization of movement.
And the humerus is connected to the forearm and follows the forearms initialization of movement.

So you put the sensor onto the forearm to get the parent motion, then you put sensors on the humerus and hand to get the children position.

And if your fancy, you put sensors on the fingers,
and get the primary wiggling fingers position,
then the secondary wiggling fingers position.
These would be children to the hand position parent in hierarchy.

But for now finger position is too fancy, just strap three sensors to the arm, 1 on the humerus, one on the forearm, one on the hand and now you have arms in virtual reality,

not just floating hands which look weird.

[android78]

It seems like you're doing some sort of inverse kinematics, or maybe forward kinematics given that you are talking about measuring the joints (http://en.wikipedia.org/wiki/Inverse_kinematics), but I still am not sure what sensors you would use, or how they would work.
Is your idea to have some sort of potentiometer that would measure the angle and rotation of the shoulder, elbow and wrist joint?

You say 'just strap three sensors to the arm, 1 on the humerus, one on the forearm, one on the hand and now you have arms in virtual reality' - wouldn't you need to also measure the shoulder position?

[JDuncan]

I'm not doing any kinematics and I don't know what a potentiometer is.

Wouldn't they need to measure the shoulder position? Not necessarily.
The reason is the arms are dynamic motion, so they must relate to a static motion.

Then from the way they change their position compared to the static position, they show their position.

Look at the gametrak;

https://www.youtube.com/watch?v=nObCJFLvqrQ

The base stays on the ground then the person moves the arms, and this movement translates to in game movement.
This is what I mean by there needing to be a static compared to a dynamic.
The static in the gametrak is the base, the dynamic is the cords moving because they are tied to the hands glove.

I thought about doing something like the gametrak and tying the base on the neck using a collar. Then the wires are tied to the chin via a mask covering the chin, and then when the head moves this translates to the gametrak sitting on the neck and creates head tracking.

I said this to show the gametrak is tied to the neck and stays still.

If 1 wireless controller on is the humerus,
1 wireless controller on the forearm,
1 wireless controller on the hand;
the controllers change their position to the base on the neck,
and the effect of this like the gametrak, the wireless controllers on the arm act as the moving cords on the gametrak.

Now the head is found by the neck collar, and the arms and hand position is found by the neck collar.
The head uses wires so it's just a gametrak with the head acting as the hands that move around.
And the wireless controllers show how they change position compared to the still neck collar and this shows a point that is where the arms originate, and the movement of the arms.

I think that about covers the idea.

[android78]

So your idea uses a form of absolute position tracking, such as the Razer Hydra, for all of the points you mention? The main problem with this is the cost of the trackers is still quite expensive. Something similar could be achieved with the soon-to-be-released sixense STEM system tracking the hands and feet, and using inverse kinematics to get a reasonable estimation for the rest of the limb positions. Ultimately, a position sensor at each and every joint would be the goal, but probably not really feasible at this stage.
You could use IR dot camera tracking, however this has to deal with optical occlusion as well as heavy processing.

[JDuncan]

"So your idea uses a form of absolute position tracking, such as the Razer Hydra, for all of the points you mention?"

Yes, I think the sixense product stem would work well here, you only need four sensors and the base on the neck.

I say four sensors, the sensor in the shape of a collar tied around the neck touches the top of the humerus bone on the shoulder area.

So the first sensor is on the elbow, and this measures the distance from the top of the humerus touching the neck collar, and the elbow.

The second sensor is on the wrist, this measures the length of the forearm.

The sensors on the fingers would be next, but that's too expensive for all 10 sensors, 1 per finger.

Then the sensor on the wrist and sensor on the elbow give length dimension to the forearm, and then the forearm in the hierarchy is the parent.
The hierarchy children would be, the wrist to the hand, and the elbow to the top of the humerus.

The wrist to the hand, is letting the wrist sensor be the parent to the sensors on the fingers.
But again, there's no sensors on the fingers because ten sensors would be too expensive.

Right now the sixense can use five sensors or controllers I think.

So the one controller on the head, measuring the neck, to the controller on the head, the effect is something like the gametrak I showed previously.

And four controllers for the arms, one per elbow and one per wrist.

Then you have the top of the humerus by found be touching the neck collar and this lets the arms move from this spot.

So the neck collar to mark the head tracking and arm tracking is important.

I said the top of the humerus is what the arm is tied too as it moves, but doesn't this make the top of the humerus the parent? No because this is the static position, the neck I mean.
By the forearm being the parent the hands and humerus is tied to a moving part.

SO that's how the sixense stem can be used for hand and head tracking in VR.

As far as having one stem controller somehow show the whole arm using " inverse kinematics", I don't know anything about that. But it would be cool to see in action.