Meant to be Seen

Posted: **Mon Jan 07, 2013 7:23 pm**

He meant download for the software, I'm sure.

Posted: **Mon Jan 07, 2013 7:32 pm**

Dycus wrote:He meant download for the software, I'm sure.

The software is at github:
https://github.com/kylemcdonald/FaceTracker

readme.md:

# FaceTracker

FaceTracker is a library for deformable face tracking written in C++ using OpenCV 2, authored by [Jason Saragih](http://jsaragih.org/) and maintained by [Kyle McDonald](http://kylemcdonald.net/).

It is available free for non-commercial use, and may be redistributed under these conditions. Please see `license.md` for complete details.

Wrappers are available for:

* [openFrameworks](http://www.openframeworks.cc/): [ofxFaceTracker](https://github.com/kylemcdonald/ofxFaceTracker)
* [Cinder](http://libcinder.org/): [ciFaceTracker](https://github.com/Hebali/ciFaceTracker)
* Python: [pyfacetracker](https://bitbucket.org/amitibo/pyfacetracker)

## Installation

These instructions are for compiling the code on OS X only. Compilation on other Unix-type architectures should be similar.
---<snip>---

license.md:

Summary: FaceTracker is available for non-commercial use only. For other licenses, contact the author [Jason Saragih](jason.saragih@csiro.au) directly. The complete license follows.

---

Copyright (C) 2010, Jason Mora Saragih, all rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

* The software is provided under the terms of this license strictly for academic, non-commercial, not-for-profit purposes.
* Redistributions of source code must retain the above copyright notice, this list of conditions (license) and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice, this list of conditions (license) and the following disclaimer in the documentation and/or other materials provided with the distribution.
* The name of the author may not be used to endorse or promote products derived from this software without specific prior written permission.
* As this software depends on other libraries, the user must adhere to and keep in place any licensing terms of those libraries.
* Any publications arising from the use of this software, including but not limited to academic journal and conference publications, technical reports and manuals, must cite the following work: `J. M. Saragih, S. Lucey, and J. F. Cohn. Face Alignment through Subspace Constrained Mean-Shifts. International Conference of Computer Vision (ICCV), September, 2009.`

---<snip>---

Posted: **Sun Jan 13, 2013 2:48 pm**

Thanks for the tip. Definitely going to look at this. I am still unsure how much use this will have only tracking the mouth tho.

Only time will tell. I think it would be great to have a way of achieving a good sense of eye contact while in VR-meetings.

So far, I have been thinking about having your eyes snap to the nearest person to your straight line of sight and guestimate where your real eyes are fixed.

Posted: **Sun Jan 13, 2013 6:50 pm**

I really liked this technology at first, then realized that you can't really track the face while wearing the Rift. Who knows if there is any mouth tracking software out there that can work without the eyes and nose as reference points.

The eye snapping sounds like a good idea though, especially if mouth tracking works.

Posted: **Sun Jan 13, 2013 7:04 pm**

Naru wrote:I really liked this technology at first, then realized that you can't really track the face while wearing the Rift. Who knows if there is any mouth tracking software out there that can work without the eyes and nose as reference points.

If the camera is fixed beneath the Rift then you don't need any reference points, as you already know where the camera is in relation to the face.

Posted: **Sun Jan 13, 2013 7:28 pm**

If it was possible to mount a camera to capture the eyes, and one for the mouth, it should be possible to stream that into the game as a video texture.
I dont remember if I linked to this before, but Forbidden Siren is one of the games that do that, only part of the mesh that actually moves
is the jaw. Most of the face except for the nose is flat and has a animated texture projected onto it.

[youtube]http://www.youtube.com/watch?v=fWp_n-IEClk[/youtube] its kinda weird but also lifelike in a unique way.

Posted: **Sun Jan 13, 2013 7:45 pm**

you'd have to correct for the rifts lens' distortion as well because since it's right next to the eye the eye can only be imaged through it.

Posted: **Sun Jan 13, 2013 8:15 pm**

For the mouth I considered mounting the camera underneath the Rift with maybe some led lights to brighten things up in dark rooms, but hadn't really thought of how the algorithm would work for detection. I think it wouldn't be too much trouble to simply stitch the upper portion of a face onto the video feed to get a face image that could be used with currently existing algorithms. I am one to prefer performance and so I would much rather use some face detection library that can do 30-60fps rather than code my own mouth only algorithm.

Posted: **Mon Jan 14, 2013 3:10 am**

This may be way off the mark, but wouldn't it be possible to sort of 'calibrate' the software (not wearing the Rift), so it can record/understand what your eyes/nose/eyebrows do naturally when you smile/frown/laugh etc.

Then sort of extrapolate this data when you are wearing the Rift, so it knows what your whole face is doing, when it 'sees' your mouth smile etc.

Like I said....most likely way off the mark!

Posted: **Mon Jan 14, 2013 10:19 am**

You might be right tho, I have thought about this too.

Posted: **Mon Jan 14, 2013 11:40 am**

i've seen things about cartooning that says the eyes are far more important than the mouth in expression. just try some of the different ones you can do with a neutral mouth.

edit: here's actors using and discussing their eyebrows. http://www.youtube.com/watch?v=kjFvpgVYiHI

Posted: **Tue Jan 15, 2013 5:58 am**

The Emotiv EPOC is supposed to be able to read facial expressions however I am skeptical about the level of detail it has for the mouth seeing as how the electrodes are so high up on the face. It should read the eyes well enough but maybe when used in conjunction with a webcam below the Rift that should work.

Although I don't know how I could strap the Rift on at the same time. Difficult but not impossible

edit: the EPOC is pure crap for identifying facial expressions, disregard it as an option for that

Posted: **Fri Jan 25, 2013 2:35 pm**

Benjimoron wrote:You may hate me for saying this, or you may love me for saving you alot of time! But, I think this already exists

That actually looks pretty fantastic. If the Rift is implemented into the Source engine I would say we should use this. Otherwise we have to stick with Unity or Unreal.

Posted: **Fri Jan 25, 2013 2:55 pm**

This looks like a very cool project : )
Just make sure you focus on the small easy parts first to get the basics up and running and later you can start implementing the advanced stuff.
Seems like you have a lot of big ideas : p

Posted: **Fri Jan 25, 2013 3:02 pm**

Dont worry

Posted: **Wed Jan 30, 2013 11:11 am**

Hey guys, long time lurker first time poster, etc... just wanted to be part of the fun here

Anyway, there is a way to get data from above the mouth without cramming cameras into the Rift when trying to track facial expression. This could be done by putting a sensor array along the foam padding that comes in contact with the face, since I assume it covers an area close enough to the eyes to be relevant. I'm sure that there is enough data to be mined this way via distortions to get a decent idea of what goes on around the eyes. Although, it wont be as easy as throwing half a dozen electrodes and calling it a day; the type of sensor used is crucial.

Without abusing Occams razor, I think this could be accomplished with simple, inexpensive pressure sensitive pads. I figure that most facial expressions that current engines are capable of emulating in real time wont need data from subtle changes in facial geometry. Therefore, contractions in the muscles right below the eyes (upper, middle cheek) and right above the eyes (eyebrows), as well as data from the camera pointed at the mouth should provide enough of a description for the engine to figure out whats happening on your face, and make any in-game observer believe what they're seeing.

Although I'm not one to comment on specifics since I don't code and have little working knowledge of electronics, I hope my 2 cents helps the cause

Posted: **Wed Jan 30, 2013 11:30 am**

Thanks for your input.
Sounds good for expressions, but it still does not solve the issue of eye tracking (where are you looking).
Therefore maybe it can be used in combination with a small camera tracking eye movement.
Without a camera the eyes could lock onto the nearest persons eyes, but it would be cool if it had actual eye tracking too.

Posted: **Wed Jan 30, 2013 7:46 pm**

Yeah I didn't think of that part...

Oh well, I guess its inevitable that in order for this to happen were going to have to glue a camera in there somehow.

By the way, I'm still trying to figure out what's going on with your dog there. Its face is very difficult to comprehend for some reason

Posted: **Thu Jan 31, 2013 12:46 am**

Just catched a snack, and the mouth is about to close.
High speed photo sort of.

Posted: **Sun Feb 10, 2013 7:13 am**

Sorry to dissapoint everybody, but regardless of garrys mod I have continued development of my virtual cinema and lounge.
I had the warp, IPD and horizontal transformation set for 5.6" and then 7" happened.

Now I have managed to shift the warp, towards the center of the display and made partial overlap with the idea our eyes are more towards the center of the whole display instead of center of each sbs half.

Let me show you a short clip where you can see the head tracking, playing a MP4 video and the new warp + shifted view.
It might be wrong as I have not assemble my own DIY rift yet, but I do have all the parts now.

to sum up; I can select and play any format I have codecs for now, including mkv files with h.264.
I am doing positional tracking in this video with Trackir5 but x,y,z is very limited to mimic how the rift headtracking would work.
I can project the video texture to also light up the room but that is disabled at the moment.

[youtube-hd]http://www.youtube.com/watch?v=85KtcZ4NNL4[/youtube-hd]

Posted: **Sun Feb 10, 2013 7:26 am**

Namielus wrote:
[youtube-hd]http://www.youtube.com/watch?v=85KtcZ4NNL4[/youtube-hd]

HOLY poop.

Great work Namielus! This is exactly what I was picturing, and I think it's a great showcase of the simple ways VR is going to change things.

Posted: **Sun Feb 10, 2013 7:27 am**

Namielus wrote:Sorry to dissapoint everybody, but regardless of garrys mod I have continued development of my virtual cinema and lounge.

Huh? Disappointed? I would only be disappointed if you STOPPED developing this GREAT application! And what does "garrys mod" have to do with it (unless it was sucking up your time better spent on THIS app)?

Namielus wrote:... to sum up; I can select and play any format I have codecs for now, including mkv files with h.264.
I am doing positional tracking in this video with Trackir5 but x,y,z is very limited to mimic how the rift headtracking would work.
I can project the video texture to also light up the room but that is disabled at the moment. ...

That is AWESOME! Where can I download your demo? How hard would it be to get this working on my Nexus 7 with my fresnel lens stack?

Posted: **Sun Feb 10, 2013 7:37 am**

wow very good job so far! cant wait to try this out!

Posted: **Sun Feb 10, 2013 7:41 am**

Its not long until we have a demo ready

There is some scripts/dll files I don't to publicize yet so I need to figure out how to solve that.

First i need to know if the picture looks good in a diy 7"-er

Posted: **Sun Feb 10, 2013 7:43 am**

haha this is so great! Now we only need online multi player with direct communication thus people can annoy you talking and laughing during the exciting parts of the Movies. And please add cellphones that dazzle you in the dark room.. oh and Tortillas with abnormal loud chewing sounds thus you can't concentrate on the Movie because some one next to you is eating some. Overpriced Popcorn and Maxi Drinks would be nice too since that is what I actually a like about cinema (beside the big screen). Oh and before I forget a toilet where your Avatar hast to go 30minutes before the end of the Movie (only if he buyed an Overpriced Maxi-Drink). What do you think about an Free2Play Concept, but players can buy all that stuff mentioned above for real money? ;D

Seriously I like what you are doing there. What features will you implement next or are definitely planned?

Posted: **Sun Feb 10, 2013 7:44 am**

geekmaster wrote:That is AWESOME! Where can I download your demo? How hard would it be to get this working on my Nexus 7 with my fresnel lens stack?

I can accomodate different screen sizes by shifting the warp and the viewports.

Is your lenses set dead center on each half of the screen or actually outside?

Posted: **Sun Feb 10, 2013 7:52 am**

Namielus wrote:
geekmaster wrote:That is AWESOME! Where can I download your demo? How hard would it be to get this working on my Nexus 7 with my fresnel lens stack?
I can accomodate different screen sizes by shifting the warp and the viewports.

Is your lenses set dead center on each half of the screen or actually outside?

Their centers are somewhere between the IPD and half the screen width. I am not worried about distortion at this time. I just want to get it running on my Nexus 7. How hard would it be to port it to android (or linux)? Or do I need to use SpashTop to run it on a host PC while viewing it on my Nexus 7?

However I do it, I want to test it so I can get a feel for this experience (before I get my Rift)...

Posted: **Sun Feb 10, 2013 7:54 am**

Dakor wrote:haha this is so great! Now we only need online multi player with direct communication thus people can annoy you talking and laughing during the exciting parts of the Movies. And please add cellphones that dazzle you in the dark room.. oh and Tortillas with abnormal loud chewing sounds thus you can't concentrate on the Movie because some one next to you is eating some. Overpriced Popcorn and Maxi Drinks would be nice too since that is what I actually a like about cinema (beside the big screen). Oh and before I forget a toilet where your Avatar hast to go 30minutes before the end of the Movie (only if he buyed an Overpriced Maxi-Drink). What do you think about an Free2Play Concept, but players can buy all that stuff mentioned above for real money? ;D

Seriously I like what you are doing there. What features will you implement next or are definitely planned?

I dont know what payment model I am going to use, I would love for it to be free but I have had to spend lots of money on it myself as im not an expert programmer.

But its going to have a really nice price and maybe even free2play.

Planned features/stuff (from what I can remember at the moment):

A) Stream urls from the web, webcams and capture cards. (confirmed)
B) Multiplayer with avatars / watching the same content (both need the file locally)
C) Possibility to enable funny NPC's that behave comicly
D)Hand tracking with Leap Motion
E) Nicer graphics interface with movie covers etc custom made for oculus rift.
F) 3D movies that pop out
D) An entire building with stuff going on it will be a seperate app but will feature the cinema within the building.

Posted: **Sun Feb 10, 2013 7:55 am**

geekmaster wrote:
Namielus wrote:
geekmaster wrote:That is AWESOME! Where can I download your demo? How hard would it be to get this working on my Nexus 7 with my fresnel lens stack?
I can accomodate different screen sizes by shifting the warp and the viewports.

Is your lenses set dead center on each half of the screen or actually outside?
Their centers are somewhere between the IPD and half the screen width. I am not worried about distortion at this time. I just want to get it running on my Nexus 7. How hard would it be to port it to android (or linux)? Or do I need to use SpashTop to run it on a host PC while viewing it on my Nexus 7?

However I do it, I want to test it so I can get a feel for this experience (before I get my Rift)...

Porting to android would cost me 1500 dollars and I dont even think the video streaming would work when the game is running on android.
I think you would be better off streaming the image from your desktop to your Nexus 7 but I will look in to it for sure.

EDIT: The video streaming does work on android but there would be serious performance issues.

Posted: **Sun Feb 10, 2013 8:15 am**

Namielus wrote:
Dakor wrote:haha this is so great! Now we only need online multi player with direct communication thus people can annoy you talking and laughing during the exciting parts of the Movies. And please add cellphones that dazzle you in the dark room.. oh and Tortillas with abnormal loud chewing sounds thus you can't concentrate on the Movie because some one next to you is eating some. Overpriced Popcorn and Maxi Drinks would be nice too since that is what I actually a like about cinema (beside the big screen). Oh and before I forget a toilet where your Avatar hast to go 30minutes before the end of the Movie (only if he buyed an Overpriced Maxi-Drink). What do you think about an Free2Play Concept, but players can buy all that stuff mentioned above for real money? ;D

Seriously I like what you are doing there. What features will you implement next or are definitely planned?
I dont know what payment model I am going to use, I would love for it to be free but I have had to spend lots of money on it myself as im not an expert programmer.

But its going to have a really nice price and maybe even free2play.

Planned features/stuff (from what I can remember at the moment):

A) Stream urls from the web, webcams and capture cards. (confirmed)
B) Multiplayer with avatars / watching the same content (both need the file locally)
C) Possibility to enable funny NPC's that behave comicly
D)Hand tracking with Leap Motion
E) Nicer graphics interface with movie covers etc custom made for oculus rift.
F) 3D movies that pop out
D) An entire building with stuff going on it will be a seperate app but will feature the cinema within the building.

I'd pay for that!

Posted: **Sun Feb 10, 2013 8:23 am**

I would love some feedback from some uh.. 7"ers lol

Posted: **Sun Feb 10, 2013 8:29 am**

Looks great Namlielus! I've been following your project for awhile now, I'll be sure to buy it once you're ready to release it, and of course once I have my Rift

Posted: **Sun Feb 10, 2013 10:09 am**

looking really great. really excited about the streaming file support!

quick question: can you change the stream while in game, without leaving the cinema? it would be awesome to stay in the simulation.

Posted: **Sun Feb 10, 2013 10:11 am**

Yes you can

Posted: **Sun Feb 10, 2013 10:44 am**

so can you choose seating position? float in the air? i normally try to get eye level with screen centre both vertically and horizontally at the cinema, but that looks difficult here without being faraway. that is the case IRL as well at times tho.

Posted: **Sun Feb 10, 2013 11:05 am**

That Oculus video is being stretched horizontally.

Posted: **Sun Feb 10, 2013 11:28 am**

I had a question, sorry if this has been asked already, will the video retain the same quality if you're sitting closer to the screen in the VR theatre? Thanks!

Posted: **Sun Feb 10, 2013 11:34 am**

jf031 wrote:That Oculus video is being stretched horizontally.

Correct, the material the video is applied to is not the correct aspect ratio, theseare minor things that will be worked out by release.

Posted: **Sun Feb 10, 2013 11:42 am**

Glad to see some updates! It's not long now until we get our devkits and i'm really looking forward to trying this out. I'm sure you have lots of details to work out but will there be an option to change the screen size or aspect ratio? Disregarding the low resolution of the Rift screen it would be fun to experiment with different types of cinema screen sizes like IMAX, or Cinemascope(stitch multiple streams together for ULTRA Wide Movie viewing)!

EDIT: I wrote Cinemascope but meant Cinerama which used three synchronized 35 mm projectors to project onto a massive, curved screen!

Posted: **Sun Feb 10, 2013 11:49 am**

Havent found a way to manually set it up, but I will be able to warp you to different rooms with different types of screens.
It doesnt seem impossible to do tho, I just havent done lots of interactive materials yet.

Remember when I started this thread I had never used unity or written a line of code in my life.

Meant to be Seen

Oculus Virtual Lounge (VirtuaView)

Re: Oculus Virtual Lounge

Re: Oculus Virtual Lounge

Re: Oculus Virtual Lounge

Re: Oculus Virtual Lounge

Re: Oculus Virtual Lounge

Re: Oculus Virtual Lounge

Re: Oculus Virtual Lounge

Re: Oculus Virtual Lounge

Re: Oculus Virtual Lounge

Re: Oculus Virtual Lounge

Re: Oculus Virtual Lounge

Re: Oculus Virtual Lounge

Re: Oculus Virtual Lounge

Re: Oculus Virtual Lounge

Re: Oculus Virtual Lounge

Re: Oculus Virtual Lounge

Re: Oculus Virtual Lounge

Re: Oculus Virtual Lounge

Re: Oculus Virtual Lounge

Re: Oculus Virtual Lounge

Re: Oculus Virtual Lounge

Re: Oculus Virtual Lounge

Re: Oculus Virtual Lounge

Re: Oculus Virtual Lounge

Re: Oculus Virtual Lounge

Re: Oculus Virtual Lounge

Re: Oculus Virtual Lounge

Re: Oculus Virtual Lounge

Re: Oculus Virtual Lounge

Re: Oculus Virtual Lounge

Re: Oculus Virtual Lounge

Re: Oculus Virtual Lounge

Re: Oculus Virtual Lounge

Re: Oculus Virtual Lounge

Re: Oculus Virtual Lounge

Re: Oculus Virtual Lounge

Re: Oculus Virtual Lounge

Re: Oculus Virtual Lounge

Re: Oculus Virtual Lounge

Re: Oculus Virtual Lounge