Eye Contact in Virtual Reality - Need Help from Community

[PalmerTech]

I am doing some preliminary research into the effects of eye contact on immersion, especially in regard to feeling a connection to NPCs and other players.
It seems that eye contact is incredibly powerful in VR, something that triggers different instinctual responses depending on the context. A friendly gaze can put you at ease, a stone cold face can make you look away, and a glare can make you flinch.

Oddly enough, though, I have been unable to find many examples of eye contact in videogames, particularly multiplayer titles. It seems like such an easy thing to do (Just snap the eyes of any player looking near you to your face), but it rarely is. Mass Effect does this with NPCs to great, well, effect! Is anyone out there familiar with this topic, and do you have any interesting articles/studies/games to point us towards? What games have you seen this in?

[STRZ]

Tbh, i never paid attention to it until a few days ago when Gabe Newell talked about their research for Half Life 2. In VR it's probably a important thing which could make a big difference.

Here's the Video, at min. 28 he talks about it.

[youtube-hd]https://www.youtube.com/watch?v=t8QEOBgLBQU[/youtube-hd]

[jimbo2go]

HL2 is a great example of eye contact, even though its almost a decade old now. Check out 0:58 of this video:

[youtube]http://www.youtube.com/watch?v=Zfyr4MM3bSY[/youtube]

The Source Engine did a great job with facial expressions as well - and combined with contextual eye contact you could really feel like you were interacting with living, breathing humans. Generally during NPC scenes in Valve games, NPCs will dynamically look at you when you get near them, or at each other when they are talking, depending on how things are scripted.

We're playing around with this in Wild Skies - eye contact and very simple texture based blinking. Really fascinating stuff.

[STRZ]

Somebody did a head and guntracking mod for HL2 with the Rift in mind http://www.youtube.com/watch?feature=pl ... H2rL-lBxRs

[Linkage1992]

Red Dead Redemption is a great game to look into. The eye animation and overall expressions are extremely well done, especially when duelling.

[BOLL]

Something I also think is worth mentioning is the direction of the head/face. I have a friend who's into making machinima, and he has commented on how awesome Minecraft is because players' heads turn exactly to what they are looking at. The body kind of just follows suit when it has to, like when you turn too far or have started walking. I have to admit this introduces a whole lot of player presence, for me at least, even if the game is a graphical blocktrocity.

I wanted to mention this because a character who just looks at you without turning its head seems to be a person trapped in someone else's body looking out.

[edit] After having watched the Half-Life 2 video it is evident they spent much time doing head and eye movements work well, which is very nice But I still come to think of something that could be improved too, haha. That NPC's meet each others gazes when speaking to each other, or follow the gaze of someone looking at something while talking, simply the social behavior during conversation and/or seeing someone looking at something

[alekki]

I sent you a PM

[Namielus]

I am also very keen in finding out more on this subject. It would be lovely to make a special game with those features aimed at having autistic children practice eye contact skills.

[rhinosix]

Eye contact seems like it’s going to be a huge and interesting topic for VR, and has only just started to be explored in video games.

L.A. Noire and Half Life 2 are the better examples I can think of off the top of my head. But I think eye contact and behaviour is going to be one subset of the much larger project of creating advanced social AI.

I guess it might be more useful to look into resources from psychology, cultural studies, animal behaviour, and so on.

The amount of data you gather through eye contact in the physical world is immense.

What believable eye contact will come down to is not just having an NPC look at you, but watching how they look away, their facial expressions, and body language within the context of their dialogue and the situation.

And it will look strange if all NPCs use eye contact in the same way. People in the physical world use eye contact differently depending on which culture they were brought up in (Americans and Japanese use eye contact differently to show politeness), and on their personality (a friendly acquaintance will lie to you differently than a sociopath who believes their own lie). An introvert will tend to look to their feet more.

If you get eye contact working well, the player’s attention is going to be drawn to the next thing - facial expressions. And then the body language won’t match, and so on. So at some point it may be an issue of creating NPCs that generate their own reactions rather than pre-recorded ones.

I enjoy games that try to give NPCs their own lives and schedules (Shenmue, Watch_Dogs, Deadly Premonition, Skyrim), and I think giving NPCs their own lives will be key to creating NPCs that look right.

When writing a story the quickest way to make a character feel alive is to give them a desire - something they want within the world. You don’t want them just hanging about waiting for the protagonist, or they feel like a prop.

I would like to see something like a more advanced version of Skyrim’s Radiant A.I. that could be used by multiple game engines to create NPCs with their own personalities and culture. They could use eye contact, facial expressions, body language, and dialogue options pulled from a library according to their personality and the context of the situation.

For example:

- Create personality types for NPCs (based off something like MBTI).

- Have some parameters based off something like Hofstede’s cultural dimensions theory to spread the distribution of personality types. So you could have a society of more masculine, independent personalities, or more boisterous, or placid.

- Create a huge library of eye contact, facial expression, and body language animations which NPCs can use contextually based on their personality. For example, an ENTJ might choose an aggressive response if stared at, while an INTP would look at their feet. One NPC might tell you a lie with a straight face, or another might shuffle on their feet and look to the side.

- Have each NPC gain access to different areas of the library if they have an interaction (such as being repeatedly threatened, or witnessing an event). So their personalities and reactions can change depending on the context.

- Then generate a schedule, a set of waypoints and activities for each NPC... or keep going nuts.

So in the end you would have NPCs that have their own activities within the world, varying reactions to similar events, and different ways of making eye contact with the player based on their personality and the situation.

Like in the physical world, I guess we want to be able to look into an NPC’s eyes and instantly gather all that information from them - if they are friendly or a threat, rich or poor, local or foreign, nervous or confident, welcoming or hostile.

It will be interesting when there are AI NPCs who react like the characters in L.A. Noire in a dynamic way. You could sit on a virtual park bench, watching people and making up stories about their lives.

Sorry if I went off on a tangent, but I was thinking about this in the shower.

[cybereality]

In the book Infinite Reality they talk about a study for education using VR. Basically they modelled a virtual classroom with students, a teacher, etc. In one group they had the teacher always looking at the user. In the other the teacher would look at other students. Otherwise the class was exactly the same. They found that the eye contact group was able to retain the information better. You can probably find the study if you look at the footnotes in that book.

[android78]

I think that one of the most powerful for me is Bioshock. With the little sisters, even though you can't see their eyes, the 'gaze' in your direction as they plead with you is really powerful. I tried 'harvesting one' once, and found myself never able to do it again.
I think that was actually clever on the behalf of the developers to not have eyes in them, since the technology probably wasn't there to pull off a convincing emotion and would end up going into 'uncanny valley'.

[AussieMike]

I recently started playing resident evil 5 with my new 3d monitor. At the very start of the game you walk past a group of guys beating someone up, and was struck by how powerful the eye contact was as they glare at you when you walk up to them.

[SaviorNT]

Found this online: http://www.psychologistworld.com/bodylanguage/eyes.php

I think, perhaps, you should not look into eye contact / facial gestures / body language from a gaming perspective, but rather from a real-world, psychology perspective. Also, you should look at some of the Disney behind-the-scenes from their latest CGI movies where they discuss the gestures of their characters.

Coming from a background in CGI, creating the gestures is fairly easy once you get used to the human form and how it reacts to situations / circumstances. Admitedly, I'm not sure how this would correspond to game programming, though I can't imagine its that much different, just more time consuming.

[doktor3d]

I did my PhD thesis around this topic. You can find publications here, particularly relevant are those from 2008-2010: http://bit.ly/X26lJ1

If you PM me I can send you my thesis if you like.

[KBK]

hello Doc, and Palmer,

You're going to like this one. Directly related and indirectly related, depending on the extrapolation one may be looking for.

Brain does not process sensory information sufficiently, research team discovers


Edit:

Of course, they would do the projection thing and think of how they can hammer folks, instead of allowing folks to do the work themselves, via enabling them. ie forced entrainment over that of sensible help and correct contexts.

As the Buddhists say, 'change comes from within'.

Their statement in the article is: "An exciting question now is to what extent the alpha activity can be deliberately influenced with biofeedback".

Which is the use of cohersion and a forced situation, as a pair which could be seen as - horrifically backward. Some might say that I'm over emphasizing the situation, but the mentality that would utter those words, is what alarms me, for all the right reasons. If one is going to move into the public use and the popularization of a new methodology in science that will probably end up being widespread, don't start off on the wrong foot.

This is important with regard to the plastic aspect of mental design, as the kernel of wiring that begins the process, and is shaped by the method, this becomes the opening and shape -in the mind. This becomes the shape of the flower or wiring that evolves from the given moment. Beginning as a moment in enforced entrainment is possibly an alarm bell, that should be sounded sooner, rather than later.

Recall that this is all extrapolation based on the words spoken. That it may not be that bad, but the words spoken do not leave a good taste.

[ftarnogol]

I'm currently developing a platform for VR Psychology (AMVR -Amber-)

2 scenarios I'm contemplating are social phobias and communication/ADLs for people with autism. I'll get version 1.0 ready in the next 2.5 months (hoping to get the Rift by that time).

I'll keep you posted as soon as I start researching with AMVR and the Rift.

[Kirito]

if i think about eye contact , assasin creed comes to mind, good luck palmer

[Lookforyourhands]

Would it be possible to say have a tiny camera on the inside of the HMD that is directed
at the eyes of the user, and then transmits the data onto the avatar of the player in real time?
Basically your eyes in game would be a 'live feed' of your physical eyes but in the virtual environment.
In this manner people could actually make eye contact in VR and not just simulate it
by looking at pixels that appear to form an eye. Make sense ??
VR Skype lol.

[colocolo]

Would it be possible to say have a tiny camera on the inside of the HMD that is directed
at the eyes of the user, and then transmits the data onto the avatar of the player in real time?
Basically your eyes in game would be a 'live feed' of your physical eyes but in the virtual environment.
In this manner people could actually make eye contact in VR and not just simulate it
by looking at pixels that appear to form an eye. Make sense ??
VR Skype lol.

I have thought about that too. In general it would be a loss if the goggles prevent seeing the emotions of your friends while
experiencing a Halo or Destiny mission. A camera module nowerdays only costs about 5 bugs. It would be very nice if one camera in the Rift and one attached to your head and directed to your face, could capture your emotions.
But to do that it should be important that the goggles doesnt tighten your facial skin. Could look very strange.
What came right now to my mind and what i havent thought about it yet is that it would be so great if the system captures all your mouth/lip movement while speaking or crying about someone who has fallen in a battle against aliens.
I guess this would enhance the immersion and experience a lot.

[V8Griff]

I think the use of cameras would be difficult and could produce very strange effects while not necessarily creating the desired effect.

Something that has never caught on but in my experience is very powerful is producing in-game characters that actually look like the player. Obviously difficult when you're a suited Mech or a furry Ogre but I used to work at a company called bioVirtual in the UK and we wrote software tools that allowed you to create accurate, animated heads that blinked, had inbuilt phonemes (mouth shapes) that analysed any wav file to mimic the speech very quickly. All you needed where three or four photos to which you aligned the mesh and the software did the rest. We had importers for Unreal, Quake and the Sims.

The product was bought by quite a large number of big studios but they'd never tell us what the tools were being used for.

The point I'm making here is we used to play networked games of Unreal with characters that had our heads attached and so we actually identified with the in-game avatars down to the point of apologising to a player you'd shot if they were on your team.

So while eyecontact is nice I'd suggest that facial expression and identification are equally, if not more, important.

[MSat]

rhinosix makes some very excellent points. While interesting, having all NPCs make constant eye contact when they interact with you is not entirely compelling. A shy NPC, or one that feels guilty may just briefly look into your eyes and then look down or turn away. Other times, an NPC's gaze may bring something to your attention (why do they keep looking behind you?). I think tone of voice and body language are also equally important. Tone of voice and fidgeting of the hands could indicate that a character may be lying to you. Other times a character might be talking with their back towards you - perhaps looking at something - maybe out a window, or walking around you as they ask questions while you're sitting. Movies tend to capture the naturalness of the various kinds of human interaction. Constantly locking eyes with everyone isn't particularly natural, and might actually be quite uncomfortable, or at least prevent you from being able to use subtleties in communication that goes beyond just words.

[longshot]

I am doing some preliminary research into the effects of eye contact on immersion, especially in regard to feeling a connection to NPCs and other players.
It seems that eye contact is incredibly powerful in VR, something that triggers different instinctual responses depending on the context.

The cultural context of eye contact is extremely important. In Thailand, where I live, making prolonged eye contact is generally seen as disrespectful behavior, and will make people very uncomfortable.

Whereas in the west, people who do not make eye contact are seen as untrustworthy. So in other words, the reaction is the complete opposite, NOT making eye contact will make people uncomfortable.

Although body language is generally more universal, it is not completely. For instance, crossing your legs towards someone, in the west, would generally mean that you have positive feelings towards them. In Thailand, this is sometimes true, but I've also noticed Thai people crossing their legs away from someone, so that their feet will not be pointing at them.

However, facial expressions, are generally completely universal, across all cultures. A warm smile will get the same response across all cultures. It is also the hardest thing to model in a game.

Eye contact is a powerful thing, but since it is regarded so differently in different cultures, I think it is better to proceed cautiously with it in VR simulations, if they are to have global appeal.

[InvaderZIM]

Skyrim had the other characters always look toward you when talking or interacting.


Here's a video of face to snout interaction with a Dragon character from the game, you'll notice how the creature moves so that you always look toward it's face/eyes, especialy starting at around 2:15 seconds into the video to just past the 3 minute mark, rest of the video is straight face to face.

http://www.youtube.com/watch?v=SXkEGU7bJzA

Don't know if this helps, but there it is lol.