PC Streamable HMD with Galaxy Note 2 Android Phone ?

[budda]

Hi,

With all the recent interest in the Oculus Rift and other custom head mounted displays, it seems mobile phone technology is reaching the point where an off the shelf phone can now act like a PC remote display.

Whilst none of this is particularly new in itself, the combination of necessary elements is coming together to build a convincingly good virtual reality setup. This fact is being demonstrated by the Oculus Rift.

The main element is the display, and in this case, the phone. In my view, the Galaxy Note 2 has reached the critical threshold for display capability and practicality for a HMD, and potentially other forthcoming models of Samsung Android phones will be even better for this role.

Most of this expensive phone technology however will be wasted with the HMD, such as the touch screen and pen technology.

What is useful is the 5.5 inch, 1280x720 AMOLED display and the powerful video processing hardware to drive it. Wireless PC game video streaming is possible, so no cables are required for mobile gaming.

The main processing of the gaming video and control interfacing is done on the PC and the phone is merely the presentation device.

This has a number of advantages

1. Phone is very slim, lightweight and sustains long battery life.
2. Content supports a vast array of PC applications and games
3. Touch screen controls are not required as conventional wireless gaming controllers can be configured through the PC.
4. The PC can be upgraded as required.
5. The streamed video format is preprocessed on the PC, in side-by-side format, for instance, and distortion corrected.
6. The Android platform is more open to the developer community to improve the HMD functionality, and has big support in Samsung and Google, as opposed to Apple or Sony.

The Samsung Amoled display in the Galaxy Note 2 phone should be capable of fast refresh rates, although this rate has not been disclosed, and is likely to be constrained only by the support hardware.

None-the-less, shutterglasses support is conceivable for this phone and it would be very nice if combined with streamable S3D content in a HMD, with proper glasses synchronisation of course.

I hope that is not too much to ask.

Thanks.

[brantlew]

Don't forget that the phone inertial sensors can also be used for head tracking. Crucial is the wireless streaming video rate which some people on the forum are reporting up to 30 FPS. Is this enough for a convincing experience? Don't know. But it sounds like a great idea to try converting these devices into all-in-one VR consoles. Backtops are really clumsy and the wiring is a pain to deal with. Having a simple headset device with no wires would be really convenient for 360 free motion usage.

[cybereality]

Yeah, I think it could work. Some people have already made demos like this, check Palmer's FOV2GO.

And who could forget this?

[youtube]http://www.youtube.com/watch?v=QyrjBMAa6xE[/youtube]

[brantlew]

I don't know if it's coincidence or not, but I feel like these threads should be joined.

http://www.mtbs3d.com/phpBB/viewtopic.p ... 607#p84457


With the right lens kit and form-factor these devices could rival the Rift for presentation. The biggest issue with these devices is the low frame rates and latency introduced by streaming. If there was a groundswell movement to modify content specifically to run on this type of device, then this would be an extremely interesting alternative to the Rift since it would be completely untethered. Unfortunately Palmer's FOV2GO never really inspired as much excitement or software support as the Rift has.

[budda]

Hi,

Just a few top up ideas for the use of the SGN2 phone in a HMD.

It has a removeable backshell to change out the large replaceable battery.

This can solve three issues.

1. Use a fake phone backshell as part of the HMD frame so the phone itself can easily clip in and out of position if needed.

2. The large battery could be removed and repositioned away from the back of the phone to lower any adverse leverage effects on the head.

And 3, the battery setup or some other auxiliary power source can be used to support other accessories for the HMD, besides the phone.

Thanks.

[budda]

Hi,

These comments are in regard to the wireless bandwidth required to stream stereoscopic video to the SGN2 phone in the head mounted display.

It seems that a 720p60 video stream could be reliablly sent over 5Ghz wireless N, which the phone has.

However I wish to achieve an acceptable frame sequential rate which prevents video choppiness and shutterglasses strobing whilst being full frame resolution of 1280x720 pixels.

This will require custom application software as I believe it is not standard.

This is how I would do it.

Alternate full frames of 720p are sent at a total rate of 60Hz by video streaming on a 5 Ghz wireless N link. This gives 30 Hz of stereo pairs to the phone.

Each stereo pair is buffered in the phone and alternately displayed two, three or four times ( at the equivalent frame sequential rate of 60Hz, 90Hz or 120Hz as the case may be) so that the output image is perceived as flicker-free.

Once the next stereo pair is wirelessly received the process repeats.

The video latency is at least one stereo frameset at 30 Hz (33ms) and more likely to be two frames all up for loading and display (66ms).

I believe this arrangement is workable, but it is beyond me to make it happen.

Thanks.

[brantlew]

Hi,

These comments are in regard to the wireless bandwidth required to stream stereoscopic video to the SGN2 phone in the head mounted display.

It seems that a 720p60 video stream could be reliablly sent over 5Ghz wireless N, which the phone has.

However I wish to achieve an acceptable frame sequential rate which prevents video choppiness and shutterglasses strobing whilst being full frame resolution of 1280x720 pixels.

This will require custom application software as I believe it is not standard.

This is how I would do it.

Alternate full frames of 720p are sent at a total rate of 60Hz by video streaming on a 5 Ghz wireless N link. This gives 30 Hz of stereo pairs to the phone.

Each stereo pair is buffered in the phone and alternately displayed two, three or four times ( at the equivalent frame sequential rate of 60Hz, 90Hz or 120Hz as the case may be) so that the output image is perceived as flicker-free.

Once the next stereo pair is wirelessly received the process repeats.

The video latency is at least one stereo frameset at 30 Hz (33ms) and more likely to be two frames all up for loading and display (66ms).

I believe this arrangement is workable, but it is beyond me to make it happen.

Thanks.

If you are going with a custom design, I think a better route would be to utilize a stereoscopic compression algorithm. Stereo compression can yield massive compression rates because the information in each eye is very similar. So the data size for transmitting a stereo view is not significantly higher than transmitting a mono view. As long as the compression/decompression times are not longer than sending an entire extra frame you should be able to achieve higher throughput and frame-rates. This is also true for everybody now that is experimenting with SBS video transmission. These off-the-shelf transmitters are using generic image compression. A custom built - stereo transmitter would "probably" improve frame rates significantly.

[budda]

Hi,

I found a site which reviewed the screen characteristics between the Galaxy S3 phone and the Galaxy Note 2.

http://www.androidauthority.com/galaxy- ... le-121182/

Interestingly, they say the displays appear largely the same, but in close detail (as for a head mounted display) the Galaxy Note 2 has the superior characteristics.

Comparison photos with 5x magnification are shown ( similar to the power of lenses used in other 'Rift-like' head mounted displays )


Thanks.

[PalmerTech]

I love my Galaxy Note 2, the display is fantastic. This new pixel arrangement is even better for VR than RGB stripe.

[MSat]

Don't most high-end smart phones have enough processing power to run their own content? Sure, you can't have the same graphics quality as a decent PC, but still probably pretty cool. It would be nice to see a bit more upscale variant of the FOV2GO made for specific (fairly popular) phones. Just add a choice of bluetooth game controller and you're set. If the motion sensors in the phones are still too slow, perhaps an external tracker can also be used, and communicate back to the phone via bluetooth or usb.

[brantlew]

I think the trackers in the phones would be adequate - and probably without the latency associated with getting data from external tracking devices. Integrated packages like this are the way to go for simple and accessible VR (ie. mass acceptance). As usual Palmer was ahead of the curve with the FOV2GO but not many people (including myself) could see it at the time. If only there was as big a push for VR content creation on these devices as there has been for the Rift.

[MSat]

I think the trackers in the phones would be adequate - and probably without the latency associated with getting data from external tracking devices. Integrated packages like this are the way to go for simple and accessible VR (ie. mass acceptance). As usual Palmer was ahead of the curve with the FOV2GO but not many people (including myself) could see it at the time. If only there was as big a push for VR content creation on these devices as there has been for the Rift.

At least a couple of included good quality games , and a retail price of around $20 could possibly make it quite successful.

[PalmerTech]

The Hasbro My3D was exactly that, and it failed very badly. I just saw a pile of them in my local Goodwill for $3.99, brand new.

[brantlew]

The Hasbro My3D was exactly that, and it failed very badly. I just saw a pile of them in my local Goodwill for $3.99, brand new.

It could be a marketing problem. First of all - Hasbro. That's a bad idea because it reeks of "toy". It might actually do better if marketed as a more expensive device/gadget. Put a proper lens package and make it fit on your head like an HMD and look relatively polished. Put it in a store like The Sharper Image for 60 bucks. Just enough money so it looked like a wiz-bang piece of technology, but cheap enough so that it could be an impulse Christmas buy.

Of course there is the chicken-and-egg software content problem as well. I wonder how much investment it would take to hire a few phone developers to crank out a dozen quicky VR demos/games. And also port an older AAA title like Doom/Quake specifically for the device so at least there would be one must-have title.

[MSat]

It definitely needs to be head mountable. If performance is reasonable, It could be an inexpensive and practical way for people to test the waters with VR, which may make some want to buy something better, like the Rift.

I would buy a RIFT2GO

[budda]

Hi,

Its interesting that Samsung has an official free application that permits streaming from SOME high-end Samsung Smart-TVs to SOME Samsung phones and tablets.

The application is called 'Samsung Smart View', but it is also known as dual view, smart view or smart dual view.

The official reference site is located here : https://play.google.com/store/apps/deta ... .tab&hl=en

[About the Application]
This application is Wi-Fi remote control for Samsung Smart TV. If both your tablet PC and TV are connected to the same wireless Router, your tablet PC will link to the TV automatically where you can use your tablet PC like a remote control. The TV can also transfer live feeds to your tablet PC.

[Main Features]
-Dual View
Enjoy TV and other multimedia right from your mobile device that’s connected to Samsung Smart TV.

It seems mysterious that Samsung seems to have pulled the popular 'dual-view' support from the application for newer android tablets and phones. Customers, developers and even users of other phones are disappointed about the situation, and are looking to create ports of the application to their unsupported devices.

Here is a youtube clip showing the software streaming a movie and console game from a TV, to an older model tablet.

http://www.youtube.com/watch?v=SRkhoefPM1o

Thanks.

[budda]

Hi,

This is real example of what to expect when shutterglasses are used with an oled display.

In this demonstration, Samsung shutterglasses are used with a Samsung super-oled TV to display two separate (2D) video streams.

In a head mounted display of course, the two video streams would be seen by each respective eye, for S3D viewing.

http://www.youtube.com/watch?v=8fDZPOJHwfc

Note: The 'Dual View' for the 3D TV is different to the 'Dual View' for the wireless multimedia mirroring - its a confusing branding situation by Samsung.

Thanks.

[budda]

Hi,

A wireless streaming application from Kainy looks very promising for streaming video from PC to either tablet or phone, for head mounted displays.

http://www.kainy.com/

http://www.youtube.com/watch?v=JmLVzH72 ... e=youtu.be

Another advantage of wireless streaming vs cable is the ability to easily connect multiple devices to the source. This makes is possible for two or more people to share the same perspective of the game, as a spectator or as multiplayer.

It seems the possibilities of wireless streaming for virtual reality are growing by the week.

Thanks.

[Okta]

After checking your link to Kainy and watching some youtube clips i bought it because it looked good and the creator said spalshtop had ripped off all his ideas. It only cost $3 but unfortunately it hardly works, has about 1.5 seconds lag over lan, very buggy and crashy, and wouldn't even load a screen over 3g. I then tried free Splashtop2 over wifi lan and wow, it works a treat. Just need to pay to a month/yearly license for over 3g which i is a rip off because iphone users get it unlimited.

Im sure Kainy can work better but that was not my experience today.

[shent1080]

This seems like a great idea, cheap, wide fov, and best of all wireless.......is anyone still using this system?

What were the major cons?

[budda]

Hi,

Thanks for your interest.

I see no technical limitations to making this work with existing technology.

Only three important things need to be sorted out.

1. Parsing the input video stream into a field sequential format within the phone and displaying it in a 'sample and hold' manner. It is assumed the input video stream is uncompressed and the output video processing is low latency.

With the rumoured Galaxay Note 3 coming with a 6.3 inch screen and 1080p, the same video stream could be upscaled in the phone to maintain the consumer friendly video bandwidth requirement as for 720p.

2. The OLED screen refresh rate seems to be capped at 60 Hz as part of the 'Project Butter' Android philosophy. We need to get a 'Project Whipped Cream' going to get it to 85 Hz or better. The screen itself can certainly be revved very high and still maintain display quality.

3. The sequential frame display format needs to be compatible with the speed of the stereo shutterglasses so that minimum blanking occurs between screen refreshes. This will help to reduce the demand on the screen display by removing the need for complete blanking frames, and to maintain image quality.

I would say that Samsung or Google would be in the best position to make this technology work. There would be more demand for 'Oculus Rift'-type OLED Head Mounted Displays than OLED TV's. You can get many more panels out of the motherglass for a 6 inch HMD than for a 55 inch TV, and the profit margin would be higher as the HMD market is potentially explosive but the TV market is stagnant. Also, Samsung/Android already have market dominance in these HMD related technologies, leveraged from their mobile phone businesses.

This is a billion dollar market opportunity.

Thanks.

[shent1080]

Thanks for the info,

I only ask because i'm trying to get this setup to work for myself, i'm using a 10.1 inch tablet to test but if i can get freepie working with it solidly (with a massive amount of help from cyber villain, Mahler and others to which i am very thankful) i will get a nexus 7, it seems people have been getting great results from it.

Would you not use SBS format instead of shutterglass technology? doesn't the rift use SBS? I would have though it would be more realistic without going over the top with 3d effects, also all of the software is already in place to do this i.e. tridef.

I'm hoping that the Rift goes wireless, i've owned HMD's in the past and my biggest bug bare is that you are tethered to a box, after that it's the FOV.

I think the idea of slotting in a tablet to a head mount with lenses makes so much sense. Low setup cost, easy cheap upgradable possibilities, wireless display.......

For The SBS sytem to work i don't think it needs any help from the likes off google etc does it? all of the software is there to transform your tablet into a SBS wireless HMD.....tridef, freepie, splashtop........the only thing it lacks, for example in a FPS, is the ability to seperate mouse and head tracking movement so that a gun and view can be controlled seperately.

Do you have a tablet/phone HMD setup at the moment? what is it?

Mark

[brantlew]

The biggest issue right now is latency/frame-rates. Oculus suggests that 60 fps is the minimum speed necessary to achieve a feeling of "presence". Driving a 720p display at 60Hz wirelessly is difficult enough. Trying to achieve the higher resolutions that the Rift is targeting will be an even bigger challenge.

I'm sure that a lot of people will put together wireless systems, but there will be probably always be a tradeoff. Wired high resolution versus wireless low resolution.

[shent1080]

To be honest 720p is enough for me, I know that it's different comparing gaming on a tv and gaming on a hmd but i used yo have a great HMD for the time, the display wasn't fantastic but it blew me away, the only problem i found was that i was tethered to a breakout box.

With the discovery and commercialisation of something like this:

http://www.geekosystem.com/t-ray-wifi/

It could be the end to that particular problem.

[shent1080]

and leading up to that there is 802.11ac and wigig

[budda]

Hi,

Brantlew mentioned earlier about another similar thread discussing using a smartphone/tablet with side-by-side format and wireless.

I don't know if it's coincidence or not, but I feel like these threads should be joined.

http://www.mtbs3d.com/phpBB/viewtopic.p ... 607#p84457

The main point of this thread is to use the same full screen display so that each eye sees the full screen.

For instance anaglyph could be used to prototype this idea and opposed to the more technically advanced full-colour, full-screen field sequential display.

The key difference is the screen view is now twice as wide as a side-by-side view.

For whatever reason, the screen can be placed twice as far back with half the lens magnification as an equivalent 'Oculus-Rift' type setup. From there, the screen can be brought forward to achieve a 'better' field of view as required.

The great thing about a shutterglasses setup, is that people won't be complaining about having to wear them, if they are already built into the head mounted display.

Thanks.

[shent1080]

cool, i've posted on the other thread now, i'll take a look at anaglyph though.

I suppose the main problem with this is the fact that you Have to put the screen furthur back, will this then need a counterweight at the back of your head?

Anyway, thanks for the info.

[budda]

Hi,

Okta tried using the Kainy wireless streaming application but had some difficulties ...

After checking your link to Kainy and watching some youtube clips i bought it because it looked good ........ Im sure Kainy can work better but that was not my experience today.

Version 1.4 of Kainy was released today, and whilst it is too early to say what improvements have been made, there seems to be some significant developments for the connectivity options.

http://www.kainy.com/


Thanks.

[shent1080]

I've checked out Kainy 1.4, didn't try earlier versions but have to say that it responds better than splashtop and it's cheap at the moment on the play store.

There's less lag with Kainy in comparison to Splashtop 2, or HD.

[MarshawnMan]

LOL.. great thread!!

[budda]

Hi,

It looks like Samsung are going to create an engineering competition this year, in which the prize will be seed capital for new start-up ventures ( ... Oculus Rift anyone).

Samsung Spending $1.1 Billion to Find Next Big Thing

Article from PCMag : http://www.pcmag.com/article2/0,2817,2415085,00.asp

"This is an exciting opportunity for us to engage with entrepreneurs and innovators and empower them to leverage Samsung's technology and global brand presence to bring our collective visions to market."

........

To get things started, Samsung will host the SamsungCreate Challenge later this year, which will call on artists, engineers, entrepreneurs, and other innovators to pitch their best ideas for how best to improve lives with technology.

I suppose that includes Virtual Reality Technologies - sharpen your pencils Folks.


Thanks.

[STRZ]

I suppose that includes Virtual Reality Technologies - sharpen your pencils Folks.

It's going more down the AR route i think when it comes to big data and internet technology.