Using phone screens?

[Chriky]

What's stopping people hacking smart phone displays to be used as regular LCD displays, say taking an HDMI input? Do HTC etc keep secret to format that the cables use or something? Are the connections propriatry?

I just ask because you can get replacement screens of a pretty decent quality very cheap.

Cheers

[ERP]

Short version is driving the displays.
They don't take anything approaching a standard signal, and they are generally not well documented.
It can probably be done with an FPGA, but it would require capturing the init sequence to the display controller with a logic analyser.
If you look around you can find people who've driven older cell phone screens with FPGA's , but I don't know of anyone who has successfully driven for example an iphone retina display.

In principle they would be great for a wide FOV HMD using leap style optics, but the number of people capable of getting them running is at best limited and they apparently don't have the motivation.

[Chriky]

Thanks; you'd think that all kinds of hobby tech communities would want cheap small screens for various stuff... it's suprising there aren't enough talented guys to hack the iPhone screen.

[PalmerTech]

It is not just motivation, it is cost. FPGAs capable of driving high resolution panels well cost a lot, and they are bulky, too. Since the panels are usually pretty sensitive to timing and skew, so you cannot really extend the length of the connection between the screen and FPGA too much.

Kind of a sad state. Though I do know of some 3.2" 800x480 displays with HDMI input, tempted to try some out.

[WiredEarp]

I've been thinking about this recently, and have decided it may well be worth another look. The thing is, there are display driver circuit boards out for many LCD panels already, and while it may cost $$$ to develop the driver board, you'd think that the potential market could be there (cellphone screens are higher res and better in low light than many off the shelf ones, so 'carputer' people are interested as well), and that if a driver was achieved, we'd be able to get 800x480+ resolution per eye, wide FOV HMD's. With smartphone resolutions going through the roof, this would also mean we would be in a prime position to eventually upgrade. iPhone screens are ridiculously cheap, all the cost really is in driving it.

Unless the circuitry is proprietary or something, or there is a massive variation in componentry (IE, same phone being made different ways), surely this will be an inevitable development, once PR type HMD's get more popular when the Rift is released.

@ PalmerTech: I also found some displays of that res / size today, so very interested in checking them out. The obvious advantage would be that it would solve the aspect ratio problems, opening up a large amount of games etc. Getting them close enough together would be a PITA but doable...

[brantlew]

I'm all for this movement although I am a bit ignorant of the details.

If you attempted to directly drive the panels then you would need custom hardware and software (the FPGA solution). But don't these panels already have control hardware? The phone manufacturers have to provide something to electrically control these components, right. Would it be easier to use the existing hardware controller from a broken phone and then rework the driver and/or memory interface? Surely there is some modularity to these designs. Maybe not the iPhone, but you would think the Android manufacturers would not design motherboard/graphic interfaces from scratch. Or is the interface so entangled between the BIOS/motherboard/graphics chip/memory that you just can't tap into that hardware chain easily?

[ERP]

In a lot of ARM based phones the controller along with the GPU are in the processor.
See TI's OMAP processors as an example Broadcom and others have similar devices.

[PalmerTech]

@ PalmerTech: I also found some displays of that res / size today, so very interested in checking them out. The obvious advantage would be that it would solve the aspect ratio problems, opening up a large amount of games etc. Getting them close enough together would be a PITA but doable...

I do not know of any panels that are small enough to put side by side, the smallest decent res panels I have seen are 3.2".

Phones these days use something called an SOC, a "System-on-a-chip". It combines the RAM, CPU, GPU, display driving, and a myriad of smaller functions into a single package. That is not something you can readily tap into without highly specialized gear and expertise, these SOCs are made to interface directly to the LCDs.

That leaves either FPGAs (Bulkier, expensive per unit) and ASIC based control boards (Small, lightweight, cheap... But huge up front development and manufacturing cost). It would be great to develop a board that can take advantage of high res cell phone displays, but it would cost a lot of money, probably more than a Kickstarter project can raise. The carputer market is small and rapidly diminishing, too, since you can just use the actual phone for media, navigation, everything you used to need a computer for. I think the market is there, but raising the money would be iffy.

[WiredEarp]

I do not know of any panels that are small enough to put side by side, the smallest decent res panels I have seen are 3.2".

I'd put them at angles or to the sides and use mirrors/optics to redirect the light path. Putting them to the sides seems pretty time honored.

How much are those FPGA's?

@ brantlew: are you referring to a having a custom ROM of some sort, to do nothing but display an incoming stream onto the screen, using the existing hardware and OS functionality?

[brantlew]

I don't really know what I'm talking about. I was hoping that maybe the architecture was modular in such a way that you could "plug" into the display at multiple points. For example, if there was a separate chip that simply handled signal timing and cyclical display refresh and gathered data from a central memory unit or had it's own on-board graphics memory, maybe you could create an interface to that component instead of directly to the panel. If all the "chips" are engraved on the same die though (SOC), then that would be basically impossible.

Maybe there is a way to tap into the OS display drivers. Pull data from an external IO port and push it to graphics pipeline?

Mostly I'm just talkin' out my ass and wishful thinking though.

[PalmerTech]

There are applications out there that can stream to phones, the very best claim latency as low as 40ms.

[brantlew]

@Palmer: Stream to what interface? Are you talking over WiFi?

That would be interesting because you could solve the transmission, interface, and display problem all at once with the on-board phone hardware. Of course you would need to feed two displays simultaneously at the same data rate. Keeping the images synchronized might be a problem. If it worked it would be sweet though. Just gut two phones, load in the "streaming" driver, and then transmit the images from a rendering computer.

The other option might be through the USB (or other serial) port. If you could "stream" over serial port to the display then I would expect the latency to be less. Plus you might be able to better manage the synchronization. But you would be need a back-top setup with wired connections to both phones.

Now that I'm thinking about it more, you could manage synchronization of the images over WiFi, but it would require a custom network protocol - sort of like TCP except across two sockets. It would probably add a bit more latency though to an already laggy scenario.

Edit: Just realized...are we talking about one screen per eye or a single screen (a-la PR2)? If the latter then synchronization no problem.

[PalmerTech]

The newest version of Splashtop HD claims it has 40ms latency at 60fps under best case scenarios. In real life usage, 60ms+ latency is more realistic. Combine that with tracking latency, and you end up well over the acceptable latency limit. One display would be the best way to do it, synchronizing two phone displays is almost impossible, even when the content is rendered onboard. Sure, you can synchronize pretty close, but there is nothing keeping the refresh rates of the phones in sync! Until phones/small tablets get another resolution bump past 720p, using a display like the one in the RIFT and a wireless video link is the best solution.

Now, there are companies that make presumably FPGA based driver boards for cell phone panels. Here is a 3.2" 800x480 display with HDMI input: http://www.bhphotovideo.com/c/product/7 ... _Snap.html

As you can see, though, pretty high cost. Small markets and expensive parts tend to do that.

[brantlew]

Cool app. There's a lot of stuff going on under the covers that makes its performance so bad. For one, this is a generic VNC-type application so it has to go through all the gyrations of screen sampling, compression, network transfer, decompression, and rescaling. You could maybe prototype with something like that but real usage would probably suck - like trying to play a game over Remote Desktop.

In theory though I believe that if you wrote a custom client and server applications designed for specific hardware then you could get better performance. So for example if you wrote the PC-side of it as a display driver then you could get the video card to render at the phone resolution. No need to scale. If you went over wire then you could probably skip compression and network protocols and take a big chunk of latency out that way. The phone "driver" would just have to pull raw video off the wire and push to the display pipeline. If you wanted wireless, you might be able to cut down on the latency if you were willing to lower your frame rates. Just skip (or significantly simplify) the compression and use more bandwidth.

Sure you have to write a lot of software, but at least it's plausible. Tapping into an integrated SOC seems a lot harder (at least to me).

[WiredEarp]

60fps = 16.7ms per frame.
Now, assuming we can send a complete frame in 16.7ms to the phone (unlikely), we then have to push that to the phones display, which will probably take another 16.7ms. That gives us a 33ms lag for even naive applications. Add in overhead, and we probably will get to a 50ms lag minimum. If that math is anywhere near correct, it is not going to be worthwhile doing.

@ PalmerTech: the price on that unit is not really that bad! There was a 4" 800x480 panel I came across as well that was much cheaper (about 1/4 the price). Not sure if 4" is too large to use easily however...

[brantlew]

I'm calculating about 32 uncompressed frames per second transfer rate to an iPhone 4 resolution over a USB port. I'm thinking you could get 66% - 75% lossless real-time compression with a 2 or 3 frame lag, so that's up to around 45 FPS optimistically. Take into account rendering and display delays on both ends and you are left with what? Who knows??? Probably pretty tight but you might be able to squeeze 30 FPS out of it. That seems like it would be worth it. Also, when the USB 3.0 phones start coming out, that number could get a lot better.

[WiredEarp]

Might just be better to develop under android. Although then you may as well also just go the FOV2GO way...

[brantlew]

I feel like this would complement the FOV2GO. Offload the heavy rendering to a full size computer and use the FOV2GO + video streamer as a display device. Of course, phones are just getting more and more powerful so in a few years you might as well just render on the phone. I assume today's phones could run 12 year old titles at decent frame rates if someone were to port them. Maybe with Windows 8 we'll start seeing an easier pathway to simply recompile older titles straight to Windows phones and make this a bit simpler.

[WiredEarp]

I think you'd probably find that todays phones could probably play games from only 5 years ago at sufficient speeds. Once the cross platform tools get better this may well be a pathway that becomes more popular.

[cybereality]

Actually, mobile games are coming pretty close to rivaling current gen consoles.

I mean, if I told you this was a console or PC game from today you would probably think nothing was out of the ordinary:



And, yes, I have the game on my iPad and it does look that good.

[PalmerTech]

Yeah, mobile chipsets are amazing these days. The day where the entire console can fit in the HMD is not far!

[PalmerTech]

Yeah, mobile chipsets are amazing these days. The day where the entire console can fit in the HMD is not far!

[brantlew]

@Palmer: Something just occurred to me. What about the Playstation Vita? A high quality screen that's also a pretty good size for you. A full sensor package for motion tracking. A powerful rendering computer. Doesn't that sound like it could make an interesting HMD in the spirit of the FOV2GO or even the PRx?

[PalmerTech]

The size and resolution are a bit on the small side, better to just use the RIFT plugged into a smartphone counterweight for high FOV. I have a Vita, and it is great, but Sony won't let you write software for it without an SDK, and nobody has figured out how to run homebrew that can access all the hardware at full spec.

I would not be surprised if the next generation Nintendo console after the WiiU is a reincarnation of the Virtual Boy.

[brantlew]

Too bad it's such a closed system. If you could hardware emulate the buttons and pads you could provide a sort of head-mounted free look with current Vita games. No stereoscopic game support, but as you said - the resolution is a little low anyways. So head mount it with a monoscopic lens package, do some basic head-look d-pad emulation type stuff and you've got a quick and dirty VR gaming console.

[Guig2000]

Running 5 years old games in a smart-phone is exaggerated. for example think that supremme commander is a 5 year old game and have still sometimes FPS slowdown in a modern computer, Crysis 1 is also a five year old game. A tablet is more powerfull than a smarphone.
In fact the 5 years aproach the delay between desktop graphic cards and notebook graphic cards which if currently about 4 years.
Example: a nvidia GT555M (144 cores 590MHz (GF106), 128Bit DDR3 version), (introduced in the midle of 2011) quite equals a radeon HD3850 (introduced in November 2007).