Could you post a video that demonstrates the current limitations of the tracking? It literally looks perfect in that video I was like
I imagine it would have trouble with something occluding the camera, really violent camera movements and smooth/plain backdrops (as you've said), anything else?
Also, what kind of set up does it need? Just the "move camera sideways" set up? If so have you considered using stereo cameras - could this let you do the set up stage by comparing feeds from the two cameras?
It really looks so close to a perfect solution, hope your development continues as well as it has been going so far.