Using Virtual Reality Devices with VTK

Updated September 2016 to include Oculus Rift support

Virtual reality (VR) is a technological game changer. It allows us to look at and interact with data sets, models, and point clouds to give us a glimpse of what we can accomplish with our information. As hardware and rendering performance improve, the use of VR will only become more compelling.

With this in mind, we added two new modules into the Visualization Toolkit (VTK) to make it possible to use Oculus and OpenVR-compatible devices. The new modules vtkRenderingOculus and vtkRenderingOpenVR have been tested on the Oculus Rift and HTC Vive devices respectively.

ViveTownData

Rendering of a PointCloud within VTK using the HTC Vive (dataset courtesy of SenseFLY SA).

Getting Started

To use these new modules, download the master branch of VTK from the VTK respository on GitHub. Before configuring VTK you should make sure you have downloaded and installed the Simple DirectMedia Layer 2 (SDL2) as the Oculus Rift and HTV Vive modules require it. If you are targeting the HTC Vive you will also need the OpenVR SDK, for the Oculus Rift you will need the Oculus Windows SDK . At this point you can configure VTK and turn on the modules you desire and start the build process. To maximize performance, make sure you build an optimized version of VTK.

What’s Included

Our goal is to allow VTK programs to use the Oculus and OpenVR modules with few changes, if any. If you link your executable to the vtkRenderingOpenVR module, the object factory mechanism should replace the core rendering classes (e.g., vtkRenderWindow) with the OpenVR-specialized versions in VTK. vtkRenderingOpenVR contains the following classes as drop-in replacements.

vtkOpenVRRenderWindow

This class handles the bulk of interfacing to OpenVR. It currently supports one renderer that covers the entire window.

vtkOpenVRCamera

vtkOpenVRCamera gets the matrices from OpenVR to use for rendering. It contains a scale and translation that are designed to map world coordinates into the head-mounted display (HMD) space. Accordingly, you can keep world coordinates in the units that are best suited to your problem domain, and the camera will shift and scale the coordinates into the units that make sense for the HMD.

vtkOpenVRRenderer

The vtkOpenVRRenderer class computes a reasonable scale and translation. It sets these computations on OpenVRCamera. It also sets an appropriate default clipping range expansion.

vtkOpenVRRenderWindowInteractor

While VTK is designed to pick and interact based on X,Y mouse/window coordinates, OpenVR provides XYZ world coordinates and WXYZ orientations. vtkOpenVRRenderWindowInteractor catches controller events and converts them to mouse/window events. The class also stores the world coordinate positions and orientations for the styles or pickers that can use them. vtkOpenVRRenderWindowInteractor supports multiple controllers through the standard PointerIndex approach that VTK uses for MultiTouch.

Future Developments

These new Virtual Reality modules are currently still improving. Moving forward, we look to add support for the OpenVR overlay, which is great for displaying a user interface. We also aim to make the module faster and include more event interactions. Stay tuned to our blog for updates.

In the meantime, download VTK and test the modules for yourself. Let us know what you think by adding your experiences on the VTK mailing lists!

15 Responses to Using Virtual Reality Devices with VTK

  1. Christophe Bourcier says:

    Great news! Serious VR will be a great communication tool to show scientific results.

    Any chance Google Cardboard will be supported by OpenVR? From what I have read, it seems Unity supports both, but OpenVR supports only HTC Vive.

    • Ken Martin says:

      We are hoping to add support in the future for Google’s current VR efforts. It is something we are actively pursuing.

  2. Hello! We have developed something like that for Oculus Rift SDK 2. In particular, we developed an interface between VTK, Oculus Rift and Leap Motion device. The final aim of this research project is the emulation of a virtual orthopeadic laboratory for lower limb prosthesis design (VOLab). We will publish some videos about this research work in the next months. (web site: http://dinamico2.unibg.it/vk/)

  3. Isabella Morgan says:

    Can you change the tracking world in vtk OpenVR Rendering to avoid using the Oculus tracking system?

  4. Ivan Lee says:

    I had successfully run vtkOculusTest in my environment. I hope I can do some interaction with my romote in the virtual world.Are there any examples?Now I can not even rotate the object. Thanks!

    • Ken Martin says:

      The plan is to add event interaction for Oculus once I get my hands “literally” on a pair of touch controllers. I could have added xbox controller interactions but decided to wait for the touch controller to be released. Hopefully early next year I’ll update the event code to add support for those.

  5. Fei says:

    Hello Ken, how can i get the vtkOculusTest example? thanks a lot.

    • Ivan Lee says:

      You can choose BULID_TESTING and Mosule_vtkRenderingOculus of the latest 7.1 in Cmake.Then generate it.There are some little errors in the rendering loop.

  6. Ken Martin says:

    Just a quick update. VTK master has been updated and now includes fixes for OpenVR SDK 1.0.5 as they changed a few function signatures. I haven’t had a chance to revisit Oculus lately but I am doing another pass on the OpenVR support to clean up a few issues, add a Volume Rendering example, and add a feature or two.

  7. Ken Martin says:

    Some more updates. There are some new changes including the following

    Add floor option to show a floor

    completely rework the matrix and coordinate code to be cleaner and use standard VTK conventions (right handed)

    add support for a dashboard overlay with options to subclass and support for saving/loading camera poses and adjusting motion factors and scale factors.

    Improve the resetcameraclippingrange to work better for VR

    change how eye position is handled so that VolumeRendering will work properly

    Handle the case where people make the cubes invisible which results in a 308 error on loading the model.

    Add support for the user to specify the mapping of physical coordinates into world coordinate. Allows for different view up vectors other than the old code which was hard coded to Y.

    Add middle mouse support mapped to the grip button.

    Add some support for a hardware clipping plane.

  8. Aditya Dua says:

    Hi Ken, for some reason open selecting module vtkRenderingOpenVR in cmake, the build throw several errors saying there are unresolved references. I have both SDL and openVR SDK installed and i have tried to build several times, trying with different options but still no luck.

    Could someone from vtk provide files for pre-built vtk that includes vtkRenderingOpenVR module wrapped in python 3.6 ?

    I would much appreciate it, as several hours of google search and running builds on Visual Studio have proved completely useless.

    • Ken Martin says:

      You might try posting on the vtkuser’s list (with the specific build errors you see), I know folks out there have it building and working and often someone pops up with some advice (or maybe they have binaries). I believe my builds are all Python 2 based which doesn’t help in this case.

      • Ajay says:

        Hello Ken ,

        I am using VTK 8.0 for renderingin Oculus Rift DK2 , After building VTK 8.0 libraries using VS2015, Successfully tested VTK related sample programs, But am getting error with “vtkOculusRenderWindow” when i tried to render to oculus. Kindly help me with your suggestions. Am i missing some configuration setup or not ?

        “Error LNK2019 unresolved external symbol main referenced in function “int __cdecl invoke_main(void)” (?invoke_main@@YAHXZ)”

        • Ken Martin says:

          Oculus changed their SDK such that old code no longer compiles and we haven’t had time to update VTK to match it. In the mean time I think you can use VTK–Oculus vis steamvr using the OpenVR module in VTK.

Questions or comments are always welcome!