Using Virtual Reality Devices with VTK

Updated September 2016 to include Oculus Rift support

Virtual reality (VR) is a technological game changer. It allows us to look at and interact with data sets, models, and point clouds to give us a glimpse of what we can accomplish with our information. As hardware and rendering performance improve, the use of VR will only become more compelling.

With this in mind, we added two new modules into the Visualization Toolkit (VTK) to make it possible to use Oculus and OpenVR-compatible devices. The new modules vtkRenderingOculus and vtkRenderingOpenVR have been tested on the Oculus Rift and HTC Vive devices respectively.

ViveTownData

Rendering of a PointCloud within VTK using the HTC Vive (dataset courtesy of SenseFLY SA).

Getting Started

To use these new modules, download the master branch of VTK from the VTK respository on GitHub. Before configuring VTK you should make sure you have downloaded and installed the Simple DirectMedia Layer 2 (SDL2) as the Oculus Rift and HTV Vive modules require it. If you are targeting the HTC Vive you will also need the OpenVR SDK, for the Oculus Rift you will need the Oculus Windows SDK . At this point you can configure VTK and turn on the modules you desire and start the build process. To maximize performance, make sure you build an optimized version of VTK.

What’s Included

Our goal is to allow VTK programs to use the Oculus and OpenVR modules with few changes, if any. If you link your executable to the vtkRenderingOpenVR module, the object factory mechanism should replace the core rendering classes (e.g., vtkRenderWindow) with the OpenVR-specialized versions in VTK. vtkRenderingOpenVR contains the following classes as drop-in replacements.

vtkOpenVRRenderWindow

This class handles the bulk of interfacing to OpenVR. It currently supports one renderer that covers the entire window.

vtkOpenVRCamera

vtkOpenVRCamera gets the matrices from OpenVR to use for rendering. It contains a scale and translation that are designed to map world coordinates into the head-mounted display (HMD) space. Accordingly, you can keep world coordinates in the units that are best suited to your problem domain, and the camera will shift and scale the coordinates into the units that make sense for the HMD.

vtkOpenVRRenderer

The vtkOpenVRRenderer class computes a reasonable scale and translation. It sets these computations on OpenVRCamera. It also sets an appropriate default clipping range expansion.

vtkOpenVRRenderWindowInteractor

While VTK is designed to pick and interact based on X,Y mouse/window coordinates, OpenVR provides XYZ world coordinates and WXYZ orientations. vtkOpenVRRenderWindowInteractor catches controller events and converts them to mouse/window events. The class also stores the world coordinate positions and orientations for the styles or pickers that can use them. vtkOpenVRRenderWindowInteractor supports multiple controllers through the standard PointerIndex approach that VTK uses for MultiTouch.

Future Developments

These new Virtual Reality modules are currently still improving. Moving forward, we look to add support for the OpenVR overlay, which is great for displaying a user interface. We also aim to make the module faster and include more event interactions. Stay tuned to our blog for updates.

In the meantime, download VTK and test the modules for yourself. Let us know what you think by adding your experiences on the VTK mailing lists!

10 Responses to Using Virtual Reality Devices with VTK

  1. Christophe Bourcier says:

    Great news! Serious VR will be a great communication tool to show scientific results.

    Any chance Google Cardboard will be supported by OpenVR? From what I have read, it seems Unity supports both, but OpenVR supports only HTC Vive.

    • Ken Martin says:

      We are hoping to add support in the future for Google’s current VR efforts. It is something we are actively pursuing.

  2. Hello! We have developed something like that for Oculus Rift SDK 2. In particular, we developed an interface between VTK, Oculus Rift and Leap Motion device. The final aim of this research project is the emulation of a virtual orthopeadic laboratory for lower limb prosthesis design (VOLab). We will publish some videos about this research work in the next months. (web site: http://dinamico2.unibg.it/vk/)

  3. Isabella Morgan says:

    Can you change the tracking world in vtk OpenVR Rendering to avoid using the Oculus tracking system?

  4. Ivan Lee says:

    I had successfully run vtkOculusTest in my environment. I hope I can do some interaction with my romote in the virtual world.Are there any examples?Now I can not even rotate the object. Thanks!

    • Ken Martin says:

      The plan is to add event interaction for Oculus once I get my hands “literally” on a pair of touch controllers. I could have added xbox controller interactions but decided to wait for the touch controller to be released. Hopefully early next year I’ll update the event code to add support for those.

  5. Fei says:

    Hello Ken, how can i get the vtkOculusTest example? thanks a lot.

    • Ivan Lee says:

      You can choose BULID_TESTING and Mosule_vtkRenderingOculus of the latest 7.1 in Cmake.Then generate it.There are some little errors in the rendering loop.

  6. Ken Martin says:

    Just a quick update. VTK master has been updated and now includes fixes for OpenVR SDK 1.0.5 as they changed a few function signatures. I haven’t had a chance to revisit Oculus lately but I am doing another pass on the OpenVR support to clean up a few issues, add a Volume Rendering example, and add a feature or two.

Questions or comments are always welcome!