Taking ParaView into Virtual Reality

In an earlier blog post, we talked about using the Visualization Toolkit (VTK) and the Vive to explore your data in Virtual Reality (VR).  In this post, we are going to talk about an experimental version of ParaView you can download that adds VR support for both the Oculus Rift and HTC Vive headsets.

Before we dig into how to get your data into VR, we really want to thank the National Institutes of Health, who sponsored the work under the grant NIHR01EB014955 “Accelerating Community-Driven Medical Innovation with VTK,” and our collaborators at the Oak Ridge Leadership Computing Facility (OLCF).

Getting Some Hardware

The VR version of ParaView will work with both Oculus and the HTC Vive headsets. Currently, ParaView’s support for the HTC Vive is better, so if you are undecided, have your purchasing department buy that first. You can purchase the Vive from Amazon or directly from HTC; as of this writing, it runs about $800. If you want to go the Oculus Rift route, you can buy one directly from Oculus or from some retailers such as Best Buy (currently for $600).

A good VR experience requires very high frame rates, typically 90 Hz per eye. This means ParaView needs to render very quickly. Having a high-end CPU/GPU is critical. Both the Rift and Vive suggest at least a GTX 970 class GPU, but the higher, the better. Both vendors sell or recommend systems that are Vive-ready PCs or Oculus-ready PCs. Likewise, both vendors have tools to check if your current system is VR ready for the Vive or Oculus.

Once you have purchased a headset, make sure that it is installed per the manufacturer’s instructions and that it works with their demo software. For the Oculus, you need to make sure you authorize running “Unknown Sources” in the Settings/General menu as shown below.

 

Oculus

 

The VR version of ParaView

First, download the experimental, VR version of ParaView from the ParaView download page. It can be found under Releases with the Version of ParaView set to v5.3 for OpenVR or v5.2 for Oculus. Set the Type of Download to Experimental Demos. It is designed to run under a 64-bit version of Microsoft Windows such as Windows 10 64-bit. You can also download it for OpenVR directly or Oculus directly. Once downloaded, unzip the file and place it on your system. The zip file includes the ParaView executable along with the required DLLs to run it.

Before running this version of ParaView, make sure Oculus Home for the Rift or Steam VR for the Vive is running. Once unzipped, you can then run ParaView by double clicking on the executable. ParaView will start as usual with a window on the computer screen. Using the regular mouse/keyboard, you can load or create your visualization as you normally would in ParaView. If you are new to ParaView, you can get a quick start by loading the sample ParaView State file SampleVRTubes.pvsm, which is included in the zip file. ParaView also has a full tutorial you can download.

Once you have your visualization up in ParaView, you can send it to VR by hitting the “Send to Oculus” button for Oculus Rift or the “Send to OpenVR” button for the HTC Vive. These buttons are located at the bottom of the Properties Panel as shown in the lower-left corner of the below image.

PVVR2

 

PVVR2SendWhile in VR, you should not interact with the ParaView window. With the HTC Vive, you can interact with the data using the two touch controllers. Pulling the trigger will grab an object, allowing you to hold it and move it around. If you hold both triggers down at the same time, you can scale and translate the virtual world. Pulling your hands apart will make it larger while moving them closer together makes it smaller. Moving your hands in the same direction translates the world in that direction. Pressing down on the touch-pad will start flying you in the direction the touch-pad is pointing.

When you are done in VR, press the menu button on a touch controller on the Vive to return to ParaView or the space bar on the keyboard for the Oculus Rift. Now you can make more changes to your data in ParaView, and when you are done, hit the “Send to OpenVR/Oculus” button to return to VR. This way, you can alternate between exploring your data on the screen and in VR.

This experimental version should handle rendering polygonal surfaces and molecular models. It currently does not support volume rendering, although the underlying support in the VTK layer does. We suspect it will not work with animations either, so be warned. Many annotations will not show up. Finally, it does not support remote rendering of any kind due to the high frame rates required.

Recent additions to VTK that make this all work

Under the hood, ParaView is using the new Oculus and OpenVR rendering modules in VTK. These two modules can be found in the Rendering sub-directory and can be turned on when compiling VTK using CMake options. When VTK is built with these modules, it includes new subclasses of vtkOpenGLRenderWindow, vtkOpenGLRenderer, vtkOpenGLCamera, and vtkRenderWindowInteractor to support rendering and interacting with the Oculus or Vive, respectively. In addition, there are some new classes in RenderingCore designed to help with picking and interaction, when you have devices that support 3D positions and orientations such as the Vive and Oculus controllers. vtkRenderWindowInteractor3D extends many of the methods in vtkRenderWindowInteractor to include a Z coordinate and allow for event coordinates to be in world coordinates in additional to window coordinates. vtkPropPicker3D is designed to perform a pick using a 3D location as opposed to the more common X, Y screen location. Finally, vtkInteractorStyle3D provides the basic interaction style that the Vive controllers use to select and move objects or translate and scale the world.

If you have built VTK with either Oculus or OpenVR support, then creating an immersive visualization follows the same model as existing VTK visualizations. A simple cone example is shown below.

#include "vtkActor.h"
#include "vtkConeSource.h"
#include "vtkNew.h"
#include "vtkOpenVRCamera.h"
#include "vtkOpenVRRenderer.h"
#include "vtkOpenVRRenderWindow.h"
#include "vtkOpenVRRenderWindowInteractor.h"
#include "vtkPolyDataMapper.h"

int main(int argc, char *argv[])
{
 vtkNew<vtkActor> actor;
 vtkNew<vtkOpenVRRenderer> renderer;
 vtkNew<vtkOpenVRRenderWindow> renderWindow;
 vtkNew<vtkOpenVRRenderWindowInteractor> iren;
 vtkNew<vtkOpenVRCamera> cam;

 renderWindow->AddRenderer(renderer.Get());
 renderer->AddActor(actor.Get());
 iren->SetRenderWindow(renderWindow.Get());
 renderer->SetActiveCamera(cam.Get());

 vtkNew<vtkConeSource> cone;
 vtkNew<vtkPolyDataMapper> mapper;
 mapper->SetInputConnection(cone->GetOutputPort());
 actor->SetMapper(mapper.Get());

 renderer->ResetCamera();
 renderWindow->Render();
 iren->Start();
 return 0;
}

Future Directions

There are a number of areas we are looking to work on in the future. Oculus should be releasing their touch controllers soon, at which point we hope to add the same interaction options to the Oculus Rift that we have currently for the HTC Vive. We want to enable volume rendering and animations from ParaView to be exported to VR. ParaView has a number of 2D annotations and rendering options that would fit well on a popup sheet in VR. Lastly, 3D controllers with their real-time position and orientation open up a wide range of possibilities in terms of interactive widgets to explore and steer the visualization.

If you have an application or technology area that could benefit from visualizations in VR, feel free to contact us. There are a lot of neat opportunities in this space, and we are excited to see where it goes.

 

24 Responses to Taking ParaView into Virtual Reality

  1. Will Schroeder says:

    +100 well done

  2. Tristan Wright says:

    With the Vive can you move around in the virtual space?

    • Ken Martin says:

      Most definitely. Your head position and orientation are tracked for starters. But beyond that, using the two Vive handheld controllers you can scale and translate the world as well as fly around.

  3. It works really well! Would be great if we can add animations

  4. Axel Gerstenberger says:

    I do not have either headset, but, conceptually, would the same work with a VR Cave that has tracking? Normally, we use ParaViews VR Plugin, but this “Send to Open VR” looks like it could work with caves as well? This may allow to prepare complex scenes, while VR users look at the previous scene until you press “Send” again. Just a thought…

    +1 for animations

  5. Guillaume Jacquenot says:

    Very interesting job. It opens a wide bunch of possibilities. Is there any plan to release a full version of ParaView for Windows (i.e. with Python scripting functionnalities)? And also a version with animations?

    • Ken Martin says:

      I am working on a new paraview release with improved VR support and some additional features. I’ll try to make sure I build it with python scripting included. I do hope to get the changes merged into paraview master at some point as well.

  6. Benjamin says:

    Note that If you are running Paraview with the Vive, it will only work if the SteamVR Room Setup is configured in “Room-Scale” mode. At least for me “Standing only” mode did not work.

  7. YueHan Zhu says:

    hello! I am working on showing the processing result of openfoam to VR, and the biggest issue is exporting the 3D animation. Did the file which exported by “foamToVTK” contains all the frames from openfoam? thanks!

  8. Gregory Met says:

    Hello !

    Everything works good for me, thanks Kitware !

    Does anyone know if this is possible to see ‘animated results’ (like moving flows) with the HTC Vive ?
    Actually this is impossible, you can only see static view…

    Thanks guys !

    • yuehan zhu says:

      why it is impossible, can you give me advice, i am working on it.. thank you!

      • Ken Martin says:

        Animation is possible, but the logic for connecting the animation controls in ParaView to what is rendered in VR is not there. So it is technically possible once someone writes the code etc.

  9. David T. says:

    Hi all,

    For those of you asking for animation, we have a demo with some working now, but there are a few tricks:

    1. You can’t call vtkOpenVRRenderWindowInteractor::Start yet, but you can

    vtkOpenVRRenderWindowInteractor* ri;
    while (1) {
    ri_>DoOneEvent(renderWindow, renderer);
    // Now update to the next frame of your animation.
    }

    2. We make things more interactive by adding actors for each animation frame and then turning their visibility on/off inside the loop above.

    3. You may want to disable visibility culling if rendering speed is an issue.

    Hopefully we’ll be able to release a demo sometime soon.

  10. HAOCHENG LIU says:

    Hi All,

    may I ask do we have a plan to support PS VR? If possible, would you please point me to some PV documentation that I can try and test PV with PS VR? Thanks!

  11. I am just curious about the Linux support. Do you have plan for it? Can i install VR enabled ParaView from source?

    • Ken Martin says:

      The OpenVR changes are now merged into the main ParaView git tree so you can build it from source. I know one person has tried it on Linux and it compiles, but doesn’t seem to run correctly yet.

      • That is great news! At least, there are some progress in that direction. Do you have any time frame to have working VR enabled ParaView on Linux environment. I am looking forward to test it in earth system science domain. I think that it would facilitate better understanding of complex processes such as storm evolution, air-sea interaction etc.

  12. Ken Martin says:

    Some success from David Thompson on building for Linux. Here are his comments

    Success on x86_64 Ubuntu 16.04.2 (with some hacks, described below for the record)!

    Also, at least on my machine, the VTK OpenVR dashboard appears right-side up, but the y location of the pointer appears to be flipped (i.e., aiming at the top of the dashboard highlights the bottom row of buttons; aiming near the bottom of the dashboard highlights the top row of buttons). Maybe some transforms have changed from the release to the beta I’m running?

    David

    # Using the SteamVR beta with PV on Linux

    1. After installing SteamVR, and opting into the beta I could not start SteamVR from the desktop (it opened a dialog claiming that a download had to complete, but it lies). However, I could start it from the command line with

    ~/.local/share/Steam/ubuntu12_32/steam-runtime/run.sh ~/.local/share/Steam/steamapps/common/SteamVR/tools/bin/vrmonitor.sh

    2. But first (before running vrmonitor, anyway), I had to create symlinks to some vulkan and udev libraries included with SteamVR that were not in the system search path. This is a hack, but it works for now:

    sudo -s # i.e., run the following as root
    cd /usr/lib/x86_64-linux-gnu
    ln -s ~/.local/share/Steam/ubuntu12_32/steam-runtime/amd64/usr/lib/x86_64-linux-gnu/libvulkan.so.1.0.39 .
    ln -s libvulkan.so.1.0.39 libvulkan.so.1.0
    ln -s libvulkan.so.1.0.39 libvulkan.so.1
    ln -s libvulkan.so.1.0.39 libvulkan.so
    ln -s ~/.local/share/Steam/ubuntu12_32/steam-runtime/amd64/lib/x86_64-linux-gnu/libudev.so.0 .
    ldconfig

  13. Ken Martin says:

    There is a new Paraview 5.3 based executable with some new features (such as a dashboard, floor) for OpenVR/Vive that can be downloaded from the ParaView 5.3 Experimental section download page here http://www.paraview.org/download/

  14. VR Cranfield says:

    Hello, is there any way of creating a VR menu using Paraview VR, so the user can directly select options from there and interact with the objects? Or is it any development in that area?

Questions or comments are always welcome!