Loading scene...

World-In-Hand Navigation

An exploration into extending the Three.js Framework

About the project

Three.js is a popular JavaScript library used for 3D graphics and visualisations on the web. As part of the seminar "Extending the Three.js Framework," hosted by the Department of Computer Graphics Systems at Hasso-Plattner-Institute, our team embarked on a journey to make a different navigation approach possible. The primary goal was to explore and implement features aimed at enhancing the existing navigation capabilities, while also ensuring robustness and user-friendliness.

What is World-In-Hand Navigation?

Navigation in virtual environments

Navigation is a critical aspect of user interaction within virtual environments, allowing users to move around and explore the digital world effectively. In the realm of 3D graphics, we experience virtual environments through the lens of a virtual camera. While users may feel as though they are directly manipulating the scene, it is, in fact, the camera that is being moved. This can sometimes lead to a disconnect, as the scene's spatial information is not always directly incorporated into the user's interaction. ^{[1, 2]}

The "World-In-Hand" navigation approach aims to bridge this gap by providing users with a more intuitive and immersive navigation experience. By allowing users to interact with the virtual world as if it were a tangible object in their hands, this method enhances the sense of presence and control, making navigation more intuitive, robust, and user-friendly.

Navigation in Three.js

Three.js provides a range of navigation controls out of the box, including FlyControls, TrackballControls and OrbitControls, with the latter being the most commonly used. These controls are available as add-ons and can be easily integrated into any Three.js project. Impartial to the scene, they respond to user input and adjust the camera accordingly. Like most navigation techniques, OrbitControls offer three primary modes of interaction: rotation, zoom (techninally a dolly), and translation (panning), ^{[1, 2]}.

However, these operations are focused on one predefined target point in the scene, limiting the flexibility to interact with the scene as a whole.

Our Approach

Our approach to World-In-Hand navigation is based on the exact mouse position projected onto the scene. This allows the user to grab a point or an object and move it around (panning), rotate the scene around that point (rotating) or the point in the center of the screen, and zoom in and out to the point. Try it in our demo above or in this version to compare to OrbitControls! You can use your mouse or interact through touch on mobile devices.

How it works

We calculate the exact mouse position by transforming the screen mouse position into normalized device coordinates (NDC) and reading the depth value from the depth buffer. Three.js's Vector3.unproject(camera) method allows us to transform the NDC into world coordinates. With these world coordinates we can then calculate precise movement vectors that we apply to the camera.

Enhancing the Navigation

In addition to the core navigation features, we have implemented several basic enhancements to further improve the user experience. These efforts are based on the idea of resilient or constrained navigation: We aim to avoid potentially disorienting or confusing camera movements, such as flipping the scene upside down or moving the camera too far away, by constraining the camera's movement within certain limits. ^{[3, 4, 5]}
Our enhancements include:

• Limiting the cameras distance from the scene
• Preventing the camera from flipping upside down
• Clamping the possible scene depth for the mouse
• Adjusting the camera's movement speed when zooming based on the scene's depth
• Giving the option to prohibit rotation below the ground plane
• Allowing the user to reset the camera to its initial position

Using our navigation

Using our World-In-Hand Navigation in your Three.js project is straightforward and requires only a few extra steps compared to the standard OrbitControls. In general, we require you to use a WebGL2Renderer and render your scene into our custom WorldInHandControls.navigationRenderTarget to calculate the depth values for the mouse position:

// Last parameter is the amount of MSAA samples to use for the exposed render target
const controls = new WorldInHandControls(perspectiveCamera, renderer.domElement, renderer, scene, 4);

// When scene changes its size
scene.dispatchEvent({type: 'change'});

// If rendering on demand, listen to change events
controls.addEventListener('change', render);

// If the renderer.domElement was resized
scene.dispatchEvent({type: 'resize'});

// If manually changing camera, call this afterwards
controls.reloadCamera();

function render() {
    // Render into the exposed render target
    renderer.setRenderTarget(controls.navigationRenderTarget);
    renderer.render(scene, perspectiveCamera);
    
    // Tell the controls a render has taken place and
    // by default copy the render target to the canvas
    controls.update();
}
        

        
const controls = new OrbitControls(camera, renderer.domElement);




// If rendering on demand, listen to change events
controls.addEventListener('change', render);




// If manually changing camera, call this afterwards
controls.update();

function render() {
    // Render into the exposed render target
    renderer.setRenderTarget(null);
    renderer.render(scene, perspectiveCamera);
    
    


}
        

We have published a package on npm that you can install and use in your project. As an alternative, you can also download the source code from our GitHub repository. You can find further information on how to use our navigation under both links.

Summary & Future Work

In summary, our World-In-Hand navigation approach offers a more intuitive and immersive way to navigate virtual environments in Three.js. By allowing users to interact with the scene as if it were a tangible object in their hands, we aim to enhance the sense of presence and control, making navigation more intuitive, robust, and user-friendly. Our implementation includes core navigation features such as panning, rotating, and zooming, as well as several enhancements to improve the user experience. We have also developed a World-In-Hand-Controls Visualiser class to help developers better understand and adapt the navigation to their needs.

In the future, our approach could be further refined and expanded by adding new features and capabilities, especially regarding the resilience. Some potential areas for improvement include:

• Implementing advanced constraints for camera movement when rotating
• Detecting and avoiding collisions between scene and camera

Attribution

The building models in the demo scene are based on "Low Poly Buildings" by Irfn_93 licensed under CC-BY-4.0

References

1. Jankowski J, Hachet M. Advances in Interaction with 3D Environments. Computer Graphics Forum. 2014;34(1):152-190. doi:https://doi.org/10.1111/cgf.12466
2. Wagner F. Concept and Implementation of Navigation Techniques for Softwaremaps with GPU-based Geometry Creation. Potsdam: University of Potsdam; 2016
3. Buchholz H, Bohnet J, J. Doellner. Smart and Physically-Based Navigation in 3D Geovirtual Environments. Ninth International Conference on Information Visualisation (IV'05). Published online October 11, 2006. doi:https://doi.org/10.1109/iv.2005.117
4. Hildebrandt D, Timm R. An assisting, constrained 3D navigation technique for multiscale virtual 3D city models. GeoInformatica. 2013;18(3):537-567. doi:https://doi.org/10.1007/s10707-013-0189-8
5. Jul S. “This Is a Lot Easier!”: Constrained Movement Speeds Navigation. Accessed March 16, 2024. https://public.websites.umich.edu/~sjul/papers/chi03_leylines.pdf