Real time, cross platform visualizations with zero dependencies for the N-body package REBOUND
This paper is under review on the experimental track of the Journal of Visualization and Interaction. See the reviews and issues for this paper.
1 Introduction
Visualizations are a crucial part of the scientific process. There are many popular tools to make two or three dimensional graphs such as gnuplot (Williams and Kelley 2013), matplotlib (Hunter 2007), or yt (Turk et al. 2011) for rendering volumetric and particle data. Many specialized tools exist as well, for example (Williams et al. 2022) describe a novel approach for an interactive user interface for the WorldWide Telescope. The above examples demonstrate the diversity of visualization needs and the tools that have been developed to serve those needs. However, it would be impossible to review all the software that is available.
Developing software that involves any sort of graphical interface can be a challenge. This is especially the case if the goal is to provide a tool that works on different operating systems and supports a variety of graphics hardware. There are two ways to approach the problem of cross-platform graphics that are commonly employed:
- One can write tailor graphic routines for each platform. This requires significant resources, both for development and maintenance. Whereas this approach might be an option for large projects (think of a game developed by a large studio), it is typically not within the realm of possibilities for small scientific software packages. If resources are finite, one might end up with limited support of only a few platforms, or with some platforms enjoying more features than others. Mobile applications are an example where this approach is often used and iOS and Android versions of the same application do not necessarily share the same features.
- Alternatively, one can rely on a cross-platform graphics or user interface (UI) library such as QT, Unity, OpenGL, or Vulkan. These make cross-platform development much more straightforward but at the cost of adding a dependency. Scientific software, in particular if it involves simulating scientific processes, is mostly distributed as source code because researchers want the ability to study and modify it. Compiling source code that depends on external libraries into an executable can be a major obstacle for new developers and users alike. Furthermore, switching from one library to another is not trivial. This can become a problem when external libraries don’t get updated or become deprecated. As an example, many games and visualization tools depend on OpenGL which is now considered deprecated on MacOS. Furthermore, for commercial graphic libraries such as Unity, license agreements might change at any time. In short, the development is now highly dependent on whichever external library was chosen for the project.
This paper presents an alternative approach which we think has several important advantages over the two approaches above, especially for small scientific projects:
- Dependency free. There are no external libraries required.
- Cross platform. Any platform that has a reasonably modern browser can be used to render visualizations. This includes both desktop and mobile operating systems: Linux, MacOS, Windows, iOS, Android.
- Remote visualizations. Using port-forwarding via SSH, one can run a simulations on one computer, say a node of a computing cluster, and visualize it in real time on another workstation.
- Future-proof. Because the approach relies on open web technologies that are supported by all major browsers, it is likely that this approach will continue to work for many years to come without requiring much maintenance in the same way one can still view websites that were developed decades ago.
In the following sections, we will describe how this approach works in detail.
2 Operating modes
A key feature of our approach is its flexibility. Figure 1 show the three different operating modes that are possible. Modules with the same colour in the figure make use of the same source code. The high level of code-reuse is possible because C/C++ code can be compiled with emscripten to WebAssembly which can then be interpreted by a web browser at almost native speed. In summary, the different modes work as follows:
- In the standalone mode, the visualization is making use of OpenGL. This traditional approach has been used by both visualization tools and games for decades. It requires no browser, but in addition to a C compiler, OpenGL and GLFW libraries need to be installed. This mode provides the best performance.
- In the hybrid mode, when a REBOUND simulation is started, a web server is automatically started on a separate thread. The web server serves both visualization code and simulation data to a web browser. The visualization is then done in the browser. The server and browser do not need to run on the same machine. This mode does not require any external libraries to be installed for compilation. It provides a low entry barrier for developers and users (it just works). This mode constitutes the novel visualization concept that we describe in this paper.
- In the browser mode, not only the visualization is handled by the browser but also the simulation itself. This mode requires no external libraries nor any server. The emscripten compiler is used to bundle everything that is needed by the browser (HTML, CSS, JavaScript, WebAssembly, image data) into one single HTML file that can then be served as a static website or directly be opened locally with a web browser.
Both the standalone and the browser mode have been employed widely by other tools. The innovation of this paper is the hybrid mode which in many ways represents the best of both worlds. In the case of REBOUND, a user simply downloads the source code and compiles it using any C compiler. Because there are no dependencies on external libraries, this does not required any configure scripts, installing system-wise third party packages, setting up environment variables, or other complicated steps. The user can setup or run a simulation without even considering whether a visualization might be useful. If the user at some point decides that it might be useful, then they can simply open a web browser and point it towards the simulation’s dormant web server to start visualizing the simulation and get immediate feedback regarding the simulation.
Because we already have the visualization code running in a web browser, the additional task of running entire simulations in the web browser is straightforward. Figure 2 shows an example of what is possible by using the browser mode. After clicking on the figure, an N-body simulation of a self-gravitating disk can be seen. The simulation is running in real time in the browser. The rendering is using WebGL. The console output is also shown. The visualization is interactive: drag to rotate, shift+drag or scroll to zoom, press the space bar to pause. The figure is included in this document using an <iframe> HTML tag. A single file includes the bundled up HTML, CSS, JavaScript, WebAssembly as well as some small image assets and is less than 500 kB in size. For comparison, a simple screenshot of the simulation in PNG format would be 200 kB in size. The REBOUND documentation makes extensive use of the browser mode, offering users the ability to run all examples directly in the browser. This is a fantastic way to show case potential users the ability of a software package. Note that the exact same simulation can be compiled with a normal C compiler and run in the stand-alone mode showing the same visualization but using OpenGL instead of WebGL.
3 Web Browser
We rely on a web browser for all graphic related functionality in the hybrid and browser modes. There are different ways to render visualizations within a browser. For example, one can use HTML, CSS and JavaScript. A popular JavaScript visualization library is D3.js.
In recent years, browsers have also gained features that allow them to efficiently render high quality graphics and making use of GPU acceleration. Although the performance is not (yet) on par with native desktop implementations, it comes very close and is sufficient for most applications. We make use of these features and write GPU accelerated visualizations in the browser using WebGL, a JavaScript API that provides similar functionality to OpenGL. All major browsers support WebGL version 2.
We also make use WebAssembly, a binary instruction format that can be executed by a browser’s virtual machine at almost native speeds. Using the emscripten compiler toolchain, one can compile existing C or C++ code to WebAssembly. Emscripten also converts OpenGL to WebGL so that we don’t need to explicitly write WebGL code. With minor adjustments, many games and visualization tools that use OpenGL on a desktop can thus also be used in a browser. Similar to JavaScript libraries, programs in WebAssembly format are simply static files and don’t require the user to install anything, avoiding any of the issues that come with external libraries.
There are several advantages of focusing on WebAssembly and emscripten for the in-browser rendering code. First, we can reuse our existing C code that uses OpenGL for rendering with minimal changes. This makes it easy for existing projects to migrate to browser-based rendering. Other programming languages such as python are also beginning to be supported in browsers (Droettboom et al. 2021).
Second, not only can we reuse graphic related code, but also any other C code. In the case of REBOUND, the code running in the browser uses the same input and output routines as the N-body simulation itself. By reusing these routines, it becomes very easy to unpack visualization data from a file or data stream, thus allowing us to send back and forth data over the network - which is what we need for the hybrid mode. Specifically, we are using REBOUND’s Simulationarchive binary format (Rein and Tamayo 2017) to send data to the browser.
Third, we can even run an entire simulation in a web browser (we call this browser mode), generating data that is being visualized on-the-fly. This allows visualizations to be included in static website such as Figure 2 in this paper.
Fourth, we are not restricting ourselves to the browser ecosystem. We can also run visualizations locally, without the need for emscripten, WebGL, or a browser, by simply calling OpenGL APIs the old fashioned way. This makes development easier and allows advanced users who are comfortable installing libraries to take full advantages of the efficiencies and all the features that a native application provides.
4 Web Server
As we delegate rendering functionality to the browser, we somehow need to get the data that we wish to visualize into the browser. For small and static datasets one can simply use static files which the browser can access, or even include the data within a single HTML file (see Figure 2).
For large datasets and for tools such as REBOUND which visualize simulations in real time, it makes sense to only send the data to the browser when it is needed or when it is available. Given that the rendering takes place in a browser, we use HTTP requests and a web server for this data transfer.
There are many libraries and packages that implement a web server. Examples are the Apache HTTP Server Project, or python’s default http.server module. However, one of our initial goals was to avoid any external libraries. So instead of using an http library, we simply implement our own. This may seem like a daunting task, but the reference implementation we provide in REBOUND is only about 500 lines of C code and works natively on all major operating systems including Linux, MacOS, and Windows. This is possible because our web server doesn’t need to support security features - it is not intended for use on public networks. It just performs the task of getting data to our rendering code in the browser. Since we are implementing both the front and back end, we have tight control over what kind of requests the server needs to be able to respond to.
In addition to sending data from the simulation to the browser, the web server can also receive commands and data from the browser. For example, in REBOUND these commands allow the user to pause or quit a running simulation, or step through simulations manually. We furthermore make use of HTTP POST request to send screen captures from the browser to the server. The user may opt to save these screenshots on the server side, for example to later combine individual frames into an animation. To further facility the rendering of complex animations, we also implement a way for the server side to push the view matrix and other visualization setting to the browser and override any client side user interaction. The beginning of the simulation in Figure 2 demonstrates this: the camera automatically zooms in and rotates by 90 degrees at the beginning of the simulation before yielding control over the visualization to the user.
5 Jupyter Notebooks
Although REBOUND is written in C, it also comes with python interface, allowing users to more easily setup and run N-body simulations that do not require modifying the REBOUND source code itself. Jupyter notebooks (Kluyver et al. 2016) are a popular web-based interactive environment to edit and run python code. Given that we now have a web-based visualization tool for REBOUND, we can easily incorporate it into a Jupyter notebook. A user can start an interactive visualization of a simulation within a Jupyter notebook by simply calling the .widget() function on any simulation object. All this function does is insert an iframe in the notebook and point it to the simulation’s web server. Just as before, this workflow does not rely on any external libraries, neither on the C nor the python side. Everything is completely self-contained.
6 Conclusions
In this paper, we have presented a flexible framework to add real time visualization to a scientific simulation package. Our approach provides GPU accelerated visualizations on all major operating systems with zero dependencies on external libraries. We achieve this by relying on a web browser for rendering, and implementing a web server from scratch. Our approach is compatible with modern web-based environments such as Jupyter notebooks. It also allows interactive visualizations to be embedded in static websites such as scientific publications, documentation, and blogs.
A reference implementation of our approach is provided within the REBOUND N-body integrator. We were able to reuse the majority of the rendering code from the existing OpenGL based visualization in REBOUND by converting it to WebGL and WebAssembly with emscripten. We believe our approach is general enough that it will be applicable to a wide variety of other software packages which currently lack the possibility of real time visualizations.