O'Reilly logo

High Performance Visualization by E. Wes Bethel, Charles Hansen, Hank Childs

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Chapter 9
In Situ Processing
Hank Childs
Lawrence Berkeley National Laboratory
Kwan-Liu Ma, Hongfeng Yu
University of California, Davis and Sandia National Laboratories
Brad Whitlock, Jeremy Meredith, Jean Favre
Lawrence Livermore National Laboratory, Oak Ridge National Laboratory, and
Swiss Center for Scientific Computing
Scott Klasky, Norbert Podhorszki
Oak Ridge National Laboratory
Karsten Schwan, Matthew Wolf
Georgia Institute of Technology, Atlanta
Manish Parashar
Rutgers University
Fan Zhang
Rutgers University
9.1 Introduction ...................................................... 172
9.2 Tailored Co-Processing at High Concurrency ................... 174
9.3 Co-Processing With General Visualization Tools Via Adaptors 175
9.3.1 Adaptor Design .......................................... 178
9.3.2 High Level Implementation Issues ....................... 178
9.3.3 In Practice ............................................... 179
9.3.4 Co-Processing Performance .............................. 180
9.4 Concurrent Processing ........................................... 183
9.4.1 Service Oriented Architecture for Data Management in
HPC ..................................................... 183
9.4.2 The ADaptable I/O System, ADIOS .................... 184
9.4.3 Data Staging for In Situ Processing .................... 185
9.4.4 Exploratory Visualization with VisIt and Paraview
Using ADIOS ............................................ 186
171
172 High Performance Visualization
9.5 In Situ Analytics Using Hybrid Staging ......................... 187
9.6 Data Exploration and In Situ Processing ....................... 189
9.6.1 In Situ Visualization by Proxy .......................... 190
9.6.2 In Situ Data Triage ..................................... 191
9.7 Conclusion ........................................................ 193
References .......................................................... 194
Traditionally, visualization is done via post-processing: a simulation produces
data, writes that data to disk, and then, later, a separate visualization program
reads the data from disk and operates on it. In situ processing refers to a
different approach: the data is processed while it is being produced by the
simulation, allowing visualization to occur without involving disk storage. As
recent supercomputing trends have simulations producing data at a much
faster rate than I/O bandwidth, in situ processing will likely play a bigger
and bigger role in visualizing data sets on the world’s largest machines.
The push towards commonplace in situ processing has a benefit besides
saving on I/O costs. Already, scientists must limit how much data they store
for later processing, potentially limiting their discoveries and the value of
their simulations. In situ processing, however, enables the processing of this
unexplored data.
This chapter describes the different approaches for in situ processing and
discusses their benefits and limitations.
9.1 Introduction
The terms used for in situ processing have not been used consistently
throughout the community. In this book, in situ processing refers to a
spectrum of processing techniques. The commonality between these tech-
niques is that they enable visualization and analysis techniques without the
significant—and increasing expensive—cost of I/O. On one end of the in situ
spectrum, referred to in this book as co-processing, visualization routines are
part of the simulation code, with direct access to the simulation’s memory. On
the other end of the in situ spectrum, referred to in this book as “concurrent
processing,” the visualization program runs separately on distinct resources,
with data transferred from the simulation to the visualization program via the
network. Hybrid methods combine these approaches: data is processed and re-
duced using direct access to the simulation’s memory (i.e., co-processing) and
then sent to a dedicated visualization resource for further processing (i.e.,
concurrent processing). Table 9.1 summarizes these methods.
For approaches that process data via direct access to the simulation’s mem-
ory (i.e., co-processing and hybrid approaches), another important considera-
tion is whether to use custom, tailored code, written for a specific simulation,
or whether to leverage existing, richly featured visualization software. Tai-
In Situ Processing 173
Technique Co-processing Concurrent Hybrid
Processing
Tightly coupled Loosely coupled
Aliases Synchronous Asynchronous None
Staging
Vis runs on Data is reduced
Vis routines have dedicated, via co-
Description direct access concurrent processing
to memory of resources and and sent to
simulation code access data a concurrent
via network resource.
Memory
constraints. Data movement Complex. Also
Negative Large impact costs. Requires shares negatives
Aspects on simulation separate of other
(crashes, resources. approaches.
performance).
TABLE 9.1: Summary of in situ processing techniques
lored code is most likely to be well-optimized for performance and memory
requirements, for example by using a simulation’s data structures without
copying them into another form. However, tailored code is often not portable
or re-usable, making it tied to a specific simulation code. Utilizing existing,
richly featured visualization software is more likely to work well with many
simulations. But, typically, the price of this generality may be increased due
to the usage of resources, such as memory, network bandwidth, or CPU time.
This chapter discusses case studies for three in situ processing scenarios:
a tailored, co-processing approach at high levels of concurrency, de-
scribed in 9.2;
a co-processing approach via an adaptor layer to a richly featured visu-
alization tool, described in 9.3; and
a concurrent approach in the context of the ADIOS system, described
in 9.4.
Hybrid processing is an emerging idea, with little published work; its concepts
and recent results are discussed in 9.5. Finally, 9.6 discusses how to use in
situ processing when the visualizations and analyses to be performed are not
known a priori.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required