Parallel Streaming for In Transit Analysis with
Heterogeneous Data Layout
Author/Presenters
Event Type
Workshop
Data Analytics
Data management
SIGHPC Workshop
Visualization
TimeSunday, November 12th2:10pm -
2:15pm
Location605
DescriptionPerforming analysis or generating visualizations
concurrently with high performance simulations can yield
great benefits compared to post-processing data. Writing
and reading large volumes of data can be reduced or
eliminated, thereby producing an I/O cost savings. One
such method for concurrent simulation and analysis is in
transit - streaming data from the resource running the
simulation to a separate resource running the analysis.
In transit analysis can be beneficial since
computational resources may not have certain resources
needed for visualization and analysis (e.g. GPUs) and to
reduce the impact of performing analysis tasks to the
run time of the simulation. When sending and receiving
data in transit, data redistribution mechanisms are
needed in order to support heterogeneous data layouts
that may be required by the simulation and analysis
applications. The work described in this paper compares
two mechanisms for on-the-fly data redistribution when
streaming data in parallel between two distributed
memory applications. Our results show that it is often
advantageous to stream data in the same layout as the
sender and redistribute data amongst processes on the
receiving end than to stream data in the final layout
needed by the receiver.
