Melissa: Large Scale In Transit Global Sensitivity
Analysis Avoiding Intermediate Files
Event Type
Paper
Data Analytics
Data management
Storage
TimeThursday, November 16th4:30pm -
5pm
Location402-403-404
DescriptionGlobal sensitivity analysis is an important step for
analyzing and validating numerical simulations. One
classical approach consists in computing statistics from
the outputs of multiple simulation runs. Results are
stored to disk and statistics are computed postmortem.
Scientists are constrained to run low resolution
simulations with a limited number of probes to keep the
amount of intermediate storage manageable. In this paper
we propose a file avoiding, fault tolerant, and elastic
framework that enables high resolution global
sensitivity analysis at large scale. Our approach
combines iterative statistics and in transit processing
to compute Sobol' indices without any intermediate
storage. Statistics are updated on-the-fly as soon as
the in-transit parallel server receives results from one
of the running simulations. For one experiment, we
computed the Sobol' indices on 10M hexahedra and 100
timesteps, running 8000 parallel simulations executed in
1h27 on up to 28672 cores, avoiding 48TB of file
storage.
Download PDF:
here
