There are two phases of I/O in the Salvo algorithm: reading the trace data during initialization, and reading the velocity model and writing the migrated image during migration. During initialization, the intensive portion of the work load is the I/O while preliminary computations are a secondary load. However during the migration these roles are reversed; the migration is the primary work load and the I/O is secondary.

I/O During Initialization

The reading of the trace data can be very expensive, especially when some data sets are terabytes in size. To reduce I/O time during the initialization phase, an I/O partition has been incorporating into the Salvo algorithm.

An I/O partition is a subset of the process's computational nodes, thus these nodes are separate from the system nodes used for I/O. This allows us to program the I/O nodes to handle the particular I/O requirements of f-x migration.

The I/O nodes of the I/O partition are used to read data from disk and distribute the traces to the compute partition. The compute partition performs preliminary work required before the migration can begin. This work includes FFTs and distribution of frequencies across the compute partition. The preliminary work is "hidden" behind the I/O read time, and thus is completed at no additional cost. Other preliminary work which could be completed during initialization (and hidden behind the I/O) are decompression of trace data and phase encoding.

To the right is shown the bandwidth on the Intel Paragon at Sandia National Laboratories using Salvo. The peak bandwidth for the I/O raid system is approximately 2 MB/s/raid. By placing one 81 MB file on each of the 48 raids, an I/O test can be performed using one I/O node per raid. The red line indicates the actual bandwidth of the freads for the I/O nodes. The green line indicates the effective bandwidth which includes any idle time.

The actual bandwidth scales very well and does not diminish up to 48 raids. However the effective bandwidth does slowly degrade as the number of I/O nodes increase. This may be explained by the fact that the compute partition is performing an nearly all-to-all communication to distribute the traces. As more traces are added to the initialization phase more and more communication is necessary. When 48 raids are used, Salvo reads in 3.89 GB in just 85 seconds.

I/O During Migration

During migration, the I/O partition is used to read in the velocity model and write out the migrated image. The I/O partition handles requests from the compute partition and acts as a "smart I/O node" by completing preliminary tasks on the I/O data (interpolate the velocity model and shuffle the image to be written to files). When a velocity plane is requested, the I/O partition will interpolate the velocity model to the migration grid. This may or may not require a read because the requested plane may be interpolated from currently read planes. However the image plane must be written out at every depth step.

In the above picture, timelines for an I/O node and a compute node are shown for four depth step cycles. As can be seen, the writing of the image can take a substantial fraction of the depth step time and not necessarily the same length of time for each depth step. For this run, 13 I/O nodes were used for the 480 compute nodes. Although there is a lot of idle time on the I/O nodes, these nodes reduced the total runtime by more than 10%. Assuming perfect parallelism, more than 48 compute nodes (10% of 480) would be needed to obtain the same reduction in runtime.

Curtis C. Ober

Last modified: October 9, 1998