Portal Addressing

One-sided data movement models (e.g., shmem[4], ST[12], MPI-2[8]) typically use a triple to address memory on a remote node. This triple consists of a process id, memory buffer id, and offset. The process id identifies the target process, the memory buffer id specifies the region of memory to be used for the operation, and the offset specifies an offset within the memory buffer.

In addition to the standard address components (process id, memory buffer id, and offset), a Portal address includes a set of match bits. This addressing model is appropriate for supporting one-sided operations as well as traditional two-sided message passing operations. Specifically, the Portals API provides the flexibility needed for an efficient implementation of MPI-1, which defines two-sided operations with one-sided completion semantics.

Figure 2.4 presents a graphical representation of the structures used by a target in the interpretation of a Portal address. The process id is used to route the message to the appropriate node and is not reflected in this diagram. The memory buffer id, called the portal id, is used as an index into the Portal table. Each element of the Portal table identifies a match list. Each element of the match list specifies two bit patterns: a set of "don't care" bits, and a set of "must match" bits. In addition to the two sets of match bits, each match list element has at most one memory descriptor. Each memory descriptor identifies a memory region and an optional event queue. The memory region specifies the memory to be used in the operation and the event queue is used to record information about these operations.

Figure 2.4: Portal Addressing Structures

Figure 2.5 illustrates the steps involved in translating a Portal address, starting from the first element in a match list. If the match criteria specified in the match list entry are met and the memory descriptor list accepts the operation¹, the operation (put, get, or swap) is performed using the memory region specified in the memory descriptor. If the memory descriptor specifies that it is to be unlinked when a threshold has been exceeded, the match list entry is removed from the match list and the resources associated with the memory descriptor and match list entry are reclaimed. Finally, if there is an event queue specified in the memory descriptor and the memory descriptor accepts the event, the operation is logged in the event queue.

Figure 2.5: Portal Address Translation

If the match criteria specified in the match list entry are not met, or there is no memory descriptor associated with the match list entry, or the memory descriptor associated with the match list entry rejects the operation, the address translation continues with the next match list entry. If the end of the match list has been reached, the address translation is aborted and the incoming requested is discarded.

¹ Memory descriptors can reject operations because a threshold has been exceeded or because the memory region does not have sufficient space, see Section 3.10.