MPI Protocols

2 min read HPC

The MPI has some protocol jargon that I have not heard so far. They are the following:

  • Aggressive eager
  • Rendezvous with send push
  • Rendezvous with receiver pull
  • Rendezvous blocking (push or pull)

They are defined in the slides from MPI group on MPI performance. I got some good info on this from these slides and this, which I will reproduce here with my own additions and notes.

Some terminology

  • Message consists of envelope and data
  • Envelope contains tag, communicator, length, source information, plus impl. private data

Eager protocol

Here messages are sent assuming that the destination can store the incoming data. So the send processor is quite eager to send away all its data over the network o the receiver and it is the receivers headache to deal with the network contention. Say rank 0 eagerly send to rank 1 its data then no matching receive may exist in the receiver side i.e., rank 1 then the system has to buffer and copy the data.

Eager protocol salient features

  • Reduction in synchronization delays
  • Simplifies programming (just MPI_Send)
  • Requires significant buffering
  • May require active involvement of CPU to drain network at receiver’s end
  • May introduce additional copy (buffer to final destination)
  • Minimizes latency

Scalability of eager protocol

  • Buffering must be reserved for arbitrary senders
  • User-model mismatch (often expect buffering allocated entirely to used connections)
  • Common approach in implementations is to provide same buffering for all members of MPI_COMM_WORLD; this is optimizing for non-scalable computations
  • Scalable implementations that exploit message patterns are possible (but not widely implemented)

Rendezvous protocols

Here the sender only starts the sending when the receiver says it is ready (signals). Here the envelope of the message is delivered first and then the data is delivered when the receive buffer is available for receive. So only buffering of envelopes becomes necessary.

Rendezvous protocols salient features

  • Robust and safe (except for limit on the number of envelopes)
  • May remove copy (user to user direct)
  • More complex programming (waits/tests)
  • May introduce synchronization delays (waiting for receiver to ok send)
  • Three-message handshake introduces latency

Choosing MPI Send Modes

There is no perfect choice for a MPI send mode; however some well good performance recommendations scenarios do exist:

  • Eager is faster than rendezvous until
    • Data is unexpected: 2*latency is smaller than the time to copy from buffer
  • Ready can force Eager, but requires pre-posting of receives
    • Best when data is long but not too long (measured in terms of s/r)
  • Synchronous good
    • when MPI implementation has inadequate flow control and messages are large
MPI protocols