How do I tune large message behavior in Open MPI the v1.2 series?
Note that this answer generally pertains to the Open MPI v1.2 series. Later versions slightly changed how large messages are handled. Open MPI uses a few different protocols for large messages. Much detail is provided in this paper. The btl_openib_flags MCA parameter is a set of bit flags that influences which protocol is used; they generally indicate what kind of transfers are allowed to send the bulk of long messages. Specifically, these flags do not regulate the behavior of “match” headers or other intermediate fragments. The following flags are available: • Use send/receive semantics (1): Allow the use of send/receive semantics. • Use PUT semantics (2): Allow the sender to use RDMA writes. • Use GET semantics (4): Allow the receiver to use RDMA reads. Open MPI defaults to setting both the PUT and GET flags (value 6). Open MPI uses the following long message protocols: • RDMA Direct: If RDMA writes or reads are allowed by btl_openib_flags and the sender’s message is already register
