Nonblocking file I/O is silently lost when the aio queue fills (macOS), and one refusal disables it for the process

[eschnett]

Note: I created this issue with AI. The problem is real (originally a failure in the MPICH Fortran test suite, a reproducer is attached below.) The explanation looks plausible.

Background information

What version of Open MPI are you using?

main (2167bed), and reproduced on the MPI-standard-ABI branch of #13280
(6cb5ef1). Every file involved is byte-identical between the two.

Describe how Open MPI was installed

From a git clone, ./autogen.pl && ./configure && make install.

If you are building/installing from a git clone, please copy-n-paste the output from git submodule status

 60c0dc3ae0e964003d79777c90fd6f116e2cd1d7 3rd-party/openpmix (v5.0.10rc1-501-g60c0dc3a)
 a2d2b039a4f0de9af2b36be90283591c82cdc794 3rd-party/prrte (psrvr-v2.0.0rc1-5205-ga2d2b039a4)
 195cf1f5f532319fc9ce828ceb0857ee3a85ace7 3rd-party/pympistandard (heads/develop)
 3064f7bd191b49a5a5554170ef7be4762246b5ee config/oac (3064f7b)

Please describe the system on which you are running

• Operating system/version: macOS 26.5 (Darwin 25.5.0)
• Computer hardware: arm64, 12 cores
• Network type: not relevant; single host, file I/O only

Relevant limits on this machine:

kern.aioprocmax = 16     # what one process may have outstanding
kern.aiomax     = 90     # what the whole machine may have outstanding
_SC_AIO_MAX     = 90     # what sysconf reports

Details of the problem

A nonblocking write through a fragmented file view loses most of its data,
reports success, and leaves the process unable to do nonblocking file I/O at all
for the rest of its life.

$ mpiexec -n 4 ./ompi-aio-eagain
mca_fbtl_posix_ipwritev: error in aio_write():  Resource temporarily unavailable
mca_fbtl_posix_ipwritev: error in aio_write():  Resource temporarily unavailable
mca_fbtl_posix_ipwritev: error in aio_write():  Resource temporarily unavailable
mca_fbtl_posix_ipwritev: error in aio_write():  Resource temporarily unavailable
mca_fbtl_posix_ipreadv: error in aio_read(): errno 35 Resource temporarily unavailable
mca_fbtl_posix_ipreadv: error in aio_read(): errno 35 Resource temporarily unavailable
mca_fbtl_posix_ipreadv: error in aio_read(): errno 35 Resource temporarily unavailable
mca_fbtl_posix_ipreadv: error in aio_read(): errno 35 Resource temporarily unavailable
rank 0: 8192 of 8192 integers wrong -- the nonblocking collective write was lost
rank 1: 8192 of 8192 integers wrong -- the nonblocking collective write was lost
rank 2: 8192 of 8192 integers wrong -- the nonblocking collective write was lost
rank 3: 8192 of 8192 integers wrong -- the nonblocking collective write was lost

MPI_File_iwrite_all returns MPI_SUCCESS, MPI_Test reports the request
complete on the first call, and MPI_Wait returns MPI_SUCCESS. The only thing
in the interface that admits anything happened is MPI_Get_count on the
completed request's status, which reports 0 where a success reports every
element.

Two reproducers are attached below. The first is the four-process round trip
above. The second runs on one process, counts what survives, and separates the
three effects; mpiexec -n 1:

  (a) contiguous, control      MPI_Get_count =    64 of    64
  (b) 512 blocks of 16         MPI_Get_count =     0 of  8192   <-- lost
  (c) contiguous, after (b)    MPI_Get_count =     0 of    64   <-- lost
  (d) contiguous, once more    MPI_Get_count =     0 of    64   <-- lost
  file is 1984 bytes, a complete write leaves 65472 (16 blocks of 64 bytes landed)

(c) and (d) are the part I would flag first: they are single-request contiguous
writes, identical to the control that had just succeeded, and they now fail too.
One EAGAIN has disabled nonblocking file I/O for the process.

Diagnosis

Four things, of which the first alone accounts for the macOS symptom and the
others would still lose data anywhere the queue fills.

1. The batch is sized from the wrong limit.
mca_fbtl_posix_module_init sets ompi_fbtl_posix_max_prd_active_reqs -- the
number of aio_writes kept in flight at once -- from sysconf(_SC_AIO_MAX). On
macOS that reports kern.aiomax, the limit across every process on the machine;
one process is held to kern.aioprocmax. 90 against 16 here, so the seventeenth
aio_write of the first batch is refused. Measured by declaring the variable
extern and printing it either side of the MPI_File_open that runs
module_init: 2048, the compiled-in default, then 90.

There is no MCA parameter for it, so an environment where the system's answer is
wrong has no recourse.

2. The retry cannot retry. fbtl_posix_ipwritev.c makes ten attempts per
request with mca_common_ompio_progress() in between -- the fix for #8368. A
slot is released by aio_return, not by the operation completing, and the only
caller of aio_return is mca_fbtl_posix_progress, which
mca_common_ompio_progress reaches through req->req_progress_fn -- assigned
after the loop. So nothing in the process can reap this request's own
operations while the loop spins, and the ten attempts are ten copies of one
failure. posix-aio-limit.c below measures the aio_return behaviour with no
MPI in the way: sixteen requests that have all completed still refuse a
seventeenth, and accept it the moment they are reaped.

3. The failure is discarded, and then read as success.
common_ompio_file_write.c:456 and common_ompio_file_read.c:512 call the fbtl
as a statement. A request whose fbtl call failed carries no progress function,
and common_ompio_request.c:209 reads that as "this is a parent request", finds
req_num_subreqs == req_subreqs_completed trivially at 0 == 0, and completes it
with a status nobody has written -- MPI_ERROR is uninitialised heap. That is
where MPI_Wait's MPI_SUCCESS comes from.

This is reached through MPI_File_iwrite_all because no fcoll component
implements fcoll_file_iwrite_all, so mca_common_ompio_file_iwrite_all takes
its own "WE fake it with individual non-blocking I/O operations" branch. Nothing
aggregates, so the io array is as fragmented as the file view -- 512 entries for
the probe above.

4. The posted operations are leaked. The error path frees
data->prd_aio.aio_reqs while sixteen aio_writes are outstanding against those
control blocks -- POSIX requires an aiocb to stay valid until its operation
completes -- and calls aio_return on none of them. Every slot the process has
is retired permanently, which is (c) and (d) above, and is why the round trip
reads back nothing rather than the sixteen blocks the write did leave. A
read-only run in a fresh process against a good file recovers 256 of 8192
integers: 16 blocks, the same number again.

Scope

The blocking path is unaffected, measured: MPI_File_write_all through the same
view transfers all 8192 and says so. fbtl_posix_pwritev.c contains no aio_*
call at all, using pwritev and data sieving.

I could not find a run-time workaround. --mca fbtl_posix_priority 0,
--mca fcoll individual and --mca fcoll_vulcan_async_io 0 all still fail
(fcoll is not in this path, per the backtrace); --mca fbtl ^posix leaves no
fbtl and the job dies. There is one io component and one fbtl component to
choose from. Raising kern.aioprocmax past 90 ought to work but needs root, and
kern.aiomax would have to rise with it.

This came up as MPICH's test suite test/mpi/f08/io/i_fcoll_test failing against
Open MPI on macOS; the MPI_Type_create_darray view in that test is what makes
the io array long enough to overrun the queue.

Relation to #8368

#8368, "Exceeding the max. number of pending aio requests on MacOS", is the same
family and is closed. Its fix is the retry loop in (2), which cannot reap and so
cannot help.

posix-aio-limit.c -- the two aio facts, no MPI

// What macOS actually allows one process to have outstanding in aio, and what
// it takes to give a slot back. Two facts, neither of them Open MPI's, but both
// needed to read `ompi-aio-eagain.c`'s failure -- see MISSING.md, "a nonblocking
// collective write is lost when the aio queue fills".
//
// 1. The ceiling is `kern.aioprocmax`, per process, 16 on macOS 26. It is *not*
//    `kern.aiomax` (90 here), which is the system-wide total and is what
//    `sysconf(_SC_AIO_MAX)` reports -- and `sysconf(_SC_AIO_MAX)` is what
//    ompi/mca/fbtl/posix/fbtl_posix.c's `mca_fbtl_posix_module_init` sizes its
//    batch of concurrent `aio_write`s from.
//
// 2. A request that has *completed* still holds its slot until `aio_return` is
//    called on it. So abandoning an aiocb -- as the fbtl's error path does, by
//    free()ing the array of them -- does not merely lose that operation, it
//    retires the slot for the life of the process.
//
// No MPI. Compile and run it as an ordinary program:
//
//   cc -o posix-aio-limit posix-aio-limit.c && ./posix-aio-limit

#include <aio.h>
#include <errno.h>
#include <fcntl.h>
#include <stdio.h>
#include <string.h>
#include <sys/sysctl.h>
#include <unistd.h>

enum { N = 64, LEN = 64 };  // more requests than any plausible limit

static void report_sysctl(const char *name) {
  int val = 0;
  size_t len = sizeof val;
  if (0 == sysctlbyname(name, &val, &len, NULL, 0)) {
    printf("    %-18s = %d\n", name, val);
  }
}

int main(void) {
  printf("  the two limits, and what sysconf reports:\n");
  report_sysctl("kern.aioprocmax");
  report_sysctl("kern.aiomax");
  printf("    %-18s = %ld\n", "_SC_AIO_MAX", sysconf(_SC_AIO_MAX));

  const char *path = "posix-aio-limit.dat";
  int fd = open(path, O_CREAT | O_RDWR | O_TRUNC, 0644);
  if (-1 == fd) {
    perror("open");
    return 1;
  }

  char buf[LEN];
  memset(buf, 'x', sizeof buf);

  struct aiocb cb[N];
  memset(cb, 0, sizeof cb);

  int queued = 0;
  for (int i = 0; i < N; ++i) {
    cb[i].aio_fildes = fd;
    cb[i].aio_buf = buf;
    cb[i].aio_nbytes = LEN;
    cb[i].aio_offset = (off_t)i * LEN;
    cb[i].aio_sigevent.sigev_notify = SIGEV_NONE;
    if (-1 == aio_write(&cb[i])) {
      printf("  aio_write #%d refused: %s (errno %d)\n", i + 1, strerror(errno),
             errno);
      break;
    }
    ++queued;
  }
  printf("  (1) outstanding aio_write()s accepted: %d\n", queued);

  // Wait for every one of them to finish, but reap none.
  for (int i = 0; i < queued; ++i) {
    while (EINPROGRESS == aio_error(&cb[i])) {
      usleep(1000);
    }
  }
  printf("      all %d now report aio_error() == 0, i.e. complete\n", queued);

  struct aiocb extra;
  memset(&extra, 0, sizeof extra);
  extra.aio_fildes = fd;
  extra.aio_buf = buf;
  extra.aio_nbytes = LEN;
  extra.aio_offset = 1 << 20;
  extra.aio_sigevent.sigev_notify = SIGEV_NONE;

  int held = aio_write(&extra);
  printf("  (2) one more, with those %d complete but not aio_return()ed: %s\n",
         queued, -1 == held ? strerror(errno) : "accepted");
  if (0 == held) {
    while (EINPROGRESS == aio_error(&extra)) {
      usleep(1000);
    }
    aio_return(&extra);
  }

  for (int i = 0; i < queued; ++i) {
    aio_return(&cb[i]);
  }
  int freed = aio_write(&extra);
  printf("  (3) the same call once they have been aio_return()ed: %s\n",
         -1 == freed ? strerror(errno) : "accepted");
  if (0 == freed) {
    while (EINPROGRESS == aio_error(&extra)) {
      usleep(1000);
    }
    aio_return(&extra);
  }

  close(fd);
  unlink(path);

  // (2) failing and (3) succeeding is the point; anything else and the
  // reasoning in MISSING.md needs revisiting.
  return (-1 == held && 0 == freed) ? 0 : 1;
}

ompi-aio-eagain-probe.c -- one process, counts what survives

// The same defect as `ompi-aio-eagain.c`, on one process and instrumented, so
// that what survives can be counted rather than just called wrong. Four
// measurements, in order, all through MPI_File_iwrite_all on a file view of 512
// blocks of 16 integers at stride 32 -- the fragmentation is the whole point,
// and a vector type makes it in one line where the test's darray needs four
// processes:
//
//   (a) a contiguous nonblocking collective write, as a control: it succeeds and
//       MPI_Get_count reports every element.
//   (b) the fragmented one: Open MPI prints
//           mca_fbtl_posix_ipwritev: error in aio_write(): Resource temporarily
//           unavailable
//       MPI_File_iwrite_all returns MPI_SUCCESS, MPI_Wait returns MPI_SUCCESS,
//       and MPI_Get_count reports 0 -- the count is the only thing in the
//       interface that admits anything happened. Exactly `kern.aioprocmax`
//       blocks reach the file, 16 here, out of 512; the file is left short.
//   (c) and (d) contiguous again, single-request, identical to (a): they now
//       fail too. The failing call in (b) free()s the aiocbs of the 16 requests
//       it did queue without ever calling aio_return on them, so those 16 slots
//       -- every slot the process has -- are gone for good. One EAGAIN
//       permanently disables nonblocking file I/O for the process, which is why
//       `f08/io/i_fcoll_test` reads back nothing at all rather than the 16
//       blocks the write left behind.
//
// `posix-aio-limit.c` measures the 16 and the aio_return behaviour with no MPI
// in the way. MISSING.md, "a nonblocking collective write is lost when the aio
// queue fills", has the reading of Open MPI's source that ties them together.
//
// Prints what it finds and exits 1 if any of the four loses data, so that it
// doubles as the regression test for a fix: as `ompi-aio-eagain.c` beside it
// does, nonzero means the defect is present.
//
//   mpicc -o ompi-aio-eagain-probe ompi-aio-eagain-probe.c
//   mpiexec -n 1 ./ompi-aio-eagain-probe

#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/stat.h>

enum { NBLK = 512, BLK = 16 };  // 512 blocks of 16 ints, stride 2*BLK

static const char *path = "ompi-aio-eagain-probe.dat";

// One nonblocking collective write of `count` integers, reporting the count the
// completed request claims to have transferred.
static int write_all(MPI_File fh, const int *buf, int count, const char *label) {
  MPI_Request req;
  MPI_Status status;
  int transferred = -1;

  MPI_File_iwrite_all(fh, buf, count, MPI_INT, &req);
  MPI_Wait(&req, &status);
  MPI_Get_count(&status, MPI_INT, &transferred);

  printf("  %-28s MPI_Get_count = %5d of %5d%s\n", label, transferred, count,
         transferred == count ? "" : "   <-- lost");
  return transferred == count;
}

int main(int argc, char **argv) {
  MPI_Init(&argc, &argv);

  const int count = NBLK * BLK;
  int *buf = malloc((size_t)count * sizeof *buf);
  for (int i = 0; i < count; ++i) {
    buf[i] = i + 1;
  }

  MPI_Datatype fragmented;
  MPI_Type_vector(NBLK, BLK, 2 * BLK, MPI_INT, &fragmented);
  MPI_Type_commit(&fragmented);

  MPI_File fh;
  MPI_File_delete(path, MPI_INFO_NULL);
  MPI_File_open(MPI_COMM_WORLD, path, MPI_MODE_CREATE | MPI_MODE_RDWR,
                MPI_INFO_NULL, &fh);

  MPI_File_set_view(fh, 0, MPI_INT, MPI_INT, "native", MPI_INFO_NULL);
  int control = write_all(fh, buf, 64, "(a) contiguous, control");

  MPI_File_set_view(fh, 0, MPI_INT, fragmented, "native", MPI_INFO_NULL);
  int fault = write_all(fh, buf, count, "(b) 512 blocks of 16");

  MPI_File_set_view(fh, 0, MPI_INT, MPI_INT, "native", MPI_INFO_NULL);
  int after = write_all(fh, buf, 64, "(c) contiguous, after (b)");
  int again = write_all(fh, buf, 64, "(d) contiguous, once more");

  MPI_File_close(&fh);

  // The fragmented view spans NBLK*2*BLK integers, of which the write covers
  // the first of every pair of blocks; a complete write leaves a file ending
  // with the last block, at (NBLK-1)*2*BLK + BLK integers.
  struct stat st;
  if (0 == stat(path, &st)) {
    const long expected = (long)((NBLK - 1) * 2 * BLK + BLK) * (long)sizeof(int);
    printf("  file is %lld bytes, a complete write leaves %ld"
           " (%lld blocks of %d bytes landed)\n",
           (long long)st.st_size, expected,
           (long long)(st.st_size ? (st.st_size - BLK * sizeof(int)) /
                                            (2 * BLK * sizeof(int)) +
                                        1
                                  : 0),
           (int)(BLK * sizeof(int)));
  }

  MPI_Type_free(&fragmented);
  free(buf);

  if (!control) {
    printf("  inconclusive: the contiguous control write lost data too\n");
  } else if (fault) {
    printf("  no data lost -- the defect is not present here\n");
  } else if (after && again) {
    printf("  the fragmented write was lost, but the handle still works:"
           " the aio slots were not leaked\n");
  } else {
    printf("  the fragmented write was lost and took the file handle with it\n");
  }

  MPI_File_delete(path, MPI_INFO_NULL);
  MPI_Finalize();
  return (control && fault && after && again) ? 0 : 1;
}

ompi-aio-eagain.c -- four processes, the original round trip

// MPICH's `f08/io/i_fcoll_test` fails under Open MPI on macOS with a flood of
//
//     mca_fbtl_posix_ipwritev: error in aio_write():  Resource temporarily unavailable
//     mca_fbtl_posix_ipreadv: error in aio_read(): errno 35 Resource temporarily unavailable
//
// after which the data read back is zero: the nonblocking collective write never
// reached the file, and MPI_Wait reported success anyway.
//
// This is the same thing in C, with no Fortran: the test's own access pattern,
// which is what matters -- a 32x32x32 array of integers distributed over the
// processes with MPI_Type_create_darray, so that each process's share of the file
// is many small pieces rather than one run. A contiguous view does not reproduce
// it; the noncontiguous one issues far more asynchronous requests than macOS
// allows outstanding, `sysctl kern.aioprocmax` being 16 here.
// ompi/mca/fbtl/posix/fbtl_posix_ipwritev.c reports the EAGAIN and gives up,
// rather than retrying or falling back to a synchronous write.
//
// Prints what it finds and exits 1 if the data does not survive the round trip.
// Run on 4 processes, as the test does.
//
//   mpicc -o ompi-aio-eagain ompi-aio-eagain.c && mpiexec -n 4 ./ompi-aio-eagain

#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>

enum { SIDE = 32 };  // a 32^3 array of integers, as in the test

int main(int argc, char **argv) {
  MPI_Init(&argc, &argv);

  int rank, nprocs;
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
  MPI_Comm_size(MPI_COMM_WORLD, &nprocs);

  const int gsizes[3] = {SIDE, SIDE, SIDE};
  const int distribs[3] = {MPI_DISTRIBUTE_BLOCK, MPI_DISTRIBUTE_BLOCK,
                           MPI_DISTRIBUTE_BLOCK};
  const int dargs[3] = {MPI_DISTRIBUTE_DFLT_DARG, MPI_DISTRIBUTE_DFLT_DARG,
                        MPI_DISTRIBUTE_DFLT_DARG};
  int psizes[3] = {0, 0, 0};
  MPI_Dims_create(nprocs, 3, psizes);

  MPI_Datatype filetype;
  MPI_Type_create_darray(nprocs, rank, 3, gsizes, distribs, dargs, psizes,
                         MPI_ORDER_FORTRAN, MPI_INT, &filetype);
  MPI_Type_commit(&filetype);

  int intsize, typesize;
  MPI_Type_size(MPI_INT, &intsize);
  MPI_Type_size(filetype, &typesize);
  const int count = typesize / intsize;  // integers this process holds

  int *out = malloc((size_t)count * sizeof *out);
  int *in = malloc((size_t)count * sizeof *in);
  for (int i = 0; i < count; ++i) {
    out[i] = rank * count + i + 1;
    in[i] = 0;
  }

  const char *path = "ompi-aio-eagain.dat";
  MPI_File fh;
  MPI_Request req;

  MPI_File_open(MPI_COMM_WORLD, path, MPI_MODE_CREATE | MPI_MODE_RDWR,
                MPI_INFO_NULL, &fh);
  MPI_File_set_view(fh, 0, MPI_INT, filetype, "native", MPI_INFO_NULL);
  MPI_File_iwrite_all(fh, out, count, MPI_INT, &req);
  MPI_Wait(&req, MPI_STATUS_IGNORE);
  MPI_File_close(&fh);

  MPI_File_open(MPI_COMM_WORLD, path, MPI_MODE_RDONLY, MPI_INFO_NULL, &fh);
  MPI_File_set_view(fh, 0, MPI_INT, filetype, "native", MPI_INFO_NULL);
  MPI_File_iread_all(fh, in, count, MPI_INT, &req);
  MPI_Wait(&req, MPI_STATUS_IGNORE);
  MPI_File_close(&fh);

  int errs = 0;
  for (int i = 0; i < count; ++i)
    if (in[i] != out[i])
      ++errs;
  printf("rank %d: %d of %d integers wrong%s\n", rank, errs, count,
         errs ? " -- the nonblocking collective write was lost" : "");

  int total = 0;
  MPI_Reduce(&errs, &total, 1, MPI_INT, MPI_SUM, 0, MPI_COMM_WORLD);
  if (rank == 0) {
    if (total == 0)
      printf("No Errors\n");
    MPI_File_delete(path, MPI_INFO_NULL);
  }

  MPI_Type_free(&filetype);
  free(out);
  free(in);
  MPI_Finalize();
  return total != 0;
}

[jsquyres]

@edgargabriel Could you have a look?

[edgargabriel]

@eschnett thank you for the detailed report, that helped in devising the soluition a lot. Would you mind testing the fix in PR #14303 ? I think it should fix the issues that you reported. I can only test on Linux, I literally do not have access to any Mac anymore (neither work related nor private).

[edgargabriel]

@eschnett thank you for the detailed report, that helped in devising the soluition a lot. Would you mind testing the fix in PR #14303 ? I think it should fix the issues that you reported. I can only test on Linux, I literally do not have access to any Mac anymore (neither work related nor private).

Closing the PR #14303 for now, we might go with PR #14279 with some minor changes instead