CUDA error #700 (cudaErrorIllegalAddress): illegal memory access

Many replicas for the 4k9y, 5a9u, 5da3 systems have failed with the following error message:

Assertion failed:
Condition: stat == cudaSuccess
cudaStreamSynchronize failed. CUDA error #700 (cudaErrorIllegalAddress): an
illegal memory access was encountered.``