Troubleshooting

Training crashes

File limits

RuntimeError: received 0 items of ancdata

Please check ulimit -n on your machine and if it is of value 1024 or lower, try increasing it to 2048 or 4096. See https://github.com/pytorch/pytorch/issues/973 for further info.

CPU out of memory

OSError: [Errno 12] Cannot allocate memory

Try setting workers_per_gpu to a lower value (usually 1/2 of samples_per_gpu is sufficient).