RAID 6 mdadm + NFS sync/async
At work we have several GPU machines which until recently had separate home folders on a mdadm RAID 1. People could access their files over NFS if necessary, but most of the time people were making duplicate copies. In my opinion, it was a bit of a nightmare and made backups difficult.
Now we have a separate storage server and the
/home mount is shared across all machines. Much nicer. It seemed quite fast too until we someone tried to create a Python virtualenv and it took over half an hour. Creating a virtualenv should only take a few seconds. I experimented a bit and noticed that creating small files was incredibly slow.
sync was enabled on the
/home export but I didn't think that would be the problem, since this was never a problem for us when working over NFS before on RAID 1 mdadm exports. I decided to change it to
async, and what do you know? Less than 1MB/s jumped up to 90MB/s. Creating many small files is almost as fast as creating an equally cumulatively large single file.
I can only assume this has something to do with it being RAID 6, since we don't have this issue on RAID 1. Perhaps committing files on RAID 6 just takes a long time, and perhaps a hardware RAID controller wouldn't exhibit this behaviour?
I should think that this isn't going to be too much of a problem. I believe the issue with using
async is that you can't be sure the file has been saved before moving on. Could result in a weird race condition? I'd say only if you are writing weird code.