New ZFS Storage for the Computer Vision Lab
By my suggestion, we recently purchased a new file server in the Computer Vision Lab. This has become necessary since we are moving our GPU machines over to Slurm - no more direct login! Without centralising dataset storage, coordination of which machine a job should run on becomes just as challenging as logging into a machine directly. Our decision to move over to a scheduler was primarily due to greedy GPU usage, but it has many other advantages (in my opinion) too.
ZFS
The file server itself is a CyberStore 224S machine with an external CyberStore 212S JBOD connected via a direct attached SAS3 link. The server runs FreeBSD 11 and houses six 4TB SSDs, intended for caching (more on this later), while the JBOD contains twelve 12TB 7200RPM SAS disks. This raw 144TB of storage is divided into two pools.
NAME STATE READ WRITE CKSUM
db ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
da13 ONLINE 0 0 0
da14 ONLINE 0 0 0
da15 ONLINE 0 0 0
raidz1-1 ONLINE 0 0 0
da16 ONLINE 0 0 0
da17 ONLINE 0 0 0
da18 ONLINE 0 0 0
cache
da1 ONLINE 0 0 0
home ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
da7 ONLINE 0 0 0
da8 ONLINE 0 0 0
da9 ONLINE 0 0 0
da10 ONLINE 0 0 0
da11 ONLINE 0 0 0
da12 ONLINE 0 0 0
At this point, you may be wondering why db is configured
differently to home? My justification for this is as
follows: RAIDZ2 can be a bit of a performance killer, in terms of IOPS.
The db pool requires very fast reads and the data, in
general, is replaceable, and as such is configured as a striped volume
of two RAIDZ1 vdevs. While backed up, the data in home
would be much more annoying to recover and the I/O demand is
significantly lower, as such, a single RAIDZ2 should provide a fair
balance between IOPS and data integrity.
Currently we only have one of the six SSDs hooked up as cache to db. I will add more and keep an eye of the hit/miss rates of both L1 and L2 ARC. I am tempted to make one of the SSDs a log device for home, but we'll see how we get on. A common argument against spending money on L2ARC cache is that the money could be better spent on RAM. Well, in many cases yes, absolutely, this is true. However, there is a very specific type of workload this pool will be used for - random I/O of large(ish) datasets for the purposes of training convolutional neural networks. We'd be lucky to get one person's dataset into RAM, and it'd just get flushed out by another user's job. While the adoption so far is still quite low, we currently have an L2ARC hit rate of about 65%. This will drop slightly of course as time goes on, but at that point I will likely add a second SSD and keep an eye on performance.
Networking
Our GPU machines run CentOS. This is probably not the most popular
choice among most computer vision research groups (or perhaps our own!),
but in general it is rock solid. However, there is a cost to this
"stability". Old drivers! The ixgbe kernel module provided
for the Intel 82599 card in 3.10 is incredibly out of date, and as such
it is necessary to update this to another version.
Additionally, we have had many issues with the one particular type of
motherboard in conjunction with this card - The ASRock EP2C612 WS. Urgh.
Things seem relatively stable now, but it required disabling ASPM with
the kernel option pcie_aspm=off.
On the FreeBSD side of things, an update was also required.
Previously it had two kernel panics, since updating there hasn't been
another, so I hope this continues. Updating this driver resulted in two
more noticeable things. The first, which is great, is it resulted in all
four ix0, ix1, ix2 and ix3 devices showing up. For some reason, after
the first reboot, only two or three of these showed up. Immediately
after upgrading the driver and rebooting, the devices reappeared. The
second is that LACP load balancing for outgoing traffic went out the
window. It seems the updated driver likes to use hardware accelerated
hashing of the packets, but only pays attention to Layer 2 information.
This is no good for NFS and perhaps many other types of traffic. To
correct this, I had to recreate the lagg with the option
use_flowid, which forces the hashing to the CPU with an
insignificant performance penalty.
When updating the driver for FreeBSD, make sure you check the readme.
For this particular version it is necessary to change
hw.intr_storm_threshold to 0.
So, in general, I can't say the NIC we chose was the best choice. It certainly has some very irritating issues, most, if not all, I think I have managed to mitigate.
The more stable side of the networking seems to come from the Ubiquiti EdgeSwitch 16XG, which has been very stable. This is a very impressive switch for the price. We had been advised against this switch due to the small buffer size (2MB), which could potentially overflow and result in dropped packets. We haven't had any problems with this. I assume this kind of issue might only occur when trying to squeeze traffic from multiple hosts, into a single adapter of one host. In our use case, the opposite is true. Mostly data comes from a lagg and into many hosts. No problems so far!
Performance Optimisations
On the NFS clients, I've found that setting the NFS block size for clients to the maximum supported under CentOS to be the fastest option. This is 131072 byte for both reads and writes.
Most of the optimisations have been made on the server side:
kern.maxvnodes=17027808 - it was very easy for
vfs.numvnodes to exceed this limit, even under fairly low
load. This resulted in intense staggering of read operations across all
clients.
vfs.zfs.prefetch_disable=1 - I am still unsure about
whether this one is a sensible idea. The success rates of ZFS prefetches
appeared very low in our application. I figure this might save some
IOPS.
vfs.zfs.l2arc_write_max=134217728 and
vfs.zfs.l2arc_write_boost=134217728 - The default is
potentially too slow and prevents the L2ARC from warming up quickly
enough. I might fine this later.
zfs set atime off db - We do not need to know when
random dataset files were last modified, this was just a waste of
I/O.
Performance Testing
I submitted the following to four nodes via Slurm:
find /db/pszaj -type f -print0 | xargs -P64 -0 -I{} dd if={} of=/dev/null
The contents of my db directory is very mixed. There are
several hundred thousand large 7-10MB files, and a similar number of
small 100KB files.
Outgoing from the lagg on the file server was a conservative peak of 17Gbps, with an average of perhaps 13Gbps. So, let's say I am fairly happy with the performance. Of course, I ran this a few times first, which allowed L2ARC to warm up, but this is the expected behaviour when training a CNN so I feel comfortable with it. Of course, steady synchronous reads will be faster.
Perhaps unsurprisingly, I am not too concerned about the write performance in this case.
Related posts:
Wanting to leave a comment?
Comments and feedback are welcome by email (aaron@nospam-aaronsplace.co.uk).