You don't need to be on Sun's Linux distro/distribution to use ZFS. As a rule of thumb, in the Linux/Unix world filesystems are not tied to kernels, you can pretty much mix and match kernels and filesystems. There's nothing stopping you from using ZFS on any Linux distro. Most come with support for a wide variety of filesystems out-of-the-box, you don't need to compile anything (or even load any modules).
Core packages in the Linux world are not created by the various distros, they all come from the same source (so functionality is pretty much identical across all distros).
The differentiating factors are, in my not-entirely-humble opinion:
- community (size, momentum, helpfulness etc)
- commercial support (if you need that)
- stability and frequency-of-release (these pull in opposite directions)
I wouldn't pick RHEL for any reason, it's just added cost with nothing extra to gain for it. CentOS commercial offerings and community support pale in comparison to Ubuntu with Canonical behind it, plus they're dependent on RHEL's release cycles (which are *slow*).
I would go with Ubuntu Server for the reasons above. Plus, you distro doesn't get treated like a second-class citizen if it's free (I'm referring to the RHEL/Fedora hierarchy).
I especially like Ubuntu Server's cloud offerings (private clouds via Eucalyptus. compatible with Amazon's EC2 :
http://www.ubuntu.com/cloud). If the idea of an elastic data-center that expands/shrink to cope with demand, then out-of-the-box support might be a good idea. Like I said before, you can also get Eucalyptus (or anything else) from the upstream creator, in this case from
http://www.eucalyptus.com/
Yes, I know I sound like an Ubuntu fanboy... I just like it (disclaimer: I have zero commercial affiliations with Ubuntu/Canonical).
Bottom-line: do your research well before you decide, you'll be stuck with that choice for a long time if your deployment is sizeable.
Saidi