Ceph vs ZFS - Which is "best"?
TLDR: Ceph vs ZFS: advantages and disadvantages?
Looking for thoughts on implementing a shared filesystem in a cluster with 3 nodes.
They all serve a mix of websites for clients that should be served with minimal downtime, and some infrastructure nodes such as Ansible and pfSense, as well as some content management systems.
The intention is to have Hight Availability between the nodes, as when one is unreachable, it (seamlessly?) moves the VMs to a failover. For that, it's required to have a shared filesystem, so considering between Ceph and ZFS at the moment.
There's already disks and corresponding volumes on each of the nodes for the OS, general VMs and backups. Each of the nodes also has 2 x 200GB SAS SSDs in Raid0. These are intended to have the VMs which should be in the High Availability configuration.
So I was wondering what would be the advantages of choosing one over the other, are they even possible to compare side by side? I have read a few threads and some documentation, have had a few experiences too, and some opinions coming in. Tradeoffs? Anything you may point out that can be taken as a starting point? Would appreciate any input you may have.
Posted Dec 22, 2021, 10:51 AM UTC
If you want HA and seamless move of VMs (just like VMWare vMotion), you need your nodes to share a common FS, be ceph or NFS over zfs. The later intoduces a single point of failure, so I chose ceph for this task.
Don't get me wrong, zfs is awesome, and I use it everywhere, but it's not the right tool for this task.
Reminder, if you use SQLite at all then NFS may not be the way to go due to file locking.
SQLite? I'd rather be dead. ;-)
Same! But there are a lot of SQLite apps out there that folks run in Proxmox (think the Homelab crew) and those "arr" apps built on .NET that use SQLite for their DB backend. Those apps will crash and burn on NFS shares.
Just remember, that Ceph scales with disk count. If I interpret OPs post correctly, 3 nodes with not a lot of disks will not perform well. In a 3 node setup, a ZFS mirror for local redundancy and synchronization with a USV to ensure uptime, might be better suited
Sure, performance scales as you add disks. My setup is quite similar to op's ( 3 nodes, 1x SATA SSD per node, 10Gbps network) and performance is good imho. Using 2 SAS SSDs per node should give a performance boost for sure.
What does good mean for you? If I recall the latest benchmarks correctly, in a 2 disk SSD setup, a ZFS mirror outperforms Ceph by a factor of close to 2, at the cost of some RAM. When running multiple LXCs and VMs for example, that can make a huge difference. I am sure CEPH runs acceptable, but I would not recommend running CEPH on a small cluster for anything else but learning purposes.
Edit: forgot a word
For ha they will need shared storage. Why is zfs even in the run here?
For me, it means that my VMs and CTs were previously running on zfs, and when I set up my 3 node proxmox cluster, I migrated them to ceph and performance is enough to not consider a rollback. Of course, my needs probably are different that yours. YMMV.
What is an USV?
Ah, thats the local abbreviation, my bad. Uninterruptible Power Supply - UPS for the english folks.
yeah, USV is the german word for "
Unterbrechungsfreie Stromversorgung"
-> an UPS ;)