New Ceph Cluster - Performance question
Hello guys,
I have been reading a lot about performance issues on Ceph but the setup are always a bit different than mine so I am not sure what is my problem.
I have 3 nodes (HP G9) with Intel Xeon E3-1220 v6, 32GB DDR4 ECC, 1x 500GB SSD for Proxmox and 2x 1TB Enterprise HDDs for CEPH. They are connected in mesh with 10GB DACs, which I did setup as the cluster network, and have a 2.5GbE public network. I want 3 replicas, with a minimum of 2 so I can function when one node is down.
I added the three nodes as monitors and two as managers. I added all 6 HDDs as OSDs and created a pool with replicas = 3, min-replicas = 2
Everything seems to be working, but I am not having the performance I was expecting. I know we are writing twice to the same disk, as I don't have a seperate journaling drive, but the 10~14MB/s write speed seems incredibly low to me.
Drives themself can go >150MB/s and I have a mesh 10GB cluster network. Even if we write to all nodes at the same time and twice to each disks, we are far from saturating the drives or the network
Iperf3 test show >9Gbps on the cluster network and >2.1Gbps on the public network, so I don't think it's the problem.
I tested with and without jumbo frames (on the cluster network) and it doesn't seem to make a difference.
I know CEPH performances are best with more nodes, more OSDs and more parallel work, but I would be expecting a write speed of at least 60-70 MB/s. Maybe I am dreaming and there's not way to get more than my 10-15MB/s over HDDs...
Thanks all!
Posted Jul 21, 2023, 6:01 PM UTC
2 HDDs, and 32GB of ram per host is the bottleneck here. Even with enterprise class HDDs your replica have to replicate, that will caused mixed IO and bring HDDs to their knees. Then your limited 32GB per host on top of that. I would suggest at a min 64GB of ram and 3 HDDs per host, and consider adding an SSD for your Cache pool. Even in HCI HDDs are going to be a huge burden without proper cache backing them.
In my own CEPH setup, I found performance issues running a 3node setup with 32GB of ram each with 2 NVMe SSDs. Once i doubled to 64GB and then added a 3rd NVMe drive performance is where I expected it. its that replica set that causes this i think.
The hosts are running with around 50% RAM usage, so I didn't think the RAM was a bottleneck. I only have 8x 1TB (so I miss one to have 3 in each) so I'll look into what I can do to add at least a drive in each.
The hosts IO delay (as reported by `top` and by proxmox, are lower than 5% so I also didn't expect that to be the problem
I understand it's getting replicated and it waits for the 3 replicas to be written before confirming the write, but <15MB/s seemed like I had another problem somewhere ahah
Maybe CEPH is just not for me...
On a windows VM hosted on Ceph, download IOMeter and build out a profile on 4 workers (1 per vCPU) for 20GB(in sectors) 4 pending IO. Then two worker profiles(not at the same time) 100% reads, 4K block size, 100% random. Then the other 100% reads, 8MB block size, and 100% Seqential.
What are your IOPS and MB/s and total worker latency.
That IO Delay report on Proxmox is not the full picture, its a clue in whether or not you should be diving in. A 5% constant IO delay is about 15ms, thats when its time to dive in.
What I find weird is that when moving a disk on the CEPH pool, I get 70-80 MB/s, which is around what I am expecting in terms of write speed with my setup (around half the throughput of a single HDD). Inside a Linux VM (I tested 3) I get less than 20% of that speed
Here's the requested results from IOMeter. I don't know that app much, so I hope this is what you wanted!
For 16K 100% reads, 100% random, 4 workers, 4 outstanding IO: ~1.5 IOPS, ~ 12.5MBps, 2650ms average response time
For 8M, 100% reads, 100% sequential, 4 workers, 4 outstanding IO: ~1 IOPS, ~8.5 MBps, 3750ms average response time
I have similar results with 2 or 4 workers. I don't know if I have an issue with my config, or if CEPH just doesn't fit my usage
Thanks for your time
That high latency tells me you have remote disk access across the network and something is slowing IO fetching down. You really need another HDD in the mix per host to make it three and to add an SSD for Cache. 3second IO latency is bad. How is your networking connected? Flow control on your ports enabled, 9k MTU, what switch vendor and model? Fiber, Twinax, or RJ45 for 10G?