Source
r/ceph•
N7KnightOne

CephFS/Ceph Configuration Guidance Needed

In the near future (+/- 2 months), I will be setting up a new cluster that will consist of 6 R730XDs with HDD, SSD and NVMe storage in a HCI stack. In this configuration, I will be using VMs for legacy apps that I use, but it will most focused on containerized workloads. Most of my containerized workloads require databases and I want to place those databases on the SSDs.

Here's my problem, with only one CephFS pool supported (using Proxmox), what is the best way to utilize the SSDs for databases in my containerized workloads given the HDDs will be used for CephFS and the NVMe will be used for OS/EFI storage?

Posted Sep 23, 2020, 5:47 AM UTC

216
TheSov•

you will have performance setting up vm's on a cephfs. its not designed for that. you want rbd's for databasing since DB engines use block reads/writes, they use index's as shortcuts. this doesnt work well at file level sharing.

5
sep76•

TheSov is correct. Vm's need to be on rbd's. The performance difference is huge. Rbd also is an easier construct not beeing dependant on mds servers.

That does not means cephfs does not have uses in this scenario. I copy the disk image to cephfs mount for import, but disk move it to rbd for production.

I have 2 rbd pools, one "regular" using hdd class osd and one "fast" pool using ssd class osd.

3
N7KnightOneOP•

The CephFS pool will be backed by HDDs and used solely as a file store, and the NVMe drives will be RBD pool for VM only. What I am struggling to figure out is how to incorporate the SSDs to create a shareable location for my containerized DB workloads.

1
TheSov•

you set device classes for your osd's so your cluster treats them differently, classes for HDD and SSD already exist so you just have to make sure ceph knows about them. then you can set pools to only exist on certain device classes.

ceph osd crush set-device-class ssd osd.$OSD

so when you create your data/metadata pools for cephfs they go on device class=hdd and when you create your rbd pool it goes on SSD class.

3
insanemal•

If your doing databases in containers you should be using RBD.

I have proxmox + k8s all backed on ceph. My k8s is running in VMs on proxmox.

Do not use CephFS for databases..

3
shanehsu•

Yep. Use RBD! There probably won’t be proper driver/plugin to mount the FS to your container runtime anyway.

2
insanemal•

There is. Kubes can do both.

3
N7KnightOneOP•

Thank you! Is there some documentation that you followed to integrate RBD into your workload? At this point I feel like I need to get a better understanding of RBD and how it's intgrated.

1
insanemal•

K8s supports RBD.

Just look up the kubernetes documentation

2
porchlightofdoom•

I hope this is for a homelab? HCI is not generally recommended for Ceph as the OSD nodes like memory. Also, Ceph is not the best for small reads/writes like databases. Anything can be overcome with enough hardware, but just letting you know that this is not the best practice.

2
N7KnightOneOP•

Yepp, this is all homelab. I did not know that, thank you for letting me know.

Currently I have three R630s in a HCI configuration with VMs on NVMe drives and CephFS on SSDs. So far it's been very reliable and I am happy with the performance. The main reason for this transition is I want to move the bulk of my storage to CephFS.

1
porchlightofdoom•

Just another word of warning, a Proxmox user tried the same thing a few months ago. Something happened to Proxmox and it blew away the CephFS configuration. In affect, it made a new cluster. All data lost.

The better option is to split up your servers into a storage cluster and a compute cluster. Then use the native Ceph tools to manage the cluster.

2
N7KnightOneOP•

Yikes!

Do you happen to have that post/thread?

1
porchlightofdoom•

https://www.reddit.com/r/Proxmox/comments/hwk68v/access_data_on_ceph_osds_monitors_lost/

2
shanehsu•

Do you absolutely want NVMe for OS? I don’t really have production experience with NVMe under Ceph. But over a network for booting VMs, I don’t see much improvement there.

These are some suggestion/things to look into,

  1. For you OSD, put WAL or DB on SSD if need be
  2. CephFS (for files) to be on HDD (metadata on SSD/NVMe depending on device ratio)
  3. Multiple Block storage pool depending on your need (if your VM uses file-based images right now, you should look into importing them to Ceph as RBD)
  4. DB on NVMe, OS images on SSD
2
N7KnightOneOP•

I will say yes, as I have more NVMe drives than SSDs and I am currently using NVMe as the home for my OS vdisks in my R630 cluster. The NVMe are in a RBD pool and I am very happy with the performance.

Thank you for all of your suggestions!

1
shanehsu•

Wow more NVMe than SSDs, incredible. What kind of networking are you rolling? 10 GbE?

2
N7KnightOneOP•

Yepp, I was in the right place at the right time. Yeah, I will be running two 10G over DAC to two switches for HA.

1