How to change Ceph’s private network?
I’ve set up a successful HA and Ceph cluster. This currently uses 2 gigabit ports in LACP which works as expected. I’ve now got a 10 gig network and wanted to send the cluster traffic over that too. What options do I have?
Prox doesn’t allow you to specify another network after the cluster is created. I tried to do this once before and broke the cluster. Can this be done safely/correctly using the command line? Is it safe to change files in /etc/pve? Does there need to be a command to push new config to the cluster?
For Ceph it’s more mysterious. The official docs say to change ceph.conf but there is no OSD config in there! Prox manages the OSDs for some reason and I’m not sure how to add the private network to the existing OSDs. Can someone point me in the right direction here?
I’ve been searching but the documentation doesn’t seem to say what to do in this situation. In the end it’s just home lab stuff and I can just nuke the cluster and start over, but I’d rather not. However, I’m starting to think that might be my only choice if I want this to be correctly done. I’d appreciate any help you can provide, thanks in advance.
Posted Dec 30, 2019, 4:30 AM UTC
just add these to your ceph.conf on all cluster members and change the ip's to reflect your ip numbers.
in proxmox there are 2 copies of the ceph.conf one in /etc/ceph and one in /etc/pve/
edit: you should really consider building an external ceph cluster, as I have had nothing but anxiety and heartache using the ceph built into proxmox.
https://www.youtube.com/watch?v=5vu5ydB8FAg this video shows how to install ceph from scratch
https://www.youtube.com/watch?v=a4swML-TNXs this video shows how to add the external cluster to proxmox.
both videos are literally step by step and dont skip anything. in the proxmox connection video there is 1 error, the ceph.conf needs to also be copied to each proxmox's /etc/ceph if you wish to see cluster info, other than that its spot on.