My little home lab. Proxmox(3700x, 32gb, 1tb nvme), mikrotik rb4011 and eap1250
Posted Oct 1, 2020, 6:22 AM UTC
47364
That is a sweet seup,
I love Proxmox, how is your experience with JellyFin, I am a very long time user of Plex, dabbled with Emby for a bit, but I was already invested in Plex so never made the switch.
7
Not OP, but I run Jellyfin alongside Plex (I am a Plex Pass holder) on the same physical hardware; an actual Linux PC rather than a NAS. I find it's good enough for basic media consumption and the fact that it doesn't require access to Plex's auth servers all the time is a complete win for me.
I use Trakt and Scrobbler to sync the watched history between the two... then next month when Plex's auth servers go screwy on me I can just launch Jellyfin and I'm good.
The downside so far with Jellyfin is that the clients aren't nearly as mature. Plex has amazing clients on most platforms and things like Roku... Jellyfin has them too but they're pretty janky so far. The web client is just fine though.
Long term I might switch to Jellyfin... it does everything Plex does that I care about (i.e. being a local media server) and does nothing that I don't (I hate Plex trying to shove their channels down my and my user's mouths).
4
For my part I'm using Plex but I tried Jellyfin for a while. Unfortunately some of the bugs were deal breakers for me at the time. The last major version release was giving me issues with the Android app and hardware transcoding (quicksync on a DS218+).
I'll probably give it another shot now that I'm in the process of building my first rackmount server!
3
Jellyfin is a good startup, but it's nowhere near a suitable replacement for Plex (yet) in my opinion. I'm currently using both, but Jellyfin is still in it's early stages. Random bugs are present, client app support is limited, Chromecast on the iOS app is not supported, no smart TV apps, apps in general are far less polished than Plex etc. etc.. I can see it become a worthy alternative for Plex in the future, but right now i can't fully migrate over to it yet.
The big advantage Jellyfin has is that everything is local, it doesn't rely on a hundred online services to work (Looking at you, Plex). And you also have way more control as a server owner, i've found that the web interface is way faster than Plex's web interface as well.
3
Jellyfin is pretty good for what I’m using. I like that it’s fully open source, so I can always add features that I want if I put in the effort to contribute. I really didn’t want to pay for Plex pass or any of that, and the day I installed Jellyfin was one of the days plex’s auth servers went down. Probably a good sign to stick with Jellyfin. The Roku client is kinda immature, but I don’t really care. Like I said, if it bothers me so much I’ll just contribute to the clients.
2
I am a long time user for plex and also have the lifetime pass. I switched to Jellyfin and never been happier. It's faster more responsive (although library scanning takes longer) and overall I don't have to worry hey what happens if the internet goes out can I still use plex.
2
How are you liking that motherboard otherwise? It's super appealing to make a homelab server with a Ryzen CPU and not have to deal with a dGPU.
I'm running Plex however at home and still need to figure out hardware encoding for 4K.
4
How are you liking that motherboard otherwise? It's super appealing to make a homelab server with a Ryzen CPU and not have to deal with a dGPU.
Not OP but I'm running a 2700, X470D4U-2T and 64GB 2666 ECC. It's been rock solid and I'm really happy with the features. I went from a dual Xeon L5630 system with 32GB DDR3 that was long overdue for an upgrade.
2
Nice setup, what is your power draw on that 2700x machine?
2
It's the 2700 non X 65W TDP chip. I don't have any power usage data at the moment as it's currently got an extra dual 10GbE NIC for testing as well some drives in the HDD bay for migration. I'll do some testing and follow-up with you.
3
The idle power useage is terrible on Ryzen, my 3900X system idles at 90W stock with a negative voltage offset.
3
I don't think you need any special mother board to run Ryzen without dGPU? I use a 50£ motherboard which was the cheapest i could find with 2 x16 pcie slots. No display obviously, but works fine otherwise.
2
[deleted]
3
Interesting. Will mention that it is a gigabyte b450m ds3h for anyone looking to do something similar. I run it with 1 8x 2x 10gbit card and 1 8x 8 port SATA expander card
2
Same with an Asrock board I have in one of my systems.
I had to modify one of the x1 pcie ports so I could jam an old low power gpu in it for the system to boot (the x16 slots are in use with other cards that need the bandwidth).
1
This might sound really stupid then but how do you do the setup then? Just connect a dGPU while installing software and then ssh from there on out?
2
I installed the OS on a mobile Intel j1800 board I got around, pop the SSD into the Nas and push the power button. I have ran it this way with freenas, Ubuntu and Windows with RDP.
3
I love the ipmi. Super great to not have to setup a keyboard and monitor anytime I need to access the console. Bit more expensive than a normal motherboard, but no need for a damn gpu. The ram slots are a bit tricky with their location, so only a couple coolers will work.
1
Good setup. Really that's all you need. Proxmox is nice in that it lets you overprovision. Switch to lxc containers where you can and you'll be in a better space regarding capacity planning
3
Using LXC for pihole and vms for the rest. I need to play around with both some more.
1
I have a similar setup but went with ESXI and an LSI card passed through to a freenas Vm for my storage needs, then esx is used as my hyper visor for the VMs. Ryzen 3700x and 128gb ram is extremely beefy.
3
I’m using pretty much all my ram now. 96 or 128gb is my goal. Should be enough for the processor, but I’m looking to possibly upgrade the cpu when the new ryzens comes out. I’m impressed with how the cpu has kept up though. Not bad for $300.
1
you'd be amazed at what you can do, processor wise, from even older CPU's.
My Xeon x5650 from 2010 used to run Plex, Couchpotato, Sonarr, NZBGet, Pihole, Apache reverse proxy, pfSense, 2 Windows VMs (one for a file server, the other Active Directory). Thank god for virtualization.
AND I did all that on 16GB of RAM too, until last year. Now I run more hardware alongside that little x5650 - got tired of EVERYTHING being on one box and the 16GB of ram sucked to work with - but it's still powering Plex and anything Plex related and the Windows file server VM.
Your 3700x will last you a LONG time. If you want to upgrade the CPU, go for it. All I'm saying is, you're going to upgrade simply for the sake of upgrading.
1
If I do upgrade, it wouldn’t be for a while. If I build a nas or gaming pc, I’d put the 3700x in that and get a new ryzen to put in the server. I’m really not straining the cpu much right now aside from the occasional transcoding for Jellyfin. I’m going to be setting up some build servers soon, so that might take a bit of cpu to be compiling all the time. I’d like to add another node, but the costs of another mobo, case and psu is a little high compared to just upgrading one server.
1
I’m really not straining the cpu much right now aside from the occasional transcoding for Jellyfin
That was part of what made my Plex server run so well with the old Xeon - everything I have is either standard definition or 720p, and all my devices direct stream, so barely any load.
I noticed audio got transcoded a lot, but that barely takes any cpu resources.
I’m going to be setting up some build servers soon, so that might take a bit of cpu to be compiling all the time.
Ahhh now there's a use case for going bigger. didn't catch that in the OP.
2
The 3700x is extremely beefy for a home lab server. I was running a couple DL380 G8s and while they had more cores, the 3700x blows them away in single threaded performance and clock rate is much higher. You won’t be disappointed. I’m running around 19 VMs and they’re all extremely snappy. Unless you have like 5+ VMs with 100% cpu usage, this thing can handle a ton of VMs with ease.
2
Why Proxmox instead KVM itself. Genuinely asking
3
Not OP, but I deal with the CLI all day at work. When I’m at home “playing” with my servers, I’d rather turn my brain off and use the GUI rather than spend half my time trying to remember / look up a bunch of commands. I used pure KVM for a while, but I ended up not doing much with my lab because it felt too much like work.
5
Yeah, ease of use is a big factor. You don’t really lose much by going with proxmox instead of just kvm, but you gain a lot of features and ease of use. It’s nice to have graphs built in and backups are real easy to do.
2
but... What about Cockpit?
1
I wanted something easy to use and proxmox seemed like the best option. Nice to have the web UI when I want it but still be able to use the command line when needed. The documentation is pretty good too. If I weren’t using proxmox, I’d probably use FreeBSD and bhyve .
1
Have any pictures of how your rack is configured inside?
2
That's not a rack. It's a PC case.
8
Lol i need to stop looking at Reddit before being fully awake. Thanks!
3
It’s a case and the cable management is awful right now. Probably would go with a different case if I were to do it again.
1
Nice one!
I am looking to upgrade my starting-to-become-old labserver - this gives alot of inspiration!
2
How's the Mikrotik RB4011? I'm looking at offloading routing to one of these. Does it support OpenVPN server and parental controls? Also, would it handle Gigabit WAN?
2
The rb4011 is great. I haven’t setup a vpn yet, but I know it supports a couple different types. I really like the web interface; everything is presented right to you and you don’t have to dig through advanced and basic menus or anything. It should handle gigabit wan just fine. I’ve seen tests with it being able to do near 10 gigabit depending on the routing rules used. I plan on having it do my inter vlan routing for me once I get a 10gig switch.
2
What kind of ethernet cables are those? They look thin.
1
Mono price slimrun cat6a. I got em off amazon. They’re great. So much easier to work with.
2
Did you add a GPU? How did you get everything setup? Very powerful rig though!
1
The motherboard has a built in ipmi controller and gpu for vga output. I setup everything over ipmi. I plugged the proxmox usb stick in and configured it all in a web browser. Worked pretty damn well actually.
1
Awesome thanks!
1
What motherboard are you using? I've currently got a consumer AM4 motherboard in mine, so remotely controlling it can be a challenge.
1
The asrock rack x470d4u. They have a version with built in 10g Ethernet, but I just have the standard one. Runs for around $270. They also have a x570 version but that seems to be a bit less available. I was going to get a consumer motherboard to save money, but I didn’t want to deal with a graphics card and I really wanted ipmi.
1
I’ll be upgrading it soon. Adding either 32 or 64gb of ram and probably getting a new cooler since I can’t use all the ram slots with the stock cooler and the x470d4u motherboard. Didn’t think I was going to be doing a media server, but I ended up installing Jellyfin in a vm and used up nearly all my ssd, so I’ll be adding one or two 8TB drives from WD elements once those go on sale again. Probably a few sata ssds for vm storage eventually too. Trying to get this one node to hold me over for a while before I add another.
Running pihole, netbox, gitlab, Jellyfin and a Minecraft server right now. I’ll be adding an nginx reverse proxy so I can access my lab remotely. Might setup keycloak and sso for nginx if it isn’t too difficult. I’ll get around to setting up salt stack at some point.
I'm not sure how it's going to work out in the long run, but on my proxmox Plex vm, I mounted NFS's for my library and host the drives elsewhere, I don't see the point in having it all on proxmox, and it lets me further keep permissions secure against accidental deletion (shared as read only)
When the time comes to expand the library, I should just be able to add a few lines to fstab and mount -a
My ultimate plan is to set something like ceph or glusterfs with 1-2 HDD's per SBC in my own little storage array, and from what I've read, NFS should let me fairly seamlessly do that.
Ceph is meaningless if you use a single node to translate to NFS. If you want to go that route, I recommend at least use the ceph client in proxmox. Using proxmox to host OSDs also works pretty well, although you will need the cli when it refuses to accept formatted drives or when it borks while removing stuff.
Everything is pretty replaceable, so I don't think I'd be using any of the replication stuff for all my media, so a node crashing with a disk would be an event either way.
I have three decommed servers I got from work. Put 1 SSD for the OS and 3 old 3TB red drives in. Can I do clustering and ceph with a setup like this?
Yes. I run it on 4x optiplex 990 with 128gb boot drive, proxmox and 1tb SSD osd. I tried it with HDDs as well, but you need quite a few HDDs to get acceptable performance for vm boot drives and I only had space for 3. Would like to hear how it does with 9 though :)
That's how I do it as well, worked well for years with RAID redundancy.
I’m thinking about building a NAS at some point. If I do that I’d move any 3.5 drives over to it and probably keep the proxmox node ssd only. I just have a 400gb boot disk for my Jellyfin right now that I’ve already resized once and filled up. Really need some mass storage soon.
Nice. I love my ProxMox box. I self host a plethora of 'stuff' on it, including some security distros:
https://funkyimg.com/i/37GW3.png
https://funkyimg.com/i/37GWe.png
Still learning the ropes of ProxMox. I'm pretty sure I've just scratched the surface of what it can do.
Talk to me about reverse proxies. That sounds handy for access, but I don't like the notion of opening ports. How would you go about securing a reverse proxy with say... pfSense.
Try out the nginx proxy manager on docker. It's so easy to use.
I’ll probably be using salt to manage all my config, so I have one place to manage it all from. I’ve set up nginx plenty of times before, so it shouldn’t be too difficult. Getting SSO working might be a little difficult, so I’m not sure if I’ll be going with that or a vpn server. I want to setup remote access for Jellyfin at the least, though.
I have been using openvpn for years and works really well. Want to try out wire guard seems like new and better performance wise.
Yeah, I’ve been looking into wireguard as well. I’m not usually gone from home for long, but it would be nice to access local services remotely. My Internet is 100/6, so hopefully the 6mbps upload won’t be too much of an issue.