Plotting bottleneck

Its been about 6 months since I've added any more hard drives to my farm, but found a good deal on a 14TB so picked up and and started plotting again. I've built a new machine for personal use since then and put in one of the old nvme Inland drives I had from the first pass. Its a 1TB Inland drive and the rest of the PC has 32GB of DDR4 ram at 3600 Mhz and i7-12700K. Unfortunately, I'm getting bad plot times with madmax and it looks like its because my nvme are just not staying constant with their write speeds. Is that the case or am I missing something? Thanks. Using -r 18, -v 256 and -u 128, windows 11

Posted Apr 25, 2022, 11:55 AM UTC · edited

342

And don’t forget,after you will get truly good plotting time,there will be another bottleneck - writing time to hdd. So you should make a buffer , which could fast write your new plots and then transfer them to an hdd.

5
BWFree•

I think it’s the Inland drive that’s causing your bottleneck. If you could add a Samsung 970 pro to your system I think that would be the most cost effective fix.

Also your times will be substantially faster on Linux compared to windows. I’ve seen it be 2x faster just by getting off Windows and on to Linux.

3
lebanonjon27Chiamployee 🌱•

https://chiadecentral.com/best-nvme-for-chia-plotting-budget/

Lol the poor Samsung 980 I’m testing for the endurance video plot times are like 4000 seconds on this machine. SSD sustained write performance matters greatly, as well as mixed read/write performance since not all the reads can get cached in the kernel

3

You are the 10th person that has verbalized the same thing to me in my comments section and all have 1 thung in common / windows 11. Hmmmmm

3
AcrobaticDingo•

Seems like you NVME is getting thermally throttled. Have you checked the temps? You said this is a new build so i'm wondering how airflow is on your NVME. I had a similar experience when my plotting times tripled out of nowhere and it ended up being a thermal throttle on the NVMEs.

Might not be the case, but it's come up several times here and on some discord channels and takes just a few minutes to test out.

3
rocketrunner442OP•

I actually think this is part of the case. I checked and it was hitting 70. Swapped m2 ports with a different heat sync. Temp down to 46C and seeing some improvements, but barely anything. A write test on the drive hit the sequencing at 1.6G/s which isn't great, but I wouldn't think 100+minute slow

2
AcrobaticDingo•

Is it maxing out at 46 in the new slot during plotting vs 70 in the old slot? That's a massive difference if so. or is the 46 idle? if that's under load then i think it's massively under-utilized unless you've got a massive cooler on it.

https://www.servethehome.com/inland-premium-1tb-nvme-ssd-review-a-store-brand-surprise/3/

1
rkalla•

sustained write speed is the metric that is killing you - burst speed/peak speed is what's advertised on the box and all of them have high numbers, but once you get cranking and writing TBs of data, sws is what falls of a cliff and you get 20hr plots.

2
Bgrngod•

Any chance the "small" cores are being used and slowing things down?

2
lotrl0tr•

Your pc is just fine. Use PrimoCache if on windows, it gave me a 8min boost, I'm using 54GB as temp2 ram cache.

2
ptjunkie•

Windows 11 whyyyy

2
urufusaru•

Disabling indexing in the tmp folders could help a little bit.

2
[deleted]•

[removed]

1
AutoModeratorMOD•

This post has been removed from r/Chia because your account is less than 3 days old. Please try again when your account is older.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

1
Serious-Map-1230•

What did you get on you previous plotter that had the same Inland nvme in it? (and what were the specs of that plotter? And what are you getting now that makes you say the plot times are bad?

From what I remember those Inland drives so do pretty ok.

variations in write speed may very well be due to cache layout on those drives. But that's why I asked the questions above, need some more numbers here.

1
rocketrunner442OP•

Originally with the drive being new, I was getting a plot done in 80 minutes on an 8 year old, 6 core PC. On this newer build, I'm looking at 120 minutes.

2
Serious-Map-1230•

Ok yes that is horrible I7-12700K should be able to do much better.

As someone here already mentioned sustained write speed is key for SSD, you can check reviews on Tomshardware, they include it in their testing.

That said the Inland drives are not great, but also not super slow (at least if you have the Inland Premium tested below). So if that's the drive you have I'm a little confused you get such long plot times.

https://www.tomshardware.com/reviews/inland-premium-ssd/2

Another thing from my experience is that cranking the threads setting -r too high can also cause slowdown in fase 3. With 8 core 16 threads, I would set it between 8 and 12

1
rocketrunner442OP•

Amateur mistake on my end getting core/thread setting for what R should be. I'm going to lower down to 8 to see what impact that has. I'm still thinking my inland drives might just be degraded at this port. Might pick up a 970 evo pro or deal with the sluggishness if I don't pick up any more drives. Thanks

1

Check you cpu to confirm it is not throttling as well. Play around with different settings. Adjust -r as the e an p cores might be acting weird. Also adjust your buckets and compare different phases. For me running phase 1 at 512 and phase 3 at 256 gives me best results

1

Madmax was created to do most writes in a ram disk,you should have 128gb to use this advantage. And of course it should work under Linux, using WSL or using Windows version will have much bigger plotting time. Even if you will upgrade your nvme drive,it won’t give you as good performance advantage as expand your ram to 128gb an using it as ram disk. And btw you will save money in long run,because ram have astronomically bigger recourse.

0†
gryan315•

This is not true, mad max was created for multithreaded plotting, being able to run it in ramdisk was a secondary function. Also, plotting to ramdisk with mad max has the same filesystem overheads as plotting to good NVMe, so plotting time isn't significantly faster, even if plotting entirely in ram when you have more than 256gb ram. Generally you'll see about the same plot time plotting to a fast NVMe as you will plotting to ram + nvme with madmax.

4

Do you really compare bandwidth of ddr4 28800 with even 7000 of top pcie 4.0 writes speeds? And of course the random IOPS are incomparable . btw ram also support channeling. So bandwidth is even higher. So your statement is absolutely wrong. Check it by yourself. I used bladebit, madmax and the built in plotter. And i know what i’m talking about.

0†
gryan315•

With plotting you are double dipping on the bandwidth, so now that bandwidth needs to be shared with the real in memory operations and the filesystem you've jury rigged into place in the memory as well, plus there is the kernel level IO transfers from memory to filesystem, and filesystem to memory. storage_JM has talked in length about this exact subject, and real world examples back it up that mad max plotting in memory is NOT significantly faster than plotting to a storage device. Bladebit is so much faster because it's plotting natively in memory without filesystem translation bottlenecks, and the upcoming bladebit disk is looking to improve even filesystem plotting performance.

6
DiskoVilante•

You know your stuff. Kudos

2

When we are talking about tmpfs,it will be significantly faster.

-1
DiskoVilante•

No. Madmax was created to maximize the use of your hardware. Ramdisk is just another plotting drive option.

I’m getting 23 min plots not using a ramdisk.

3
ThirstTrapMothman•

What's your setup out of curiosity? I haven't got below 38 mins.

1
DiskoVilante•

AMD 5950x, 64 GB RAM (even though madmax only uses maybe 10gb max), 2x Samsung 970p for temp 1 and 2

2
ThirstTrapMothman•

Ah okay, I'm running 5900x. Thanks.

1
therazakael•

Why not plotting in Ram buy a cheap 3955 wx with a board and ram costs around 2 k and i need 10 mins for a plot and i habe a cheap 500 nvme as buffer to copy with a script to the hdds hit me if you need help or mor infos :)

-2
DiskoVilante•

This is terrible advice.

3
therazakael•

Can you explain why?

1
AcrobaticDingo•

plotting speed no longer matters with TNS hovering around 27EiB for the past several months, so any extra spending at this point, if one insists, should be towards space if not coins directly.

2
therazakael•

You right for small farm but i dont want to plot my whole life to fill my 4-5 pb and i also dont want to wear out nvme or ssd its faster and more energy efficient.

1
DiskoVilante•

Spending 2k on RAM for one.

1
therazakael•

2 k for a thradripper mobo and 512 gig ram is okay i guess i calculatet wirh german powercost and for me its the most efficient solution

1
rocketrunner442OP•

Thanks for the info, but I'm definitely more in this as a hobbyiest using mostly spare hardware I have around. I'm not really interested in investing 2k just to quickly plot by < 100tb farm. But I am interested on the details of my personal machine's slow down. I don't mind dropping $100 on an nvme if that will fix the problem since I can reuse that easily for personal use

1
therazakael•

Then yes i also plottet on an samsund evo 980 2 tb with an 3900x and also a buffer nvme to copy then to hdd i not remeber time but i think 60 per day

2
Serious-Map-1230•

You are right, makes no sense to invest money into a plotter to only plot 100TB, any SSD can handle that.

I can't find any results for 12700K to compare with but 11700K I think can do plots in 20-25 minutes on a good nvme ssd. So I would assume the 12700K should not be far off.

If you get a good 1TB SSD for plotting like WD SN750/SN850 or Samsung 970/980 pro

You should be able to get at least in the 30 minutes per plot range. If not, something is wrong with the system or one of the components.

2

The beat value option for is buy more ram. Ram will stand for years 24/7 work

1
therazakael•

Yes also my opinion durable and multiusage is a huge benefit

3