llama.cpp running the ai models with less ram
Hi when running the AI models i notice that the amount of ram used is a lot lot less than claimed, however the performance greatly differs based on the amount of ram the machine has. My machines have limited number of ram slots but based on the behaviour of running the models, are the models cached into ram instead? for instance if i run llama 70B i have to rent an expensive aws ec2 instead but the responses greatly differ. If i use 13B it does not answer the question and i get the question back but it does with 70B. I still would like to be able to run 70B and i am using a similar chip architecture with avx_vnni. If there isnt enough ram would it be possible to create a ram drive split across multiple machines and use 10Gb/s NICs? I have used SFP+ NICs and SFP+ slots in my switch.
Are there ways to speed up running larger models with less memory without quantising them with lower accuracy?
Posted Nov 12, 2023, 7:23 AM UTC
This is because llama.cpp uses
mmap()by default, which maps a file stream buffer into memory. The model is swapped in and out as used, with available system ram basically used as a cache. you can disable this via the command line if you want the model to be static in ram.i tried that but it still did it. What options can i do to at least run the model at a decent speed with less ram? Should i swap on other computers via networked ram drives? or does disabling that cache help? for instance the most im able to muster is 32GB of ram which isnt enough to run the 70B model, however no matter what i have tried the command to have the model static in ram never works and llama.cpp freezes whenever more ram is needed (irregardless of using the command).
I read in a git issue that the command no longer works though.
Llama.cpp can run models across multiple machines using MPI. I think there are instructions in the README in the github. It won't be very fast.
Trying to rely on any sort of swap isn't going to work well. The limited bandwith and latency of persistent storage will kill performance.
i know its not as fast as having it in ram, im just trying to see if i can distribute the memory across machines. Only 1 machine has avx_vnni.