struggling with glm 4.5 air fp8 on dual 6000 pro
# zai-org/GLM-4.5-Air-FP8
#
export USE_TRITON_W8A8_FP8_KERNEL=1
export SGLANG_ENABLE_JIT_DEEPGEMM=false
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
export NCCL_IB_DISABLE=1
export NCCL_P2P_DISABLE=1
export CUDA_HOME="/opt/cuda"
export CUDA_VISIBLE_DEVICES=0,1
uv run python -m sglang.launch_server \
--model zai-org/GLM-4.5-Air-FP8 \
--tp 2 \
--speculative-algorithm EAGLE \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--host 0.0.0.0 \
--port 8080 \
--mem-fraction-static 0.80 \
--context-length 128000 \
--enable-metrics \
--attention-backend flashinfer \
--tool-call-parser glm \
--reasoning-parser glm45 \
--served-model-name model \
--chunked-prefill-size 10000 \
--enable-mixed-chunk \
--cuda-graph-max-bs 16 \
--model-loader-extra-config '{"enable_multithread_load": true, "num_threads": 8}'
This is my config right now, and I keep running out of ram, I have messed with chunked prefill, graph max, fraction static a bunch of times and it just keeps bombing. I am using a config someone was using for 4 6000 Pros and I reduced tp to 2, and have been dropping all the parameters I mentioned above trying to get it to load. Even set them to really low values just to see if it loads. I should be able to get fp8 and full context on 192G.
Posted Nov 5, 2025, 1:58 AM UTC
255