struggling with glm 4.5 air fp8 on dual 6000 pro

Discussion
# zai-org/GLM-4.5-Air-FP8
#

export USE_TRITON_W8A8_FP8_KERNEL=1
export SGLANG_ENABLE_JIT_DEEPGEMM=false
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
export NCCL_IB_DISABLE=1
export NCCL_P2P_DISABLE=1
export CUDA_HOME="/opt/cuda"
export CUDA_VISIBLE_DEVICES=0,1
uv run python -m sglang.launch_server \
        --model zai-org/GLM-4.5-Air-FP8 \
        --tp 2 \
        --speculative-algorithm EAGLE \
        --speculative-num-steps 3 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 4 \
        --host 0.0.0.0 \
        --port 8080 \
        --mem-fraction-static 0.80 \
        --context-length 128000 \
        --enable-metrics \
        --attention-backend flashinfer \
        --tool-call-parser glm \
        --reasoning-parser glm45 \
        --served-model-name model \
        --chunked-prefill-size 10000 \
        --enable-mixed-chunk \
        --cuda-graph-max-bs 16 \
        --model-loader-extra-config '{"enable_multithread_load": true, "num_threads": 8}'

This is my config right now, and I keep running out of ram, I have messed with chunked prefill, graph max, fraction static a bunch of times and it just keeps bombing. I am using a config someone was using for 4 6000 Pros and I reduced tp to 2, and have been dropping all the parameters I mentioned above trying to get it to load. Even set them to really low values just to see if it loads. I should be able to get fp8 and full context on 192G.

Posted Nov 5, 2025, 1:58 AM UTC

255
Loading comments…