r/LocalLLaMA•
danielcar

Hardware costs to drop by 8x after bitnet and Matmul free are adopted

Discussion

Just my shower thought. What do you think?

https://arxiv.org/html/2406.02528v5

https://arxiv.org/html/2402.17764v1

Here is list of improvements:

  1. Less memory required, and or you can handle larger models
  2. 8x lower energy consumption
  3. Lower cost to train?
  4. Lower cost to serve a model
  5. Lower cost of hardware
  6. Lower Latency
  7. Improved throughput for model serving
  8. Quick speed of answers, similar to latency and throughput.

Posted Jun 27, 2024, 3:16 PM UTC

264101
Loading comments…