r/LocalLLaMaFAQ

16 archived posts · Jan 2025 – Aug 2026

Random post

We're hiring a full-time developer to build an AI-powered video generation system.

The software takes a script + voiceover and automatically generates a fully edited video.

Reference:
https://youtu.be/gJK--ssqj9c?si=5KJsFyXgTQqWoGxD

Requirements:

• Strong Python development experience
• Experience building production-ready automation systems
• Experience with AI APIs (OpenAI, Claude, Gemini, etc.)
• Experience with video generation/editing pipelines
• Comfortable working with Remotion or willing to learn quickly
• Able to work independently without supervision
• Proven track record of shipping real products

DO NOT APPLY

22

u/Huzaifa_Ahmed26•

.Hi everyone,

I'm working on a university project where we're building a Graph RAG-based troubleshooting copilot for medical equipment service manuals (CT scanners).

We have already solved the first part of the problem.

The manuals are scanned PDFs, so we used OCR to extract the text successfully.

However, we've now encountered a much bigger challenge.

The Problem

Many troubleshooting pages in the manuals are flowcharts, not normal paragraphs.

A typical page contains:

  • Symptom
  • Check
  • YES / NO decision branches
  • Next Check
  • Repair Procedure

All of these are connected with arrows.

For example:Symptom

│

▼

Check 1

├── YES ──► Check 2

│ │

10

I have Anything LLM running with ollama and qwen models, but I cannot get either LLM to help me convert my source (an OCR uploaded textbook chapter) to powerpoint slides WITH images within the source. Can anyone help me and explain it like I am 5 as I am having trouble and may just be using the wrong LLM or missing some steps after uploading to the workspace.

10

[deleted]•
10

Título:

Texto:

We ran Mistral 7B and Llama-3 8B in three precisions on an H100 SXM 80GB. The AWQ result surprised us.

The numbers (batch=128, sweet spot):

PrecisionµWh/tokenvs BF16
BF1613.8baseline
GPTQ 4-bit10.3-25%
AWQ 4-bit34.0+145%

Same result on Llama-3 8B:

PrecisionµWh/tokenvs BF16
BF1614.2baseline
GPTQ 4-bit10.7-25%
AWQ 4-bit34.8+145%

Two models. Same pattern. GPTQ saves energy. AWQ costs more.

Why we think this happens:

AWQ's dynamic dequantization introduces compute overhead that dominates at high batch sizes. The model is smaller in VRAM but the dequantization kernel runs on every forw

10

Most health AI tools don't speak Arabic. So I built one that does.

I was the AI engineer on a project called Cura Mind — a local health assistant that runs entirely on your machine using a GGUF model. No OpenAI API. No cloud. Just llama-cpp-python doing the heavy lifting.

**The hard parts nobody talks about:**

Getting a small model to give *structured* responses is painful. I didn't just want it to answer — I needed it to know when to return a nutrition table, when to give numbered steps, and when to just say yes or no. That took a custom classifier and a lot of prompt engineering.

Then Arabic happened. Auto-detecting language sounds simple until you get a sentence like "What is بر

10

been testing gemini 3.5 flash for a high-volume agent pipeline this week.

initial impression: it’s actually usable. feels less like benchmark bait and more like something that survives production traffic without completely melting your API budget. the 1m context window is a weird cheat code too. not because you should ever pass 1m tokens to a model, but because you don't have to over-engineer your RAG chunking and state management just to keep the agent from forgetting its own context loop. is anyone actually running this live yet? curious how bad the latency scaling gets once you start pushing past like 50k tokens in the context window.

10