Armym

Claude 3.7 might be good at oneshotting a whole webapp, but for those who actually want to use it to code, it's so annoying when it spits out stupid 100 line solutions for simple problems. This is just an example, but I have experienced it in the field and it annoys me so much. I will probably switch to og sonnet 3.5 (not the updated one).

Guess which screenshot is what model.

The prompt:

"xxxxxxxx - the text i exported"

I exported an sql database to csv and have this. any idea how to get the text from it? it is rtf content. any idea how to get back the formatting with the special chars?

This is an nvidia feature that upped my inference throughput for multiple models by about 43% on one L40S GPU: https://docs.nvidia.com/deploy/mps/index.html

Our locallama community is (or was) a highly technical community not talking about trends (shit like langchain, politicis etc). Right now it's mostly people showing screesnhots of deepseek chat and other hype.

I hope the hype is over soon and I start seeing highly technical content here again

I am not talking about hosting LLMs here, that's easy.

Nvidia Triton is definitely one of the best production ready inference server backends. It's in flight batching, speed, scalability and versatility is what makes it so great.

But setting anything up with it is a complete and utter mess. I am used to broken dependencies. The first tool for AI I had to learn was Docker, to handle all of that. But Nvidia triton just doesn't want to work with certain models. I was setting up a whisper + diarization pipeline in it and it's refusing to work correctly.

Whisper is discarding any token above 128 (I don't remember the exact number). Diarization with pyannote has broken dependencies. Nvidia Ne

I connected one 3090 with a riser cable to my pcie slot. The system saw the card but nvidia-smi couldn’t communicate with it or didn’t see it. The problem was probably that my riser cable was rated for Gen 3, but nvidia drivers wanted to communicate with the card in Gen 4 mode. I then headed to bios to change PCIE gen 4 to gen 3. Unfortunately, my Bios (supermicro) didn’t have the option, so I had to search for what to do. After a while of tweaking and searching, I came across the following linux commands to change the pcie slot from gen 4 to gen 3 (note this is only for one GPU slot, but could be configured for multiple):

# Set GPU
sudo setpci -s c1:00.0 CAP_EXP+0x30.W=3
# Set root por

Is there some great inference engine for whisper? I only found "whisper as a webservice" which is really not production ready and doesn't support parallel requests. I know that vLLM has whisper in the roadmap, but it's not yet available.

I work in a relatively conservative industry. I want to use Qwen 2.5 and host it with vLLM on premise. The server will not even be connected to the internet, just local. The people above told me that I can't use a Chinese model from Alibaba because it could be a trojan. It's so absurd! How would you explain to them that it doesn't matter and that it's as safe as anything else? Also, the model will be finetuned anyways, doesn't it make the model itself unrecognizable at that point?