I averaged the weights of the best open sourced coding models "pretrained" and "finetuned" weights. The results are really good.

The models are released here, because thats what everyone wants to see first:

- https://huggingface.co/collections/rombodawg/rombos-coder-v25-67331272e3afd0ba9cd5d031

But basically what my method does is combine the weights of the finetuned and pretrained models to reduce the catastrophic forgetting, as its called, during finetuning. I call my method "Continuous Finetuning" And ill link the write up bellow. So far this has been the highest quality coding model (The 32b version) that ive made so far, besides possibly the (Rombos-LLM-V2.5-Qwen-72b) model.

Here is the write up mentioned above:

- https://docs.google.com/document/d/1OjbjU5AOz4Ftn9xHQrX3oFQGhQ6RDUuXQipnQ9gn6tU/edit?usp=sharing

And here is the method I used for merging the models if you want to skip to the good part:

models:
  - model: ./models/Qwen2.5-Coder-32B-Instruct
    parameters:
      weight: 1
      density: 1
merge_method: ties
base_model: ./models/Qwen2.5-Coder-32B
parameters:
  weight: 1
  density: 1
  normalize: true
  int8_mask: false
dtype: bfloat16

Anyway if you have any coding needs the 14b and 32b models should be some of the best coding models out there as far as locally ran open source models with apache 2.0 licenses.