Hi all!

After quite a bit of work, I’ve finally completed my **Vision-Language Model** — building something this complex in a multimodal context has been one of the most rewarding experiences I’ve ever had. This model is part of my Master’s thesis and is designed to **detect product defects and explain them in real-time**. The project aims to **address a Supply Chain challenge**, where the end user needs to clearly **understand** ***why*** **and** ***where*** a product is defective, in an **explainable and transparent** way.

*Processing img ota230yckrmf1...*

I took inspiration from the amazing work of [ClipCap: CLIP Prefix for Image Captioning](https://arxiv.org/abs/2111.09734), a paper worth a reading, and modified some of his structure to adapt it to my case scenario:

For a brief explanation, basically what it does is that the image is first **transformed into an embedding using CLIP**, which captures its semantic content. This **embedding is then used to guide GPT-2** (or any other LLM really, i opted for **OPT-125** \- pun intended) **via an auxiliar mapper** (a simple transformer that can be extended to more complex projection structure based on the needs) that **aligns the visual embeddings to the text one**, catching the meaning of the image. If you want to know more about the method, this is the [original author post](https://www.reddit.com/r/MachineLearning/comments/q3xon8/p_fast_and_simple_image_captioning_model_using/), super interesting.

Basically, It **combines CLIP** (for visual understanding) **with a language model** to generate a short description and overlays showing exactly where the model “looked”, and the **method itself it's super fast to train and evaluate,** because nothing it's trained aside a small mapper (an MLP, a Transformer) which rely on the concept of the **Prefix Tuning** (A Parameter Efficient Fine Tuning technique).

What i've extended on my work actually, is the following:

* **Auto-labels images using CLIP** (no manual labels), then trains a captioner for your domain. This was one of the coolest discovery i've made and will definitely use Contrastive Learning methods to auto label my data in the future.
* Using **another LLM** (OPT-125) to generate better, intuitive caption
* Generates a **plain-language defect description**.
* A **custom Grad-CAM** from scratch based on the ViT-B32 layers, to create **heatmaps** that justify the decision—per prompt and combined, giving transparent and explainable choice visual cues.
* Runs in a simple **Gradio Web App** for quick trials.
* Much more in regard of the entire **project structure/architecture**.

Why it matters? In my Master Thesis scenario, i had those goals:

* **Rapid bootstrapping without hand labels**: I had the "exquisite" job to collect and label the data. Luckily enough, i've found a super interesting way to automate the process.
* **Visual and textual explanations for the operator**: The ultimate goal was to provide visual and textual cues about why the product was defective.
* **Designed for supply chains** setting (defect finding, identification, justification), and may be **extended to every domain** with the appropriate data (in my case, it regards the rotten fruit detection).

The model itself was trained on around **15k of images**, taken from [Fresh and Rotten Fruits Dataset for Machine-Based Evaluation of Fruit Quality](https://data.mendeley.com/datasets/bdd69gyhv8/1), which presents around \~3200 unique images and **12335 augmented** one. Nonentheless the small amount of image the model presents a surprising accuracy.

**For anyone interested, this is the Code repository:** [https://github.com/Asynchronousx/CLIPCap-XAI](https://github.com/Asynchronousx/CLIPCap-XAI) with more in-depth explanations.

Hopefully, this could help someone with their researches, hobby or whatever else! I'm also happy to answer questions or hear suggestions for improving the model or any sort of feedback.

Following a little demo video for anyone interested (could be also find on the front github repo page if reddit somehow doesn't load it!)

*Processing video fgjdz2xjrrmf1...*



Thank you so much!

Tried building an explainable Vision-Language Model with CLIP to spot and explain product defects!