Multimodal Model + Technical Drawings

I have no ML/AI background but I would like to understand on a high-level how one would go about training models which are capable of certain tasks. Aswell I am interested in how to decide which available model is best for the tasks.

• The model should be able to take an image as an input (the higher the supported resolution the better) and describe it in a certain style (that's where the fine-tuning comes into place)

The model should describe the image in detail, include all rooms, materials, descriptions, measurements etc.

The more detailed, the better.

The idea is to create an embedding for the generated description so it is possible to retrieve drawings by describing it. Does that make sense?

I find a lot of promising multimodal models on huggingface but I also discovered MiniCPM-V which at first glance looks quite powerful.

How would you go about solving such a task? What model to pick? How many datasets do you assume would be required to get decent results?

Appreciate all the inputs!