Multimodal Vision-Language Model & CLIP Architecture
Designing an automated data generation pipeline and fine-tuning generative and contrastive vision-language models for visual question answering and image captioning on spatial video game environments.
Overview
In this project for Advances in Deep Learning at UT Austin, I built end-to-end multimodal systems capable of understanding spatial scenes from SuperTuxKart gameplay data. The work centered around two main architectures: fine-tuning a generative Multimodal Large Language Model (MLLM) for visual question answering and constructing a contrastive CLIP model for multi-choice visual reasoning.
A core challenge was developing an automated data-engineering pipeline to extract raw 3D scene data and synthesize balanced question-answer pairs and captions for downstream training.
Data pipeline breakdown
Because model capability is tied to dataset diversity, I expanded raw 3D scene metadata into dual pipelines to support both generative and contrastive architectures:
1. Generative VQA Data
Parses scene info into 5 question categories evaluating spatial counts, distances, and object identification.
Target: (Question, Answer, Image)2. Contrastive CLIP Data
Synthesizes detailed scene captions describing active karts, track elements, and view indices.
Target: (Caption, Image)Sample JSON structure output from the automated QA pipeline:
{
"image": "data/train/00042_0.png",
"question": "How many karts are in front of the ego Kart?",
"answer": "2"
}
]
My approach
Generative VLM Fine-Tuning
Built generate_qa.py to auto-generate question/answer labels across 3D gameplay scenes. Using PEFT/LoRA adapters, I fine-tuned the base vision-language model to directly process raw images alongside text prompts and output short-form reasoning answers.
Contrastive CLIP Model
Extracted the VLM's vision model as an image encoder and its underlying LLM as a text encoder. After generating thousands of paired image-caption samples with generate_captions.py, I trained the joint embedding space to perform multi-choice visual selection based on similarity scores.
Challenges
The generated question-answer pairs needed to be accurate before training the models. I took some time to look through and really understand the game and dataset to better tune the data pipeline. I skimmed through some of the images associated with incorrect question-answer pairs to see why the detection was off, then modified my functions accordingly. This resulted in 100% accuracy with both the generated question-answer pairs and the generated image captions.
Results
Evaluating fine-tuned models against held-out validation sets demonstrated massive improvements over zero-shot baselines across visual question answering and multi-choice spatial reasoning tasks.