Skip to content
Marko Ivanovski

Multimodal hateful meme detection

Tests whether a multimodal model can detect hateful memes, comparing joint fusion against two separately trained models.

category
academic
year
2026
status
completed
role
One of two on the project
  • Hugging Face Transformers
  • Python

What it is

A controlled comparison for Intelligent Information Systems, with one colleague: does reading a meme's image and text together beat combining two separate models? Early fusion was Gemma 4 E4B fine-tuned with LoRA. Late fusion averaged the probabilities of a frozen BERT on the text and a frozen ViT on the image. Both trained on 500 memes from the Hateful Memes dataset, which was built with confounders to punish naive combinations, and were tested on 1,000.

Early fusion scored the highest AUROC, 0.649. BERT alone reached 0.607 and ViT alone 0.481, close to chance, while averaged late fusion fell to 0.529, below the better single model. The pattern reported for very large models held for open models small enough to fit on one Colab T4.

figure 1 · AUROC on 1,000 held-out memes, 490 of them hatefulOnly the jointly fine-tuned Gemma 4 model separated hateful memes clearly better than chance, at 0.649 AUROC (95% CI 0.616 to 0.684). Averaging the text and image models gave 0.529, below the text model alone at 0.607, and the image model alone sat at chance with 0.481.
The numbers
MethodAUROC95% CI
BERT, text only0.607none
ViT, image only0.481none
Late fusion, averaged scores0.529none
Gemma 4 E4B, joint fusion with LoRA0.6490.616 to 0.684

Recomputed from the saved test-set predictions, matching the notebook outputs.

  • Uses the hateful memes dataset from Meta's earlier public challenge.
  • Compares Gemma 4 doing joint fusion against a late-fusion pair of BERT and ViT.

What I learnt

  1. Pick the checkpoint by validation AUROC, not the last epoch. One run fell from 0.639 at epoch four to 0.487 at epoch five.
  2. LoRA on only the language side of a multimodal model is not enough to make it use the image. The adapter has to reach the vision tower and its connector too.
  3. Averaging a strong signal with a near-random one pulls the result down, not up.
  4. On a hard test set the F1-maximising threshold can drop so low that accuracy and F1 stop meaning much, so AUROC is the metric to report.

What I would do next

Swap the frozen ViT for CLIP, and learn the fusion weights instead of a fixed 50/50 average.

Regularise the classification heads and train on more than 500 memes.

Evaluate the confounder memes on their own once a dataset version labels them.