Multi-modal Semantic Communication for Visual Question Answering

Intent Lab, IIT Kanpur

Abstract

Semantic communication systems transmit task-relevant information rather than raw data, making them well-suited for bandwidth-constrained environments. We investigate a multi-modal semantic communication framework for visual question answering (VQA) built on the MU-DeepSC architecture, extending it with Vision Transformer (ViT)-based encoders to improve semantic feature extraction from images. The system is trained and evaluated under noisy channel conditions to assess robustness in realistic transmission scenarios. Compared to CNN-based semantic extractors, the ViT-based approach yields consistent improvements in VQA accuracy, demonstrating that richer visual representations lead to more effective semantic compression. This work highlights the potential of transformer-based encoders in end-to-end learned communication systems for multi-modal tasks.

Summary

I worked on a semantic communication framework for visual question answering using MU-DeepSC. I extended the system with Vision Transformer-based encoders and evaluated performance under noisy channel conditions. My work showed improved accuracy when using transformer-based semantic extractors compared to CNN-based baselines.