This is a Streamlit-based multimodal campus assistant that supports image input, voice input, and text queries. It uses CLIP + FAISS for image retrieval, Whisper for speech-to-text transcription, DistilBERT for intent classification, and a structured JSON knowledge base for final response generation.
The system allows users to upload campus images, ask questions using voice or text, and receive structured responses including location name, description, opening hours, events, and navigation guidance. It is designed as a multimodal system where each input type can work independently or together through a fusion layer.
Dataset and Models: https://drive.google.com/file/d/1Zea-orC28m3mUs5vcJC-35lALjosJWqt/view?usp=sharing
After downloading:
Campus_Assistant/ dataset/ models/ audio_samples/ results/ notebook/
Install dependencies using: pip install -r requirements.txt
pip install git+https://github.com/openai/CLIP.git
Run the Streamlit application: streamlit run app.py
Then open: http://localhost:8501
KMP_DUPLICATE_LIB_OK=TRUE streamlit run app.py
Build Docker image: docker build -t campus-assistant .
Run container: docker run -p 8501:8501 campus-assistant
app.py contains the Streamlit interface and backend logic. The dataset folder stores campus images and audio samples. Models include CLIP, Whisper, and DistilBERT pipelines. requirements.txt lists all dependencies required to run the system.
Whisper is used only for transcription and not training. CLIP + FAISS performs zero-shot image retrieval. DistilBERT is fine-tuned on synthetic campus queries. The system does not store any user images or voice data, ensuring privacy and compliance with data protection principles.