- Model Weight is in models/
- Parameter sweep is in sweep_runs/results.json
- Website and Code for Demo is in web/
- We use the MNIST handwritten digit dataset for both training and evaluation, following the standard setup introduced by LeCun et al. 1998. This dataset provides us with text labels and corresponding image data.
- For 3D point cloud of the digits, we utilize the methodology for 2D to 3D lifting provided by Su et al. 2024.
- Chiranjeevi Hegde: Baseline Results, Report Writing, Demo
- Ashwani Rathee: Methodology, Evaluation
- Akhilesh Thite: Dataset Finalization, Hyperparameter sweep, Presentation
We want to align the data from different modalities in the same feature space for downstream tasks. This aligns these concepts in same embedding space and we can calculate similarity between similar concepts in different modalities.
Step 1
git clone https://github.com/ashwanirathee/contrastive-mnist.git
cd constrastive-mnist
Step 2: Dataset Generation
python -m venv .venv
source .venv/bin/activate
pip install torch torchvision h5py numpy tqdm sklearn umap-learn seaborn matplotlib onnx
python src/dataset_creation_3D.py
Step 3: Hyperparameter sweep
python src/parameter_sweep.py
Step 4: Training
python src/train.py
Step 5: Evaluation
python src/evaluation.py
Step 6: Visualize Features
python src/visualize_features.py
The tri-modal backbone couples three lightweight encoders—a CNN stack for 28×28 images, a masked-average text encoder built on 64-d token embeddings, and a PointNet-style MLP for 3D point clouds—each producing L2-normalized feature vectors of shared dimensionality. During training, batches from MNISTDataset (with point clouds limited to 2,048 points and normalized MNIST images) flow through the encoders, and the model optimizes a symmetric InfoNCE loss over every pair of modalities (image↔text, image↔point cloud, text↔point cloud). The train.py loop runs Adam with dropout, temperature scaling, and weight decay, tracks validation loss every epoch to retain the best checkpoint, and finally extracts embeddings from the held-out test split for downstream use
We are doing training from scratch
We did parameter sweep to identify these parameters by training on the training set for 3 epochs and identifying good configs based on validation loss.
- Batch Size: 128
- Learning Rate: 0.0001
- Temperature: 0.05
- Embedding Dim: 64
- Dropout: 0.25(within the image encoder)
We evaluate the results mainly by analysing the zero shot top-k classification results for image-text and pointcloud-text pairs.
| Methodology | Modality | Top-1 | Top-3 | Top-5 |
|---|---|---|---|---|
| Baseline: OpenShape | Image - Text | 61.5 | 78.27 | 90.28 |
| Point Cloud - Text | 23.9 | 46.4 | 66.4 | |
| Ours | Image - Text | 96.00 | 99.80 | 99.90 |
| Point Cloud - Text | 81.40 | 95.80 | 98.90 |
- Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, november 1998
- Hang Su, Subhransu Maji, Evangelos Kalogerakis, Erik Learned-Miller (2024). Dataset: 3D-MNIST. https://doi.org/10.57702/z4z5be0h
- Minghua Liu, Ruoxi Shi, Kaiming Kuang, Yinhao Zhu, Xuanlin Li, Shizhong Han, Hong Cai, Fatih Porikli, Hao Su. OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understanding. NeurIPS 2023
- Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. ICML 2021
Incorporate audio and video modality into the same embedding space to get a good understanding all the various modalities.