딥러닝 공부방

Prototypical Contrastive Learning of Unsupervised Representations Junnan Li, Pan Zhou, Caiming Xiong, Steven C.H Hoi, arXiv 2020 PDF, SSL By SeonghoonYu August 11th, 2021 Summary Clustering + NCE Loss 를 결합하여 self-supervised learning을 수행합니다. 기존의 contrastive learning의 문제점은 instance discrimination을 수행하기 때문에 비슷한 특징을 지닌 instance들을 negative로 정의하여 서로 밀어냈었습니다. 즉, low-level semantics를 포착하여 discriminative..

Unsupervised Representation Learning by Pre-diction Image Rotations Spyros Gidaris, Praveer Singh, Nikos Komodakis, arXiv 2018 PDF, SSL By SeonghoonYu August 4th, 2021 Summary The ConvNet is trained on the 4-way image classification task of recognizing one of the four image rotation(0, 90, 180, 270). The task of predicting rotation transformations provides a powerful surrogate supervision signel..

Invariant Information Clustering for Unsupervised Image Classification and Segmentation Xu Ji, Joao F.Henriques, Andrea Vedaldi, arXiv 2018 PDF, Clustering By SeonghoonYu July 30th, 2021 Summary This paper presents IIC model which acieves SOTA performance on Image clustering and Image segmentation by maximizing the mutual information between the original image and the transformed image from orig..

Big Self-Supervised Models are Strong Semi-Supervised Learners Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, Geoffrey Hinton arXiv 2020 PDF, SSL By SeonghoonYu July 26th, 2021 Summary This paper achieves SOTA performance by combine the pre-trained model on self-supervised learning with knowledge distilation. Namely, They show that using pre-trained model on SSL as teacher model fo..

Unsupervised Learning of Visual Representations using Videos Xiaolong Wang, Abhinav Gupta, arXiv 2015 PDF, Video By SeonghoonYu July 23th, 2021 Summary This paper use hundreds of thousands of unlabeled videos from the web to learn visual representations. They use the first frame and the last frame in same video as positive samples and a random frame from different video as negative sample. They ..

Learning Deep Representations by Mutual Information Estimation and Maximization R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, arXiv 2018 PDF, SSL By SeonghoonYu July 21th, 2021 Summary This paper updates model's parameters by maximizing mutial information between immediate feature maps and flattened last feature maps obtained from ConvNet. To do this, they use Jensen-Shannon divergence(..

Understanding the Behaviour of Contrastive Loss Feng Wang, Huaping Liu, arxiv 2020 PDF, Self-Supervised Learning By SeonghoonYu July 15th, 2021 Summary There exists a uniformity-tolerance dilemma in unsupervised contrastive learning. and the temporature plays a key role in controlling the local separation and global uniformity of embedding distribution. So the choice of temperature is important ..

안녕하세요, 오늘 읽은 논문은 An Empirical Study of Training Self-Supervised Vision Transformers 입니다. 해당 논문은 MoCov1/2보다 좋은 성능을 갖는 MoCov3을 제안하고, 이 MoCov3을 Vision Transfermers(ViT)에 적용하는 실험을 합니다. CNN 구조에 SSL을 적용하는 많은 연구가 이루어져 있지만 ViT 모델에는 어떻게 self-supervised learning을 적용해야하는지에 대해 많은 연구가 이루어지지 않았습니다. 저자는 다양한 실험을 통해 self-supervised ViT의 효과를 조사합니다. 또한 기존 self-supervised transformer 보다 좋은 성능을 보여줍니다. 실험을 위해 batch s..

안녕하세요, 오늘 읽은 논문은 Exploring Simple Siamese Represent 입니다. 두 representation을 비교하기 위해 Siamese network 구조는 자연스러운 선택입니다. Siamese network는 모든 출력값이 상수로 무너지는 문제점이 존재하는데, 이 문제점을 해결하기 위해 많은 노력이 존재합니다. 예를 들어, SimCLR은 different image(negative pair)는 밀어내고 positive pair는 서로 당기도록 학습합니다. 해당 논문에서는 Siamese network의 출력값이 상수로 무너지는 문제점을 해결하기 위하여 (1) same encoder, (2) prediction MLP, (3) stop-gradient 방법을 제안합니다. 다른 방..

안녕하세요, 이번 포스팅에서는 MoCov2를 Google Colab 환경에서 PyTorch로 구현해보도록 하겠습니다. 논문 리뷰와 전체 코드는 아래 주소에서 확인하실 수 있습니다. [논문 읽기] MoCov2(2020), Improved Baselines with Momentum Contrastive Learning 안녕하세요, 오늘 읽은 논문은 Improved Baselines with Mometum Contrastive Learning 입니다. 해당 논문은 MoCo v1에서 SimCLR의 두 가지 아이디어를 적용한 모델입니다. SimCLR은 contrastive learning.. deep-learning-study.tistory.com Seonghoon-Yu/MoCov2_Pytorch_tutorial..