Data Machina #175
Contrastive learning, CLOOB model, Google Flan-T5 model, evaluating LLMs, visualising federated learning, PyTorch symbolic, zero initialisation of neural nets, transfer learning with Flax
On the resurgence of Contrastive Learning . Contrastive learning has been around since 2019, when @FacebookAI team published MoCo. Now it’s resurging strongly.
Contrastive learning is a type of self-supervised visual representation learning. Its aim is to learn low-dim representations of data by contrasting between similar and dissimilar samples. This is a good, short Introduction to Contrastive Learning.
Transformers, text-to-Image, and diffusion models, etc have triggered a boom for ginormous datasets for pre-training humongous large models with billions of parameters. But labelling, curating large-scale datasets and supervised pre-training of large models is extremely expensive.
So the idea is how to leverage vast amounts of unlabelled data (e.g. images) with self-supervised learning to efficiently pre-train large models. A team @StanfordAI dives deeper into this in: Understanding Deep Learning Algos that Leverage Unlabelled Data: Contrastive Learning.
In this great post, Tony @USBerkeleyML elaborates on why contrastive learning for unlabelled data, and on the power of contrastive learning + supervised learning. See: Learn without Labels: A Summary of Recent Advances in Contrastive Learning
In that sense, learning unbiased models from biased data has become a hot research topic. Just days ago, a team @UniofTurin & France Telecom published a new framework for Unbiased Supervised Contrastive Learning.
A team @UniofCambridge,Language Tech Lab has been doing deep research on Large Language Models, and on whether these LLMs are really anisotropic. (i.e. exhibit properties with different values when measured in different situations.) Weeks ago they concluded that Contrastive Search Is What You Need For Neural Text Generation.
And subsequently, it took a few days for a team @Huggingface (obviously who else!) :-) to implement Generating Human-level Text with Contrastive Search in Transformers.
CLIP is a foundation model that is widely used in LLMs due to its SoTA performance. But just a week ago, a team @LIT AI Lab & IARAI has released CLOOB, a new contrastive learning method that overcomes the problems of CLIP, and achieves zero-shot learning SoTA on several large datasets.
Have a nice week.
10 Link-o-Troned
[Workshop] Emerging Research & Apps of Large Language Models
Speed up & Reduce Training Costs of Generative AI by 6x times
A Pythonista *Experience*
Scripting aRt
Deep & Other Learning Bits
ResearchDocs
ZerO Initialization: Initializing NNs with only Zeros & Ones
[Google] Scaling Transformer Inference Efficiency (paper & code)
Algorithmic Potpourri
El Robótico
data v-i-s-i-o-n-s
DataEng Wranglings
startups -> radar
ML Datasets & Stuff
Postscript, etc
Tips? Suggestions? Feedback? email Carlos
Curated by @ds_ldn in the middle of the night.