Data Machina #181
The latest on LLMs. Thinking like transformers. An alt to backpropagation. Neural search frameworks. DL for math reasoning.Reversible Column Networks.
Language Models Firecrackers. As the festive season rushed in, AI researchers kept cracking on, shooting out new stuff non-stop. Here are my notes on LMs from the last 10 days:
@MSResearch et al: Language Models as Inductive Reasoners, in which, for the first time, researchers describe how well pretrained LMs can induce natural language rules from natural language facts.
@MetaAI: Adapting Language Models to Reasoning Tasks. A new method for assessing LMs reasoning ability on complex tasks that require reasoning skills to solve.
@UniofUIC: Reasoning in Large Language Models, 2022. A survey of papers, techniques, and resources on LMs reasoning.
@Deepmind & @CornellUni: Pretraining Without Attention. A new way to replicate BERT pretraining results without attention, that can be extended to long-form pre- training of 4096 tokens.
@Anthropic: Discovering LMs Behaviours with Model-Written Evaluations. The impact of RLHF and evidence on the behaviours that LMs exhibit.
@BigScience: An easy way to run 100B+ LMs without high-end GPUs. Up to 10x faster than offloading.
@PhilWang: A PyTorch implementation of RLHF on top of Google PaLM. ChatGPT but with PaLM.
@Huggingface: the latest, up-to-date, OLM version of BERT and RoBERTa transformers, trained on a cleaned October 2022 snapshot of Common Crawl & Wikipedia.
@Alpa: Free, unlimited text gen with a version of Meta AI OPT-175B. I’ve tried it and it seems way behind ChatGPT. Also some comedy, crazy replies.
@BigCode: A 1.1B parameter LM model for code generation in Python, Java & JavaScript. Checkout SantaCoder (try the demo + code)
Have a nice week, enjoy the festive season!
10 Link-o-Troned
A Pythonista *Experience*
Scripting aRt
Deep & Other Learning Bits
ResearchDocs
El Robótico
data v-i-s-i-o-n-s
DataEng Wranglings
AI startups -> radar
ML Datasets & Stuff
Postscript, etc
Tips? Suggestions? Feedback? email Carlos
Curated by @ds_ldn in the middle of the night.