Data Machina #186

LLMs: The latest. Stanford Advances in Foundation Models. Neuro-Symbolic LLM. TransformerXL explained. Trustable & auto-retrainable ML models. Open problems in Deep Learning.

[Maybe] The Latest in Language Models? A Tour de Force. Let me tell you: I don’t know about you but it's bit challenging for me to keep up with the latest in LLMs. I use this handy, annotated, Large Language Models spreadsheet. It’s not fully comprehensive but it’s cool.

Open AI released InstructGPT, a new LM that is better than GPT-3 at following English instructions. More here: Aligning Language Models to Follow Instructions

StandfordNLP published a new LM framework for composing search and LMs with up to 120% gains over GPT-3.5. TL;DR: Use imperative code and forget about prompt engineering. Link: Demonstrate–Search–Predict (DSP.)

CarperAI released a set of diff models for editing/ generating code. These diff models are autoregressive LM models that were trained on millions of GitHub commits. Checkout: Diff Models – A New Way to Edit Code

Meta AI keeps delivering amazing LLM research. They just released: 1) A new model for generating high-fidelity music from text descriptions: MusicLM (paper, demo) and 2) A new method for generating 3D dynamic scenes from text descriptions: MAV3D Make-A-Video3D (paper, demo)

The Inverse Scaling Prize awarded 7 prizes to researchers who identified important tasks on which LMs perform worse the larger they are (“inverse scaling”.)

Speaking of scaling LMs, Jason @GoogleBrain gave a great talk on why Scaling Unlocks Emergent Abilities in Language Models (slides.)

Many professionals in media, entertainment & arts are all up in arms. They feel threatened by generative AI LLMs. This has triggered some researchers to find a way to algorithmically mark and detect text generated by AI. Paper: A Watermark for Large Language Models

LangChain is becoming the de facto library for building LM apps. This is a great post on Getting Started with LLMs using LangChain.

If you are interested in building apps with LangChain, @lostintangent developed a one-click dev environment for building LLM apps with LangChain.

If you need inspiration and examples on agents and chains for building LM apps, checkout the very new LangChain Hub.

Researchers @CarnegieMellonUni ask: Why Do Nearest Neighbor Language Models Work? They show that retrieval-augmented, kNN-LMs perform better than standard parametric LMs (Jan 2023).

LMs have many limitations. Neuro-Symbolic AI to the rescue? Oblivious to the AI Marketing Storm from the Tech Goliaths, a tiny little group of researchers @LIT_AI_Lab in Austria, built SymbolicAI API: a compositional, neuro-symbolic framework that combines LLMs with Differentiable Programming. This framework extends & augments LLMs with magic powers. And it’s beautifully documented. Awesome!

The v4.0 of Talking About Large Language Models (Jan 25, 2023) is out. It’s a great paper by Murray @ImperialCollege.

Remember ELIZA, the very 1st AI therapist? This startup has developed Serena, a chatbot that uses LLMs for Mental Therapy

Feels miserable outside, like in London? Here are some indoors Language Models entertainment suggestions:

• Amusing: Ask anything to GPT, and an alive portrait replies to your query

• Bookworm fun: Pick a book from a library to talk to. The library is a bit tailored for those energetic Silicon Valley hustlers. But it’s OK

• Pretty amazing: Instruct Pix2Pix - Load an image, write some text to edit the image on the fly

Have a nice week.

10 Link-o-Troned

1. [Free] Stanford Foundation Models Seminar (2023)

2. A Review of the Techniques Behind ChatGPT

3. Just Know Stuff. Or, How to Achieve Success in a ML PhD

4. Replacing a SQL Analyst with 26 Recursive GPT Prompts

5. How to Build Trust in ML Models, The Sane Way

6. A First Look… ChatGPT + WolframAlpha Combined

7. AI or Not - New Hugging Face Competitions (Like Kaggle)

8. Auto Retraining for ML Models: Tips & Lessons Learned

9. [Free course] Deepmind - Math for ML & Data Science

10. AI Product Index - Discover Awesome AI-Powered Products

---

the ML Pythonista

1. The Innovations of TransformerXL Explained (PyTorch code)

2. Writing a Tokenizer with ChatGPT & Python

3. Train Transformer LMs with Reinforcement Learning

the ML codeR

1. Fine-tuning Transformers for Text Data from within R

2. Googly++: Win Probability Using DL & Player Embeddings

3. SHAP + XGBoost + Tidymodels = LOVE

Deep & Other Learning Bits

1. Open Problems in Applied Deep Learning (Jan 2023)

2. Stanford CS234 Topics in Advances in Foundation Models

3. Code / Examples for: A Survey on Active Learning SoTA

AI/ DL ResearchDocs

1. Return of the GAN: SoTA in Text2Image with StyleGAN-T

2. Imitating Human Behaviour with Diffusion Models

3. DetectGPT: Zero-Shot Machine Generated Text Detection

El Robótico

1. Modelling a Robot with Generative AI Tools

2. [Free e-book] Programming Cognitive Robots

3. [Free course] Probabilistic, Autonomous Mobile Robotics

data v-i-s-i-o-n-s

1. Reuters: On the Brink - The Extinction of 1 Million Species

2. Where are Piero della Francesca’s Masterpieces?

3. [iPynb] A Map of Arts and Culture in London

DataEng Wranglings

1. 8 Alternatives to pandas for Processing Large Datasets

2. Snowflake is a Table Game, Google BigQuery is a Casino

3. Root Cause Analysis, ELT & Bayesian Networks

AI startups -> radar

1. Moveworks - Conversational AI for Solving Issues at Work

2. Monster API - Unused Crypto Compute for Generative AI

3. Aurora - AI for Selling & Designing Solar Projects

ML Datasets & Stuff

1. A Comprehensive English Football League DB (1888-2022)

2. The MusicCaps Dataset: 5,521 labelled music examples

3. BLASTNet: A Network of Datasets for Scientific Big Data

Postscript, etc

Tips? Suggestions? Feedback? email Carlos

Curated by @ds_ldn in the middle of the night.