Data Machina #196

AutoGPTs. Autonomous LLM Agents. Long Learning. Smol GPT in your CPU. StackLLaMA. RLAIF. Berkeley AI Koala. Google ViT 22B Params. Meta AI Segment Anything. Auto-GPT. babyagi. JARVIS

AutoGPTs: Autonomous LLM-Based Agents. There is a whole new, emerging breed of powerful, autonomous AI agents based on LLMs. I guess if we had 10 AI experts in a room, they’d never agree on what exactly is an autonomous LLM-based agent. But just to simplify matters, let’s call these agents AutoGPTs. Let me explain:

The Core Idea: Auto Prompting + LLM Patterns. First, you provide the AutoGPT with an identity/role, a task to complete with goals and objectives, and also context and background on what to accomplish. You may also offer rewards to the agent if it completes the task. All this is achieved with sophisticated prompt engineering.

Then the AutoGPT follows different LLM patterns, and using auto prompting or self-prompting, it finds its way to complete the task autonomously. There are many patterns, but I’d say these three LLM patterns are the foundations of AutoGPTs:

1. Chain-of-Thought (CoT) Prompting (Jan 2022): First introduced by Google Brain in Chain-of-Thought Prompting Elicits Reasoning in LLMs. Sufficenty large LMs can generate chains of thought if demonstrations of chain-of-thought reasoning are provided in the exemplars for few-shot prompting. This paper unleashed a research trend on CoT reflected in these 40 papers.

2. MRKL (pronounced “miracle”) Sytems (May 2022): First introduced by AI21 Labs in MRKL Systems: A modular, neuro-symbolic architecture that combines LLMs, external knowledge sources and discrete reasoning. MRKL consists of an extendable set of ’modular experts,’ and a ‘router’ that routes every incoming NL input to a module that can best respond to the input.

3. The ReAct Pattern (Oct 2022): First introduced by Google Brain in ReAct: Synergizing Reasoning and Acting in Language Models (v3, Mar 2023.) ReAct is a pattern by which the model generates both reasoning traces and task-specific actions in a synergic, synchronised manner. The reasoning traces help the model induce, track, and update action plans, while actions allow it to interface with external sources to gather additional information.

Now, let’s get more practical. Let me show you 6 amazing AutoGPTs:

babyagi: This is a “plan, execute” autonomous agent that leverages GPT-4, Pinecone vector search, and LangChainAI framework to autonomously create and perform tasks based on an objective. You can read more in the paper: A Task-driven Autonomous Agent by @yoheinakajima

Microsoft HuggingGPT [a.k.a JARVIS] (repo, paper): An autonomous agent that acts as a controller to solve tasks. Given a task, JARVIS first performs task planning. Second, it selects the best model for completing the task from a pool of expert models in Hugginface. Then it executes the task and generates the response.

Auto-GPT (repo, demo): Watch the demo. Absolutely mind-blowing! An open-source app powered by GPT-4. The app autonomously develops and manages businesses to increase net worth. IMO This app perhaps is the prototypical AutoGPT, implementing the best of CoT, ReAct, MRKL, and LLMs augmentation and composability.

AutoGPT Website (repo, demo): Inspired by Auto-GPT. Set up the initial role and goals for your AI buddy, without human's supervision, it will automatically leverage all of the resources it has to achieve your goal.

ChatArena (repo, demo): An agent that autonomously facilitates communication and collaboration between multiple LLMs, player agents, and humans to play games. Try the online demo. Brilliant!

Here’s another awesome example of AutoGPTs. Watch @mckaywrigley using only his voice and his GPT-4 coding assistant to build and deploy a fully working app with Vercel, Supabase, Next.js. Without touching his keyboard. This is insane indeed!

Prompt engineering is here to stay. And AutoGPTs are the next wave in AI innovation. @LangChain just raised $10 M. I guess because -like LlamaIndex- it enables the development of AutoGPTs. They recently published a blog post on building custom LLM-based agents.

Having a lazy Sunday. Here’s some AI entertainment for you:

• The fun test: Test the limits of Cheetah (a Whisper & GPT-based app) for passing remote s/w engineering interviews

• The long talk: Sebastian, Sr. Principal ML Researcher @MSR, elaborates on his journey researching Sparks of AGI: early experiments with GPT-4

• The long discussion : Yann Le Cun & Andrew Ng discuss why the proposal of a 6-month moratorium on AI is a bad idea

Enjoy the Easter Break. Have a nice week.

10 Link-o-Troned

1. Eight Things to Know about LLMs

2. Standford ML Research: From Deep to Long Learning?

3. Edge AI Just Got Faster. How We Did It with llama.cpp

4. Software 3.0: Building a Neural Computer

5. An Intro to Zero-Knowledge ML (ZKML)

6. Boosting Tabular Data Predictions with LLMs

7. The Tale of Bloom Embeddings and Unseen Entities

8. A Recipe for Training Good Generative Models

9. Medical Chat : Advanced AI Chatbot for Health Professionals

10. Berkeley Koala - New LLaMA-based Chatbot for Academic Research

---

the ML Pythonista

1. A Developer-First Guide to LLM APIs (Mar 2023)

2. FLAML - A Fast Library for AutoML (Supports GPT Models)

3. Smol GPT: A Multimodal Text Gen Model on your CPU, 14GB RAM!

the ML codeR

1. Tips for Using ChatGPT to Learn R

2. askgpt: A Chat Interface that Helps You to Learn R

3. Self-Guided Learning through a Wordle Guess Generator

Deep & Other Learning Bits

1. StackLLaMA: A Hands-on Guide to Train LLaMA with RLHF

2. Intro to Imitation Learning from Language Feedback

3. Towards Reinforcement Learning with AI Feedback (RLAIF)

AI/ DL ResearchDocs

1. Meta AI - Segment Anything (paper, demo, code, blog)

2. Google - Scaling Vision Transformers to 22 Billion Params

3. Instruction Tuning with GPT-4 (paper, code, data)

El Robótico

1. BerkeleyAIR: Autonomous Vehicles Interactive Fleet Learning

2. DribbleBot & Soccer: Dynamic Legged Manipulation in the Wild

3. ULIP: A Unified 3D Model for Language, Images & Point Cloud

data v-i-s-i-o-n-s

1. How I Created a Dataviz with 0 Coding Skills with GPT-4

2. A Visual Analysis of Brazilian Amazon Fire Spots

3. Plants & People: Interactive Touch Maps to Tell Stories

MLOps Untangled

1. MLOps and MLflops

2. A Git-native, Open Source Tool for Simplifying MLOps

3. Build Reliable ML Pipelines with Continuous Integration

AI startups -> radar

1. GradientJ - Build LLM Apps in Minutes

2. Sanctuary AI - Building Machines that Think Like People

3. AWS Generative AI Startup Accelerator

ML Datasets & Stuff

1. Meta AI - Segment Anything 1 Billion (SA-1B) Dataset

2. Graph Deep Learning Datasets

3. One Billion Word Benchmark Dataset for LLMs

Postscript, etc

Tips? Suggestions? Feedback? email Carlos

Curated by @ds_ldn in the middle of the night.