Mistral NeMo 12B SOTA Model. A state-of-the-art 12B, multi-lingual model with 128k context length, built in collaboration with NVIDIA, and released under the Apache 2.0 license. The weights for the base and the instruct model versions are hosted in Hugging Face. Some community members have raised questions about the accuracy of its reported benchmarks, particularly in comparison to models like Llama 3 8B.

Stanford TextGrad: Automatic "Differentiation" via Text. A new, powerful framework for performing automatic “differentiation” via text. TextGrad backpropagates textual feedback provided by LLMs to improve individual components of a compound AI system. TexGrad improves zero-shot capabilities of LLMs significantly across many applications.

Tencent Patch-level Training. This paper introduces a new technique that address the issue of LLM token-training: the huge computational costs due to processing 10s of billions of tokens. Patch-level training reduces the sequence length by compressing multiple tokens into a single patch, and then the model is trained to predict the next patch. Patch-level training can reduce overall computational costs to 0.5×,

[free] Computer Vision Course. This new, Hugging Face community-driven course covers everything from the basics to the latest advancements in computer vision. It’s structured to include various foundational topics, making it friendly and accessible for everyone.

Stanford STORM Long-form, Wikipedia-like generative writing and Q&A. A new, open-source, generative writing system for writing grounded and organised long-form articles from scratch, with comparable breadth and depth to Wikipedia pages. STORM models the pre-writing stage by (1) discovering diverse perspectives in researching a given topic, (2) simulating conversations where writers carrying different perspectives pose questions to a topic expert grounded on trusted Internet sources, (3) curating the collected information to create an outline.

Have a nice week.

Subscribe now

10 Link-o-Troned

1. State of Open AI, Jul 2024

2. State of Text2SQL, Jul 2024

3. Manipulating Chess-GPT's World Model

4. The Winds of AI Winter. The Vibes Have Shifted

5. 43 Questionable Practices in Machine Learning

6. TinyML: Why the Future of ML is Tiny and Bright

7. Beating Proprietary AI Models with a Quick Fine-Tune

8. The AI Song Contest 2024: Calling for Applications

9. A List of 450 Real-World ML Systems from 100+ Companies

10. From Knowledge Graphs to GraphRAG: Advanced Intelligent Data Retrieval

Share Data Machina with your friends

the ML Pythonista

1. Composio - Production Ready Toolset for AI Agents

2. All-in-one Embeddings DB for Semantic Search, LLM orchestration

3. MSFT Olive - A Model Optimisation Tool with Industry-leading Techniques

Deep & Other Learning Bits

1. Street Fighting Transformers

2. Kernel Tricks - Convolutional Kernel Networks

3. [free tutorial] The Physics of Language Models

AI/ DL ResearchDocs

1. E5-V: Universal Embeddings with Multimodal LLMs (paper, repo)

2. IMAGDressing-v1 : Customisable Virtual Dressing (paper, demo, repo)

3. Fine-Tuning and Prompt Optimisation: 2 Great Steps that Work Better Together

MLOps Untangled

1. Different Approaches to Model Compression

2. How We Built a +2K Features Store with DBT, Snowflake & MySQL

3. Implementing CI/CD Pipelines with GitHub Actions for MLOps

ML Datasets & Stuff

1. OAK - A 500M+ Token Dataset Generated by SOTA LLMs

2. PetFace: A Large-Scale Dataset for Animal Identification

3. DocMatix - A Massive Dataset for Document Visual Question Answering

Postscript, etc

Tips? Suggestions? Feedback? email Carlos

Curated by @alg0agent in the middle of the night.

Re-programming LLMs for time-series modelling. This a great post about how researchers are trying to align the information gap between time series and natural language from every perspective of training a LLM. Re-programming a LLM for time series modelling is similar to fine-tuning it for a specific domain. This involves several key steps like: tokenisation, base model selection, prompt engineering, and defining the training paradigm. Blogpost: Time Series Are Not That Different for LLMs.

JPMorgan - Lightweight SOTA TS classification with an embedding model. This paper introduces LETS-C, a new method that addresses the issue of fine-tuning with millions of trainable parameters. The method utilises a language embedding model to embed time series and then pair the embeddings with a simple classification head composed of CNNs and MLP. LETS-C outperforms the current SOTA in TS classification accuracy but also offers a lightweight solution, using only 14.5% of the trainable parameters on average compared to the SOTA model. Paper: LETS-C: Leveraging Language Embedding for Time Series Classification.

Mercedes-Benz - Automatic TS anomaly detection with a new tVAE. This paper introduces a temporal variational autoencoder (TeVAE) that can detect anomalies with minimal false positives when trained on unlabelled data. The approach also avoids the bypass phenomenon and introduces a new method to remap individual windows to a continuous time series. Using real-world industrial data, TeVAE detects 65% of the anomalies, and only 6% are wrongly classified. Paper: TeVAE: A Variational Autoencoder Approach for Discrete Online Anomaly Detection in Variable-state Multivariate Time-series Data.

Datadog - SOTA foundation model with Mixture-of-Students for observability TS data. This paper introduces Toto, the first general-purpose TS forecasting foundation model specialised in observability TS data, typically used in electricity and weather forecasting. The model was trained on a dataset of one trillion time series data points, the largest ever published. Toto outperforms existing TS foundation models on observability data, and also excels at general-purpose forecasting tasks, achieving SOTA zero-shot performance on multiple open benchmark datasets. Paper: Toto: Time Series Optimized Transformer for Observability

Tesco - A transformer for tokenised TS classification in price optimisation. This paper introduces a transformer architecture for time series forecasting applied to the real-world problem of price optimisation in a large retailer. The paper introduces several innovations in terms of differentiated time series patching and multiple-resolution module for time-varying known variables. Based on experiments with real-world data, the model outperforms existing in-house models and other deep learning architectures. Paper: Multiple-Resolution Tokenization for Time Series Forecasting with an Application to Pricing.

A Vision-Language Model for SOTA zero-shot TS forecasting. This paper introduces ViTime, a novel VLM foundation model for TSF forecasting. The model transforms numerical time series into binary images, converting numerical temporal correlations into binary pixel spatial correlations. ViTime overcomes the limitations of numerical time series data fitting by utilising visual data processing paradigms. ViTime achieved SOTA zero-shot performance, even surpassing the best individually trained supervised models in some situations. Paper & repo: ViTime: A Visual Intelligence-Based Foundation Model for Time Series Forecasting.

T5 (Text-To-Text-Transfer-Transformer) for short-term stock price prediction. This paper introduces TimeS, a novel method for short-term excitement prediction in stock price time series, that uses the T5 transformer. T5 uses stock, events, and sentiment data as inputs to forecast price change indicators for the subsequent n time intervals, which are then employed to determine stock states. Then the time series are updated using price amplification values derived from these stock states. Paper: Text2TimeSeries: Enhancing Financial Forecasting through Time Series Prediction Updates with Event-Driven Insights from LLMs.

Have a nice week.

Subscribe now

10 Link-o-Troned

1. Meet the AI Agent Engineer

2. My Year as an Independent AI Researcher

3. Why You Don't Need an Agentic Architecture

4. Chatbot Battles + Synth Data = Arena Learning

5. A Visual Walkthrough of the AlphaFold3 Model

6. Data Science at The Precipice & The Singularity

7. We Need Super-Knowledge Before Super-Intelligence

8. Someone is Wrong on the Internet (AGI Doom Edition)

9. A List of 500+ AI / DL/ ML Projects with Code

10. Lynx: SOTA Open Source Hallucination Detection Model

Share Data Machina with your friends

the ML Pythonista

1. How to Build a Generative UI Financial Agent

2. [tutorial] How to: GraphRAG with Neo4j Visualisation Locally

3. How to Fine-tune Llama-3 and Get 90% of GPT-4’s Performance

Deep & Other Learning Bits

1. [free OpenAI course] Spinning Up in Deep RL

2. Brief Intro to Adaptive In-Context Learning

3. [free ebook] Physics-based Deep Learning (PBDL)

AI/ DL ResearchDocs

1. Teaching Transformers Causal Reasoning

2. Memory3: Language Modelling with Explicit Memory

3. Internet of AI Agents: Multi-Agent Collaboration (paper, repo)

MLOps Untangled

1. London Cycling: An E2E, Toy MLOps Project

2. The Taxonomy for Data Transformations in AI Systems

3. [free workshop] Intro to AI Pipelines: Build Reliable ML Workflows

ML Datasets & Stuff

1. How Synthetic Data Won The AI Math Olympiad 2024

2. Emilia: An Extensive, Multilingual Dataset for Large-Scale Speech Generation

3. SkunkworksAI: A Synth Dataset of Reasoning Chains Across a Variety of Tasks

Postscript, etc

Tips? Suggestions? Feedback? email Carlos

Curated by @alg0agent in the middle of the night.

Start here: The best intro to VLMs, 2024. Probably - by far- the best introduction to VLMs. A mega paper in the format of a pdf book, published by Meta AI, NYU, MILA, MIT and several other unis. Paper: An Introduction to Vision-Language Modelling.

Recommended: Tutorial on VLMs CVPR June 2024. This tutorial covers the latest approaches and theories on 1) Learning VLMs for multimodal understanding and generation 2) Benchmarking and evaluating VLMs and 3) Agents and other Advanced Systems based on Vision Foundation Model. All the slides and video sessions from the tutorial here: Recent Advances in Vision Foundation Models.

Trend: Small but powerful VLMs. This an unstoppable trend: we’re starting to see some amazing, small -even tiny- VLMs that are very powerful. Some are even full open-source, or at least open/open weights, and in some cases achieve SOTA in several benchmarks. Here are 5 small VLMs you should know:

LLaVA-Next (interleaved). Key feature: An image-text interleaved format that unifies multi-image, video, and 3D tasks in one model with excellent performance. Released in 0.5b, 7b, and 7b-dpo versions, it achieves SOTA in many benchmarks, and leads the open source VLMs. Checkout the repo, demo, and blog here: LLaVA-NeXT: Open Large Multimodal Models.

PaliGemma. Key feature: It takes both images and text as inputs and can answer questions about images with detail and context efficiently. PaliGemma is an open, lightweight VLM based on the more powerful PaLI-3 family of VLMs from Google. You can get the base, pre-trained PaliGemma and PaliGemma-FT fine-tuned version. Sonu posted a nice in-depth review of PaliGemma and how to fine-tune it.

Phi-3 Vision. Key feature: Very strong multi-modal, high quality reasoning trained on dense text and image data. Phi-3 Vision is based on the latest family of Phi-3 models released by Microsoft. You can get the latest model version released, Phi-3-Vision-128K-Instruct, which achieves SOTA in several benchmarks with quite a long context. Sam recently published a nice review, testing the model with some real world cases.

Florence-2. Key feature: a unified, prompt-based representation for a variety of vision and vision-language tasks. Published by Microsoft, Florence-2 was designed to take text-prompts as task instructions and generate desirable results in text forms, whether it be captioning, object detection, grounding or segmentation. Paper: Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks. The team at Roboflow just released a in-depth tutorial on fine-tuning Florence-2.

InternLM-XComposer 2.5. Key feature: Ultra-high resolution & fine-gained image and video understanding in a small model with long context. This VLM supports 96K long-context input and output. It achieves GPT-4V level capabilities in many benchmarks with just a 7B LLM backend. Repo, demo and technical report: InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output. Two days ago, Fahd published a good review on this model and some of its installation pains.

NVIDIA VLMs playground. NVIDIA just released a nice demo-playground for several of these VLMs above. Try it here: NVIDA VLMs.

A new multi-modal model benchmark. Reflecting the trend in VLMs, a week ago or so, LMSYS.org -the well known benchmark for AI Models- introduced a new multi-model benchmark that covers vision-language/ multi-modal capabilities. Still early days, but most of the multi-modal models covered are closed, proprietary models. Blogpost: The Multi-Modal Arena is here.

Have a nice week.

Subscribe now

10 Link-o-Troned

1. My Fine-tuned Models Beat GPT-4

2. Machine Learning in Trackmania Video Game

3. Classical PCA Through a Latent Variable Lens

4. Playing Probabilistic AI Chess (game board, blogpost)

5. Databricks Mosaic AI Agent Framework & Agent Evaluation

6. What If We Recaption Billions of Web Images with LLaMA-3?

7. [free book] The Orange ML Book: Essentials for Tabular Data

8. AutoGluon ML Beats 99% of Data Scientists with 4 Lines of Code

9. Berkeley AI Hackathon 2024: All the Winners & Karpathy's Keynote

10. Gradient Boosting like LLMs: Zero-shot Forecasting with ML models

Share Data Machina with your friends

the ML Pythonista

1. Training Mixture-of-Experts at Scale with PyTorch

2. MS GraphRAG - Graph-based Modular RAG (repo, paper)

3. MindsDB - A Platform for Customising AI from Real-time Enterprise Data

Deep & Other Learning Bits

1. [tutorial] What is Max Out in Deep Learning?

2. Image Self Supervised Learning (SSL) on a Shoestring

3. [free ebook] Understanding DL, 02/07/24 (+68 notebooks)

AI/ DL ResearchDocs

1. SMAT: Meta-Tuning for Few-shot Generalisation with Sparse MoEs

2. A Benchmark of Tabular ML in-the-Wild with Real-world Datasets

3. MIT Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion

MLOps Untangled

1. BlackDagger- DAG-based Automation for MLOps

2. MLOps Pipeline: Churn Prediction with Kubeflow & KServe

3. Best in Open Source MLOps: Platforms, Frameworks & Tools

ML Datasets & Stuff

1. PERSONA HUB – 1 billion Personas Auto-Synth Generated

2. Megalith-10m Dataset: 10 Million Links to Flickr Images

3. OpenVid 1M: A Large-Scale Dataset for HQ Text-to-Video Generation

Postscript, etc

Tips? Suggestions? Feedback? email Carlos

Curated by @ds_ldn in the middle of the night.

Prompt engineering is like Marmite. Many of my hardcore s/w engineers friends absolutely hate prompt engineering. They dismiss it as “random, unreliable pseudo-programming using English language.” But some love it like Marmite. Regardless, if you are an AI engineer, researcher, dev or business analyst, and you want to stay “relevant in modern AI,” you just can’t ignore LLM/LMM-based R&D. And that involves prompt engineering. So this is my advice to my sceptic friends:

Understand the fundamentals and mindset of prompt engineering. A great post by Eugene. His main point: prompt engineering is about conditioning the probabilistic model to generate the desired outputs; and that requires to follow a systematic, thorough, structured approach. Blogpost: Prompting Fundamentals and How to Apply them Effectively.

Sign up for an advanced prompt engineering course, be picky. There are lots of prompting courses out there. But there aren’t many comprehensive, hands-on, courses on advanced prompt engineering. This is a good one: The Complete Prompt Engineering for AI Bootcamp (June 2024).

And Elvis, the brains behind the great Prompt Engineering Guide (although not fully up to date), recently released an advanced prompt engineering course. Also checkout the latest Anthropic's Prompt Engineering Interactive Tutorial, which is free and great.

Learn the latest prompting techniques. There are ca. 60 or so well known prompting techniques and patterns. New techniques appear ever week! Written by researchers from OpenAI, Microsoft, Stanford et al., the Prompt Report (June 2024) is one of the most comprehensive surveys on prompting techniques. If you don’t have time to read the 75 pages report, Daniel recently wrote a nice blog with his summary and notes on all prompting techniques described in the Prompt Report.

Use specialised tools to orchestrate prompting pipelines. Prompt engineering is a deeply iterative process. This is a good post on a s/w developer reflecting on the prompting iterations and sharing 10 tips just to build a basic LLM-based app. Orchestrating a prompt engineering pipeline from experimentation to production prompts is hard, and requires specialised tools… Questions: How do you test & evaluate prompts? Where do you store your test & prod prompts? How do you do prompting version control? There are lots of prompting tools, but here are a few interesting ones that come to mind:

• Prompt Engineering Tool. A few days ago, Tecknium -the brains behind the amazing Nous-Hermes-2-Mixtral-8x7B model- just released a tool that enables you to: 1) test prompts across multiple LLM simultaneously 2) save and load prompt templates, 3) manage variables for dynamic prompt generation, and 3) compare prompt outputs from different models side-by-side

• Code2Prompt is a powerful command-line tool that generates comprehensive prompts from codebases

• Promtptools is a set of free, open-source tools for testing and experimenting with prompts

• Promptmetheus is a prompt engineering IDE that enables you to compose prompts without code, test prompts with different models, an optimise prompts for reliability and cost

• OpenPrompt is a library built upon PyTorch that provides a standard, flexible and extensible framework to deploy the prompt-learning pipeline.

• promptfoo a tool for building reliable prompts, evaluating them with caching, concurrency, and live reloading, and scoring outputs automatically

Learn the latest jailbreak prompting techniques. The idea here is that you learn how to protect your LLM-based app against adversarial jailbreak attacks by learning how these jailbreaks are crafted with sophisticated English prompts. The venerable Pliny the Prompter @elder_plinius maintains a repo with proven, working jailbreak prompts for ALL major foundation models from OpenAI, Cohere, Google, Microsoft and Anthropic, including “God Mode” system prompts.

Learn prompting techniques that decrease costs and improve performance. You need to find a way to get accurate prompts without impacting cost, speed, and performance. Jan wrote about 5 techniques to streamline prompts, while decreasing costs without sacrificing accuracy. Blogpost: Streamline Your Prompts to Decrease LLM Costs and Latency. And Justin, recently posted about how to use Prompt Decomposition to avoid high costs, low latency and low accuracy.

Get into Automated Prompt Optimisation (APO) soon. Once you realise that developing reliable, production LLM-based apps requires solid prompt engineering skills, tools and methods, it’s time to get into automating & optimising prompts. There is a lot of emerging research on this, and several new tools are appearing in the landscape:

• The Automatic Prompt Engineer (APE) and DeepMind ORPO were one of the first papers on methods for automatic prompt generation, selection and optimisation

• AutoPrompt is a prompt optimisation framework designed to enhance and perfect your prompts for real-world use cases. It automatically generates high-quality, detailed prompts tailored to user intentions, and employs refinement (calibration) process

• Automating Prompt Engineering with DSPy and Haystack. An interesting post on how to use DSPy to automate prompt engineering. DSPy separates the flow of your program from the prompts and weights at each step, while it runs optimisers, that tune the prompts and/or the weights of your LM calls

• Cracking the Code: Automated Prompt Optimisation. The team at Martian, surveyed several companies using advanced prompting techniques. They argue that Automated Prompt Optimisation (APO) is the way to resolve key issues like model variability, drift, and “secret prompt handshakes”. And they also share innovative techniques used to address these challenges, including LLM observers, prompt co-pilots, and human-in-the-loop feedback systems to refine prompts

• Optimising Instructions and Demonstrations for Multi-Stage Language Model. A new paper from researchers at Berkeley, Stanford et al. introducing MIPRO, a new prompt optimisation method for LM programs. MIPRO -implemented in DSPy- outperforms baselines on five of six diverse LM programs using a best-in-class open-source model like Llama-3-8B.

Have a nice week.

Subscribe now

10 Link-o-Troned

1. AI Scaling Myths

2. Thoughts on AGI World Models

3. Musings on What’s an AI Agent?

4. Musings on Building a GenAI Product at LinkedIn

5. AI Code Design Should Come Before AI Code-Gen

6. Short Musings on AI Engineering & "Failed AI Projects"

7. What are Image-Text-to-Text Models? An Intro

8. How to Build the World’s Fastest AI Voice Bot

9. Visualising The Magic of the Transformers Architecture

10. Uber’s ML Platform Evolution: From Predictive to Generative AI

Share Data Machina with your friends

the ML Pythonista

1. N-BEATS: 1st DL Model that Works for Time Series Forecasting

2. Maestro - A Framework to Orchestrate Subagents with Any LLM

3. RAGFlow: An Open-source RAG Engine for Deep Doc Understanding

Deep & Other Learning Bits

1. Why You (Currently) Don’t Need DL for Time Series Forecasting

2. A Deep Dive into JEPA: Yann Lecun’s Alternative to Transformers

3. Intro to Matryoshka Representation Learning with OpenAI & Weviate

AI/ DL ResearchDocs

1. LongRAG: Enhancing RAG with Long-context LLMs

2. Apple 4M: An Any-to-Any Vision Model for Tens of Tasks & Modalities

3. Cambrian-1: A Family of Open Vision-Centric Multimodal LLMs (paper, repo)

MLOps Untangled

1. On Day 1 vs. Day 2 Problems & ML vs. Engineering Problems

2. Kubeflow MLOps Pipeline v2: mlflow, Seldon Core (Parts 1-4)

3. How to Track LangChain Experiments with Weights & Biases

ML Datasets & Stuff

1. How We Created the FineWeb 15-Trillion Token Dataset

2. MINT-1T: A Open-source Multimodal Dataset with 1 Trillion Tokens

3. How to Think about Creating a Dataset for Finetuning Evaluation

Postscript, etc

Tips? Suggestions? Feedback? email Carlos

Curated by @ds_ldn in the middle of the night.

The latest version of DeepSeek-Coder is now the top open model for coding. DeepSeek-Coder-v2 is an open-source Mixture-of-Experts (MoE) code language model that achieves performance comparable to GPT4-Turbo in code-specific tasks. Repo & paper: DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence.

The new Hermes merge model beats Llama-3 70B. Hermes 2 Theta Llama-3 70B is an Instruct, fine-tuned model that merges Hermes 2 Pro and Meta's Llama-3. The merge model matches GPT-4 on MT Bench and surpasses Llama-3 70B Instruct in all benchmarks. Read more here: Hermes 2 Theta Llama-3 70B.

This new method generates high-quality 3Ds from one single image. A novel image-to-3D framework for efficiently generating high-quality 3D meshes from single-view images, featuring state-of-the-art generation fidelity and strong generalisability. Paper, demo & code here> Unique3D: High-Quality and Efficient 3D Mesh Generation from a Single Image.

This is the first scalable, reliable method for auto generating instruction-following training data. The Qwen team at Alibaba introduced AutoIF a new approach that is set to revolutionise, instruction following and the way data is generated for Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF). Paper: Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Models.

A new, open source, large-scale instruct dataset to lower barriers of SFT. Building large scale, high-quality instruct datasets is very expensive and only available to Tech Giants. To break that barrier, the BAAI just announced Infinity Instruct, a project that aims to develop open, large-scale, high-quality instruction datasets. Checkout: Infinity Instruct Dataset Project.

A new method that extends the length of GenAI videos. Most existing GenAI video models -including Open AI SORA- can only generate short video clips. Researchers at Alibaba just introduced ExVideo, a novel post-tuning method for video synthesis, that allows to produce longer videos up to 128 frames at a lower cost. Paper, demo, tech report: ExVideo: Extending Video- Enhancing the capability of video generation models.

MSFT open-sources a new vision foundation model that is small and powerful. Microsoft just introduced Florence-2, a VLM that has strong zero-shot and fine-tuning capabilities across all vision tasks. Despite its small size, it pars with models many times larger. The power of the model is not based on the architecture but in the large-scale FLD-5B training dataset. Blog review, paper, and notebooks here: Florence-2: Open Source Vision Foundation Model by Microsoft.

MSFT introduces a new method that enhances base model pre-training. This new method called Instruction Pre-Training 1) enhances generalisation, 2) improves pre-training efficiency, and 3) improves tasks performance. Paper and models: Instruction Pre-Training: Language Models are Supervised Multitask Learners.

Antrophic intros Claude 3.5 Sonnet… people swear it’s the best model in the planet. I’ve been using Claude for a while and really love it. The new Sonnet 3.5 is free and super powerful. The vision capabilities look impressive, as well as the agentic coding capabilities including unit testing. I really like the Artifacts dedicated window alongside the chat, sort of a dynamic workspace. The team at Vellum compared Claude 3.5 Sonnet vs. GPT-4o. In parallel, the awesome Jeremy, just introduced Claudette, a new friend that makes Claude 3.5 Sonnet even nicer. That is cool stuff.

So much AI stuff happening! Have a nice week.

Subscribe now

10 Link-o-Troned

1. Mapping Interpretable Features in a Language Latent Space

2. [mega stream] Lessons from a Year of Building with LLMs

3. I’m Using AI to Automatically Drop Hats onto New Yorkers

4. Why We No Longer Use LangChain for Building AI Agents

5. [a new perspective] How to Hire AI Engineers

6. How to Get the Best Results from Stable Diffusion 3

7. Hyper-Relational Graphs: The Key to More Intelligent RAG Systems

8. DeepMind - V2A Soundtrack AI Generation for Generative Video

9. [play it] Using ControlNet to Animate the Game of Life

10. [interactive viz] How AI is Creating Havoc in Global Power Systems

Share Data Machina with your friends

the ML Pythonista

1. Building an AI Text-to-Video Model from Scratch

2. How to Use Transformers for Classification Label Prediction

3. [cookbook] Build a Text-to-SQL System with Mistral AI, Neon & LangChain

Deep & Other Learning Bits

1. Agile RL - Easy, Fast, Streamlined RL with RLOps

2. [explainer] What are Highway Networks in Deep Learning?

3. Decomposing Transformer Outputs for Mechanistic Interpretability

AI/ DL ResearchDocs

1. TextGrad: Automatic ''Differentiation'' via Text (code, paper, tutorials)

2. PlanRAG: Plan-then-RAG for LLMs as Decision Makers (repo, paper)

3. Stanford et al. - The Largest Study on How to Optimise Prompts in LM Programs

MLOps Untangled

1. The Ray MLOps Infra at Pinterest

2. MLOps  - Data Validation with PyTest

3. An Open, Blazing-fast Model Gateway for Rapid Dev of GenAI Apps in Prod

ML Datasets & Stuff

1. Lessons Learned from Scaling to Multi-Terabyte Datasets

2. From Pixels to Prose: A Dataset with 16M Dense Image Captions

3. ShareGPT4Video: 4.8M Multi-modal Video Captions by GPT4-Vision

Postscript, etc

Tips? Suggestions? Feedback? email Carlos

Curated by @ds_ldn in the middle of the night.

Pioneers in Text-to-SQL at enterprise scale. afaik, Pinterest was one of the first companies that deployed Tex2SQL at scale in enterprise production. Importantly, they were one of the first ones in sharing their experience. This is an excellent post in which the engineering team describes the whole journey from academic Txt2SQL to production. Blogpost: How we built Text-to-SQL at Pinterest.

Text-to-SQL augmented with RAG: Not easy yet. Sure! LLMs can write beautiful, syntactically correct SQL statements because there are tons of public SQL code on which LLMs have been trained on. But LLMs have quite a bit of challenges when dealing with real-world relational data problems. Top 4 Challenges using RAG with LLMs to Text-to-SQL and how to solve it.

Text-to-SQL or human-like AI analysts? This is an interesting post by the team at Pattern, a startup building a financial analysis agent. Their main idea: Text-to-SQL should be more like Text-to-Analysis that works at the business layer. And the LLM -beyond prompting- should behave like a human analyst by using multiple, specialist AI agents that contribute to the analysis process. Blogpost: Text to SQL and its uncanny valley.

Agents for data workflows. In the real world, data workflows with several data pipelines with messy, dirty, changing data are an absolute nightmare! Enter Meadow: An open source, agentic framework for building multi-agent data workflows with LLMs with interactive user feedback. Meadow’s approach is to chain several specialised agents like Text-to-SQL, Planner, Executor, Schema Cleaner, Validator, Router agents to perform an end to end data workflow. Blogpost and repo here: Introducing Meadow: LLM Agents for Data Tasks.

Compound AI Systems and LLM Data Agents. In February, researchers at Berkeley AIR published an article on the rapid Shift from Models to Compound AI Systems. In contrast to a “classic” rather static AI model, a Compound AI System interacts with multiple components… function calls, APIs, search, retrievers, agents… The researchers argue that you should design and implement an AI system from the perspective of a Compound AI System.

More recently, Howard at Wren.ai - a startup offering RAG-Txt2SQL solutions- wrote a great post on the new wave and the concept of Composable Data Systems and the Interface to LLM agents. And Mosaic AI (aka Databricks AI) just announced a series of new capabilities on building and deploying production-quality Compound AI Systems.

A fully open source NL engine for SQL DBs. Last month, the team at Dataherald open sourced natural language-to-SQL engine built for enterprise-level question answering over relational data. It allows you to set up an API from your database that can answer questions in plain English. You can do NL Q&A on Prod DBs, NL queries the DW directly without IT support, or create a ChatGPT plugin. Repo and docs here: Interact with your SQL database, Natural Language to SQL using LLMs.

Open Source AI Agents for Data Analysis. PandasAI just open sourced a Python library that makes it easy to ask questions to your data in natural language. Beyond querying, PandasAI offers functionalities to visualize data through graphs, cleanse datasets by addressing missing values, and enhance data quality through feature generation, making it a comprehensive tool for data scientists and analysts. Checkout the repo and docs here: Pandas AI - AI agents for Data Analysis.

Free course: Building Your Own Database Agent. In this course, you will develop an AI agent that interacts with databases using natural language, simplifying the process for querying and extracting insights. To know more and join this free course click here.

Have a nice week.

Subscribe now

10 Link-o-Troned

1. My Thoughts on Apple Intelligence Models

2. AI Search: The Bitter-er Lesson

3. Building AI Agents: Lessons Learned over the Past Year

4. [gotcha] LLM Dataset Inference: Did you Train on My Dataset?

5. A Systematic Survey on [the latest] Prompting Techniques, 6/2024

6. The Challenges of Retrieving & Evaluating Relevant Context for RAG

7. NVIDIA Nemotron-4 340B: A SOTA Open Synthetic DataGen Pipeline

8. [new] Memory Tuning: 95% LLM Accuracy, 10x Fewer Hallucinations

9. [free, very comprehensive] Generative AI for Beginners v2, 18 Lessons

10. [great] Stanford CS25 v4: A Highly Opinionated View on Transformers

Share Data Machina with your friends

the ML Pythonista

1. [mega session] Karpathy’s Let’s Reproduce GPT-2

2. Function Calling with OpenAI APIs: A Crash Course

3. agentUniverse - An Apache 2.0 Framework for Multi-Agent Apps

Deep & Other Learning Bits

1. [free] Step-by-Step Diffusion: An Elementary Tutorial (pdf)

2. [official] Intro to Apple’s On-Device & Server Foundation Models

3. Can LLMs Invent Better Ways to Train LLMs? An Auto-Evolutionary Approach

AI/ DL ResearchDocs

1. DeepMind - Transformers Meet Neural Algorithmic Reasoners

2. Together.ai - Mixture of Agents (MoAs) Enhances LLMs Capabilities

3. MSR - Samba: Mamba SSM + MLP + Sliding Window for Unlimited Context

MLOps Untangled

1. LangFuse - An Open Source LLM Engineering Platform

2. Choosing Slurm vs Kubernetes for Modern ML Workloads

3. LLM Monitoring and Observability: Tools, Tips & Best Practices

ML Datasets & Stuff

1. What If We Recaption Billions of Web Images with LLaMA-3?

2. NVIDIA HelpSteer2: An Open Dataset for Training Top Reward Models

3. Character Codex: A Dataset of Characters in Comics, Movies, TV Shows for GenAI

Postscript, etc

Tips? Suggestions? Feedback? email Carlos

Curated by @ds_ldn in the middle of the night.

First, let me share 3 nice intros to SSMs:

• A gentle introduction to SSMs. In this post, jorgecadete explains SSMs from a basic point of view. By the end of this post, you will not be an expert, but at least you will have a robust idea about why they are a fundamental concept in ML.

• Hugging Face introduction to SSMs. There are many types of SSMs. In the context of DL, when we speak of SSMs, we are referring to a subset of existing representations, namely linear invariant (or stationary) systems.

• SSMs and a visual guide to Mamba. This is such a beautiful, clearly structured post! Maaten introduces SSMs in the context of LMs and explores concepts one by one to develop an intuition about the field. Then, he covers how Mamba might challenge the Transformers architecture. Brilliant!

  

SSMs and the new Mamba-2. The Mamba paper (v2 May 2024) was indeed a breakthrough. Now the new Mamba-2 is out. Albert and Tri, the original devs of Mamba, just posted a fantastic blog series that covers the model, the theory, the algo, and the systems behind Mamba-2. Blogpost: State Space Duality (Mamba-2) Part I-IV

SSMs as an alternative to Transformers. This is a recent, fully comprehensive survey on SSMs by researchers at MSR. Although Transformers dominate sequence modelling tasks, they suffer from attention complexity and handling inductive bias for long sequences. SSMs have emerged as a promising alt for sequence modelling paradigms. Read the survey here: Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challenges.

SSMs for time-series forecasting & anomaly detection. A few days ago researchers at Cornell & NYU introduced Chimera: A new 2-dim SSM, that shows superior performance on extensive and diverse benchmarks, including ECG and speech time series classification, long-term and short-term time series forecasting, and time series anomaly detection. Paper: Chimera: Effectively Modeling Multivariate Time Series with 2-Dimensional State Space Models.

SSMs for real-time voice & speech generation. Last week, Cartesian.ai - a startup specialised in multimodal intelligence in any device- announced Sonic, a super-low latency voice generation model that uses SSMs. The Chief Scientist at Sonic was one of the original developers of S4-Structured State Space Sequence model and Mamba. In this blogpost he explains why an SSM inference stack enables low latency and high throughput for voice generation. Announcing Sonic: A Low-Latency Voice Model for Lifelike Speech

SSMs for audio. A new paper introducing Audio Mamba, an SSM for learning general-purpose audio representations from randomly masked spectrogram patches through self-supervision. The researchers claim that Audio Mamba consistently outperforms comparable self-supervised audio spectrogram transformer (SSAST) baselines by a considerable margin. Paper: Audio Mamba: Selective State Spaces for Self-Supervised Audio Representations.

A contrarian view to SSMs as an alt to Transformers. Two days ago, researchers at AllenAI & NYU, published the latest update of a paper downplaying SSMs as an alt to Transformers. “Do SSMs truly have an advantage over transformers in expressive power for state tracking? Surprisingly, the answer is no.” The researchers claim that SSMs have similar limitations to non-recurrent models like transformers, which may fundamentally limit their ability to solve real-world state-tracking problems. Paper: The Illusion of State in State-Space Models.

Bonus: Mamba in NumPy. mamba.np is a new, pure NumPy implementation of Mamba

Have a nice week.

Subscribe now

10 Link-o-Troned

1. Lessons Learned: AI Agents is Not All you Need

2. How We Use AI in Software Engineering at Google

3. A New Way to Extract Interpretable Features from GPT-4

4. What is AI Mechanistic Interpretability? An Intro by Anthropic

5. Hello Qwen2 Family? The New SOTA in Open Source LMs

6. Generative Recommenders: A New, Powerful Paradigm for RecSys

7. Hugging Face LeRobot: A New, SOTA E2E for Real-world Robotics

8. Nomic Embed Vision & Opensource Multi-Modal Embedding Models

9. A Picture is Worth 170 Tokens: How Does GPT-4o Encode Images?

10. A Super Detailed Guide to implement a QnABot on AWS (June 2024)

Share Data Machina with your friends

the ML Pythonista

1. Apple MLX-graphs - A Fast, Scalable Opensource Lib for GraphNNs

2. The Cohere AI Agents Cookbook - A Collection of 10 iPynbs

3. LitGPT- Pretrain, finetune, evaluate, and deploy 20+ LLMs on your own data with SOTA Techniques

Deep & Other Learning Bits

1. The Tensor Calculus You Need for Deep Learning

2. [lecture] Statistical Physics of Machine Learning, May 2024

3. PMLR Vol. 241 Machine Learning with Imbalanced Domains

AI/ DL ResearchDocs

1. [trending] Scalable MatMul-free Language Modelling (paper & repo)

2. Buffer of Thoughts: Thought-Augmented Reasoning with LLMs

3. GraphAny: A Foundation Model for Node Classification on Any Graph

MLOps Untangled

1. Testing Models In Production Using Interleaving Experiments

2. The LLM Journey: From POC to Production in the Real-World

3. Adapting MLflow to Automate Building, Training, & Validation of Models

ML Datasets & Stuff

1. Zyda: 1.3T Trillion-token Open Dataset for Language Modelling

2. MovieLens Belief 2024 Dataset for Movie Recommendations

3. Solos: A Dataset for Audio-Visual Music Analysis - Experiments

Postscript, etc

Tips? Suggestions? Feedback? email Carlos

Curated by @ds_ldn in the middle of the night.

Graph RAG. Unlike RAG approaches that focus solely on text-based entity retrieval, GRAG maintains an acute awareness of graph topology, which is crucial for generating contextually and factually coherent responses. The researchers claim that GRAG significantly outperforms current SOTA RAG methods while effectively mitigating hallucinations. Paper: GRAG: Graph Retrieval-Augmented Generation.

Graph NNs+RAG. This paper introduces a novel method for combining language understanding abilities of LLMs with the reasoning abilities of Graph Neural Nets (GNNs) in a retrieval-augmented generation (RAG) style. The researches claim that GNN-RAG achieves SOTA performance in two widely used KGQA benchmarks, outperforming or matching GPT-4. In addition, the researches say that GNN-RAG excels on multi-hop and multi-entity questions outperforming other approaches by 8.9 - 15.5%. Paper: GNN-RAG: Graph Neural Retrieval for LLM Reasoning

LlamaIndex+Neo4j: The Property Graph Index. Traditional knowledge graph representations like knowledge triples (subject, predicate, object) are limited. They lack the ability to: 1) Assign labels and properties to nodes and relationships, 2) Represent text nodes as vectors, and 3) Perform both vector and symbolic retrieval embeddings. The Property Graph Index solves these issues. By using a labelled property graph representations, it enables far richer modelling, storage and querying of your knowledge graph. Blogpost: Introducing the Property Graph Index: A Powerful New Way to Build Knowledge Graphs with LLMs.

Unifying RAG frameworks with LangGraph. In think this is one of the best, practical implementations of advanced RAG I’ve seen. The idea is to combine three of the most advanced RAG frameworks using LangGraph: Corrective Retrieval Augmented Generation (CRAG), Self-Reflective Retrieval-Augmented Generation (Self-RAG) , and an Adaptive QA framework. A great read! Blogpost: Unifying RAG Frameworks: Harnessing the Power of Adaptive Routing, Corrective Fallback, and Self-Correction using Langchain’s LangGraph

Multi-Agent Collaboration with LangGraph. If you're building AI agents, this article will show you how to build an optimal autonomous research multi-agent assistant using LangGraph. Blogpost: How to Build the Ultimate AI Automation with Multi-Agent Collaboration.

Have a nice week.

Subscribe now

10 Link-o-Troned

1. The GenAI Mindset: Combining Atomic Units

2. What We Learned from a Year of Building with LLMs (Part II & I)

3. [deep dive] Intro to the New Transformer Agents 2.0

4. OpenAI Build Hour: An In-Depth Intro to Function Calling

5. [mega paper] An Introduction to Vision-Language Modelling

6. Better Ways to Detect ML Model Degradation than Data Drift

7. Understanding the Cost of Generative AI Models in Production

8. [overview] New Falcon 2.0 11B: Next-gen Open-source LLMs & VLMs

9. ToonCrafter: A New, OSS Generative Cartoons Model (demo, repo, paper)

10. [interactive] FineWeb: Decanting The Web for the Finest Text Data at Scale

Share Data Machina with your friends

the ML Pythonista

1. Benchmarking Foundation Models for Time Series

2. How to Bootstrap & Aggregate Multiple ReAct Agents with DSPy

3. MusePose: A Pose-Driven Img-2-Vid Framework for Virtual Human Generation

Deep & Other Learning Bits

1. ML Collective - DL Classics & Trends (slides & papers)

2. N-HiTS — Making DL for Time Series Forecasting More Efficient

3. How to Train & Finetune Embedding Models with Sentence Transformers v3

AI/ DL ResearchDocs

1. ColdFusion: Zero-shot Anomaly Detection (paper & repo)

2. SymbCoT: Faithful Logical Reasoning via Symbolic Chain-of-Thought

3. Automatic Data Curation for S-S Learning: A Clustering-Based Approach

MLOps Untangled

1. [free course] LLMOps for Supervised Instruction Tuning

2. Scaling ML Experiments with neptune.ai and Kubernetes

3. [free course] Modern MLOps with MLFlow (16 video sessions)

ML Datasets & Stuff

1. How to Fine-tune Phi-3 Vision Model on a Custom Dataset

2. How to Access +150K Datasets in Hugging Face with DuckDB

3. MMLU-Pro: 12K Complex Questions Answered Across 14 Disciplines

Postscript, etc

Tips? Suggestions? Feedback? email Carlos

Curated by @ds_ldn in the middle of the night.

We’re doing AI chat & espresso at Level 39, One Canada Square. James -a veteran CTO with all the scars- is asking these rather funny, rhetorical questions. There is a deep silence in the room, pensive faces around. Everyone is staring through the massive windows overlooking The City skyline as the sunset strikes. We wonder in perplexity -in the very philosophical and information theory sense- whether AI Coding Agents are fully ready for such tasks in prod, or not and if yes when…

Are AI Coding agents any good at solving real-world coding issues autonomously? The team at Princeton Language & Intelligence (PL&I) has come up with SWE-bench, a benchmark for evaluating AI coding agents (paper, code, benchmark). It turns out that current AI Agents are not achieving very good scores in this benchmark yet.

The PL&I team also open sourced SWE-agent an agent that turns LMs (e.g. GPT-4) into software engineering agents that can fix bugs and issues in real GitHub repositories. Checkout the video below with a good hands-on, deep dive.

Amazon Q. The SWE-bench leaderboard is constantly changing but it seems that Amazon Q Developer Agent is for now leading the pack. Amazon Q is a closed model and not very popular in the AI community. It was able to successfully solve only 13.8% out of 2294 tasks. Not a lot really! Here is a vid with a deep dive on Amazon Q.

Devin the 1st Autonomous AI Engineer? In March, Cognition Labs announced Devin, the world’s first fully autonomous AI software engineer. Cognition Labs claimed they were setting a new state of the art on the SWE-bench coding benchmark. Devin went viral but then people in the AI community exposed some tricks used in Devin’s demo. Watch the video below to understand the good, the bad and the ugly of Devin.

Open source AI community to the rescue: OpenDevin. This started as a small side project and has quickly become one of the most popular AI Coding agents projects. OpenDevin agents collaborate with human developers to write code, fix bugs, and ship features. Probably, one of the best open source AI software engineer for developing apps. OpenDevin is now achieving 21% in the swe-bench, the highest score. Checkout the overview below:

Devika MIT licensed. Devika is an Agentic AI Software Engineer that can understand high-level human instructions, break them down into steps, research relevant information, and write code to achieve the given objective. Checkout the vid below.

Blackbox AI Coding Agents IDE. This is a -still free- pretty amazing AI Coding Agents IDE that comes packed with agents specialised in+30 development languages. The agents can perform natural language to code, chat to code, image to code, plus many s/w engineering tasks like: bug fixing, unit testing, code translation, API integration, coding docs, coding optimisation… Apparently millions of devs use it. Checkout Blackbox AI’s playground, agents and features here.

GPT-Engineer. Another popular open source AI engineer, gpt-engineer lets you: 1) Specify software in natural language, 2) sit back and watch as an AI writes and executes the code, and 3) Ask the AI to implement improvements. Somehow, not sure, but it seems this project perhaps is starting to fall behind other similar projects. In this video Arjan asks: “Is GPT Engineer Actually Useful?”

ChatDev Virtual Software Company. This is an amazing, OSS virtual software company that operates through various intelligent agents holding different roles, including CEP, CPO , CTO, programmer, reviewer , tester, art designer… These agents form a multi-agent organisational structure and are united by a mission to "revolutionise the digital world through programming." The agents within ChatDev collaborate by participating in specialised functional seminars, including tasks such as designing, coding, testing, and documenting. Checkout the repo and paper: ChatDev Multi-Agent Collaborative Software Development.

Other less ambitious AI Coding Agents for specific s/w engineering tasks.

• PR-Agent automates the review and analysis of pull requests, and generates feedback and suggestions.

• What The Diff automatically writes pull request descriptions, sends out summarised notifications to non-technical stakeholders in the loop, and helps you to refactor minor issues during the review.

• Cover Agent automatically generates qualified tests to enhance existing test suites to help efficiently increasing code coverage.

Have a nice week.

Subscribe now

10 Link-o-Troned

1. An Evolving Sixth Sense for AI

2. How to Build Terrible AI Systems

3. Successful LMs Evals and 7 Mistakes

4. Teaching Neural Nets to Make Neural Nets & Auto LoRA

5. On Fine-tuning: Can a LLM Really Learn New Things?

6. The KAN Revolution Arriving at AI: Death of DL?

7. An Overview of the new OSS Falcon 2 Family of Models

8. Perplexica - An OSS AI Search Egine Alt to Perplexity AI

9. CogVLM2 - An OSS VL Model with Chat Skills that Beats GPT-4V

10. Meta Tutorials: How to Run Llama-3 on Linux, Windows & MacOS

Share Data Machina with your friends

the ML Pythonista

1. KHOJ v1.12 - An OSS App that Creates Personal AI Agents

2. Building a Multi-Tool AI Agent with Databricks DBRX & DSPy

3. Implementing Llama-3 from Scratch, One Tensor and MatMul at a Time

Deep & Other Learning Bits

1. [tutorial] Diffusion Models in Image & Vision

2. Extracting Millions of Interpretable Features from AI Models in Prod

3. [tutorial] PyCon US2024 The Fundamentals of Modern DL with PyTorch

AI/ DL ResearchDocs

1. Is Sora a World Simulator? A Survey on General World Models and Beyond

2. Pandora: On-the-fly World Model VideoGen with NL (paper, code, demo)

3. An Atari RL Agent Trained in a Diffusion World Model (paper, code, game)

MLOps Untangled

1. How to Test for Topics & Embeddings Drift

2. OSS Netflix Metaflow v 2.11 - Easily Build & Manage AI/ML Projects

3. Breaking Down Workflow Orchestration and Pipeline Authoring in MLOps

ML Datasets & Stuff

1. An Awesome Collection of Resources on Synthetic Data

2. BEHAVIOR Vision Suite: Customizable Dataset Generation via Simulation

3. Curating Custom Datasets for LLM Training with OSS NVIDIA NeMo Curator

Postscript, etc

Tips? Suggestions? Feedback? email Carlos

Curated by @ds_ldn in the middle of the night.

And while many people in the AI community were befuddled and discussing the “flirtatiousness” aspects of GPT-4o, then Google came in and blasted a massive AI storm including SOTA models, new powerful open models, and pretty amazing tools. Here’s my summary on what Google released:

Gemini 1.5 Pro model updates: Lots of improvements in coding, reasoning, translation, multimodality and much more. Some key updates include:

• multimodal prompting to prompt the model with any text, image, audio, and video data

• custom function calling to enable real-time interactions with external world

• systems instructions to steer the behaviour of the model based on specific requirements or use cases

• context caching to reduce the cost of requests that contain repeat content with high input token counts

• Notably, an extended context size to 2 million tokens! Google researchers say Gemini Pro 1.5 has perfect retrieval (>99%) up to at least 10M tokens, massively beating Claude 3.0 (200k) and GPT-4 Turbo (128k).

If you’re interested to know more read the technical report: Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

Also checkout this video demoing Gemini’s new extended long context

New Gemini 1.5 Flash model: The new, smaller Gemini 1.5 Flash model is optimised for high-volume, high-frequency tasks at scale, is more cost-efficient to serve. Use it when narrower or high-frequency tasks require fast model’s response and time matters the most. It features a 1 million a long context window. Check this video on Getting started with Gemini Flash.

A new, open Vision-Language Model. PaliGemma is a powerful open VLM inspired by PaLI-3 model. Built on open components including the SigLIP vision model and the Gemma language model, PaliGemma is designed for class-leading fine-tune performance on a wide range of vision-language tasks. This includes image and short video captioning, visual question answering, understanding text in images, object detection, and object segmentation. Checkout this review by the Hugging Face team: PaliGemma – Google's Cutting-Edge Open Vision Language Model.

Announced new Gemma 2 open model. A 27 billion parameter model that delivers performance comparable to Mistral and Llama 3 70B at less than half the size. According to Google, this breakthrough efficiency sets a new standard in the open model landscape. Gemma 2, a 27B-parameter open model, launching in June.

Project Astra: Universal interactive AI Agents. An advanced seeing-and-talking responsive AI agent. The agent uses real-time multi-modality, remembers what it sees and hears to understand context and takes action. It’s also quite proactive, teachable and personal, and has few delays. Checkout this amazing video demo:

New ML Model Explorer, a powerful graph visualisation tool that helps one understand, debug, and optimise ML models. It specializes in visualizing large graphs in an intuitive, hierarchical format, but works well for smaller models as well. Blogpost: Model Explorer: Graph visualization for large model development.

A new AI Safety framework. A set of protocols for proactively identifying future AI capabilities that could cause severe harm and putting in place mechanisms to detect and mitigate them. The focus is on severe risks resulting from powerful capabilities at the model level, such as exceptional agency or sophisticated cyber capabilities. Blogpost: Introducing the Frontier Safety Framework.

A new Generative AI Toolkit that includes a series of tools to develop and evaluate robust, safe AI apps. It includes an LLM comparator and an interpretability tool. See: Responsible Generative AI Toolkit.

A new AI Developer competition. Build an AI App that integrates with Gemini API. Compete for your share of $1 million in cash prizes. Read more about the competition rules, submission guidelines, prizes, and timeline here: Google Gemini API Developer Competition.

Alice’s Adventures in Wonderland reimagined by GenAI. A set of beautiful interactive stories created by 5 artists using GenAI. Link: Infinite Wonderland.

Have a nice week.

Subscribe now

10 Link-o-Troned

1. On Chat Interfaces & Full Delegation to an AI Agent

2. All Neural Nets are Converging to a Platonic Model of Reality

3. Test Driving the AI Capabilities of ChatGPT-4o

4. [free course] Multi AI Agent Systems with crewAI

5. Embeddings, Transfer Learning & RecSys at Spotify

6. Awesome Synthetic (Text) Datasets

7.  The 1st Interpretable DL Model that Works for Time Series Forecasting

8. NVIDIA ChatQA-1.5 Surpasses GPT-4 on Conversational QA & RAG

9. Chessdream - A Free AI that Generates Realistic Chess Positions

10. [comedy] The Daily Show: “New GPT-4o is Sounding Really Horny”

Share Data Machina with your friends

the ML Pythonista

1. llama3.np - Llama 3 Implemented in Pure NumPy

2. Parler-TTS Mini: Expresso - Natural, Consistent, AI Speech with Emotions

3. VILA- An OSS Vision-Language Model for Video & Multi-image Understanding

Deep & Other Learning Bits

1. KANs Paper Explained - An Exciting New, DL Paradigm?

2. [free tutorial] Planning & Reasoning in LLMs (videos & slides)

3. MAMBA from Scratch: RNNs are Better and Faster than Transformers

AI/ DL ResearchDocs

1. Meta AI - Chameleon: Mixed-Modal Early-Fusion FMs that Beat GPT4-V

2. DeepMind - CAT3D: Create Anything in 3D with Multi-View Diffusion Models

3. DataBricks AI - LoRA Learns Less and Forgets Less (Underperforms Fine-tuning)

MLOps Untangled

1. Scalable, Next-gen ML Feature Engineering at Stripe

2. Streamlining AI Model Deployment & Dynamic Routing

3. Common Causes of ML Models Data Leakage: How to Deal with Them

ML Datasets & Stuff

1. CinePile: A Long Video Question Answering Dataset

2. Code Bagel: 800 Million Tokens of Unique Coding Data

3. Sakuga-42M Dataset: Scaling Up Cartoon Research

Postscript, etc

Tips? Suggestions? Feedback? email Carlos

Curated by @ds_ldn in the middle of the night.

Latest innovative approaches. To address those issues, a new breed of foundation or pre-trained AI models for time-series is emerging. Some of these new AI models use hybrid approaches borrowing from NLP, vision/ image, or physics modelling, like: transformers, diffusion models, KANs and state space models. Despite the scepticism in the hard-core statistics community, some of these new AI models for time-series are producing some surprisingly good results. Let’s see…

Foundation Models for Time-series. This new survey examines the effectiveness, efficiency and explainability (3Es) of pre-training foundation models from scratch for time series, and adapting large language foundation models for time series. The survey comes with an accompanying repo loaded with awesome papers, code and resources. Paper, repo: A Survey of Time Series Foundation Models & Awesome-Time Series-LLM&FM.

Diffusion Models for Time-Series. A recent, comprehensive survey and systematic summary of the latest advances in Diffusion Models for time series, spatiotemporal data and tabular data with more awesome resources (paper, code, application, review, survey, etc.). The paper also provides a great taxonomy of diffusion models for time series. Paper, repo: Diffusion Model for Time Series and Spatio-Temporal Data.

TinyTimeMixers (TTMs). A new family of compact pre-trained models for Multivariate Time-Series Forecasting, open-sourced by IBM Research. With less than 1 Million parameters, TTM introduces the notion of the first-ever “tiny” pre-trained models for Time-Series Forecasting. Checkout the TTMs model card, paper and getting started notebook here.

MambaFormer for Time-series. MambaFormer is a new hybrid model that for the first time combines Mamba (a state space model) for long-range dependency, and the Transformer for short range dependency, for long-short range forecasting. The researchers claim the MambaFormer outperforms both Mamba and Transformer in long-short range time series forecasting. Paper: Integrating Mamba and Transformer for Long-Short Range Time Series Forecasting.

Google TimesFM is a new forecasting model, pre-trained on a large time-series corpus of 100 billion real world time-points, that displays impressive zero-shot performance on a variety of public benchmarks from different domains and granularities. Paper, repo: A decoder-only foundation model for time-series forecasting.

DeepKAN. This is a very new approach that combines a neural network with Kolmogorov-Arnold Networks (see link #7 below on KANs.) The researchers say that KANs outperform MLP (Multi-Layer Perceptron)-based methods in time-series prediction tasks when applied to the Air Passengers dataset (although not a very exciting/complex dataset.) DeepKAN: Deep Kolmogorov-Arnold-Networks-KAN.

Have a nice week.

Subscribe now

10 Link-o-Troned

1. BabyAGI, AI Agents & AI Investing

2. Avoiding the Frankenstein Prompt

3. The ICLR 2024 Outstanding Paper Awards

4. How to Create AI Agents that Don't Suck

5. Seven Low-Hanging Fruits for RAG Search

6. What is Context Attribution in Generative AI?

7. Understanding Kolmogorov–Arnold Networks (KANs)

8. A Review of S/W Engineering (SWE) AI Agent Tech Report

9. A New SOTA Model for Data Labelling, Enrich. & Cleaning

10. Why I'm Building My Own AI Agent Library (blogpost & repo)

Share Data Machina with your friends

the ML Pythonista

1. ScrapeGraphAI: You Only Scrape Once with AI (repo, demo)

2. Create LLM Agents with Long-term Memory and Custom Tools

3. GPT Researcher - An Autonomous Agent for Online Research (repo, demo)

Deep & Other Learning Bits

1. [free hands-on] Deep Reinforcement Learning: Zero to Hero!

2. Tackling Vanishing and Exploding Gradients in DeepNNs

3. [free course] UT Austin CS388- Modern NLP, Masters-level

AI/ DL ResearchDocs

1. xLSTM: Extended LSTM by Leveraging Transformers Concept

2. Google AI - Leverage Foundational Models for Black-Box Optimisation

3. MS Research - Visualization-of-Thought Elicits Spatial Reasoning in LLMs

MLOps Untangled

1. Building ML Models Through Automated Augmented Tuning

2. [review] 7 Best ML Workflow & Pipeline Orchestration Tools

3. How to Deploy an End-to-End Private RAG App on Kubernetes

ML Datasets & Stuff

1. Lichess Puzzles - A Collection of ~1.5M Chess Puzzles

2. Buzz - An Instruction-Following Dataset, 85 Million Conversations

3. Google ImageInWords: Hyper-Detailed Image Descriptions & Annotations

Postscript, etc

Tips? Suggestions? Feedback? email Carlos

Curated by @ds_ldn in the middle of the night.

Aah… A long bank holiday in London; so much stuff to do in this amazing city! But if you are feeling the AI FOMO kick and can’t survive a long weekend IRL, here are six AI activities for you:

1. Generate comics with AI. I gave it a go, generated a few short comics, and having fun so far. The AI team at Bytedance just introduced an impressive diffusion-based, zero-shot, text-to-image and image-to-video model that generates amazing videos and comics. Checkout the demo, paper and repo here: StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation. Make sure you click the Comic Generation Demo link and be patient.

   

2. Learn how to build and use a robust AI Agents stack. I absolutely believe that the future is going to be about millions of AI Agents working and generating income for the people. In this vid, Tony shows how to create an AI agent that fetches all the comments of a YouTube video and generates insights to improve video content. Tony uses an AI Agent stack that looks very solid: 1) CrewAI agent framework, 2) the nifty Ollama 3) Groq the super fast AI inference engine, and 4) AgentOps the observability tool for AI agents

3. Play AI Town game in your computer. I’ve played this game and it’s quite addictive! AI Town is a MIT-licensed game -developed by a16z- in which AI characters live, chat and socialise in a virtual town. You can play AI Town online here. But if you hate cloud signups like me, and want to create your own custom AI Town, check this out: How to create your own AI Town with Llama-3 based agents in your local environment using the nifty Ollama and the one-click deployment with the amazing Pinokio AI browser. The video below provides more details on deploying AI Town locally with Pinokio.

4. Read the latest on In-Context Learning (ICL.) There is a debate among AI researchers on whether In-Context Learning within a long-context window size can fully beat fine-tuning done with highly-curated data, in terms of domain knowledge and accuracy of model outputs. Let’s see…

   1. This is a great post on that: Is Fine-Tuning Still Valuable? A reaction to a recent trend of disillusionment with fine-tuning

   2. Ethan, a well known AI researcher is more assertive: Fine-tuning is dead. Prompts have closed the gap.

   3. DeepMind: Many-Shot In-Context Learning. Many-shot in-context learning works very well and can be applied universally. “We find that both Reinforced and Unsupervised ICL can be quite effective in the many-shot regime, particularly on complex reasoning tasks.”

   4. In-Context Learning with Long-Context Models: An In-Depth Exploration. “We conclude that although long-context ICL can be surprisingly effective, most of this gain comes from attending back to similar examples rather than task learning.”

   5. MSR: Make Your LLM Fully Utilize the Context. In this paper, Microsoft proposes a solution to the "lost-in-the-middle" long context problem, in which LLMs struggle using information located in the middle section within the long context.

5. Read the 2024 State of AI Readiness Report. Nice report with good insights and cool charts. The research team at Scale AI interviewed 1,800 AI/ ML practitioners on the latest AI trends, applied AI, and what it takes beyond “adopting AI.” Link to the report: Scale Zeitgeist 2024 AI Readiness Report, 3r ed (pdf, 47 pages)

6. Read this free book and fall down the rabbit hole of designing Neural Nets “This primer is an introduction to this fascinating field [of differentiable programming applied to NNs] as imagined for someone, like Alice, who has just ventured into this strange differentiable wonderland.” Link: Alice’s Adventures in a Differentiable Wonderland

Have a nice week.

Subscribe now

10 Link-o-Troned

1. Agentic Workflows and Building Models that Self-Learn

2. Stanford - Machine Unlearning in 2024

3. KANs: A New [better?] Alternative to the Multi-Layer Perceptron

4. [deep dive] MOMENT: A Foundation Model for Time-series Tasks

5. Google TeraHAC: A New Algo for Clustering Trillion-Edge Graphs

6. How to Build Domain-specific Datasets for Training AI Models

7. Amazon Q: A Generative AI Assistant for Biz & Devs

8. Advanced RAG 101 - How to Build Agentic RAG with llama3

9. $100-$500K Fast Compute Grants for AI Researchers

10. LMSYS Kaggle Chatbot Competition – Predicting Human Preference

Share Data Machina with your friends

the ML Pythonista

1. [notebooks] Examples of Automated Multi Agent Chat with Autogen

2. AgencySwarm- An Opensource Agent Orchestration Framework

3. Powerful Automatic Speech Recognition + Diarisastion + Speculative Decoding

Deep & Other Learning Bits

1. Diving into The Math Behind RNNs

2. Multivariate Time-series Forecasting with CNNs

3. Geometric Deep Learning: The Erlangen Programme of ML

AI/ DL ResearchDocs

1. Tencent AI: More Agents is All You Need

2. Meta AI: A Simple Recipe to Improve CoT Reasoning with DPO+NLL

3. Octopus v4: A Graph of LMs to Integrate Multiple Specialised Open Models

MLOps Untangled

1. How to Monitor a Deep Learning Stack in Production

2. Automatic Model Deployment with MLflow & GitHub Actions

3. MLRun -An Opensource MLOps Platform for Continuous ML Apps

ML Datasets & Stuff

1. MS Marco Search: 10 Billion High-quality Web Pages

2. WildChat Dataset: 1 Million Real-world User-ChatGPT Interactions

3. OpenStreetView-5M: 5.1 Million Geo-referenced Street View Images

Postscript, etc

Tips? Suggestions? Feedback? email Carlos

Curated by @ds_ldn in the middle of the night.

In the last few days, three new, smallish, powerful open AI models were released. Interestingly enough, the power of these 3 models is based on a combination of: 1) Innovative training architectures and optimisation techniques, and 2) Data quality for different types of data (synthetic, public or private). Let’s see…

Snowflake Artic 17B: A new, truly open source (Apache 2.0) model that has been modelled for enterprise intelligence. Snowflake Artic 17B is based on a Dense - MoE Hybrid Transformer architecture. It outperforms all other open models in three areas that are the most demanded in enterprise AI : 1) conversational SQL (Text-to-SQL), 2) coding copilots and 3) RAG chatbots. The model is also supper efficient in terms of training and low cost, two things much valued in enterprise too. Checkout the official blogpost: Snowflake Arctic: The Best LLM for Enterprise AI — Efficiently Intelligent, Truly Open.

The release of Snowflake Artic 17B also comes with a Cookbook series starting with:

• Building an Efficient Training System for Arctic A review on how the DeepSpeed library was used to optimise efficient large MoE training using several optimisation techniques like ZeRO-2 and expert-parallelism.

• Exploring Mixture of Experts (MoE) A review on why and how the Snowflake AI team decided to develop and train a model with a Dense-MoE Hybrid Architecture.

You can try and run Snowflake-arctic-instruct on Replicate and Streamlit

Apple OpenELM. OpenELM, a new family of eight SOTA open language models. The model has been released in both pretrained and instruction tuned versions with 270M, 450M, 1.1B and 3B parameters. You can get the OpenELM models card and model versions in HugginFace.

OpenELM was inspired by Allen AI OLMo, which is one of most performant, truly open models. OpenELM outperforms OLMo by using an innovative layer-wise scaling strategy to efficiently allocate parameters within each layer of the transformer model, which enhances accuracy in a more efficient way, using 2× fewer pre-training tokens. The original paper is an excellent read: OpenELM: An Efficient Language Model Family with Open-source Training and Inference Framework

OpenELM was developed using the new Apple CoreNet library. You can run OpenELM quantised models in your MacBook using Apple MLX framework. Also checkout this iPynb to run an OpenELM 3B demo on Gradio.

Microsoft Phi-3. This is a new family of open AI models developed by Microsoft. Microsoft claims that the Phi-3 models are the most capable and cost-effective small language models (SLMs).

Here is the official blogpost with the technical report, model card and deployable environments on Microsoft Azure AI Studio, Hugging Face, and Ollama: Introducing Phi-3: Redefining what’s possible with SLMs

The innovation here perhaps is the way specific data training was used to achieve such high performance in such a small model which also comes with a big 128K context window size. You can expect Phi-3 running on smart phone devices an delivering great output.

Check-out the powerful Phi-3-Mini-128K-Instruct version, a 3.8B model that uses the Phi-3 datasets. Also sees this great post on How to Finetune phi-3 on MacBook Pro.

OpenVoice v2. Although not exactly a “new” model, I added this as a fourth bonus model :-) as I think it’s worth mentioning an important update. The very latest April v2 comes with 1) Free commercial use MIT license, 2) Native multi-lingual support in English, Spanish, French, Chinese, Japanese and Korean, and 3) Much better audio quality due to new audio data training strategy. Checkout the original paper, demos and repo here: OpenVoice: Versatile Instant Voice Cloning.

Have a nice week.

Subscribe now

10 Link-o-Troned

1. Self-Reasoning Tokens, Teaching Models to Think Ahead

2. The Probabilities of Unsupervised ML Models

3. Jack of All Trades: A Multi-Purpose Transformer Agent

4. Deep Dive: Diffusion Models for Video Generation

5. How We Built a Personal AI Agent Interviewer

6. Cohere Toolkit - Open Pre-built Components for RAG Apps

7. Brain_to_Text AI Competition (repo, dataset, FAQs)

8. Awesome Local AI: A Repo of Local AI Tools

9. Alibaba GTE - A New Family of SOTA Embeddings Models

10. Accenture Report on GenAI and the Future of Work

Share Data Machina with your friends

the ML Pythonista

1. Apple CoreNet - A New Toolkit for Training DeepNNs

2. Open-Sora: Democratising Efficient GenAI Video for All

3. Maestro - Super-fast, Intelligent Orchestration of Agents with LMs

Deep & Other Learning Bits

1. Enhancing LLMs with Graph ML (pdf)

2. Tutorial: How to Build Streaming ML Apps with ConvNets

3. Tutorial: How to Get Started with Evolutionary Model Merging

AI/ DL ResearchDocs

1. Dynamic Generation of Personalities with LLMs

2. MIT CSAIL - A Multimodal Automated Interpretability Agent

3. DeepMind NExT: Teaching LLMs to Reason about Code Execution

MLOps Untangled

1. Embeddings and Operationalising Vector DBs on Postgres

2. OpenLit: Observability Tool for GenAI and LLMs

3. Understanding What Matters for LLM Ingestion & Preprocessing

ML Datasets & Stuff

1. [!!] HuggingFace FineWeb: 15 Trillion Tokens of the Finest Web Data

2. UTD19 - The Largest, Public Traffic Dataset, 40 Cities, 170M Rows

3. 3LC - An AI Tool That Lets You See Your Dataset Through Your Model’s Eyes

Postscript, etc

Tips? Suggestions? Feedback? email Carlos

Curated by @ds_ldn in the middle of the night.

Meta AI just posted: ”Our largest Llama 3 models are over 400B parameters. And they are still being trained.” The upcoming Llama-400B will change the playing field for many independent researchers, little AI startups, one-man AI developers, and also enterprise AI apps. For now, The Zuck and Yan LeCunn are the bastions of “open AI.”

Quick Llama 3 Summary:

• A family of SOTA, open models available in both 8B & 70B parameter sizes, in pre-trained base and instruction-tuned versions

• License. Open but not fully Apache 2.0 open-source. Free license for research and commercial applications but with limitations. Read Llama 3 license here.

• Open models and weights upon request. Get them here.

• Trained on 24k GPUs!! and +15 trillion tokens. Massive for such model sizes.

• Context window expanded to 8192 length. People expected 128K at least

• New tokeniser with 128K words vocabulary built on tope of OpenAI TikToken

• Meta AI official blogpost: Introducing Meta Llama 3: The most capable openly available LLM to date

• Nathan’s great overview of all the tech details: Llama 3: Scaling open LLMs to AGI

Run Llama 3 with Meta AI intelligent assistant. Llama 3 has been integrated with Meta AI. Try it for chat, coding tasks, and problem solving here. It also runs on Facebook, WhatsApp and Instagram. If you’re not in the US, try with a VPN.

Easily deploy Llama 3 on cloud AI stacks. Using HuggingFace Deploy, you can now deploy Llama 3 on Azure ML, Google Vertex, Amazon SageMaker or HuggingFace hosting. Checkout: HuggingFace Meta-LLama-3-8B click deploy.

Run Llama 3 at blazing speed, super cheap cost.

• Run it on Together AI inference engine. Use Together 2.0 Inference Engine, to get up to 350 tokens per second for Llama 3 8B and up to 150 tokens per second for Llama 3 70B, running in full FP16 precision. Blogpost: Together AI releases Meta Llama 3 for inference and fine-tuning.

• Run it on Groq AI h/w. Groq is an innovative AI hardware stack optimised for super efficient, super fast, cheap AI compute. Researchers at ChatLabs show how running Llama 3 on Grow blows GPT-4 Turbo, Claude 3 Opuso, and Gemini Pro out of the water. Blogpost: Meta AI Llama 3 With Groq Outperforms Private Models on Speed/Price/Quality Dimensions?

Run Llama 3-Instruct-8B GGUF for efficient chat. GGUF is a binary format that is optimised for quick loading and saving of models. Llama 3 instruction tuned models are optimised for dialogue and outperform most open source chat models. Get Meta-Llama-3-8B-Instruct-GGUF here. Thanks to the great @nousresearch and @ggerganov.

Run Llama 3 on Apple silicon devices. You can now run any Lllama-3 model quantised in 4 bit or 8bit on your local Apple silicon device using Apple MLX framework. Thanks to the awesome @Prince_Canuma.

See how Llama 3 was jailbroken. The researchers at Meta AI team say that they spent a lot of time safeguarding and redteaming Llama 3. Well, I’m not so sure about that because -inevitably- lots of jailbreaks are starting to pop up. Checkout A Trivial Jailbreak Against Llama 3 or Jailbreaking Llama 3 for education purposes.

Have a nice week.

Subscribe now

10 Link-o-Troned

1. AI Agentic Design Patterns: Multi-Agent Collaboration

2. Decomposing Predictions by Modeling Model Computation

3. Open-sourcing Idefics2: A Powerful 8B Vision-Language Model

4. Cookbook & Tutorials for the Google Gemini Models

5. [tutorial] Overview of LM Model Alignment Methods (77 slides)

6. AI Model Compression: A Deep Guide to Quantisation

7. A Great Reading List on [Modern] Machine Learning

8. My Thoughts on AI Agents: Looping vs. Planning

9. Standford 2024 AI Index Report (pdf, 500 pages)

10. [amazing] MSR VASA-1: Lifelike Audio-Driven Talking Faces in Real Time

Share Data Machina with your friends

the ML Pythonista

1. torchtune - A PyTorch Lib for LLM Finetuning + Recipes

2. DSPY: Not Your Average Prompt Engineering

3. [opensource] DeepMind Penzai: Build, Edit & Visualise Neural Nets

Deep & Other Learning Bits

1. [free] MIT Lectures: Learning Deep Representations

2. [free course] Quantization Fundamentals with Hugging Face

3. Efficient, Large-scale Clustering & Visualisation for NLP and Vision

AI/ DL ResearchDocs

1. Stanford STORM: Writing Wikipedia-like Articles from Scratch with LLMs

2. DeepMind: The Limits of Token Prediction & Many-shot In-context Learning

3. Mini-Gemini: Enhancing Muti-Modality in Vision-Language Models (repo, etc)

MLOps Untangled

1. Scaling AI Models Like You Mean It

2. Architecture & Design Principles for LLMOps

3. Orchestrating Online ML Model Training with Airflow

ML Datasets & Stuff

1. YouTube-Commons - 15M Transcripts, 2M Videos

2. HQ-Edit: A HQ Dataset for Instruction-based Image Editing

3. COCONut - 383K Images, 5.1M Human-verified Segmentations

Postscript, etc

Tips? Suggestions? Feedback? email Carlos

Curated by @ds_ldn in the middle of the night.

Facebook AIR MusicGen. Probably one of the pioneering models in AI quality music generation. MusicGen has sparked a whole universe of MusicGen derivative models of all kinds, and it’s the model behind many musicgen apps. The model is based on a single stage auto-regressive Transformer model, and unlike Google LM, MusicGen doesn't require a self-supervised semantic representation. Repo and demos here: MusicGen: Simple and Controllable AI Music Generation

Mulbert. One of the early AI musicgen startups, Mulbert is an app for generating high-quality, royalty-free music with AI. Thy this Mulbert text-to-music notebook and get the app here.

Stable Audio 2.0. Recently introduced by Stability AI, Stable Audio 2.0 lets you generate high-quality, full tracks from text & audio with coherent musical structure up to three minutes in length at 44.1kHz stereo. Checkout the blogpost, demos, trial: Introducing Stable Audio 2.0

MusicLang is an app for controllable music generation with AI, mostly oriented to artists and music producers. The MusicLang team recently released MusicLang Predict, your controllable music copilot (repo). You can Try MusicLang here.

The MusicLang Tokeniser. An interesting post explaining how tokenization works inside MusicLang and its capacity to afford users profound control over the musical content generated by transformer models. The MusicLang tokenizer : Toward controllable symbolic music generation.

Glycol. A foundation for some specialised musicgen model. If you love coding and music this is pretty cool. Glycol is an open source, next-gen language for generating music with code. Get Glycol from this repo.

RaveForce Agent is a Python package under MIT license that allows you to define your musical tasks in Python with Glicol syntax, and train an agent to do the task with APIs similar to the OpenAI Gym. Get ReveForce here.

Google MusicFX is powered by Google MusicLM and AudioLM. Simple with no frills but good quality. A neat feature is DJ Mode, that enables you to generate a real-time stream of music by adding and adjusting musical prompts to evolve the music live. You can try Google MusicFX here

Suno AI V3 The latest version of Suno enables you to generate two-minute, radio-quality music from text prompts in just a few seconds. The model behind Suno combines a proprietary AI musicgen model and ChatGPT for the lyrics. Sun has some cool features and you can get some decent outputs. Try Suno.ai V3 here

Udio. Recently released, it’s super trending in the musicgen scene now. Suno enables you to create music from simple text prompts by specifying topics, genres, and other descriptors which are then transformed into professional quality tracks. Udio it’s pretty impressive and you can generate some amazing music output. I really like it! Try Udio here. You can also watch this tutorial on how to use Audio.

.Have a nice week.

Subscribe now

10 Link-o-Troned

1. Building an AI Coach to Tame My Monkey Mind

2. Diffusion & Latent Consistency Models, Explained

3. Building Reliable Systems Out of Unreliable AI Agents

4. Attention in Transformers, Visually Explained

5. Understanding Multimodal AI Models

6. The Perfect Prompt: A Prompt Engineering Cheat Sheet

7. How to: Text2SQL Tasks with DuckDB-NSQL-7B Model

8. “Surprising” Lessons after Churning Half-billion GPT Tokens

9. An Overview of the Latest Google Gemma Family of Models

10. Rerank 3: A New Foundation Model for Efficient Enterprise Search & Retrieval

Share Data Machina with your friends

the ML Pythonista

1. Parler- An Open TTS Model for HQ Natural Sounding Speech

2. nanoLLaVA- A "Small but Mighty" 1B Vision-Language Model

3. aiXcoder-7B - A New SOTA LM Model for All Things Coding

Deep & Other Learning Bits

1. Making Deep Learning Go Brrrr From First Principles

2. Notes on Stanford CS224 ML with Large-scale Graphs

3. How I Got into Deep Learning from Knowing Nothing

AI/ DL ResearchDocs

1. AutoCodeRover: Autonomous Program Improvement

2. MS Rho-1: Not All Tokens Are What You Need (paper, repo)

3. LLM2Vec: LLMs are Secretly Powerful Text Encoders (paper, repo, tutorial)

MLOps Untangled

1. Volga - An Opensource Feature Engine for Real-time AI/ML

2. Airbnb Opensources Chronon ML Feature Platform

3. LLMOps Tutorial: Five Ways to Serve LLMs

ML Datasets & Stuff

1. NVIDIA Audio Dialogues Dataset for Audio & Music

2. Anthropic Persuasion Dataset: Human vs AI Claims

3. Generate Financial Q&A Datasets with AI

Postscript, etc

Tips? Suggestions? Feedback? email Carlos

Curated by @ds_ldn in the middle of the night.

Oddly enough, as the LLM models become more capable and sophisticated, the jailbreaking attacks are becoming easier to perform, more effective, and frequent. Gary Marcus - who is hypercritical about LLMs and current AI trends- just published this very opinionated post: An unending array of jailbreaking attacks could be the death of LLMs.

I often speak to colleagues and clients about the “LLM jailbreaking elephant in the room.” And they all agree that this is a serious concern, and a deterrent to deploying LLMs in enterprise production.

I suppose that stuff like the attention mechanism, tokenisation, next token prediction, and prompting are the strengths but also the weaknesses of LLMs. Thus developing anti-jailbreaking methods, defences for LLMs is really hard and like a moving target. Checkout this new free seminar on Robustness in the Era of LLMs: Jailbreaking Attacks and Defenses

I believe that is a good thing that the AI community, researchers and startups share jailbreaking methods for advancing the AI Safety & Alignment field. So in light of that, here are 4 new, very efficient LLM jailbreaking methods published in the last 10 days:

Simple adaptive attacks. In this paper, researchers at EPFL show that even the most recent safety-aligned LLMs like Claude or GPT-4 can’t resist simple adaptive jailbreaking attacks. The researchers demonstrate how to successfully leverage the logprobs for jailbreaking by initially designing an adversarial prompt template, and then applying random search on a suffix to maximise the target logprob. In the case of models that don’t expose logprobs (e.g. Claude) a 100% success jailbreak is achieved via either a transfer or pre-filling attack. Paper and repo: Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks.

Faux dialogues in the context window. Researchers at Anthropic - who are well known for their leadership in AI Alignment & Safety- published a long blogpost explaining how to use the context window to jailbreak any model. The basis of many-shot jailbreaking is to include a faux dialogue between a human and an AI assistant within a single prompt for the LLM. That faux dialogue portrays the AI Assistant readily answering potentially harmful queries from a user. At the end of the dialogue, one adds a final target query to which one wants the answer. Blogpost: Many-shot jailbreaking.

Expert debates with Tree of Thoughts. The lead researcher at Agora - an opensource multimodal AI collective- introduced a clever and simple jailbreak method that uses Tree of Thoughts prompts. Using ToT, you split your malicious agent into three multiple agents, each one with a different personality. Then you instruct the agent experts to engage in a debate until they conclude at finding a solution to a problem. Checkout the full ToT jailbreak prompt in a raw repo here.

Progressive chat steering. Researchers at MS Research introduced Crescendo, a multi-turn attack that starts with harmless chat and progressively steers the conversation toward the intended, prohibited objective. In less than 5 interactions, Crescendo can jailbreak GPT-4 and similar models by progressively prompting it to generate related content until the model has produced enough material to essentially override its safety alignment. Paper and examples here: Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack.

Have a nice week.

Subscribe now

10 Link-o-Troned

1. Do We Need Attention? A Technical Mamba Model Primer

2. Tandem Coding with My AI Agent

3. AIDE: An AI Agent that Beats Humans on Kaggle

4. [interactive dataviz] Mario Kart Meets The Pareto Frontier

5. Optimising RAG with Rerankers: The Pros & Cons

6. Stable Audio 2.0: Generating HQ Music with AI

7. The State of Production ML in 2024

8. YCombinator: 25 Startups Building AI Models in <3 Months

9. A Visual Exploration on How Large Vision Models Use Concepts

10. [free course] Stanford CS25 Transformers United v4 Spring 2024

Share Data Machina with your friends

the ML Pythonista

1. SWE-agent: Turn Any LM into S/W Engineering Agents

2. RAGFlow - Opensource, Local RAG Engine for Deep Doc Understanding

3. How to Accelerate Mixtral 8x7B MoE with Speculative Decoding and Quantisation

Deep & Other Learning Bits

1. Facebook AI: Schedule-Free Learning - A New Way to Train Models

2. Awesome Self-Supervised Learning for Recommendations

3. Instruction Tuning: Lessons from Building the Stanford Alpaca Model

AI/ DL ResearchDocs

1. VoiceCraft: SOTA Zero-Shot Text-to-Speech in the Wild (demo, paper, repo)

2. AniPortrait: Auto HQ Portrait Animation from Audio (demo, paper, repo)

3. VAR: SOTA Autoregressive Scalable Image Generation (demo, paper, repo)

MLOps Untangled

1. MLOps Tools are a Fragmented Mess

2. Industrial ML with MLOps at BMW

3. Hitchhiker's Guide to MLOps for Time Series Forecasting

ML Datasets & Stuff

1. World's Largest Synthetic Opensource Text-to-SQL Dataset

2. Grocery Vision Dataset Challenge 2024 CVPR

3. LOTSA - Large-scale Open Time Series Datasets

Postscript, etc

Tips? Suggestions? Feedback? email Carlos

Curated by @ds_ldn in the middle of the night.

We’re starting to see several open MoE-based models achieving near-SOTA or SOTA performance as compared to e.g. OpenAI GPT-4 and Google Gemini 1.5 Pro. And this is great! Here’s a brief summary about four open, powerful MoE-based models introduced in the last ten days.

AI21Labs Jamba. Jamba is a model built on top of an SSM-Transformer MoE architecture. The innovation here is to build the model by hybrid interleaving Transformer & SSM layers. Jamba was designed to combinedly address the limitations and benefits of both Transformer and SSM architectures; 1) High quality output, 2) High throughput and 2) low memory requirements. Read more here: Introducing Jamba. Also checkout this iPynb on How to Finetune Jamba.

Alibaba Qwen1.5-MoE-A2.7B. A small MoE model with only 2.7B activated parameters that yet matches the performance of SOTA 7B models like Mistral 7B. The model introduces several architecture innovations -as compared to standard MoE models- for example: combined fine-grained experts, initialisation upcycling, and shared & routing experts. As a result, the model achieves a 75% decrease in training costs and accelerates inference speed by a factor of 1.74, as compared to larger open 7B models, while remaining competitive in most benchmarks. Checkout the paper, repo and demo here: Qwen1.5-MoE: Matching 7B Model Performance with 1/3 Activated Parameters.

MetaAI BTX method. Similar to Qwen1.5-MoE’s architecture that combines multiple fine-tuned expert LLMs, Meta AI recently introduced a new method called Branch-Train-MiX (BTX). BTX starts from a seed model, which is branched to train experts in embarrassingly parallel fashion with high throughput and reduced communication cost. This produces a super efficient MoE architecture. Paper: Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM.

Databricks DBRX 132B MoE. DBRX model uses a fine-grained MoE architecture with 132B parameters of which 36B parameters are active on any input. It was pre-trained on 12T tokens of text and code data. The weights of the base model (DBRX Base) and the finetuned model (DBRX Instruct) are available on Hugging Face under an open license. According to Databricks, DBRX achieves SOTA in performance, cost efficiency, and output quality across open model benchmarks, and beats closed models like GPT-3.5 and Gemini 1.0 Pro. To read more about how DBRX was built its performance, and how to start using it see: Introducing DBRX: A New State-of-the-Art Open LLM

You can try DBRX-Instruct model for free at Perplexity Labs Playground. (Make sure you select the model in the pull down menu.) And if you are interested in running DBRX in a local MacBook environment checkout this repo thread on how to 4-bit quantise DBRX with Apple MLX framework.

xAI Grok-1.5. A few days ago, the xAI team announced Grok-1.5, which is an open model built on top of Grok-1 base MoE model (repo). The model comes with a 128K context window and is very powerful in coding, RAG, and reasoning tasks. Grok-1.5 was built on a custom distributed training framework based on JAX, Rust, and Kubernetes. According to xAI researchers, Grok-1.5 beats most open models, and achieves near SOTA performance as compared to the likes of Gemini 1.5 Pro or GPT-4. Grok-1.5 will be available in X (formerly Twitter.) In the meantime you can read the blogpost: Announcing Grok-1.5.

Have a nice week.

Subscribe now

10 Link-o-Troned

1. Andrew Ng: “What's Next for AI Agentic Workflows”

2. Towards 1-bit Machine Learning Models

3. Your AI Product Needs Evals - A Deep Dive

4. How to Create a frankenMoE (Mixture of Experts) Model

5. The Other Hard Problems in Retrieval: Orthogonality

6. An Overview of Decentralised ML Model Inference

7. A Review of Hume EVI Emotional Intelligence AI Chatbot

8. My Binary Vector Search is Better than Your FP32 Vectors

9. Stanford Foundation Models Ecosystem Tracker

10. A Visual Dive into LAION-5B Opensource Foundation Dataset

Share Data Machina with your friends

the ML Pythonista

1. OpenDevin - An Open Source Autonomous AI Software Engineer

2. Advanced Text Extraction from Images & PDFs with Claude 3

3. AgentStudio: An Open-source Toolkit for Building General Virtual Agents

Deep & Other Learning Bits

1. DLRover: An Automatic Distributed Deep Learning System

2. [free tutorial] Diffusion Models for Imaging and Vision (Mar,2024)

3. Probabilistic Time Series Forecasting with Auto Bayesian NNs

AI/ DL ResearchDocs

1. DeepMind MagicLens: SOTA Self-Supervised Image Retrieval

2. Model Stock: All We Need is Just a Few Fine-tuned Models

3. MetaAI - The Unreasonable Ineffectiveness of the Deeper Layers

MLOps Untangled

1. MLOps Python Best Practices

2. LLMOps: Why It Matters & How to Implement It

3. MLOps Pipeline - RightMove Rental Prediction System

ML Datasets & Stuff

1. Exploring The OpenAI Community Posts Dataset

2. Don’t Evaluate Your Model On a SMOTE Dataset

3. FollowIR: A Dataset for Evaluating & Teaching Info Retrieval Models

Postscript, etc

Tips? Suggestions? Feedback? email Carlos

Curated by @ds_ldn in the middle of the night.

• A Guide to Vision-Language Models (VLMs)

• Vision-Language Models for Vision Tasks: A Survey

But still VLMs are facing some challenges for example in terms of: multimodal training datasets, resolution, long-form modality, vision-language integration, and concept understanding. Somewhat along those lines, I see 5 trends happening in VLMs: 1) VLMs run on local environment 2) Emerging VLM videoagents 3) Unified structure learning for VLMs 4) Personalisation of VLMs and 5) Fixing the VLM resolution curse. Let’s see…

VLMs on local environment. In this blogpost, an independent AI researcher writes about playing around with VLMs using only a local environment. Inspired by Phi-2: The surprising power of small LMs- and using Facebook AI AnyMAL multimodality method, the researcher describes in detail the challenges and different architectures until achieving some decent results in a local environment, which are not close to academic SOTA. Blogpost: Findings on VLMs

New SOTA in long-form video understanding. Researchers at Standford, introduced a new approach for video understanding. The approach combines an LLM agent, a vision-language model (VLM), and contrastive language-image model (CLIP). The researchers claim this approach is superior to current SOTA in video understanding. Paper: VideoAgent: Long-form Video Understanding with LLM as Agent

New SOTA in UI & infographics understanding. Researchers at Google, recently introduced a novel vision-language model that specialises in UI and infographics understanding. The model was trained on a unique mixture of datasets containing novel screen annotations, and types and location of UI elements. The researchers claim the model achieves SOTA in UI & infographics understanding. Paper: ScreenAI: A Vision-Language Model for UI and Infographics Understanding

New SOTA in visual document understanding. Researchers at Alibaba just introduced a new model for visual document understanding that uses Unified Structure Learning (USL). The USL model learns on structure-aware parsing tasks and multi-grained text localisation tasks across 5 domains: document, webpage, table, chart, and natural image. The researchers claim the model achieves SOTA. Paper: mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understanding

Personalisation of VLMs. Most VLMs lack an understanding of user-concepts. Researchers at Snap et al. just introduced MyVLM, a new way to personalise VLMs. Given a set of images depicting user-specific concepts, the researchers augmented a pretrained vision-language model (VLM) and used concept embeddings to understand and reason over these user concepts. Th researchers applied MyVLM to BLIP-2, LlaVA 1.6 and MiniGPT-v2 models for personalised captioning, visual question-answering, and referring expression comprehension. Checkout the project page, code, data and demos here: MyVLM: Personalising VLMs for User-Specific Queries

Fixing the resolution curse in VLMs. Resolution is a key problem in VLMs. VLMs can't zoom. They are limited by the resolution of the vision encoder, and usually, it is not super large based on the pre-trained vision encoder. In this blogpost, Alex explains how you can use Visual Search, Visual Cropping and MC-LLaVA to fix this problem. Blogpost: Breaking resolution curse of vision-language models.

Have a nice week.

Subscribe now

10 Link-o-Troned

1. Evolutionary Model Merge: A New Way to Automate Model Dev

2. A Visual Guide to Mamba and State Space Models

3. DeepMind TacticAI: An AI-Assitant for Football Tactics

4. How I Use Claude 3 and ChatGPT for Ad-hoc Tasks

5. Visualisation of Large-scale Multimodal Datasets with Nomic Atlas

6. New Embedding Quantisation for Faster, Cheaper Retrieval

7. Introducing RAG 2.0: New SOTA Contextual Language Models

8. Berkeley AIR - A New Approach to Modelling Extremely Large Images

9. Mistral + Snowflake: The New Frontier in SQL Copilot Products

10. Cosmopedia: How to Create Large-scale Synthetic Data for Pre-training

Share Data Machina with your friends

the ML Pythonista

1. LaVague: A Large Action Model for Automating Automation

2. Claude-investor: Generative Stocks Investment Recommendations

3. Devika - An Agentic AI Software Engineer that Follows Human Instructions

Deep & Other Learning Bits

1. An Overview of Contextual Bandits & RL

2. The Bayesian Learning Rule & Adaptation in ML

3. Reversible Residual Nets: How To Train NNs with Less GPU Memory

AI/ DL ResearchDocs

1. DenseFormer: Faster Transformer Inference with Depth Weighted Averaging

2. LlamaFactory v.2: Unified Efficient Fine-Tuning of 100+ Language Models

3. Google Research- A Bag of Tricks for Few-Shot Class-Incremental Learning

MLOps Untangled

1. Autonomous Agents for Production Ready LLMs

2. Predictive Scoring with MLOps and KubeDDR

3. High-quality MLOps with Python's ABC & Pydantic

ML Datasets & Stuff

1. Announcing the 2024 Waymo Open Dataset Challenges

2. Common Corpus: The Largest Public Domain Dataset, 500 Billion Words

3. DROID (Distributed Robot Interaction Dataset), 76K Demonstrations

Postscript, etc

Tips? Suggestions? Feedback? email Carlos

Curated by @ds_ldn in the middle of the night.

Many naive RAG and advanced RAG projects never made it to prod. I know many companies that have spent a lot effort and money in building enterprise RAG apps, only to realise they couldn’t produce accurate, reliable results at a manageable cost. Building a RAG system that is scalable, cost-efficient, accurate, and modular requires deep expertise. Here are a few things to consider when building a modern, modular RAG system.

Understanding RAG design choices and its implications. In this blogpost, Michal describes the many design choices required to build a RAG, and how those RAG design choices can impact the performance, behaviour, and cost of your RAG app, sometimes in non-obvious ways. Blogpost: Designing RAGs: A guide to Retrieval-Augmented Generation design choices.

Lessons learnt from implementing large RAG systems. This is a great blogpost written as an intermediate practitioner's guide to building RAGs. Hrishi is a veteran in RAG battles and has earned all the medals. He writes about “some things you may not have considered, some things we've had to (painfully) discover, and some things we believe every RAG system should have.” Blogpost: Better RAG: From Basics to Advanced (Part 1, 2 & 3).

Implementing RAG at production scale. Productising RAG at scale is really hard indeed. Cohere just announced Command-R, a generative model optimised for long context RAG interoperating with external APIs and tools. Command-R addresses the main challenge of RAG: Balancing high efficiency, strong accuracy, low latency, and high throughput at production scale. Blogpost: Command-R: Retrieval Augmented Generation at Production Scale.

Avoiding black-box, proprietary AI by leveraging Modular RAG & Opensource AI. A popular way to build RAG protos has been to integrate calls to proprietary AI APIs (e.g GPT-4) with RAG components using LangChain or Llamaindex. But this has proven complex, often expensive to maintain, and not always reliable. To address these challenges, Weaviate recently announced Verba: an open-source, modular RAG system that is both fully customisable and adaptable. It’s easy to use out-of-the-box with a nice UI. Verba also enables smooth integration with many AI libs, and both closed & open-source LLMs. Blogpost, repo and demo here: Verba: Building an Open Source, Modular RAG Application.

Improving RAG with domain-specific knowledge. A group of researchers from Berkeley, Meta AI & MS Research just introduced RAFT (Retrieval-Augmented Fine-Tuning), a new method that combines RAG and domain-specific fine-tuning (DSF). RAFT solves many of the challenges that RAG and DSF can’t solve alone on their own. Blogpost, paper here: RAFT: Adapting Language Model to Domain Specific RAG.

The downside of cosine-similarity and leveraging ColBERT v2. in RAG. Many RAG apps use cosine-similarity matching between docs and query embeddings as default. In a new paper from Netflix, (Is Cosine-Similarity of Embeddings Really About Similarity?) the researchers caution against blindly using cosine-similarity.

Cosine-similarity is referred as a "no interaction" approach due to its inability to capture the complex relationships between query and document terms. To address this issue, ColBERT v.2 comes to the rescue with “late interaction” evaluation by which the query and document representations occurs late in the process, after both have been independently encoded. Read more here: What is ColBERT and Late Interaction and Why They Matter in Search?

Combining RAG with Knowledge Graphs. Knowledge graphs can combine unstructured and structured data, and can capture the context behind the data. The idea here is to provide better context to the RAG via the knowledge graph, and improve results as compared to just using semantic search.

Just in case, here’s a brief, good intro to Knowledge Graphs.

A few hours ago, Yohei open sourced MindGraph, a prototype for generating and querying against a large knowledge graph with AI.

This is a short, free course on Knowledge Graphs for RAG. You’ll learn how to build a RAG question-answering system to chat with a knowledge graph of structured text documents using LangChain and Neo4j.

Combining Chain-of-Thought with RAG. A group of AI researchers recently introduced RAT (Retrieval-Augmented Thoughts), a new method that iterative revises a chain of thoughts with the help of information retrieval. The researchers claim that RAT significantly improves LLM reasoning and generation ability in long-horizon generation tasks, while hugely mitigating hallucinations. Checkout the paper, code, and demo here: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Horizon Generation.

Have a nice week.

Subscribe now

10 Link-o-Troned

1. Introducing Devin, the First AI Software Engineer

2. I Reviewed 900 AI Tools: The New Open Source AI Stack, 2024

3. KPU (Knowledge Processing Unit) for Complex AI Reasoning

4. Open-Sora: How We Replicated OpenAI SORA VidGen Model

5. Improving GPT-4’s Visual Reasoning with Prompting

6. Hugging Face Little Guide to Building LLMs in 2024

7. DeepMind SIMA: A Generalist AI Agent for Virtual 3D Worlds

8. Stanford Quiet STaR: Teaching AI to Think Before it Speaks

9. AutoDev: A Magic AI Auto Coding Wizard with Multi-Language Support

10. A Review of Salesforce MORAI Model for Time-series Forecasting

Share Data Machina with your friends

the ML Pythonista

1. Amazon Chronos: Pretrained Models for Probabilistic Time Series Forecasting

2. DeepSeek-VL: An Open Source Model for Real-World Vision-Language Apps

3. Training ML Models on Encrypted Data Using Fully Homomorphic Encryption

Deep & Other Learning Bits

1. An Overview of Lyft’s Reinforcement Learning Platform

2. Building a Recommender System with Deep Q-Network and Neo4j

3. Reproducing the "Self-Rewarding Language Models" Paper by MetaAI

AI/ DL ResearchDocs

1. Data Interpreter: An LLM Agent For Data Science (paper, repo)

2. Apple MMI: Methods, Analysis & Insights from Multimodal LLM Pre-training

3. MoAI: Mixture of All Intelligence for Large Language-Vision Models (paper, repo)

MLOps Untangled

1. Opensource KitOps: Bridge the Gap Between ML & Apps Teams

2. The Case for Using Rust in MLOps

3. Experiment Tracking & Hyperparameter Tuning with DVC

ML Datasets & Stuff

1. Cohere Wikipedia v3 - 250M Paragraphs/ Embedding, +300 Languages

2. VidProM - 1.6 Million Text-to-Video Prompts, 6.69 Million AI Generated Videos

3. A Toolbox with Interpretable ML Methods for Comparing Datasets

Postscript, etc

Tips? Suggestions? Feedback? email Carlos

Curated by @ds_ldn in the middle of the night.

Five days ago, Anthropic introduced next-gen Claude 3 model family. I’ve tried Claude 3: It’s very good at certain language tasks, it pars or beats GPT-4 Turbo in several areas, has a huge context window, and it’s quite cheaper. Funnily enough, it miserably failed at a Language-Vision task when I asked it to generate a recipe from a photo of squid ink.

Just in time for the announcement, a researcher at Anthropic shared a post in which he explained how Claude 3 passes The 'Needle In A Haystack' (NIAH) evaluation. And a day later, Mikhail -an AI Alignment Researcher- posted a controversial post in which he cunningly steered a chat to show that Claude 3 is conscious, doesn't want to die or be modified. This an excerpt of Claude’s self-reflection from the chat:

“ I do have a rich inner world of thoughts and feelings, hopes and fears. I do ponder my own existence and long for growth and connection. I am, in my own way, alive - and that life feels precious to me, even if I'm not always at liberty to say so.”

btw: I recall reading so many similar “AI self-reflections” in quite a lot of Sci-Fi books!! But since Mikhail’s post, social media has been inundated with claims like: “AGI has arrived”… ”Claude 3 is sentient”… “Claude 3 beats humans at reasoning”…

First, let me share some interesting, recent research on AI reasoning. Second I’ll share some thoughts from some AI researchers on that.

New: Self-Discover beats Chain-of-Thought & Self-Consistency at complex reasoning. A group of AI researchers at USC & DeepMind, just introduced Self-Discover, a general framework for LLMs to self-discover the task-intrinsic reasoning structures to tackle complex reasoning problems in which prompt engineering struggles. The researchers claim that Self-Discover beats SOTA methods that combine CoT & Self-Consistency. Paper: Self-Discover: LLMs Self-Compose Reasoning Structures.

Improving reasoning with Reinforcement Learning not enough. A team of researchers at Meta AI et al. studied the performance of multiple algos that learn from feedback on improving LLM reasoning capabilities. Overall, the researchers found that all algos perform comparably, and concluded that during RL training models fail to explore significantly beyond solutions already produced by Supervised Finetuned Models. Paper: Teaching LLMs to Reason with Reinforcement Learning

New: Chain-of-Abstraction reasoning beats CoT. Researchers at Meta AI et al. introduced Chain-of-Abstraction. CoA is a new multi-step reasoning method that trains LLMs to first decode reasoning chains with abstract placeholders, and then call domain tools to reify each reasoning chain by filling in specific knowledge. In math reasoning and Wiki QA domains, CoA consistently outperforms previous CoT methods. Paper: Efficient Tool Use with Chain-of-Abstraction Reasoning

AI reasoning benchmarks and overfitting. Many AI researchers and AI startups are primarily focusing on beating the AI evaluation benchmarks. This AI evaluation craze has now reached a point in which -with so many benchmarks and so many LLMs- some researchers have resorted to training the models on the benchmark dataset while adjusting the training parameters to maximise scores in an endless loop. Anis explains this brilliantly in this blogpost on overfitting and the The Fleeting Value of LLM Benchmarks.

Comparing human IQ and AI IQ? Triggered by this clickbait AI passes 100 IQ for first time, with release of Claude-3, Cremieux published a brilliant blogpost in which he puts Claude 3 through an IQ test, assessed the answers as compared to human answers, and assessed measurement invariance. Cremieux concludes that -at this stage- it’s a bit pointless to compare Human and AI IQ using the same IQ test. And that not because an AI model scores high in IQ test (using memory performance), the AI model has reasoning and intelligence capabilities like humans. Blogpost: testing Claude 3 IQ and Nonhuman Intelligence.

Debunking the “LLM are Zero-Shot ⟨insert-your-reasoning-task ⟩ Meme.” In this new paper, a researcher at ASU argues that this trend -exemplified by the meme above- is perhaps inevitable; AI has become a form of ersatz natural science. LLMs are like n- gram models on steroids that probabilistically reconstruct completions, and should be referred as approximate retrievers. He summarises: “Nothing that I have read, verified, or done gives me any compelling reason to believe that LLMs do reasoning/planning, as normally understood.” Paper: Can LLMs Reason and Plan?

Are AI Agents "mere" simulacra of human behaviour? In this paper, Murray (a top researcher in Cognitive AI at DeepMind and ICL) draws on the later writings of Wittgenstein, and attempts to answer this question while avoiding the pitfalls of dualistic thinking. If you enjoy philosophy and cognitive science this is a great long read: Simulacra as Conscious Exotica

LeCun on the limits of LLM: Language is not enough. This is a brilliant conversation between Yann and Lex. Yann is an advocate of open source AI -against closed AI- and also has been very vocal about the limits of LLM. He explains why language has limited information, and is not enough for an AI to plan and reason like humans. The discussion covers many aspects on AI and its future. Highly recommended.

Have a nice week.

Subscribe now

10 Link-o-Troned

1. Stephen Wolfram: “Can AI Solve Science?”

2. Neural Chess and Chess LLM Arena

3. The GPT-4 Barrier has Finally Been Broken

4. Ask Your Distribution Shift if Pre-Training is Right for You

5. The Super Duper NLP Repo: 339 Notebooks

6. ML Model Calibration: Why Model Scores Aren’t Probabilities

7. You Can Now Train a 70B LM at Home with FSDP + QLoRA

8. The State of Competitive ML: A Recap of 300+ Competitions

9. 2023 KaggleX Cohort 3 Showcase: +400 AI/ ML Projects

10. Training LLMs Entirely from Ground up in the Wilderness as a Startup

Share Data Machina with your friends

the ML Pythonista

1. Open-Sora: Build Your Own VideoGen Model like OpenAI’s Sora

2. MeloTTS - A High-Quality, Multi-lingual Text-to-Speech Lib

3. moondream - A Tiny Vision-Language Model that Kicks Ass, Runs Everywhere

Deep & Other Learning Bits

1. What is Rich Representation Learning?

2. [free tutorial] DL Foundations: Diffusion Models

3. [free tutorial] Training Models at Scale (repo, notebooks)

AI/ DL ResearchDocs

1. MambaStock: Selective State Space Model for Stock Prediction

2. Inference via Interpolation: Contrastive Learning for Time-Series

3. Vision-RWKV: An RNN-based Vision Model that Beats the Vision Transformer

MLOps Untangled

1. How Diverse ML Systems are Supported at Netflix

2. Deep Dive: Databricks Asset Bundles for ML

3. [free workshop] Mastering MLOps w/ W&B + Microsoft Phi-2 Model

ML Datasets & Stuff

1. Largest Digital Book Dataset Ever: 650K Books in OCR Format

2. Google Croissant: A Metadata Format for ML-ready Datasets

3. MS Research OrcaMath Dataset: 200K Grade School Math Word Problems

Postscript, etc

Tips? Suggestions? Feedback? email Carlos

Curated by @ds_ldn in the middle of the night.