Data Machina #216
AI Coding. Code LLama. WizardCoder 34B. Phind 34B. Lemur Models. Auto Train Advanced. Prompt2Model. Graph of Thoughts. GigaGAN. Meta AI Seamless M4T SOTA Speech & Text Translation
AI Coding: The Llamas Beating GPT-4? About 6 months ago or so, many AI pundits and AI experts were quick to praise Open AI’s massive moat, and its unsurpassable lead in “all things AI…” Fast forward today, we are starting to watch Open AI’s models being challenged by many open source, smaller, Llama-based models across several fronts. I suspect most companies will start “doubting” black-box AI APIs soon and start investing in Llama-based models because:
The open-sourcing of Llama for research and commercial use, has unleashed a massive ecosystem of Llama-based models and apps
New super efficient, computational methods… powerful Llama inference in C … quantisation, model compression, knowledge distillation… train Llama 65B in 1 GPU… massive context size enablement like the new MS DeepSpeed Ulysses
New, open-source, improved instruction and fine-tuning methods for training models on specialised tasks and domain knowledge specialisation. Checkout this interesting new paper with lots of insights: Instruction Tuning for LLMs: A Survey
Code Llama: SOTA Coding Models. Three days ago Meta AI released Code Llama, a new family of SOTA Coding models. This has pretty much revolutionised the landscape of coding LLMs in a matter of days. Key points:
Three models: 1) Code Llama, the foundational code model, 2) Code Llama - Python specialised for Python; and 3) Code Llama - Instruct, which is fine-tuned for understanding natural language instructions.
Three versions: 7B, 13B, and 34B. There is no 65B or 70B version.
Capable of generating code, and natural language about code, from both code and natural language prompts
Support for 100K context size, which is pretty huge and very useful for coding
Free for research and commercial use!
Read more details here: Introducing Code Llama, a state-of-the-art large language model for coding. The original paper: Code Llama: Open Foundation Models for Code.
Phind models and fine-tuning. A few days ago, a team of researchers at a startup called Phind, just came up with a clever approach to match GPT-4 coding performance by fine-tuning Code LLama models. They fine-tuned the models on a proprietary dataset of ~80k high-quality programming problems and solutions. Unlike most other coding models approaches, they Instead used code completion examples, from a dataset with instruction-answer pairs. Their approach uses native fine-tuning and doesn’t use LoRA. Read more: Beating GPT-4 on HumanEval with a Fine-Tuned CodeLlama-34B
WizardCoder models and instruction fine-tuning. Back in June, researchers at MSR published a paper called: WizardCoder: Empowering Code LLMs with Evol-Instruct. WizardCoder, is a new family of Llama-based coding models. The models are empowered with complex instruction fine-tuning, by adapting the Evol-Instruct method to the domain of code. Just days ago, the team has released the new WizardCoder-Python-34B-V1.0 which surpasses GPT4, ChatGPT-3.5, and Claude2 in the HumanEval benchmark. Want to know more? Checkout the video below:
Lemur models: balancing coding & text. Until recently, most open source coding LMs, were focused mostly in coding but not in text. A few days ago, researchers at XLang Lab, open-sourced Lemur, a new family of SOTA coding models that balance coding and text capabilities. This could be very useful for developing complex language apps, that require understanding, reasoning, planning, coding, and context grounding capabilities.
Some AI humour for the w/e. AI Dad Jokes is an AI that writes humorous content inspired by images. Throw an image to it, get a joke. LOL!
Have a nice week.
10 Link-o-Troned
the ML Pythonista
Deep & Other Learning Bits
AI/ DL ResearchDocs
data v-i-s-i-o-n-s
MLOps Untangled
AI startups -> radar
ML Datasets & Stuff
Kelvin Legal- the Largest Legal Training Dataset, 150B Tokens
Public Dataset with 3K Clinical MRIs of Patients with Acute Stroke
Postscript, etc
Tips? Suggestions? Feedback? email Carlos
Curated by @ds_ldn in the middle of the night.