MatMul & Tensors All The Way Down. My friend -who is a PhD in Physics- says that matmul and tensors are just regular stuff in physics; she doesn’t quite seem to grasp why this is so new or exciting for the AI/ML community.
She tells me that I should watch The Theoretical Minimum, the Stanford’s video course taught by the world renowned physicist Leonard Susskind. “You can learn how tensors are so pervasive in physics,” she clobbered me.
Days ago, Deepmind published a paper on a new, super fast matrix multiplication method for very large matrices. In this post they explain how they used AlphaTensor - a reinforcement learning agent- to discover a faster matmul approach.
Yannic has published a great deep dive and video explainer of AlphaTensor. Check it out. Recommended.
A lot of people in the AI community say that AlphaTensor is a “game changer.” Because until now the Strassen’s Algo was the fastest matmul approach.
Fabian argued in his post On AlphaTensor’s new matrix multiplication algorithms that AlphaTensor is for large matrices that are expensive to compute. But at the end of the day, he writes: “if you’re multiplying 4×4 matrices of scalars, the standard algo remains the way to go.”
Back in 2018, a team @Caltech developed StrassenNets: Deep Learning with a Multiplication Budget, which re-discovered Strassen's algo, and reduced matmul by 99.5% while maintaining accuracy. This team now argues that “AlphaTensor may not be fully stable.”
In parallel, Meta AI has open sourced AITemplate a new, efficient, super fast, revolutionary inference engine. AIT breaks away from NVIDIA’s proprietary, black box TensorRT dependencies and enables a unified inference system for both AMD and NVIDIA.
Have a nice Sunday.
10 Link-o-Troned
A Pythonista *Experience*
Scripting aRt
data v-i-s-i-o-n-s
Distributed de-Entangler
ResearchDocs
Algorithmic Potpourri
El Robótico
Deep & Other Learning Bits
startups -> radar
ML Datasets & Stuff
Postscript, etc
Tips? Suggestions? Feedback? email Carlos
Curated by @ds_ldn in the middle of the night.