Data Machina #174
Transformers for text summarisation, fine-tuning LLMs, conformal predictions, neural forecasting, NeRFs, kNN-LM...
On Transformers and Text Summarisation. Long and abstractive text summarisation is a challenging task in natural language modelling, generation and understanding.
Unlike extractive summarisation (an information-retrieval task that copies sections of the source text into the summary,) abstractive text summarisation aims to understand the full document and generate paraphrased text to summarise the salient points.
Hallucinations? Transformer-based summaries can be a bit repetitive, inconsistent, containing some grammar and factual errors. See Problems with Existing Abstractive Text Summarisation Models- Even SoTA. And they’re also limited in terms of text length due to the attention, sequence model constraints.
In How Far are We from Robust Long Abstractive Summarisation? the team @MonashUni et al. goes deep & wide on what defines a robust abstractive summarization system. They also generate several summaries with Longformer- based, pre-trained BART and PEGASUS transformers. And finally they score the results against the ROUGE benchmark. Below some links relevant to the paper:
Allen AI Longformer - Pre-trained transformer for long documents
Google AI Pegasus - SoTA transformer for abstractive summarisation
Facebook AI BART - A denoising autoencoder for text comprehension
ROUGE - A package for auto evaluation of text summaries
Here is a nice intro on Transformers BART Model for Text Summarisation
Just a few weeks ago, a team @UniofIllinois published HETFORMER: Heterogeneous Transformer with Sparse Attention for Long-Text Extractive Summarisation. The team claims they achieved SoTA on extractive summarisation tasks in Rouge F1 benchmark while using less memory and fewer parameters.
Have a nice week.
10 Link-o-Troned
A Pythonista *Experience*
Scripting aRt
Deep & Other Learning Bits
ResearchDocs
Algorithmic Potpourri
El Robótico
data v-i-s-i-o-n-s
DataEng Wranglings
startups -> radar
ML Datasets & Stuff
Postscript, etc
Tips? Suggestions? Feedback? email Carlos
Curated by @ds_ldn in the middle of the night.