Comparative Analysis of Transformer Architectures for Abstractive Text Summarization: Achieving Superior Performance with PEGASUS

Abstract

We present a comparative analysis of four transformer-based architectures—BERT, T5, BART, and PEGASUS—for abstractive text summarization, focusing on their ability to generate concise and semantically meaningful summaries from large textual documents. The study employs a dataset of approximately 300,000 document-summary pairs derived from CNN/Daily Mail, XSum, and scientific article repositories. Our methodology involves a comprehensive preprocessing pipeline, tokenization with transformer-specific tokenizers, and fine-tuning of each model using cross-entropy loss and the AdamW optimizer. The evaluation relies on ROUGE metrics to quantify n-gram overlap and longest common subsequence similarity. Experimental results demonstrate a clear performance hierarchy: BERT achieves ROUGE-1 of 42.3, ROUGE-2 of 19.4, and ROUGE-L of 39.1; T5 improves these scores to 45.8, 22.6, and 42.5, respectively; BART further raises them to 47.2, 23.9, and 44.1; and PEGASUS attains the highest scores with 49.6, 25.8, and 46.7. Therefore, the novelty of this work lies in systematically identifying PEGASUS as the superior architecture for abstractive summarization, attributed to its gap-sentence pretraining strategy. The significance of this research is its potential to mitigate information overload across domains such as news aggregation, scientific literature, legal analysis, and legal document analysis. Our findings collectively advance the development of intelligent document management systems by providing a clear benchmark for model selection in real-world summarization tasks.