A Performance Evaluation of BERT-Based Models for Text Classification Tasks

Abstract

BERT (Bidirectional Encoder Representations from Transformers) has revolutionized natural language processing with its ability to capture contextual information through deep bidirectional representations. This paper evaluates BERT and its variants (DistilBERT and RoBERTa) on a suite of text classification tasks including sentiment analysis, topic classification, and spam detection. Datasets include IMDB reviews, AG News, and Enron spam email corpus. Models are fine-tuned with task-specific heads and compared to LSTM and CNN baselines. BERT outperforms all baselines, achieving 94.2% accuracy on IMDB and 96.8% on AG News. RoBERTa slightly surpasses BERT in most tasks but requires more training time and memory. DistilBERT offers competitive performance with 40% fewer parameters, making it suitable for edge deployments. We examine hyperparameter sensitivities, training stability, and inference latency across models. Results indicate that BERT's pretraining depth allows for greater generalization across diverse tasks with minimal tuning. However, resource requirements remain high, particularly in low-latency environments. This study affirms the dominance of transformer-based models in text classification while providing comparative insights into their trade-offs. Our analysis informs practitioners choosing between accuracy, speed, and computational cost when deploying BERT-like models in real-world NLP applications.