The detection xof surface anomalies, such as potholes and cracks, in a timely and accurate manner is crucial for road safety. In this paper, we have proposed a deep learning based approach. We have used a pre-trained VGG-16 model for robust feature extraction and a custom CNN for classification. Our model has an accuracy of 93% for detecting road anomalies and shows better performance than conventional methods. The proposed system has promising potential to be integrated into the navigation of autonomous vehicles and infrastructure of smart city in real-time analysis. In this study, a system based on a Convolutional Neural Net-work (CNN) has been developed to detect and classify road anomalies in real-time. The system identifies cracks, potholes, and normal road surfaces, while supporting safer autonomous driving. CNN model was trained using transfer learning. The VGG-16 model was further trained for road anomaly classification. The model was trained using a dataset of images with three types of road surface viz. cracks, potholes and normal surfaces. The model achieved an accuracy rate of 93.0% in distinguishing among these road conditions. Performance metrics were evaluated to confirm the effectiveness of the model. Matthew’s correlation coefficient (MCC) was found to be 0.8934, while Cohen’s kappa (CK) was 0.896. The mean square error (MSE) was calculated to be 0.0315 and the peak signal-to-noise ratio (PSNR) was 63.147 dB. Multiple evaluation criteria were used to validate the performance of the model. The obtained results were compared with various deep learning algorithms. Transfer learning was found to be an effective and reliable approach for automating road anomaly detection. This method provides a cost-effective way to make self-driving cars safer by allowing for the accurate detec-tion of road surface anomalies in real-time.