The issue of urban expansion encroaching on Egypt’s limited agricultural land poses significant economic and social challenges. Despite efforts made by the nation to curtail the movement of illegal constructions under the law, the dissemination of illegal constructions continues to be difficult because thousands of aerial photographs need manual monitoring. We present an automated solution in this paper to overcome this challenge by daily visual inspection of illegally built buildings from aerial space to detect illegal buildings on an automated basis. The proposed system incorporates a two-step process by way of image segmentation with a U-Net model and using a convolutional neural network (CNN) to spot buildings. The aerial images are first segmented into 500x500 meter zones, which later are segmented into 100x100 meter subregions. This optimized segmentation is processed by CNN for illegal building classification and finding out what is illegal. The proposed approach adequately alleviates issues like data imbalance and classification accuracy and achieves an F1 score of 94.94%, an accuracy of 91.47%, and a recall of 95.2%. When applied to eight Egyptian governorates namely those of the key agricultural areas of the Delta and Alexandria — the system performed well and showed a high level of discriminative ability in the separation of legal and illegal structures. Comparative analysis with other models highlighted the system’s superior segmentation and recognition accuracy. Drawing on recent developments in artificial intelligence (AI) and GA, the system provides a powerful, scalable and efficient tool for policymakers to monitor illegal constructions while maintaining the protection of Egypt’s agricultural resources and encouraging sustainable land use.