In recent years, digital communication captured massive attention among people, and it plays a significant role in various streams such as banking, healthcare departments, industries, information technologies, and more. At present, all data transfers are transmitted through the internet, which requires a high level of security to transfer the original message until it reaches the destination. Cryptography and Steganography are the two important functionalities that ensure data security over open internet sources. Steganography is the procedure that deals with hiding secret text, audio, and video within massive data. It is found to be a useful source as well as it paves the way for secured communication between two groups to hide the information. However, existing methods are not capable enough to provide efficient results and key creation frequently depend on predictable, deterministic techniques, which might not fully account for anomalies or inefficiencies in the key selection procedure. Hence, proposed method introduces effective key generation using an Optimized Genetic Algorithm (OGA) as it produces complex keys along with Enhanced Rivest – Shamir- Adleman (RSA) encryption and Enhanced RSA decryption algorithm. Enhanced Discrete Wavelet Transform (E-DWT) is employed for the compression and decompression process, and Improvised Lifting Wavelet Transform (I-LWT) is used for the data embedding process. The Genetic algorithm with the Rabin Miller Primality test (RMPT) is mainly proposed for the complex key generation process with secured communication. The private and public keys are generated using an optimized genetic algorithm. Performance metrics are employed to evaluate and analyse the capability of the proposed method considering the Peak signal noise ratio (PSNR), Single to noise ratio (SNR), and Mean Squared Error (MSE) metrics. The proposed model resulted in of MSE rate of 0.00000000042036 and an SNR value of 99.98 %.