The optimization of residential heating, ventilation, and air conditioning (HVAC) systems is crucial for reducing energy consumption and maintaining user comfort. As urbanization increases, there is a demand for smart buildings with energy-consuming appliances. Researchers are developing HVAC control strategies, especially for commercial buildings with complex load patterns. This paper presents a novel model-free deep reinforcement learning (RL) approach. RL allows systems to learn and adapt to individual occupant preferences, creating customized comfort Using the Deep Deterministic Policy Gradient (DDPG) algorithm to design an optimal control strategy for multi-zone residential HVAC systems. DDPG aims to reduce energy costs while maintaining occupant comfort. The DDPG reinforcement learning technique is applied to control a multi-zone commercial HVAC system, aiming to minimize energy costs while maintaining occupant comfort. The DDPG method achieves a 56% faster convergence time compared to linear HVAC control methods and a 15% improvement over linear reinforcement learning models. The mean steps required for DDPG and Linear RL models are 9.9 and 115.3, respectively.