The development of a novel hybrid algorithm (AOA) with the aid of simulated annealing technique is discussed in this paper. The novel algorithm, named modified arithmetic optimization algorithm (mAOA), is proposed as an effective tool for optimizing power system stabilizer (PSS) adopted in a single-machine infinite-bus power system. To perform the assessments, MATLAB/Simulink software was used. The evaluations on the proposed algorithm are initially performed using several benchmark functions that have unimodal and multimodal natures. The results are then compared with five of the other competitive approaches (arithmetic optimization algorithm, simulated annealing algorithm, genetic algorithm, particle swarm optimization and gravitational search algorithm). The comparisons with respect to those algorithms demonstrate the great promise of the constructed hybrid mAOA algorithm. This shows the greater balance between global and local search stages achieved by the mAOA algorithm. The performance of the developed mAOA algorithm is also assessed through designing an optimally performing PSS for further evaluation which allows the observation of its capability for complex real world engineering problems. To do so, PSS damping controller is formulated as an optimization problem and the constructed mAOA algorithm is used to search for optimal controller parameters to demonstrate the applicability and the greater performance of the proposed hybrid algorithm for such a complex real-world engineering problem. The obtained results for the latter case are compared with the sine-cosine and symbiotic organisms search algorithms as they are the best performing reported algorithms. The comparisons have demonstrated the superiority of the mAOA algorithm over reported best performing algorithms in terms of PSS design, as well.