The driving pattern has an important influence on the parameter optimization of the energy management strategy (EMS) for hybrid electric vehicles (HEVs). A new algorithm using simulated annealing particle swarm optimization (SA-PSO) is proposed for parameter optimization of both the power system and control strategy of HEVs based on multiple driving cycles in order to realize the minimum fuel consumption without impairing the dynamic performance. Furthermore, taking the unknown of the actual driving cycle into consideration, an optimization method of the dynamic EMS based on driving pattern recognition is proposed in this paper. The simulation verifications for the optimized EMS based on multiple driving cycles and driving pattern recognition are carried out using Matlab/Simulink platform. The results show that compared with the original EMS, the former strategy reduces the fuel consumption by 4.36% and the latter one reduces the fuel consumption by 11.68%. A road test on the prototype vehicle is conducted and the effectiveness of the proposed EMS is validated by the test data.