In this paper, the appropriate rated power of battery energy storage system (BESS) and the operating limit capacity of wind farms are determined considering power system stability, and novel output control methods of BESS and wind turbines are proposed. The rated power of BESS is determined by correlation with the kinetic energy that can be released from wind turbines and synchronous generators when a disturbance occurs in the power system. After the appropriate rated power of BESS is determined, a novel control scheme for quickly responding to disturbances should be applied to BESS. It is important to compensate the insufficient power difference between demand and supply more quickly after a disturbance, and for this purpose, BESS output is controlled using the rate of change of frequency (ROCOF). Generally, BESS output is controlled by the frequency droop control (FDC), however if ROCOF falls below the threshold, BESS output increases sharply. Under this control for BESS, the power system’s stability can be improved and the operating limit capacity of wind farms can be increased. The operating limit capacity is determined as the smaller of technical limit and dynamic limit capacity. The technical limit capacity is calculated by the difference between the maximum power of the generators connected to the power system and the magnitude of loads, and the dynamic limit capacity is determined by considering dynamic stability of a power system frequency when the wind turbines drop out from a power system. Output of the dynamic model developed for wind turbine is based on the operating limit capacity and is controlled by blade pitch angle. To validate the effectiveness of the proposed control method, different case studies are conducted, with simulations for BESS and wind turbine using Power System Simulation for Engineering (PSS/E).