One of the established unstable power swing (out-of-step) detection algorithms in micro grid/smart grid power systems uses a trajectory of apparent impedance in the R-X plane. However, this algorithm is not suitable for fast out-of-step conditions and it is hard to detect out-of-step conditions exactly. Another algorithm for out-of-step detection is using phasor measurement units (PMUs). However, PMUs need extra equipment. This paper presents the out-of-step detection algorithm using the trajectory of complex power. The trajectory of complex power and generator mechanical power is used to identify out-of-step conditions. A second order low pass digital filter is used to extract the generator mechanical power from the complex power. Variations of complex power are used to identify equilibrium points between stable and unstable conditions. The proposed out-of-step algorithm is based on the modification of assessment of a transient stability using equal area criterion (EAC). The proposed out-of-step algorithm is verified and tested by using alternative transient program/electromagnetic transient program (ATP/EMTP) MODELS.