Voltage source converter (VSC) has been extensively applied in renewable energy systems which can rapidly regulate the active and reactive power. This paper aims at developing a novel optimal nonlinear adaptive control (ONAC) scheme to control VSC in both rectifier mode and inverter mode. Firstly, the nonlinearities, parameter uncertainties, time-varying external disturbances, and unmodelled dynamics can be aggregated into a perturbation, which is then estimated by an extended state observer (ESO) called high-gain perturbation observer (HGPO) online. Moreover, the estimated perturbation will be fully compensated through state feedback. Besides, the observer gains and controller gains are optimally tuned by a recent emerging biology-based memetic salp swarm algorithm (MSSA), the utilization of such method can ensure a desirably satisfactory control performance. The advantage of ONAC is that even though the operation conditions are constantly changing, the control performance can still be maintained to be globally consistent. In addition, it is noteworthy that in rectifier mode only the reactive power and DC voltage are required to be measured, while in inverter mode merely the reactive power and active power have to be measured. At last, in order to verify the feasibility of ONAC in practical application, a hardware experiment is implemented.