The absence of electricity is among the gravest problems preventing a nation’s development. Hybrid renewable energy systems (HRES) play a vital role to reducing this issue. The major goal of this study is to use the non-dominated sorting genetic algorithm (NSGA)-II and hybrid optimization of multiple energy resources (HOMER) Pro Software to reduce the net present cost (NPC), cost of energy (COE), and CO2 emissions of proposed power system. Five cases have been considered to understand the optimal HRES system for Kutubdia Island in Bangladesh and analyzed the technical viability and economic potential of this system. To demonstrate the efficacy of the suggested strategy, the best case outcomes from the two approaches are compared. The study’s optimal solution is also subjected to a sensitivity analysis to take into account fluctuations in the annual wind speed, solar radiation, and fuel costs. According to the data, the optimized PV/Wind/Battery/DG system (USD 711,943) has a lower NPC than the other cases. The NPC obtained by the NSGA-II technique is 2.69% lower than that of the HOMER-based system.