Simultaneous satisfaction of both thermal and visual comfort in buildings may be a challenging task. Therefore, this paper suggests a comprehensive framework for the building energy optimization process integrating computational fluid dynamics (CFD) daylight simulations. A building energy simulation tool, IDA Indoor Climate and Energy (IDA-ICE), was coupled with three open-source tools including GenOpt, OpenFOAM, and Radiance. In the optimization phase, several design variables i.e., building envelope properties, fenestration parameters, and Heating, Ventilation and Air-Conditioning (HVAC) system set points, were selected to minimize the total building energy use and simultaneously improve thermal and visual comfort. Two different scenarios were investigated for retrofitting of a generic office building located in Oslo, Norway. In the first scenario a constant air volume (CAV) ventilation system with a local radiator in each zone was used, while an all-air system equipped with a demand control ventilation (DCV) was applied in the second scenario. Findings showed that, compared to the reference design, significant reduction of total building energy use, around 77% and 79% in the first and second scenarios, was achieved respectively, and thermal and visual comfort conditions were also improved considerably. However, the overall thermal and visual comfort satisfactions were higher when all-air system was applied.