Modeling and assessing the sustainability of machining systems has been considered to be a crucial approach to improving the environmental performance of machining processes. As the most common machining system, the computer numerical control (CNC) milling system is a typical man−machine cooperative system where the activities of the machine tool and operator generate material and energy consumption. However, the energy consumption of the operator in the CNC milling system has often been ignored in most existing research. Therefore, existing methods fail to provide a comprehensive understanding of the sustainability of the CNC milling system. To fill this gap, an exergy loss assessment method is proposed to investigate the sustainability of the CNC milling system, where the energy consumption of the operator, the energy consumption of the machine tool, and material consumption are taken into consideration. The key performance indexes of the energy consumption of the operator, the energy consumption of the machine tool, the exergy loss, and the specific exergy loss (SEL) are analyzed and modeled. To demonstrate the feasibility of the proposed method, a case study was carried out on a three-axis machining center (XH714D), in which the energy consumption of the operator, the energy consumption of the machine tool, the exergy loss of energy consumption, the exergy loss of chips, the exergy loss of compressed air, the exergy loss of cutting tool wear, the exergy loss of cooling liquid dissipation, and the SEL were found to be 169,750 J, 758,211 J, 603,131 J, 2,031,404 J, 22,023 J, 301,868 J, 2673 J, and 88.04 J/mm3, respectively. The proposed method is effective to assess the sustainability of the CNC milling system, and the established exergy loss models build a good basis for exergy efficiency optimization.