Integrating inverter-based generators in power systems introduces several challenges to conventional protection relays. The fault characteristics of these generators depend on the inverters’ control strategy, which matters in the detection and classification of the fault. This paper presents a comprehensive machine-learning-based approach for detecting and classifying faults in transmission lines connected to inverter-based generators. A two-layer classification approach was considered: fault detection and fault type classification. The faults were comprised of different types at several line locations and variable fault impedance. The features from instantaneous three-phase current and voltages and calculated swing-center voltage (SCV) were extracted in time, frequency, and time−frequency domains. A photovoltaic (PV) and a Doubly-Fed Induction Generator (DFIG) wind farm plant were the considered renewable resources. The unbalanced data problem was investigated and mitigated using the synthetic minority class oversampling technique (SMOTE). The hyperparameters of the evaluated classifiers, namely decision trees (DT), Support Vector Machines (SVM), k-Nearest Neighbors (k-NN), and Ensemble trees, were optimized using the Bayesian optimization algorithm. The extracted features were reduced using several methods. The classification performance was evaluated in terms of the accuracy, specificity, sensitivity, and precision metrics. The results show that the data balancing improved the specificity of DT, SVM, and k-NN classifiers (DT: from 99.86% for unbalanced data to 100% for balanced data; SVM: from 99.28% for unbalanced data to 99.93% for balanced data; k-NN: from 99.64% for unbalanced data to 99.74% for balanced data). The forward feature selection combined with the Bag ensemble classifier achieved 100% accuracy, sensitivity, specificity, and precision for fault detection (binary classification), while the Adaboost ensemble classifier had the highest accuracy (99.4%), compared to the other classifiers when using the complete set of features. The classification models with the highest performance were further tested using a new dataset test case. They showed high detection and classification capabilities. The proposed approach was compared with the previous methodologies from the literature.