Vis-NIR and XRF spectroscopy are widely used in monitoring heavy metals in soil due to their advantages of being fast, non-destructive, cost-effective, and non-polluting. However, when used individually, XRF and vis-NIR may not meet the accuracy requirements for Cd determination. In this study, we focused on the impact area of a non-ferrous metal smelting slag site in Gejiu City, Yunnan Province, fused the pre-selected vis-NIR and XRF spectra using the Pearson correlation coefficient (PCC), and identified the characteristic spectra using the competitive adaptive reweighted sampling (CARS) method. Based on this, a quantitative model for soil Cd concentration was established using partial least squares regression (PLSR). The results showed that among the four fusion spectral quantitative models constructed, the model combining vis-NIR spectral second-order derivative transformation and XRF spectral first-order derivative transformation (D2(vis-NIR) + D1(XRF)) had the highest coefficient of determination (R2 = 0.9505) and the smallest root mean square error (RMSE = 0.1174). Compared to the estimation models built using vis-NIR and XRF spectra alone, the average computational time of the fusion models was reduced by 68.19% and 63.92%, respectively. This study provides important technical means for real-time and large-scale on-site rapid estimation of Cd content using multi-source spectral fusion.