Representative volume element (RVE) is an important parameter in numerical tests of mechanical properties of heterogeneous geomaterials. For this study, a digital image processing (DIP) technology was proposed for estimating the RVE of heterogeneous geomaterials. A color image of soil and rock mixture (SRM) with size of 400 × 400 mm2 taken from a large landslide was used to illustrate the determination procedure of the SRM. Six sample sizes ranging from 40 × 40 mm2 to 240 × 240 mm2 were investigated, and twelve random samples were taken from the binarized image for each sample size. A connected-component labeling algorithm was introduced to identify the microstructure. After establishing the numerical finite difference models of the samples, a set of numerical triaxial tests under different confining pressures were carried out. Results show that the size of SRM sample affects the estimation of the mechanical properties, including compressive strength, cohesion, and internal friction angle. The larger the size of the samples, the less variability of the estimated mechanical properties. The coefficient of variation (CV) was applied to measure the variability of mechanical properties, and the RVE of the SRM was determined easily with a predefined acceptance threshold of the CV. The results show that a DIP-based modeling method is an effective method got the RVE determination of heterogeneous geomaterials.