Shale oil stored in the shale system occurs mainly in adsorbed and free states, and ascertaining the amount of adsorbed crude oil in shale is a method of ascertaining its free oil content, which determines the accuracy of shale oil resource evaluation. Both inorganic minerals and kerogen have the ability to adsorb crude oil, but there is controversy surrounding which plays the greatest part in doing so; clarifying this would be of great significance to shale oil resource evaluation. Therefore, in this study, the evolution states of inorganic minerals and kerogen in shale were changed using pyrolysis, and the adsorbents were prepared for crude oil adsorption experiments, to explore the effects of inorganic minerals and kerogen on the crude oil adsorption of shale. The results showed that the differences in kerogen’s structural units and content in organic-rich shale (TOC = 1.60−4.52%) had no obvious effects on its crude oil adsorption properties. On the contrary, inorganic minerals, as the main body of shale, played a dominant role in the adsorption of crude oil. The composition and evolution of the inorganic minerals controlled the surface properties of shale adsorbents, which is the main reason for the different crude oil adsorption properties of the different types of adsorbents. The results of this study are helpful in improving our understanding of the performance and mechanisms of shale in adsorbing crude oil and promoting the development of shale oil resource evaluation.