Olive stones are an abundant lignocellulose material in the countries of the Mediterranean basin that could be transformed to bioethanol by biochemical pathways. In this work, olive stones were subjected to fractionation by means of a high-temperature dilute-acid pretreatment followed by enzymatic hydrolysis of the pretreated solids. The hydrolysates obtained in these steps were separately subjected to fermentation with the yeast Pachysolen tannophilus ATCC 32691. Response surface methodology with two independent variables (temperature and reaction time) was applied for optimizing D-xylose production from the raw material by dilute acid pretreatment with 0.01 M sulfuric acid. The highest D-xylose yield in the liquid fraction was obtained in the pretreatment at 201 °C for 5.2 min. The inclusion of a detoxification step of the acid prehydrolysate, by vacuum distillation, allowed the fermentation of the sugars into ethanol and xylitol. The enzymatic hydrolysis of the pretreated solids was solely effective when using high enzyme loadings, thus leading to easily fermentable hydrolysates into ethanol. The mass macroscopic balances of the overall process illustrated that the amount of inoculum used in the fermentation of the acid prehydrolysates strongly affected the ethanol and xylitol yields.