In 2020, residential sector loads reached 25% of the overall electrical consumption in Europe and it is foreseen to stabilise at 29% by 2050. However, this relatively small increase demands, among others, changes in the energy consuming behaviour of households. To achieve this, Demand Response (DR) has been identified as a promising tool for unlocking the hidden flexibility potential of residential consumption. In this work, a holistic incentive-based DR framework aiming towards load shifting is proposed for residential applications. The proposed framework is characterised by several innovative features, mainly the formulation of the optimisation problem, which models user satisfaction and the economic operation of a distributed household portfolio, the customised load forecasting algorithm, which employs an adjusted Gradient Boosting Tree methodology with enhanced feature extraction and, finally, a disaggregation tool, which considers electrical features and time of use information. The DR framework is first validated through simulation to assess the business potential and is then deployed experimentally in real houses in Northern Greece. Results demonstrate that a mean 1.48% relative profit can be achieved via only load shifting of a maximum of three residential appliances, while the experimental application proves the effectiveness of the proposed algorithms in successfully managing the load curves of real houses with several residents. Correlations between market prices and the success of incentive-based load shifting DR programs show how wholesale pricing should be adjusted to ensure the viability of such DR schemes.