Buildings account for a significant portion of our overall energy usage and associated greenhouse gas emissions. With the increasing concerns regarding climate change, there are growing needs for energy reduction and increasing our energy efficiency. Forecasting energy use plays a fundamental role in building energy planning, management and optimization. The most common approaches for building energy forecasting include physics and data-driven models. Among the data-driven models, deep learning techniques have begun to emerge in recent years due to their: improved abilities in handling large amounts of data, feature extraction characteristics, and improved abilities in modelling nonlinear phenomena. This paper provides an extensive review of deep learning-based techniques applied to forecasting the energy use in buildings to explore its effectiveness and application potential. First, we present a summary of published literature reviews followed by an overview of deep learning-based definitions and techniques. Next, we present a breakdown of current trends identified in published research along with a discussion of how deep learning-based models have been applied for feature extraction and forecasting. Finally, the review concludes with current challenges faced and some potential future research directions.