Irradiation-induced swelling plays a key role in determining fuel performance. Due to their high cost and time demands, experimental research methods are ineffective. Knowledge-based multiscale simulations are also constrained by the loss of trustworthy theoretical underpinnings. This work presents a new trial of integrating knowledge-based finite element analysis (FEA) with a data-driven deep learning framework, to predict the hydrostatic-pressure−temperature dependent fission swelling behavior within a CERCER composite fuel. We employed the long short-term memory (LSTM) deep learning network to mimic the history-dependent behaviors. Training of the LSTM is achieved by processing the sequential order of the inputs to do the forecasting; the input features are fission rate, fission density, temperature, and hydrostatic pressure. We performed the model training based on a leveraged dataset of 8000 combinations of a wide range of input states and state evaluations that were generated by high-fidelity simulations. When replicating the swelling plots, the trained LSTM deep learning model exhibits outstanding prediction effectiveness. For various input variables, the model successfully pinpoints when recrystallization first occurs. The preliminary study for model interpretation suggests providing quantified insights into how those features affect solid and gaseous portions of swelling. The study demonstrates the efficacy of combining data-driven and knowledge-based modeling techniques to assess irradiation-induced fuel performance and enhance future design.