Multiscale systems biology and systems pharmacology are powerful methodologies that are playing increasingly important roles in understanding the fundamental mechanisms of biological phenomena and in clinical applications. In this review, we summarize the state of the art in the applications of agent-based models (ABM) and hybrid modeling to the tumor immune microenvironment and cancer immune response, including immunotherapy. Heterogeneity is a hallmark of cancer; tumor heterogeneity at the molecular, cellular, and tissue scales is a major determinant of metastasis, drug resistance, and low response rate to molecular targeted therapies and immunotherapies. Agent-based modeling is an effective methodology to obtain and understand quantitative characteristics of these processes and to propose clinical solutions aimed at overcoming the current obstacles in cancer treatment. We review models focusing on intra-tumor heterogeneity, particularly on interactions between cancer cells and stromal cells, including immune cells, the role of tumor-associated vasculature in the immune response, immune-related tumor mechanobiology, and cancer immunotherapy. We discuss the role of digital pathology in parameterizing and validating spatial computational models and potential applications to therapeutics.