Mathematical models are powerful tools to investigate, simulate, and evaluate potential interventions for infectious diseases dynamics. Much effort has focused on the Susceptible-Infected-Recovered (SIR)-type compartment models. These models consider host populations and measure change of each compartment. In this study, we propose an alternative patch dynamic modeling framework from pathogens' perspective. Each patch, the basic module of this modeling framework, has four standard mechanisms of pathogen population size change: birth (replication), death, inflow, and outflow. This framework naturally distinguishes between-host transmission process (inflow and outflow) and within-host infection process (replication) during the entire transmission-infection cycle. We demonstrate that the SIR-type model is actually a special cross-sectional and discretized case of our patch dynamics model in pathogens' viewpoint. In addition, this patch dynamics modeling framework is also an agent-based model from hosts' perspective by incorporating individual host's specific traits. We provide an operational standard to formulate this modular-designed patch dynamics model. Model parameterization is feasible with a wide range of sources, including genomics data, surveillance data, electronic health record, and from other emerging technologies such as multiomics. We then provide two proof-of-concept case studies to tackle some of the existing challenges of SIR-type models: sexually transmitted disease and healthcare acquired infections. This patch dynamics modeling framework not only provides theoretical explanations to known phenomena, but also generates novel insights of disease dynamics from a more holistic viewpoint. It is also able to simulate and handle more complicated scenarios across biological scales such as the current COVID-19 pandemic.