Psychological distress caused by decreased physical activity (PA) is a growing concern among the elderly due to public health measures since the coronavirus disease (COVID-19). We aimed to (1) assess how public health restrictions impact PA, subjective well-being (SWB), and health-related quality of life (HRQoL) of community-dwelling elderly, and (2) investigate risk factors that lead to a decline in PA. Self-administered questionnaires assessed the changes in PA, SWB, HRQoL. Multivariate logistic regression analysis was performed to identify significant associated risk factors for decreased PA. Of 165 participants (valid response rate, 41.3%; mean age, 78.5 +/- 8.0 years), 47.3% became less active, 23.0% became more active, and 29.7% maintained PA levels. There was a significant decrease in SWB at baseline and follow-up after COVID-19 restrictions in the less active group (p < 0.01). Higher levels of moderate or strenuous exercise/sports activity at baseline (odds ratio [OR], 1.12; 95% confidence interval [CI], 1.01-1.24), and lower mental component HRQoL scores at baseline (OR, 0.96; 95% CI, 0.93-0.99) were associated with an increased risk of decreased PA. Public health restrictions impact the PA of the elderly, especially those who had higher levels of exercise/sports activity and lower HRQoL before COVID-19. Decreased PA was strongly associated with lower SWB.