Background: Male sex is related to increased COVID-19 severity and fatality although confirmed infections are similarly distributed between men and women. The aim of this retrospective analysis was to investigate the impact of sex hormones on disease progression and immune activation in men with COVID-19. Patients and Methods: We studied for effects of sex hormones on disease severity and immune activation in 377 patients (230 men, 147 women) with PCR-confirmed SARS-CoV-2 infections hospitalized at the Innsbruck University Hospital between February and December 2020. Results: Men had more severe COVID-19 with concomitant higher immune system activation upon hospital admission when compared to women. Men with a severe course of infection had lower serum total testosterone (tT) levels whereas luteinizing hormone (LH) and estradiol (E2) levels were within the normal range. tT deficiency was associated with elevated CRP (rs = - 0.567, p < 0.001), IL-6 levels (rs = - 0.563, p < 0.001), lower cholesterol levels (rs = 0.407, p < 0.001) and an increased morbidity and mortality. Men with tT levels < 100 ng/dL had a more than eighteen-fold higher in-hospital mortality risk (OR 18.243 [95%CI 2.301 - 144.639], p = 0.006) compared to men with tT levels > 230 ng/dL. Moreover, while morbidity and mortality showed a positive correlation with E2 levels at admission, we detected a negative correlation with the tT/E2 ratio upon hospital admission. Conclusion: Hospitalized men with COVID-19 present with rather low testosterone levels linked to more advanced immune activation, severe clinical manifestations translating into an increased risk for ICU admission or death. The underlying mechanisms remain elusive but may include infection driven hypogonadism as well as inflammation mediated cholesterol reduction causing gonadotropin suppression and impaired androgen formation. Finally, in elderly late onset hypogonadism might also contribute to lower testosterone levels.