Rationale: Given the rapid spread of COVID-19, an updated risk-stratify prognostic tool could help clinicians identify the high-risk patients with worse prognoses. We aimed to develop a non-invasive and easy-to-use prognostic signature by chest CT to individually predict poor outcome (death, need for mechanical ventilation, or intensive care unit admission) in patients with COVID-19. Methods: From November 29, 2019 to February 19, 2020, a total of 492 patients with COVID-19 from four centers were retrospectively collected. Since different durations from symptom onsets to the first CT scanning might affect the prognostic model, we designated the 492 patients into two groups: 1) the early-phase group: CT scans were performed within one week after symptom onset (0-6 days, n = 317); and 2) the late-phase group: CT scans were performed one week later after symptom onset (>/=7 days, n = 175). In each group, we divided patients into the primary cohort (n = 212 in the early-phase group, n = 139 in the late-phase group) and the external independent validation cohort (n = 105 in the early-phase group, n = 36 in the late-phase group) according to the centers. We built two separate radiomics models in the two patient groups. Firstly, we proposed an automatic segmentation method to extract lung volume for radiomics feature extraction. Secondly, we applied several image preprocessing procedures to increase the reproducibility of the radiomics features: 1) applied a low-pass Gaussian filter before voxel resampling to prevent aliasing; 2) conducted ComBat to harmonize radiomics features per scanner; 3) tested the stability of the features in the radiomics signature by several image transformations, such as rotating, translating, and growing/shrinking. Thirdly, we used least absolute shrinkage and selection operator (LASSO) to build the radiomics signature (RadScore). Afterward, we conducted a Fine-Gray competing risk regression to build the clinical model and the clinic-radiomics signature (CrrScore). Finally, performances of the three prognostic signatures (clinical model, RadScore, and CrrScore) were estimated from the two aspects: 1) cumulative poor outcome probability prediction; 2) 28-day poor outcome prediction. We also did stratified analyses to explore the potential association between the CrrScore and the poor outcomes regarding different age, type, and comorbidity subgroups. Results: In the early-phase group, the CrrScore showed the best performance in estimating poor outcome (C-index = 0.850), and predicting the probability of 28-day poor outcome (AUC = 0.862). In the late-phase group, the RadScore alone achieved similar performance to the CrrScore in predicting poor outcome (C-index = 0.885), and 28-day poor outcome probability (AUC = 0.976). Moreover, the RadScore in both groups successfully stratified patients with COVID-19 into low- or high-RadScore groups with significantly different survival time in the training and validation cohorts (all P < 0.05). The CrrScore in both groups can also significantly stratify patients with different prognoses regarding different age, type, and comorbidities subgroups in the combined cohorts (all P < 0.05). Conclusions: This research proposed a non-invasive and quantitative prognostic tool for predicting poor outcome in patients with COVID-19 based on CT imaging. Taking the insufficient medical recourse into account, our study might suggest that the chest CT radiomics signature of COVID-19 is more effective and ideal to predict poor outcome in the late-phase COVID-19 patients. For the early-phase patients, integrating radiomics signature with clinical risk factors can achieve a more accurate prediction of individual poor prognostic outcome, which enables appropriate management and surveillance of COVID-19.