The global Coronavirus disease 2019 (COVID-19) pandemic caused by SARS-CoV-2 has affected more than eight million people. There is an urgent need to investigate how the adaptive immunity is established in COVID-19 patients. In this study, we profiled adaptive immune cells of PBMCs from recovered COVID-19 patients with varying disease severity using single-cell RNA and TCR/BCR V(D)J sequencing. The sequencing data revealed SARS-CoV-2-specific shuffling of adaptive immune repertories and COVID-19-induced remodeling of peripheral lymphocytes. Characterization of variations in the peripheral T and B cells from the COVID-19 patients revealed a positive correlation of humoral immune response and T-cell immune memory with disease severity. Sequencing and functional data revealed SARS-CoV-2-specific T-cell immune memory in the convalescent COVID-19 patients. Furthermore, we also identified novel antigens that are responsive in the convalescent patients. Altogether, our study reveals adaptive immune repertories underlying pathogenesis and recovery in severe versus mild COVID-19 patients, providing valuable information for potential vaccine and therapeutic development against SARS-CoV-2 infection.