The bone marrow (BM) is key to protective immunological memory because it harbors a major fraction of the body's plasma cells, memory CD4+ and memory CD8+ T-cells. Despite its paramount significance for the human immune system, many aspects of how the BM enables decade-long immunity against pathogens are still poorly understood. In this review, we discuss the relationship between BM survival niches and long-lasting humoral immunity, how intrinsic and extrinsic factors define memory cell longevity and show that the BM is also capable of adopting many responsibilities of a secondary lymphoid organ. Additionally, with more and more data on the differentiation and maintenance of memory T-cells and plasma cells upon vaccination in humans being reported, we discuss what factors determine the establishment of long-lasting immunological memory in the BM and what we can learn for vaccination technologies and antigen design. Finally, using these insights, we touch on how this holistic understanding of the BM is necessary for the development of modern and efficient vaccines against the pandemic SARS-CoV-2.