The COVID-19 caused by SARS-CoV-2 has spread globally and caused tremendous loss of lives and properties, and it is of utmost urgency to understand its propagation process and to find ways to slow down the epidemic. In this work, we used a coarse-grained model to calculate the binding free energy of SARS-CoV-2 or SARS-CoV to their human receptor ACE2. The investigation of the free energy contribution of the interacting residues indicates that the residues located outside the receptor binding domain are the source of the stronger binding of the novel virus. Thus, the current results suggest that the essential evolution of SARS-CoV-2 happens remotely from the binding domain at the spike protein trimeric body. Such evolution may facilitate the conformational change and the infection process that occurs after the virus is bound to ACE2. By studying the binding pattern between SARS-CoV antibody m396 and SARS-CoV-2, it is found that the remote energetic contribution is missing, which might explain the absence of cross-reactivity of such antibodies.