The rankings you might see for physics refer to PhD programs, not undergraduate programs.  While it is true that our undergraduates are taught by the same professors who teach our graduate students, the importance of these rankings for our undergraduates is more subtle. 

We think students learn how to be outstanding physicists through participation in research with outstanding physicists. Therefore, we encourage all of our undergraduates to participate in cutting-edge research with our internationally recognized faculty. Through this participation our students learn what it takes to be outstanding physicists and, as a consequence, have the opportunity to show their talent for physics and commitment to making progress in the field. This becomes important when students apply to graduate school or for jobs -- people will want to know how they conduct themselves and whether they have talent.  As a result of their reputations for excellence, recommendations from our faculty are taken very seriously by their peers at other universities, as well as in government and industry. 

That is why the rankings are important.