I went to the highest ranking program I went to (VHRM), following advice from my mentors. In retrospect, however, this isn't the best decision in my life. If your research interests aren't mainstream in the program -- even if you can find several faculty members that sort of "match" -- you're going to feel marginalized in the community.
Occasionally I wonder what my experiences would be like in relatively lower ranked program (top 10-20) where my interests would be mainstream. In addition to feeling less marginalized, I would probably get better exposure to the topics I'm *most* interested in; and, better funding.
One difficulty is that prior to graduate school, it may be hard to differentiate between topics you are interested in and topics you'd like to spend years doing. I wish I thought harder about that issue and hadn't trusted the decision so much to other people's advice, especially those based on perceived prestige. The decision might still be the same or may be different, but the thought process would be invaluable.