Sure, social Darwinism had a role in white European imperialism. I suspect most expansionist empires are racist, though. There was racism in Japan in WWII, for example. The Nazis were certainly social Darwinians.

I wasn't aware of that quotation from Darwin. It's from Descent of Man, though, and it's somewhat incidental. He was talking about how species diverge, although he seems to assume that savage races are closer to lower primates than are civilized humans. In any case, Herbert Spencer was the big social Darwinian.