No, I don't think it does sound stereotypically black or strange on a white boy. I think it probably used to, but it's really taken off in popularity since 2000, when it broke into the top hundred, and has been there since. This would not be true if only black people are using it. Now you may run into an older person here and there who has not kept up with the times and will continue to believe
Xavier is "black", the same way my mother-in-law persisted in believing that
Emily is "old lady" and
Rebecca is "Jewish." But you can't let that stop you from using a name.