It seems weird to me to say that it's silly to name a black girl Ivory or a white girl Ebony.
Sienna is sort of popular with white people and no one says, "how silly! her skin's not orange/red/brown enough for that name"...same goes for other popular color names like Violet or Gray.
That said, it does make me think of soap.