...and besides that it isn't much different than naming your child Lily, Rose, Violet, or Iris in my eyes and in today's world, those names are VERY popular.
Magnolia is fine, and it even has great nicknames like Nola, Maggie, Nell, and Lia. It isn't mainstream but that doesn't mean it is bad, just as is the case with Primrose.