It is a difficult one, I think.
However, you do know how it is pronounced and are attempting to pronounce it as accurately as possible. That doesn't bother me because I know that names on my list that I can't pronounce quite right so to say 'you shouldn't use it then' to you would be hypocritical.
On the other hand, had you been insisting on pronouncing
Aisling as AYZ-ling or insisting that
Benoit was BEN-oh-it then I would be annoyed :)