It doesn't work well, people will always mispronounce it as Am-eh-ree and then it will get confused with
Emily,
Emory,
Amélie and all kinds of other names.
If you want the pronunciation you listed try
Amery. It is closer to
Avery. I still think there will be pronunciation issues, though.