I have a couple of audio elements that appear in the body of my page. They look like this.
This is tricky. I would try replacing the whole element instead of just changing its source. This way, the new audio element hasn't been added to the page, so it will be forced to load the file.