I created an audio tag without a download button with
This was successful.
Now,
The new controlslist
attribute in the HTML5 standard can only accept three settings:
"nodownload", "nofullscreen" and "noremoteplayback".
There are none to control specific controls beside from that. The only (current) workaround for this is to build your own custom interface for the audio object and provide only the controls you want the user to access via the UI.
The attribute is currently only supported in Chrome and Opera, as well as Android. This of course being another incentive to build a custom UI for the audio (or use a library as suggested by Johannes).