Update, because I think no one thought it what that easy. Just read it, if you want to use it for private usage (you know, youtube terms and stuff like that).
After some research work with different APIs and libraries and researching source code, I just tried it by myself and wolla.
Just saying, audio only links are in the page source of each video.
You just have to know what you are looking for ;)
Open a youtube video, view page source and search for "mime%3Daudio". There should be 5 matches, full url for example:
"https%3A%2F%2Fr2---sn-35cxanpbo5a-8pxl.googlevideo.com%2Fvideoplayback%3Fsource%3Dyoutube%26signature%3D30EC556F55533FBFD9003767730D10556681F33A.B12E021CA54CCB9E225F58A4430E9BB528081FB5%26requiressl%3Dyes%26expire%3D1527306011%26clen%3D3064602%26initcwndbps%3D1152500%26ipbits%3D0%26mime%3Daudio%252Fmp4%26dur%3D192.911%26fvip%3D5%26lmt%3D1524946334873350%26key%3Dyt6%26sparams%3Dclen%252Cdur%252Cei%252Cgir%252Cid%252Cinitcwndbps%252Cip%252Cipbits%252Citag%252Ckeepalive%252Clmt%252Cmime%252Cmm%252Cmn%252Cms%252Cmv%252Cpl%252Crequiressl%252Csource%252Cexpire%26itag%3D140%26gir%3Dyes%26ip%3D81.217.53.239%26id%3Do-ALNxMeQYw4LLc1FAjxt4h795wKTdqJnzc_SBgzEJVBxR%26c%3DWEB%26keepalive%3Dyes%26mm%3D31%252C29%26mn%3Dsn-35cxanpbo5a-8pxl%252Csn-2gb7sn7r%26ei%3Du4IIW7DfC46rgAfrrpnICw%26ms%3Dau%252Crdu%26mt%3D1527284230%26pl%3D16%26mv%3Dm"
Just decode URL and you got what you want.