Found it after all, after turning off WiFi on the device I was using. It seems that when debugging over WiFi you also need the
<uses-permission android:name="android.permission.ACCESS_WIFI_STATE" />
in the manifest. Don't know if it was just my case or it's a general thing, but I guess it should be in the documentation somewhere (it's not).
So if you are reading this and having similar problems try either disabling WiFi or adding the above permission alongside the others.