No, the file which references the manifest is always itself cached. From the spec:
The resource that declares the manifest (with the manifest attribute) will always get taken from the cache, whether it is listed in the cache or not, even if it is listed in an online whitelist namespace.