If you're open to using an external program, you can use openssl to attempt to decrypt the data. As long as you provide a non-blank password on unencrypted data it will return with a "bad magic number" error. Not sure yet exactly how openssl is generating that message (if I could you could do that in your code), but it's a start.
$ openssl enc -d -aes-256-cbc -in /Applications/Wireshark.app/Contents/MacOS/Wireshark
enter aes-256-cbc decryption password:
bad magic number
Another method would be to check the first 8 characters of the file are "Salted__" and the following 8 characters are the salt (provided the encryption was salted which the openssl binary does by default but can be disabled).
0000000: 5361 6c74 6564 5f5f 70d6 3655 ae12 58de Salted__p.6U..X.