CLEAN SOLUTION FOUND
I found a very clean solution which really renders this whole question pointless, and I am certain it existed when I asked this
In the contentScript or whatever is running in the web site (i dont know much about sdk), check if (window.frameElement == true)
if its true than its a frame then you want to cancel the attach.
edit:
maybe window.frameElement wont work from within the frame. if it doesnt work do this if (window.top != window)
then its a frame
Here's a link to the Mozilla documentation you can use the attachTo setting. The default setting of 'all' includes iframes, so you should do something like the following
var pageMod = require("sdk/page-mod");
pageMod.PageMod({
include: "*",
contentScript: "",
attachTo: ["existing", "top"],
onAttach: function(worker) {
console.log(worker.tab.url);
}
});