I can't imagine a situation where I truly can't make an estimate at all--more often there's the case where I can imagine multiple scenarios which would result in vastly different timeframes for the project, depending on various things that could reasonably crop up. And I don't want to lie--the worst thing you can do with your boss is to just make stuff up.
So I explain each of the possibilities. Of course, this only works with an understanding boss, but if your boss is so ignorant or foolish that he refuses to listen to the full explanation, you have other problems.
For example, here's how I did it for a recent case where I actually had to do exactly this.
x264, the video encoder I work on, implements a very primitive form of interlaced coding chosen solely for the reason that it was very easy to implement. We wanted to implement the full form of this coding, but I had no idea how many of the assumptions made for the simplified version would fail in such a case.
So I thought through the various levels of things that might have to be changed, and made the estimate a range--well, at best, it might already be nearly working, but that's doubtful. And at worst there's a whole ton of stuff that needs to be changed. So, I told my boss, it was probably better to assume the worst here, since the spec was very complicated and despite not knowing about any of that complexity, I suspected that given the major lack of related code in the program, nearly none of that complexity was actually implemented. In the end I was right--the changes required ended up being quite complicated, and they outsourced the project to a contractor with more expertise in the complexities of H.264's interlaced coding.
