Why can't a class extend a static nested class occurring within it?

Question

This class:

public class OuterChild extends OuterChild.InnerParent {
    public static class InnerParent {
    }
}

Fails to compile:

<

Answer 1

This appears to be a fairly nefarious corner-case, as there are a number of bugs related to cyclic inheritance, often leading to infinite loops, stack overflows, and OOMs in the compiler. Here are some relevant quotes that may offer some insight:

Bug 4326631:

This example is not legal, and this is made clear in the forthcoming 2nd edition of the Java Language Specification. Classes simultaneously related by both inheritance and enclosure are problematical, however the original innerclasses whitepaper did not adequately address the issue, nor did the pre-1.3 compilers implement a consistent policy. In JLS 2nd edition, the rule against cyclic inheritance has been extended to prohibit a class or interface from "depending" on itself, directly or indirectly. A type depends not only on types that it extends or implements, but also on types that serve as qualifiers within the names of those types.

Bug 6695838:

The two class declarations are indeed cyclic; accordingly to JLS 8.1.4 we have that:

Foo depends on Foo$Intf (Foo$Intf appears in the implements clause of Foo)
Foo$Intf depends on Moo$Intf (Moo$Intf appears in the extends clause of Foo$Intf)
Foo$Intf depends on Foo (Foo appears as a qualifier in the extends clause of Foo$Intf)

For transitivity, we have that Foo depends on itself; as such the code should be rejected with a compile-time error.

Bug 8041994:

Stepping back, the directly-depends relationship for classes and interfaces was introduced in JLS2 to clarify JLS1 and to cover superclasses/superinterfaces that are nested classes (e.g. A.B in the Description).

Bug 6660289:

This problem is due to the order in which javac perform attribution of type-variable bounds wrt class attribution.

1) Attribution of class Outer<T extends Outer.Inner>
1a) Attribution of Outer triggers attribution of Outer's type variable
2) Attribution of Outer.T
2a) Attribution of Outer.T triggers attribution of its declared bound
3) Attribution of class Outer.Inner<S extends T>
3a) Attribution of Outer.Inner triggers attribution of Outer.Inner's type variable
4) Attribution of Outer.Inner<S>
4a) Attribution of Outer.Inner.S triggers attribution of its declared bound
5) Attribution of Outer.T - this does nothing but returning the type of T; as you can see, at this stage T's bound has not been set yet on the object representing the type of T.

At a later point, for each attributed type variable, javac performs a check to ensure that the bound of a given type variable does not introduce cyclic inheritance. But we have seen that no bound is set for Outer.T; for this is the reason javac crashes with a NPE when trying to detect a cycle in the inheritance tree induced by the declared bound of Outer.Inner.S.

Bug 6663588:

Type-variable bounds might refer to classes belonging to a cyclic inheritance tree which causes the resolution process to enter a loop when looking up for symbols.

To your specific question of "what is the problematic dependency?" it appears to be a complex compile-time symbol resolution edge case, and the solution introduced in JLS2 was to simply ban cycles introduced by qualifier types as well as actual supertypes.

In other words this could theoretically be made to work with appropriate improvements to the compiler, but until someone comes along and makes that happen it's more practical to just ban this unusual relationship in the language specification.

Answer 2

An educated SWAG: Because the JVM must first load the parent class, which includes a command to load the inner class. The inner class is defined by the CL after the outer class is defined so any refs to the outer class' fields or methods are resolvable. By trying to extend the outer by the inner, it asks the JVM to compile the inner before the outer, thus creating a chicken and egg problem. The problem has its roots in the fact that an inner class may reference its outer class' field values provided rules around scope and instantiation (static v. non-static) are followed. Because of this possibility, the JVM would need to be guaranteed that at no time anything in the inner class will try to access or mutate any field or object references in the outer class. It can only find this out by compiling both classes, outer first, but needs this information prior to compilation to be certain there won't be a scope or instance problem of some kind. So it's a catch-22.