Why does my Jest test of React propTypes break when using multiple unacceptable prop values?

Question

In an answer to a different SO question about how to test propTypes in React using Jest, I proposed a solution that mocks console.error, something othe

Answer 1

Here's a bit of detail on the why, to expand on Andrew Willems' detailed description of the conditions causing the problem.

Note that my explanation here applies to the prop-types library that Facebook extracted from React recently, specifically when checking props with the checkPropTypes function it exports.

The checkPropTypes.js module keeps mutable state in the form of an object called loggedTypeFailures on line 16:

 var loggedTypeFailures = {};

Lines 47-50 show how it is used and explain the thinking:

      if (error instanceof Error && !(error.message in loggedTypeFailures)) {
        // Only monitor this failure once because there tends to be a lot of the
        // same error.
        loggedTypeFailures[error.message] = true;

When a propType check fails, the message is set as a key on loggedTypeFailures before logging the error, to track that it has already been logged. The next time a propType fails with the same message, the error.message in loggedTypeFailures check succeeds and logging is skipped.

Since loggedTypeFailures is private to the module, there is no way to access or reset it.

---

If you need to detect propType failures in a more robust way (eg., for unit tests), I published a tiny library that helps: check-prop-types. It checks props the same way as checkPropTypes.js, but returns the error instead of logging it.

Answer 2

Short Answer

The short answer is that, for at least some complex propTypes like for myProp2, you can only include a single non-null test prop type of any type. This is an inherent limitation to this approach, and comes from a built-in characteristic of React/Jest that causes multiple error conditions that might otherwise have produced identical error messages to be reported only once. In fact, even for simple propTypes like for myProp1, a related but slightly more hidden limitation also exists: you can include any number of different unacceptable non-null test prop values, but never more than one of the same type. e.g. ['', {}, []] will work but ['', 'x', {}, []] will not (note that there are two strings).

This limitation is important as there doesn't currently seem to be a better way of testing propTypes in React. However, the constraints this places on this approach seem manageable, allowing propTypes to still be reasonably tested.

Details

There are currently some significant limitations on how you can use this approach (i.e. mocking console.error) which, if over-stepped, could be the source of some hard-to-trace testing bugs.

The source of these limitations is a behaviour of React (or perhaps Jest in response to React?) that is built-in to its error reporting mechanism. It currently only seems to produce multiple console.error messages, at least for propTypes issues, when those messages are distinct from each other. Presumably the rationale behind this is that repeated identical messages provide no new information and so such a message should only be shown the first time, which seems reasonable. The implication of this, though, is that two distinct error conditions which, when they occur separately, produce identical error messages only result in a single error message when they occur within the same test run. All subsequent identical messages are suppressed. This has several further implications for strategies such as this one that necessarily expect the output from console.error to exactly correlate with test-failing conditions, as follows.

Currently, for simple propTypes (like that for myProp1 in the example code), distinct error messages seem to appear for all unacceptable prop values of different data types, e.g.

"Warning: Failed prop type: Invalid prop 'myProp1' of type 'string'
supplied to 'MyComponent', expected 'number'."

However, the error message for a second test prop value of the same type seems to have its expected error message suppressed. Thus an array of unacceptable values all of different data types, such as [123, [123], '123'], will cause each to produce an error message as expected, allowing the test to work. However, an array of unacceptable values in which some have the identical data types, such as [123, 456, [123], '123'], will not work, as only the first value of any particular data type will produce the expected error message, and so the presumed error messages for any subsequent similarly-typed values will be suppressed, breaking the test. Thus, for simple propTypes, you can use as many unacceptable test prop values as you want as long as they are all of different data types. This doesn't seem like a big problem, as you shouldn't need to test more than one value of a particular data type anyway, but you need to be aware of this caveat.

Perhaps even more importantly, however, some complex propTypes (like that for myProp2) paradoxically (though perhaps reasonably) result in even more simplified error messages, e.g.

"Warning: Failed prop type: Invalid prop 'myProp2' supplied to 'MyComponent'."

This means, though, that only the first non-null unacceptable test prop value of any data type produces an error message, with all subsequent tests using other unacceptable values having their error messages suppressed. Thus, with such complex propTypes, unfortunately you can include only a single unacceptable prop value for a particular prop for a particular component in your test suite. This does not include a null or undefined value for a required complex propType, as these values produce a distinct error message which is thus not suppressed. Note, however, that you still can test as many acceptable values as you want, e.g. allowing you to test multiple possible acceptable prop values for myProp2 in the example code. Even more confusingly, this limitation holds not just within a test but between tests. Thus, any test suite can only contain a single unacceptable prop type test for a particular prop for a particular component. This does seem like a significant limitation, and it is absolutely necessary to keep it in mind. However, it actually might not be that limiting as it initially might sound, as one wouldn't typically expect to test unacceptable values for a particular prop for a particular component multiple times anyway. Thus this approach, i.e. mocking console.error, still seems reasonable, and still seems the only real way of testing propTypes anyway.

Note that this limitation could change in the future, any time that React and/or Jest changes how it uses console.error in response to inappropriate prop values.

Note also that I haven't investigated exactly how React/Jest perform error reporting for other complex propTypes, e.g. React.PropTypes.arrayOf, ... .objectOf, ... .shape, etc. Presumably, to test these robustly you would first need to investigate what kind of error messages are produced for inappropriate values as well as which if any such messages are suppressed.