It's not only about public or private methods or functions, it is about implementation details. Private functions are just one aspect of implementation details.
Unit-testing, after all, is a white box testing approach. For example, whoever uses coverage analysis to identify parts of the code that have been neglected in testing so far, goes into the implementation details.
A) Yes, you should be testing implementation details:
Think of a sort function which for performance reasons uses a private implementation of BubbleSort if there are up to 10 elements, and a private implementation of a different sort approach (say, heapsort) if there are more than 10 elements. The public API is that of a sort function. Your test suite, however, better makes use of the knowledge that there are actually two sort algorithms used.
In this example, surely, you could perform the tests on the public API. This would, however, require to have a number of test cases that execute the sort function with more than 10 elements, such that the heapsort algorithm is sufficiently well tested. The existence of such test cases alone is an indication that the test suite is connected to the implementation details of the function.
If the implementation details of the sort function change, maybe in the way that the limit between the two sorting algorithms is shifted or that heapsort is replaced by mergesort or whatever: The existing tests will continue to work. Their value nevertheless is then questionable, and they likely need to be reworked to better test the changed sort function. In other words, there will be a maintenance effort despite the fact that tests were on the public API.
B) How to test implementation details
One reason why many people argue one should not test private functions or implementation details is, that the implementation details are more likely to change. This higher likelyness of change at least is one of the reasons for hiding implementation details behind interfaces.
Now, assume that the implementation behind the interface contains larger private parts for which individual tests on the internal interface might be an option. Some people argue, these parts should not be tested when private, they should be turned into something public. Once public, unit-testing that code would be OK.
This is interesting: While the interface was internal, it was likely to change, being an implementation detail. Taking the same interface, making it public does some magic transformation, namely turning it into an interface that is less likely to change. Obviously there is some flaw in this argumentation.
But, there is nevertheless some truth behind this: When testing implementation details, in particular using internal interfaces, one should strive to use interfaces that are likely to remain stable. Whether some interface is likely to be stable is, however, not simply decidable based on whether it is public or private. In the projects from world I have been working in for some time, public interfaces also often enough change, and many private interfaces have remained untouched for ages.
Still, it is a good rule of thumb to use the "front door first" (see http://xunitpatterns.com/Principles%20of%20Test%20Automation.html). But keep in mind that it is called "front door first" and not "front door only".
C) Summary
Test also the implementation details. Prefer testing on stable interfaces (public or private). If implementation details change, also tests on the public API need to be revised. Turning something private into public does not magically change its stability.