Should QA test from a strictly black-box perspective?

Question

Assuming that unit tests are handled by development, is there any reason for QA to have knowledge of the details of how a product works? By which I mean, do they need to kn

Answer 1

I think it depends on the role that your QA team plays on a given project. I think you can make an argument that situations that arise from specific values being present in the database should be represented by test cases, and if they can be represented in that way, then developers should write (should have written) unit tests for those situations.

If you've also used code inspections to identify and fix defects, it may not be necessary to expose QA to anything behind the scenes. I suppose there are projects where it might be helpful for them to test code outside the user experience, but I would probably use a QA team for black-box testing rather than white- or clear-box testing.

Answer 2

I think a hybrid approach works well. If you use a combination of white-box testing (unit tests) and black-box testing, you end up with better coverage. Each has its pros and cons, but they do partially cover weaknesses in the other.

Understanding the inner workings of code will cause you to test in a certain way, which is not always the best way to uncover certain problems.

Answer 3

On projects that I've been involved with, QA tested from the user's perspective and their tests were from a standpoint of meeting requirements. Their testing was black box testing. White box testing was done by the devs. We never expected a QA person to open up a DB query tool and manually change values. This was the responsibility of the dev's unit tests.

Answer 4

I would say there is every reason for a QA person to not have detailed inside knowledge of how your application works. The QA staff should have just about the same level of computer competency as your target audience.

The reason for this is simple: The more a QA person knows about how your application is built, the more likely they are to avoid normal usability problems that your regular users will run into.

QA isn't just about testing whether the application works. It should also be about testing whether it is understandable by your average user and therefore actually usable.

UPDATE
This was getting too long to fit into a comment box.

Regarding @testerab's questions.

I define QA as the person or group responsible to ensure 1. business requirements are met; and, 2. the functions of the application with regards to those requirements are error free. Hence the moniker "Quality Assurance". A third item I believe belongs with QA is simply usability.

They must have an understanding of the business requirements which means they should work hand in hand with any Business Analysts and the End Users (if any are available). The best QA members I've worked with have been hired from those areas. The worst QA members I've seen had been developers, or were trained as such. People who moved from Tech Support also tend to make good QA members as they know exactly the type of garbage an end user will try.

There are many different classes of applications. Far and away the most common of which is glorified data entry. You have some screens that information is entered into and buttons are pressed. The information is then stored and/or routed to wherever it needs to go. Everything from MS Word to CRMs fall into this category.

Therefore a QA person's job is to first make sure that the app will accept the desired inputs and perform the desired functions. A secondary step is to submit bad inputs and verify the app responds in a desired manner. e.g. it doesn't blow up or allow bad information to go through. Automated testing tools work great in these cases. A final part is deciding whether that function should be buried three menu levels deep or is it something that should be easier to get to.

None of the above requires the QA person to read code or twiddle with the bits.

Some systems do not have a UI component. For these it is up to the developer to provide a testing harness that will allow a QA team to submit data and review the results. This covers web services and any type of EDI you can imagine.

Other systems are API's in themselves. These require either a QA person who is knowledgeable enough in order to implement such API calls themselves OR for the developers to provide ways to easily call them and record the results. In this case it is preferable that the QA team can do their own programming but, again, not have access to the underlying code.

The problem with reviewing the actual code is that it tends to cause a person to focus on just what they are seeing. This is bad. Instead they should be mentally free of those artificial limits so that they can blindly type "ABC" into a text box that expects numeric entry.

---

As far as "seeing" the code in order to know what to test, this is a whole different problem and one that is better solved by trained developers in a code review setting. The purpose here is to identify possible errors, best practices, security failures, etc. QA people are rarely capable of this level of analysis as it requires someone to be an active developer.

---

Regarding SDETs: If your product has, or is, an API or a foundation that devs use to build other things then you might want one or more people dedicated to implementing software around it in order to support the regular QA group. I'm on the fence as to the necessity of this role and believe it might be a project by project decision.

I believe there are two groups that are better qualified to test APIs. These are the end users who are already implementing and other devs in the company that built it. Dog fooding is now a common practice and is very cost effective. After all, if it doesn't work you can be sure it will be fixed quickly. For end users, as long as they see it as a real conversation in which the dev team is responsive then they will happily "test" it for you.

I've been in all three situations and as an end user I feel having access to the original devs goes a long ways to resolving issues. Especially when they are also using the product. The amount of mistranslation to the QA people normally associated with these projects is just horrible.

---

In summary, I've worked with all sorts of QA people. From ones that you wondered how they managed to drive to work to superstars that were highly adept at finding those corner cases which caused apps to choke.

Traits of the best ones were: 1. were never programmers; 2. understood the business; 3. understood exactly what the end users did on a day to day basis. 4. Honestly CARED about those subjected to the software.

Traits of the worst ones included: 1. had been programmers or thought they were; 2. did not understand the business; 3. never met an end user; 4. used the word "idiot" way too often; 5. caught up in the mechanics of how it worked instead of whether it was even usable.

Answer 5

Surely it depends on the architecture. I worked on a project where the db tier was developed, managed and tested by a completely separate team in a different building. Their QA definitely wriggled around with data to see whether the procedures, queries and the like all ran in a range of test conditions.

If you are at the UI end then there are two levels, one is simple functional testing for which the QA need no working knowledge of the application (and all of which should prbably be automated) and then there is the QA which says whether the app does what it is supposed to do. For the second kind it really helps if the QA team know how it works. It saves a lot of time rejecting silly bugs for a start, but more importantly they need to behave like users and have end to end use cases which try out some more complex overlaying scenarios. To design such tests they have to have a good knowledge of the application.

Answer 6

It depends on the approach and the kind of software you are writing. There are different kinds of QA. If the software should be fault-tolerant, QA should simulate faults. Also, knowing how a product works can help QA think of potentially problematic cases and test them more thoroughly.

On the other hand, knowing how a product works might prevent the QA from testing completely from the user's point of view. So maybe first the basic tests should be designed without knowing the internals, and then more in-depth tests based on potential problems.