When Contexts Collide, Part II

In my last post I described the events leading up to a meeting we had to talk about how to incorporate some new code that a couple of our functional testers had written. It enabled some new types of testing within our test automation framework.

I kicked off the meeting by reiterating the differences between Project Context and Product-Line Context automation. Project Context automation is focused on delivering the automation and results that are needed to complete a development project. Product-Line Context automation focuses on building automation that can be run completely hands-off for a long period of time across multiple projects/releases.

Then, myopically, I started to talk about what needed to be done to finish their new code for the Product-Line Context. Once again, I was trapped in my own context and not stopping to think about the needs of the Project Context. Fortunately, one of my colleagues stopped me almost immediately and suggested that we should define completion criteria for both the Project Context and the Product-Line context.

This lead to a very good discussion. We eventually reached consensus on two different sets of "Done Criteria." Below are the detailed criteria, from the post-meeting email that I sent out.

Project Context: when something is “done” for the Project Context, it is ready to be checked into the ITE source tree and shared among other ITE users, but it is not ready to migrate into the lab to be run fully hands-off by the Automation team.


We agree to the following as “Done Criteria” for the Project Context:

• Must be backwards compatible with the ITE and existing tests—if other people start using the new code, it isn’t suddenly going to cause a bunch of false test results or other problems.
• It must have a good suite of unit tests. Both the new unit tests and the existing unit tests should run and pass.
• It must be able to submit pass/fail/abort test results to the ITE results database.
• It must be checked into the ITE code tree (in some appropriate location, might not be on the main branch).
• There must be sufficient documentation to allow other people can set it up and use it.
• Other ITE users must be able to build/deploy it to their ITE installations.

Product-Line Context: when something is “done” for the Product Line Context, it is ready for the Automation team to start running it in the lab. No hands-on modifications or monitoring will be needed.


We agree to the following as “Done Criteria” for the Project Context:

• It can be set up using f5iteconfig (f5iteconfig is the tool we use to set up a new automation environment).
• Our subjob health-checks can monitor all the hardware in the test harness and generate alerts if one of them fails (health-checks are done periodically during a test run to verify that all the systems in the test harness are functioning).
• The controller can identify the harness and use appropriate meta-data to figure out which tests can run on it (our tests are tagged with meta-data denoting the hardware/software releases for which the test is valid).
• We can retrieve data (log files/core files/whatever) from all the hardware in the harness.
• We can configure/provision/license all the DUT in the harness.
• We can allocate resources/configure services on all the hardware in the harness.