Tuesday, December 27, 2011

Functional Testing Tools

Modern software is extremely complex, and testing today's software is especially challenging because testers must be able to design and execute a set of tests that will provide project leaders with the appropriate information that highlights potential risks and qualifies important attributes and capabilities of the software. Testers must define a finite set of tests from all possible tests that provides the organization with high levels of confidence that testers have exposed critical issues and appropriately evaluated the important features of the product.

One way to increase the effectiveness of our tests and gather important information about certain aspects of the product is to employ functional testing techniques. Functional techniques are systematic procedures that can help us perform a comprehensive investigation of the software's functional attributes and capabilities. Functional techniques are generally applied from the user interface but can be used to design tests from both black box and white box test perspectives. When we use functional testing techniques correctly and in the appropriate context, they can help us achieve a more thorough analysis of the system under test and reduce the susceptibility to the pesticide paradox.{taken from book 'How We Test Software at Microsoft'}

Functional testing bases its test cases on the specifications of the software component under test. It verifies a program by checking it against design documents or specifications. Functional tests are supplemented with information that allow us to improve their ability to find and correct deficiencies in the system in a cost effective way. Additionally, understanding the physical principles behind the system under test is helpful in developing a test procedure that truly verifies the system.

Functional techniques provide testers with systematic approaches that can help achieve a more comprehensive investigation of individual features and components. There are various open source tools functional testing available. Most of those tools are based Behaviour-Driven Development(BDD). BDD is an evolution of test-driven development (TDD) and acceptance-test driven design, and is intended to make these practices more accessible and intuitive to newcomers and experts alike. It shifts the vocabulary from being test-based to behaviour-based, and positions itself as a design philosophy. The most impressive open source frameworks for functional testing are:

  • Cucumber: Cucumber is a tool that executes plain-text functional descriptions as automated tests. The language that Cucumber understands is called Gherkin.
    While Cucumber can be thought of as a “testing” tool, the intent of the tool is to support BDD. This means that the “tests” (plain text feature descriptions with scenarios) are typically written before anything else and verified by business analysts, domain experts, etc. non technical stakeholders. The production code is then written outside-in, to make the stories pass.
  • JBehave : JBehave is a framework for BDD. User write a story and maps the steps to Java. User stories are configured and run as unit tests. It also provide reports in HTML which can used to verify user stories.
  • Concordian: Specifications are written in simple HTML. Developers instrument the concrete examples in each specification with commands (e.g. "set", "execute", "assertEquals") that allow the examples to be checked against a real-life system.
    The instrumentation is invisible to a browser, but is processed by a Java fixture class that accompanies the specification and acts as a buffer between the specification and the system under test. The fixture is also a JUnit test case, so it's easy to run and integrate into an automated build. The results of running the examples are exported with green and red indicating successes and failures.

No comments: