Software Test Design and Execution

Testing is necessary in the manufacturing of any industrial product and in the development of IT projects.

Who would sell a vacuum cleaner without being sure that it vacuums properly?

Who would sell a digital radio without having checked that it can tune into channels?

In the same way, a software application cannot reach the hands of the end user with errors, especially if they are visible and clear enough to have been detected by the developer. This would result in a lack of professionalism and would decrease user trust, which could cause problems for future opportunities. Any member of an IT project team can make mistakes. Furthermore, errors can occur in any phase of the project (analysis, design, coding...). Some of these flaws will be more decisive than others and will have more or fewer implications for the future development of the project.

The phases of a project's development are:

  • Requirements gathering.
  • Analysis.
  • Design.
  • Development.
  • Testing.
  • Completion.
  • Handoff.

What is testing?

Testing is a phase of software project development, and it is essential to perform it before finishing the project and transferring it to the end user.

Imagine if the project didn't even work as the client wants...

Considerations

  • All phases established in software development are important. Regardless, we should consider the testing phase as a crucial phase and give it much more importance than it is sometimes given.
  • The absence or poor execution of any phase by any member of the team can cause the rest of the project to carry forward one or more errors that will be more or less decisive for the company's success.

Taking into account the usual phases in software development (requirements gathering, analysis, design, development, testing, completion, and handoff), the typical profiles of a team in a software development project are:

  • Project manager.
  • Analyst.
  • Designer.
  • Programmers.
  • Testers.

Possible errors

Example 1:

The project manager plans the hours spent on interface design poorly, and in the end, it turns out that the designer spends more than double.

What will happen?

→ The impact on the project will only be at the time and cost level. It might be compensated for in a later task.

Example 2:

The DB creator omitted a key field from a main table and its link to another table.

What will happen?

→ Depending on when the error is detected, it might be fixed by modifying a few lines of code, or a huge amount of work might need to be redone, meaning we worked many hours for nothing.

Studies show that an error not detected at the beginning of a project's development can require up to 50 times more effort to fix than if it had been detected in time.

Validation

It is the activity responsible for ensuring that the resulting software application is the one initially specified and the one that was intended to be built.

Validation will confirm that the application functions as the client requested.

Does the system do what the client wants?

Verification

It is the activity that verifies the correct operation of the programmed code. It checks that the software application has been developed correctly from a technological standpoint.

Does the system do what it is supposed to do?

Considerations

  • These concepts are directly related to tests that formally verify the correctness of a software application.
  • Throughout the software development project, the system needs to be verified and validated at each step of the process, starting from the verification and validation of the previous step and establishing adequate documentation to be able to re-validate and re-verify in the next step.

In the following image, you can observe the differences between what the user requests, what is programmed, and what is delivered to the user, which in the end is what will be used.

Classification of tests

  • Unit testing: Performed as each module is completed.
  • Black-box testing.
  • White-box testing.
  • Reviews.

Integration testing: Performed when all modules are finished and need to be integrated.

  • Bottom-up integration testing.
  • Top-down integration testing.
  • Combined integration testing.
  • Big bang integration testing.

Load testing: Performed when the application is already finished and integrated, putting it into operation with a load of real data.

Acceptance testing: When the client already has the application and performs the tests themselves.

System testing: Performed to validate the integration of the software into the user's existing system (performance/robustness/endurance/security/usability).

Unit testing

Black-box testing

  • They test the operation of a module through its interface without analyzing its internal workings.
  • At no point is the code evaluated or accessed. Only the function's responses to multiple given inputs can be seen. To verify if the module checks correctly, tests must be carried out by entering:
  • Correct data.
  • Erroneous data.
  • Equivalent data.
  • Upper and lower boundary values.

Techniques for establishing the test suite:

  • Equivalence partitioning. Choose test cases for each type or set of input and output data.
  • Boundary value analysis.
  • Study of typical errors.
  • Graphical user interface testing (windows, buttons, icons, etc.).

Advantages:

  • They are language-independent. Therefore, they are valid for all programming methods used.

White-box testing

They focus on how a specific implementation of a module or application has been coded. These tests examine the internal implementation details of the code.

To examine the inside of the module, aspects such as the following must be verified:

  • Statements, conditions, loops, etc.
  • Code structure.
  • Code optimization.

Techniques for establishing the test suite.

  • Control flow coverage. The control flow diagram is derived in reverse from how it is programmed, ensuring that each path in the program is traversed at least once. From the flowchart, all possibilities should be tested.
  • Simple path.
  • Conditions.
  • Simple loops.
  • Nested loops.

Advantages

  • If all suggestions for designing test cases for each element (simple path, conditions, etc.) of the programmed code are followed, a minimum guarantee of effectiveness will be achieved.

Reviews

They are based on the direct inspection of the code, either module by module or completely. Unlike the previous ones, this does not test the code at runtime. Reviews are also important for tasks related to quality control, validation, and system verification.

Code review is conducted systematically following standard rules. Generally, it is done by the programmer themselves.

Group techniques:

  • Meetings: The review is conducted by the developers themselves.
  • External channels: The review is done by expert reviewers.
  • Inspections: The review is carried out by developers and external reviewers guided by a moderator.

Advantages.

  • Code documentation.
  • Separation of graphical user interfaces and code implementation.
  • Simplification of integration between modules, as errors in independent modules will be less likely.

Integration testing

Bottom-up

These tests begin by combining units or modules into groups to test them. Different level groupings are tested progressively until all levels are integrated.

To achieve this objective, test drivers or controllers must be designed to simulate the behavior of higher-level modules.

These should be easy to build because they only share the same interface without executing the full function.

Their purpose is to enable execution of the module or group of modules being tested.

Top-down

First, top-level modules are tested, and then lower-layer components are progressively integrated, and so on.

In these cases, it is necessary to design stubs (dummy modules), test simulators that emulate the behavior and share the same interface as the real ones. They are considerably harder to build because both the interface and function of each module must be created.

Combined

These tests combine bottom-up and top-down integration. Individual modules are tested with drivers as well as stubs, and then upper and lower layers are tested by replacing the drivers and stubs with modules that have already been tested. Usually, in combined integration, a top-down strategy is used for higher levels while a bottom-up strategy is applied to lower levels.

Big bang

First, all methods are tested individually, and then all components or modules of the system are integrated at once to perform the tests.

Although it is a fairly simple strategy to apply, it has the disadvantage that when an error is discovered, it is very difficult to locate and fix.

Load testing

Objectives

Check the performance and integrity of the finished application using real data. The goal is to simulate the application's runtime environment. To play it safe and minimize possible errors during the acceptance phase, when the product is to be used by a large number of clients or end users, the following techniques are usually applied:

Alpha testing

These consist of inviting clients to test the system in a development environment. Being in a controlled environment, the client always has the support of an expert to help them use the system or analyze results.

Beta testing

These take place after alpha testing in the client's environment. The client is alone with the product and tries to find errors to report back to the developers.

Acceptance testing

Obtain the client's approval regarding the operational quality of the developed and tested system.

System testing

The objective is to validate the application once it has been accepted by the user and integrated with the rest of their system.

Types of testing

  • Performance testing.
  • Endurance testing.
  • Robustness testing.
  • Security testing.
  • Usability testing.
  • Installation testing.

Regression testing

The objective is to detect potential new errors or issues that may arise when introducing changes or enhancements to the software. These typically correspond to different releases/versions of the same application.

Smoke testing

Describes the validation of code changes in the software before those code changes are recorded in the project documentation. Therefore, a code review is usually performed before running smoke tests.

Test cases

Test cases define how tests will be carried out, specifying among other things:

the type of tests, test inputs, expected results, or the conditions under which they should be executed.

Test cases also follow a classic lifecycle:

  • Definition of test cases.
  • Creation of test cases.
  • Selection of values for the tests.
  • Execution of test cases.
  • Comparison of obtained results with expected results.

Test procedures

Test procedures specify how test cases or parts of them can be carried out independently or jointly, establishing the relationships between them and the order in which they must be addressed.