Static Testing
The Verification activities fall into the category of Static Testing. During static testing, you have a checklist to check whether the work you are doing is going as per the set standards of the organization. These standards can be for Coding, Integrating and Deployment. Review's, Inspection's and Walkthrough's are static testing methodologies.
Static testing is a form of software testing where the software isn't actually used. This is in contrast to Dynamic testing. It is generally not detailed testing, but checks mainly for the sanity of the code, algorithm, or document. It is primarily syntax checking of the code or and manually reading of the code or document to find errors. This type of testing can be used by the developer who wrote the code, in isolation. Code reviews, inspections and walkthroughs are also used.
From the black box testing point of view, static testing involves review of requirements or specifications. This is done with an eye toward completeness or appropriateness for the task at hand. This is the verification portion of Verification and Validation.
Bugs discovered at this stage of development are less expensive to fix than later in the development cycle.
Static code analysis is the analysis of computer software that is performed without actually executing programs built from that software (analysis performed on executing programs is known as dynamic analysis). In most cases the analysis is performed on some version of the source code and in the other cases some form of the object code. The term is usually applied to the analysis performed by an automated tool, with human analysis being called program understanding or program comprehension.
The sophistication of the analysis performed by tools varies from those that only consider the behavior of individual statements and declarations, to those that include the complete source code of a program in their analysis. Uses of the information obtained from the analysis vary from highlighting possible coding errors (e.g., the lint tool) to formal methods that mathematically prove properties about a given program (e.g., its behavior matches that of its specification).
Some people consider software metrics and reverse engineering to be forms of static analysis.
A growing commercial use of static analysis is in the verification of properties of software used in safety-critical computer systems and locating potentially vulnerable code.
Formal methods is the term applied to the analysis of software (and hardware) whose results are obtained purely through the use of rigorous mathematical methods. The mathematical techniques used include denotational semantics, axiomatic semantics, operational semantics, and abstract interpretation.
Dynamic Testing:
Dynamic Testing involves working with the software, giving input values and checking if the output is as expected. These are the Validation activities. Unit Tests, Integration Tests, System Tests and Acceptance Tests are few of the Dynamic Testing methodologies.
Dynamic testing (or dynamic analysis) is a term used in software engineering to describe the testing of the dynamic behavior of code. That is, dynamic analysis refers to the examination of the physical response from the system to variables that are not constant and change with time. In dynamic testing the software must actually be compiled and run; this is in contrast to static testing. Dynamic testing is the validation portion of Verification and Validation.
Some of dynamic testing methodologies include:
1. Unit Testing
2. Integration Testing
3. System Testing
4. Acceptance Testing
Blackbox Testing:
Black box testing takes an external perspective of the test object to derive test cases. These tests can be functional or non-functional, though usually functional. The test designer selects valid and invalid input and determines the correct output. There is no knowledge of the test object's internal structure.
This method of test design is applicable to all levels of software testing: unit, integration, functional testing, system and acceptance. The higher the level, and hence the bigger and more complex the box, the more one is forced to use black box testing to simplify. While this method can uncover unimplemented parts of the specification, one cannot be sure that all existent paths are tested.
Testing Strategies/Techniques
* black box testing should make use of randomly generated inputs (only a test range should be specified by the tester), to eliminate any guess work by the tester as to the methods of the function
* data outside of the specified input range should be tested to check the robustness of the program
* boundary cases should be tested (top and bottom of specified range) to make sure the highest and lowest allowable inputs produce proper output
* the number zero should be tested when numerical data is to be input
* stress testing should be performed (try to overload the program with inputs to see where it reaches its maximum capacity), especially with real time systems
* crash testing should be performed to see what it takes to bring the system down
* test monitoring tools should be used whenever possible to track which tests have already been performed and the outputs of these tests to avoid repetition and to aid in the software maintenance
* other functional testing techniques include: transaction testing, syntax testing, domain testing, logic testing, and state testing.
* finite state machine models can be used as a guide to design functional tests
* According to Beizer the following is a general order by which tests should be designed:
1. Clean tests against requirements.
2. Additional structural tests for branch coverage, as needed.
3. Additional tests for data-flow coverage as needed.
4. Domain tests not covered by the above.
5. Special techniques as appropriate--syntax, loop, state, etc.
6. Any dirty tests not covered by the above.
Black Box Testing Strategy
Black Box Testing is not a type of testing; it instead is a testing strategy, which does not need any knowledge of internal design or code etc. As the name "black box" suggests, no knowledge of internal logic or code structure is required. The types of testing under this strategy are totally based/focused on the testing for requirements and functionality of the work product/software application. Black box testing is sometimes also called as "Opaque Testing", "Functional/Behavioral Testing" and "Closed Box Testing".
The base of the Black box testing strategy lies in the selection of appropriate data as per functionality and testing it against the functional specifications in order to check for normal and abnormal behavior of the system. Now a days, it is becoming common to route the Testing work to a third party as the developer of the system knows too much of the internal logic and coding of the system, which makes it unfit to test the application by the developer.
In order to implement Black Box Testing Strategy, the tester is needed to be thorough with the requirement specifications of the system and as a user, should know, how the system should behave in response to the particular action.
Various testing types that fall under the Black Box Testing strategy are: functional testing, stress testing, recovery testing, volume testing, User Acceptance Testing (also known as UAT), system testing, Sanity or Smoke testing, load testing, Usability testing, Exploratory testing, ad-hoc testing, alpha testing, beta testing etc.
These testing types are again divided in two groups:
a) Testing in which user plays a role of tester and
b) User is not required.
Functional Testing:
In this type of testing, the software is tested for the functional requirements. The tests are written in order to check if the application behaves as expected.
Stress Testing:
The application is tested against heavy load such as complex numerical values, large number of inputs, large number of queries etc. which checks for the stress/load the applications can withstand.
Load Testing:
The application is tested against heavy loads or inputs such as testing of web sites in order to find out at what point the web-site/application fails or at what point its performance degrades.
Ad-hoc Testing:
This type of testing is done without any formal Test Plan or Test Case creation. Ad-hoc testing helps in deciding the scope and duration of the various other testing and it also helps testers in learning the application prior starting with any other testing.
Exploratory Testing:
This testing is similar to the ad-hoc testing and is done in order to learn/explore the application.
Usability Testing:
This testing is also called as ‘Testing for User-Friendliness’. This testing is done if User Interface of the application stands an important consideration and needs to be specific for the specific type of user.
Smoke Testing:
This type of testing is also called sanity testing and is done in order to check if the application is ready for further major testing and is working properly without failing up to least expected level.
Recovery Testing:
Recovery testing is basically done in order to check how fast and better the application can recover against any type of crash or hardware failure etc. Type or extent of recovery is specified in the requirement specifications.
Volume Testing:
Volume testing is done against the efficiency of the application. Huge amount of data is processed through the application (which is being tested) in order to check the extreme limitations of the system.
User Acceptance Testing:
In this type of testing, the software is handed over to the user in order to find out if the software meets the user expectations and works as it is expected to.
Alpha Testing:
In this type of testing, the users are invited at the development center where they use the application and the developers note every particular input or action carried out by the user. Any type of abnormal behavior of the system is noted and rectified by the developers.
Beta Testing:
In this type of testing, the software is distributed as a beta version to the users and users test the application at their sites. As the users explore the software, in case if any exception/defect occurs that is reported to the developers.
Black Box Testing Example
In this technique, we do not use the code to determine a test suite; rather, knowing the problem that we're trying to solve, we come up with four types of test data:
1. Easy-to-compute data
2. Typical data
3. Boundary / extreme data
4. Bogus data
Whitebox Testing:
White box testing is a security testing method that can be used to validate whether code implementation follows intended design, to validate implemented security functionality, and to uncover exploitable vulnerabilities.
This section introduces white box testing for security, how to perform white box testing, and tools and techniques relevant to white box testing. It brings together concepts from two separate domains: traditional white box testing techniques and security testing. It assumes the reader to be familiar with general concepts of software security. Refer to other content areas on this portal to learn different aspects of software security.
This section will help security developers and testers understand white box testing for security and how to effectively use the approach, tools, and techniques applicable to white box testing.
The section is organized into separate sections dealing with what white box testing is, how to perform white box testing, what results to expect, the business case to justify white box testing, skills and training required to perform white box testing, and a brief case study.
The purpose of any security testing method is to ensure the robustness of a system in the face of malicious attacks or regular software failures. White box testing is performed based on the knowledge of how the system is implemented. White box testing includes analyzing data flow, control flow, information flow, coding practices, and exception and error handling within the system, to test the intended and unintended software behavior. White box testing can be performed to validate whether code implementation follows intended design, to validate implemented security functionality, and to uncover exploitable vulnerabilities.
White box testing requires access to the source code. Though white box testing can be performed any time in the life cycle after the code is developed, it is a good practice to perform white box testing during the unit testing phase.
White box testing requires knowing what makes software secure or insecure, how to think like an attacker, and how to use different testing tools and techniques. The first step in white box testing is to comprehend and analyze source code, so knowing what makes software secure is a fundamental requirement. Second, to create tests that exploit software, a tester must think like an attacker. Third, to perform testing effectively, testers need to know the different tools and techniques available for white box testing. The three requirements do not work in isolation, but together.
Black box testing is based on the software’s specifications or requirements, without reference to its internal workings. Gray box testing combines white box techniques with black box input testing [Hoglund 04]. This method of testing explores paths that are directly accessible from user inputs or external interfaces to the software. In a typical case, white box analysis is used to find vulnerable areas, and black box testing is then used to develop working attacks against these areas. The use of gray box techniques combines both white box and black box testing methods in a powerful way.
The general outline of the white box testing process is as follows:
1. Perform risk analysis to guide the whole testing process.
2. Develop a test strategy that defines what testing activities are needed to accomplish testing goals.
3. Develop a detailed test plan that organizes the subsequent testing process.
4. Prepare the test environment for test execution.
5. Execute test cases and communicate results.
6. Prepare a report.
In addition to the general activities described above, the process diagram introduces review cycles, reporting mechanisms, deliverables, and responsibilities.
Error handling
The most neglected code paths during the testing process are error handling routines. Error handling in this paper includes exception handling, error recovery, and fault tolerance routines. Functionality tests are normally geared towards validating requirements, which generally do not describe negative (or error) scenarios. Even when negative functional tests are created, they don’t test for non-normative behavior or extreme error conditions, which can have security implications. For example, functional stress testing is not performed with an objective to break the system to expose security vulnerability. Validating the error handling behavior of the system is critical during security testing, especially subjecting the system to unusual and unexpected error conditions. Unusual errors are those that have a low probability of occurrence during normal usage. Unexpected errors are those that are not explicitly specified in the design specification, and the developers did not think of handling the error. For example, a system call may throw an ”unable to load library” error, which may not be explicitly listed in the design documentation as an error to be handled. All aspects of error handling should be verified and validated, including error propagation, error observability, and error recovery. Error propagation is how the errors are propagated through the call chain. Error observability is how the error is identified and what parameters are passed as error messages. Error recovery is getting back to a state conforming to specifications. For example, return codes for errors may not be checked, leading to uninitialized variables and garbage data in buffers; if the memory is manipulated before causing a failure, the uninitialized memory may contain attacker-supplied data. Another common mistake to look for is when sensitive information is included as part of the error messages.
Unit Testing:
In computer programming, unit testing is a procedure used to validate that individual units of source code are working properly. A unit is the smallest testable part of an application. In procedural programming a unit may be an individual program, function, procedure, etc., while in object-oriented programming, the smallest unit is a method; which may belong to a base/super class, abstract class or derived/child class.
Ideally, each test case is independent from the others; mock objects and test harnesses can be used to assist testing a module in isolation. Unit testing is typically done by developers and not by Software testers or end-users.
Six Rules of Unit Testing
1. Write the test first
2. Never write a test that succeeds the first time
3. Start with the null case, or something that doesn't work
4. Don't be afraid of doing something trivial to make the test work
5. Loose coupling and testability go hand in hand
6. Use mock objects
Requirements Testing:
Requirements seem to be ephemeral. They flit in and out of projects, they are capricious, intractable, unpredictable and sometimes invisible. When gathering requirements we are searching for all of the criteria for a system's success. We throw out a net and try to capture all these criteria. Using Blitzing, Rapid Application Development (RAD), Joint Application Development (JAD), Quality Function Deployment (QFD), interviewing, apprenticing, data analysis and many other techniques, we try to snare all of the requirements in our net.
Regression Testing:
Regression testing is any type of software testing which seeks to uncover regression bugs. Regression bugs occur whenever software functionality that previously worked as desired, stops working or no longer works in the same way that was previously planned. Typically regression bugs occur as an unintended consequence of program changes.
Common methods of regression testing include re-running previously run tests and checking whether previously fixed faults have re-emerged.
Experience has shown that as software is developed, this kind of reemergence of faults is quite common. Sometimes it occurs because a fix gets lost through poor revision control practices (or simple human error in revision control), but often a fix for a problem will be "fragile" in that it fixes the problem in the narrow case where it was first observed but not in more general cases which may arise over the lifetime of the software. Finally, it has often been the case that when some feature is redesigned, the same mistakes will be made in the redesign that were made in the original implementation of the feature.
Therefore, in most software development situations it is considered good practice that when a bug is located and fixed, a test that exposes the bug is recorded and regularly retested after subsequent changes to the program. Although this may be done through manual testing procedures using programming techniques, it is often done using automated testing tools. Such a 'test suite' contains software tools that allow the testing environment to execute all the regression test cases automatically; some projects even set up automated systems to automatically re-run all regression tests at specified intervals and report any regressions. Common strategies are to run such a system after every successful compile (for small projects), every night, or once a week. Those strategies can be automated by an external tool, such as BuildBot.
Regression testing is an integral part of the extreme programming software development method. In this method, design documents are replaced by extensive, repeatable, and automated testing of the entire software package at every stage in the software development cycle.
Regression Test Generation
Effective regression tests generate sufficient code execution coverage to exercise all meaningful code branches. Therefore, software testing is a combinatorial problem. Every boolean decision statement requires at least two tests: one with an outcome of "true" and one with an outcome of "false". As a result, for every line of code written, programmers often need 3 to 5 lines of test code.
Traditionally, regression testing has been performed by a software quality assurance team after the development team has completed work. However, defects found at this stage are the most costly to fix. This problem is being addressed by the rise of developer testing. Although developers have always written test cases as part of the development cycle, these test cases have generally been either functional tests or unit tests that verify only intended outcomes. Developer testing compels a developer to focus on unit testing and to include both positive and negative test cases.
When regression test generation is supported by a sustainable process for ensuring that test case failures are reviewed daily and addressed immediately, the end result is a regression suite that evolves with the application, and becomes more robust and more intelligent each day. If such a process is not implemented and ingrained into the team's workflow, the application may evolve out of sync with the generated test suite—- increasing false positives and reducing the effectiveness of the test suite.
Types of Regression
* Local - changes introduce new bugs.
* Unmasked - changes unmask previously existing bugs.
* Remote - Changing one part breaks another part of the program. For example, Module A writes to a database. Module B reads from the database. If changes to what Module A writes to the database break Module B, it is remote regression.
Mitigating Regression Risk
* Complete test suite repetition
* Partial test repetition based on traceability and analysis of technical and business risks
* Customer or user testing
o Beta - early release to both potential and current customers
o Pilot - deploy to a subset of users
o Parallel - users use both old and new systems simultaneously
* Use larger releases. Testing new functions often covers existing functions. The more new features in a release, the more "accidental" regression testing.
* Emergency patches - these patches are released immediately, and will be included in future maintenance releases.
Uses of Regression Testing
Regression testing can be used not only for testing the correctness of a program, but it is also often used to track the quality of its output. For instance in the design of a compiler, regression testing should track the code size, simulation time and compilation time of the test suite cases.
Strategies and Factors
Some strategies and factors to consider during regression testing include the following:
* Test fixed bugs promptly. The programmer might have handled the symptoms but not have gotten to the underlying cause.
* Watch for side effects of fixes. The bug itself might be fixed but the fix might create other bugs.
* Write a regression test for each bug fixed.
* If two or more tests are similar, determine which is less effective and get rid of it.
* Identify tests that the program consistently passes and archive them.
* Focus on functional issues, not those related to design.
* Make changes (small and large) to data and find any resulting corruption.
* Trace the effects of the changes on program memory.
Some of Basic Testing:
# Error Handling Testing
Error handling refers to the anticipation, detection, and resolution of programming, application, and communications errors. Specialized programs, called error handlers, are available for some applications. The best programs of this type forestall errors if possible, recover from them when they occur without terminating the application, or (if all else fails) gracefully terminate an affected application and save the error information to a log file.
In programming, a development error is one that can be prevented. Such an error can occur in syntax or logic. Syntax errors, which are typographical mistakes or improper use of special characters, are handled by rigorous proofreading. Logic errors, also called bugs, occur when executed code does not produce the expected or desired result. Logic errors are best handled by meticulous program debugging. This can be an ongoing process that involves, in addition to the traditional debugging routine, beta testing prior to official release and customer feedback after official release.
A run-time error takes place during the execution of a program, and usually happens because of adverse system parameters or invalid input data. An example is the lack of sufficient memory to run an application or a memory conflict with another program. On the Internet, run-time errors can result from electrical noise, various forms of malware or an exceptionally heavy demand on a server. Run-time errors can be resolved, or their impact minimized, by the use of error handler programs, by vigilance on the part of network and server administrators, and by reasonable security countermeasures on the part of Internet users.
# Manual Support Testing
Manual testing is the oldest and most rigorous type of software testing. Manual testing requires a tester to perform manual test operations on the test software without the help of Test automation. Manual testing is a laborious activity that requires the tester to possess a certain set of qualities; to be patient, observant, speculative, creative, innovative, open-minded, resourceful, unopinionated, and skillful.
Steps for Manual Testing
A manual tester would typically perform the following steps for manual testing:
1. Understand the functionality of program
2. Prepare a test environment
3. Execute test case(s) manually
4. Verify the actual result
5. Record the result as Pass or Fail
6. Make a summary report of the Pass and Fail test cases
7. Publish the report
8. Record any new defects uncovered during the test case execution
There is no complete substitute for manual testing. Manual testing is crucial for testing software applications more thoroughly. Test automation has become a necessity mainly due to shorter deadlines for performing test activities, such as regression testing, performance testing, and load testing.
# Intersystem Testing
There is software , some built in the OS and many after market that can do in system "house cleaning " and testing. One example is PC Alert. It shows up as a small open eye in task bar and when We move mouse over it, it reads out my CPU temp. If I right click it and go to the main page, I can test the inter system of every function of my computer as well as all the external systems as well. An inter system test, for example, could be a memory map, showing what is in memory at that moment. Another is a test of my fans, speed, efficiency etc. It basically is the way you find out if the inter system is working with out any problem and is a big help when you do have a problem in finding a fix for it.
Testing Start Process
Testing is sometimes incorrectly thought as an after-the-fact activity; performed after programming is done for a product. Instead, testing should be performed at every development stage of the product. Test data sets must be derived and their correctness and consistency should be monitored throughout the development process.
If we divide the lifecycle of software development into “Requirements Analysis”, “Design”, “Programming/Construction” and “Operation and Maintenance”, then testing should accompany each of the above phases. If testing is isolated as a single phase late in the cycle, errors in the problem statement or design may incur exorbitant costs. Not only must the original error be corrected, but the entire structure built upon it must also be changed. Therefore, testing should not be isolated as an inspection activity. Rather testing should be involved throughout the SDLC in order to bring out a quality product.
Testing Stop Process
This can be difficult to determine. Many modern software applications are so complex, and run in such as interdependent environment, that complete testing can never be done. "When to stop testing" is one of the most difficult questions to a test engineer. Common factors in deciding when to stop are:
* Deadlines ( release deadlines,testing deadlines.)
* Test cases completed with certain percentages passed
* Test budget depleted
* Coverage of code/functionality/requirements reaches a specified point
* The rate at which Bugs can be found is too small
* Beta or Alpha Testing period ends
* The risk in the project is under acceptable limit.
Practically, we feel that the decision of stopping testing is based on the level of the risk acceptable to the management. As testing is a never ending process we can never assume that 100 % testing has been done, we can only minimize the risk of shipping the product to client with X testing done. The risk can be measured by Risk analysis but for small duration / low budget / low resources project, risk can be deduced by simply: -
* Measuring Test Coverage.
* Number of test cycles.
* Number of high priority bugs.
Test Strategy
How we plan to cover the product so as to develop an adequate assessment of quality.
A good test strategy is:
* Specific
* Practical
* Justified
The purpose of a test strategy is to clarify the major tasks and challenges of the test project.
Test Approach and Test Architecture are other terms commonly used to describe what I’m calling test strategy.
Example of a poorly stated (and probably poorly conceived) test strategy:
"We will use black box testing, cause-effect graphing, boundary testing, and white box testing to test this product against its specification."
Error Guessing:
Error Guessing is a test case design technique where the tester has to guess what faults might occur and to design the tests to represent them.
Contents:
Ability to guess based on previous experience in Software Testing environment.
Adhoc method to identify tests likely to expose errors based on experience and intuition. Some areas to guess are Empty or null strings, Zero instances, occurrences, Blank or null characters in strings, Negative numbers
* Purpose
The purpose of error guessing is to focus the testing activity on areas that have not been handled by the other more formal techniques, such as equivalence partitioning and boundary value analysis. Error guessing is the process of making an educated guess as to other types of areas to be tested. For example, educated guesses can be based on items such as metrics from past testing experiences, or the tester's identification of situations in the Functional Design Specification or Detailed Design Specification, that are not addressed clearly.
* Examples
Though metrics from past test experiences are the optimum basis for error guessing, these may not be available. Examples of error prone situations include:
- initialization of data, (e.g., repeat a process to see if data is properly removed),
- wrong kind of data, (e.g., negative numbers, non-numeric versus numeric),
- handling of real data, (i.e., test using data created through the system or real records, because programmers tend to create data that reflects what they are expecting),
- error management, (e.g., proper prioritization of multiple errors, clear error messages, proper retention of data when an error is received, processing continues after an error if it is supposed to),
- calculations, (e.g., hand calculate items for comparison),
- restart/recovery, (i.e., use data that will cause a batch program to terminate before completion and determine if the restart/recovery process works properly),
- proper handling of concurrent processes, (i.e., for event driven applications, test multiple processes concurrently).