View source for Regression testing

{{Short description|Checking whether changes to software have broken functionality that used to work}}
{{About|software development|the statistical analysis process|Regression analysis|}}
{{Software development process}} 
'''Regression testing''' (rarely, ''non-regression testing''<ref>{{cite book |last1=Pezzè |first1=Mauro |last2=Young |first2=Michal |title=Software testing and analysis: process, principles, and techniques |date=2008 |publisher=Wiley |url=https://www.google.com/search?q=Mauro+%22non-regression%22+%22regression+testing%22 |quote=Testing activities that focus on regression problems are called (non) regression testing. Usually "non" is omitted}}</ref>) is re-running [[functional testing|functional]] and [[non-functional testing|non-functional tests]] to ensure that previously developed and tested software still performs as expected after a change.<ref>{{cite book|last=Basu|first=Anirban| title=Software Quality Assurance, Testing and Metrics| year=2015| publisher=PHI Learning| isbn=978-81-203-5068-7| url=https://books.google.com/books?id=aNTiCQAAQBAJ&pg=PA150}}</ref> If not, that would be called a ''[[software regression|regression]]''.

Changes that may require regression testing include [[software bug|bug]] fixes, software enhancements, [[configuration file|configuration]] changes, and even substitution of [[electronic component]]s ([[Computer hardware|hardware]]).<ref>[[National Academies of Sciences, Engineering, and Medicine|National Research Council]] Committee on Aging Avionics in Military Aircraft: [https://www.nap.edu/catalog/10108/aging-avionics-in-military-aircraft ''Aging Avionics in Military Aircraft'']. The National Academies Press, 2001, page 2: ″Each technology-refresh cycle requires regression testing.″</ref> As regression [[test suite]]s tend to grow with each found defect, test automation is frequently involved. Sometimes a [[change impact analysis]] is performed to determine an appropriate subset of tests (''non-regression analysis''<ref>{{cite book |last1=Boulanger |first1=Jean-Louis |title=CENELEC 50128 and IEC 62279 Standards |date=2015 |publisher=Wiley |isbn=978-1-119-12248-7 |url=https://books.google.com/books?id=IbZNCAAAQBAJ&pg=PA149}}</ref>).

==Background==
As software is updated or changed, or reused on a modified target, emergence of new faults and/or re-emergence of old faults is quite common.

Sometimes re-emergence occurs because a fix gets lost through poor [[revision control]] practices (or simple [[human error]] in revision control). Often, a fix for a problem will be "[[software brittleness|fragile]]" in that it fixes the problem in the narrow case where it was first observed but not in more general cases which may arise over the lifetime of the software. Frequently, a fix for a problem in one area inadvertently causes a [[software bug]] in another area.

It may happen that when a feature is redesigned some of the same mistakes that were made in the original implementation of the feature also occur in the redesign. In most software development situations, it is considered [[Best Coding Practices|good coding practice]], when a bug is located and fixed, to record a test that exposes the bug and re-run that test regularly after subsequent changes to the program.<ref name="kolawa">{{cite book | last = Kolawa | first = Adam | author2 = Huizinga, Dorota | title = Automated Defect Prevention: Best Practices in Software Management | url = http://www.wiley.com/WileyCDA/WileyTitle/productCd-0470042125.html | year = 2007 | publisher = Wiley-IEEE Computer Society Press | page = 73 | isbn = 978-0-470-04212-0 | archive-date = 2012-04-25 | access-date = 2010-07-20 | archive-url = https://web.archive.org/web/20120425232401/http://www.wiley.com/WileyCDA/WileyTitle/productCd-0470042125.html }}</ref>

Although this may be done through [[manual testing]] procedures using programming techniques, it is often done using [[automated testing]] tools.<ref>[http://safari.oreilly.com/0201794292/ch08lev1sec4 Automate Regression Tests When Feasible], Automated Testing: Selected Best Practices, Elfriede Dustin, Safari Books Online</ref> Such a [[test suite]] contains software tools that allow the testing environment to execute all the regression [[Test case (software)|test case]]s automatically; many projects have automated [[Continuous integration]] systems to re-run all regression tests at specified intervals and report any failures (which could imply a regression or an out-of-date test).<ref>{{cite web| url=http://www.drdobbs.com/tools/change-code-without-fear/206105233| title=Change Code Without Fear: Utilize a Regression Safety Net|last=daVeiga|first=Nada|work=[[Dr. Dobb's Journal]]| date=2008-02-06}}</ref>

Common strategies are to run such a system after every successful compile (for small projects), every night, or once a week. Those strategies can be automated by an external tool.

Regression testing is an integral part of the [[extreme programming]] software development method. In this method, design documents are replaced by extensive, repeatable, and automated testing of the entire software package throughout each stage of the [[software development process]]. Regression testing is done after functional testing has concluded, to verify that the other functionalities are working.

In the corporate world, regression testing has traditionally been performed by a [[software quality assurance]] team after the development team has completed work. However, defects found at this stage are the most costly to fix. This problem is being addressed by the rise of [[unit testing]]. Although developers have always written test cases as part of the development cycle, these test cases have generally been either [[functional testing|functional tests]] or [[unit testing|unit tests]] that verify only intended outcomes. Developer testing compels a developer to focus on unit testing and to include both positive and negative test cases.<ref>{{cite web|url=http://www.sys-con.com/read/47359.htm|title=Developer Testing Is 'In': An interview with Alberto Savoia and Kent Beck|last=Dudney|first=Bill|date=2004-12-08|access-date=2007-11-29}}</ref>

== Techniques ==
The various regression testing techniques are:

=== Retest all ===
This technique checks all the test cases on the current program to check its integrity. Though it is expensive as it needs to re-run all the cases, it ensures that there are no errors because of the modified code.<ref name=":0">{{Cite conference|date=2008-03-29|title=Understanding Regression Testing Techniques|first1=Gaurav|last1=Duggal|first2=Bharti|last2=Suri|conference=National Conference on Challenges and Opportunities|location=Mandi Gobindgarh, Punjab, India|citeseerx=10.1.1.460.5875}}</ref>

=== Regression test selection ===
Unlike Retest all, this technique runs a part of the [[test suite]] (owing to the cost of retest all) if the cost of selecting the part of the test suite is less than the Retest all technique.<ref name=":0" />

=== Test case prioritization ===
Prioritize the test cases so as to increase a test suite's rate of fault detection. Test case prioritization techniques schedule test cases so that the test cases that are higher in priority are executed before the test cases that have a lower priority.<ref name=":0" />

==== Types of test case prioritization ====
* General prioritization – Prioritize test cases that will be beneficial on subsequent versions
* Version-specific prioritization – Prioritize test cases with respect to a particular version of the software.

=== Hybrid ===
This technique is a hybrid of regression test selection and test case prioritization.<ref name=":0" />

== Benefits and drawbacks ==
Regression testing is performed when changes are made to the existing functionality of the software or if there is a bug fix in the software. Regression testing can be achieved through multiple approaches; if a ''test all'' approach is followed, it provides certainty that the changes made to the software have not affected the existing functionalities, which are unaltered.<ref name=":1">{{cite journal |last1=Yoo |first1=S. |last2=Harman |first2=M. |title=Regression testing minimization, selection and prioritization: a survey |journal=Software Testing, Verification and Reliability |date=2010 |volume=22 |issue=2 |pages=67–120|doi=10.1002/stvr.430}}</ref>

In [[agile software development]]—where the software development life cycles are very short, resources are scarce, and changes to the software are very frequent—regression testing might introduce a lot of unnecessary [[Overhead (business)|overhead]].<ref name=":1" />

In a software development environment which tends to use [[black box]] components from a third party, performing regression testing can be tricky, as any change in the third-party component may interfere with the rest of the system (and performing regression testing on a third-party component is difficult, because it is an unknown entity).<ref name=":1" />

==Uses==
Regression testing can be used not only for testing the ''[[correctness (computer science)|correctness]]'' of a program but often also for tracking the quality of its output.<ref>{{cite web |url=http://www.wrox.com/WileyCDA/Section/id-291252.html |title=Regression Testing, Programmer to Programmer |last=Kolawa |first=Adam |author-link=Adam Kolawa |work=Wrox }}</ref> For instance, in the design of a [[compiler]], regression testing could track the code size and the time it takes to compile and execute the test suite cases.

{{Quotation |text=Also as a consequence of the introduction of new bugs, program maintenance requires far more system testing per statement written than any other programming. Theoretically, after each fix, one must run the entire batch of test cases previously run against the system to ensure that it has not been damaged in an obscure way. In practice, such ''regression testing'' must indeed approximate this theoretical idea, and it is very costly.|author=[[Fred Brooks]]|title=''[[The Mythical Man Month]]''|source=p. 122}}

Regression tests can be done at any level, from [[unit testing|unit]] through to [[System integration testing|system integration]]. Functional tests exercise the complete program with various inputs. These tests are often automated due to the need for repetition, and may be done by test tools that are not part of the compiler suite. Non-functional tests assess aspects such as performance, security, or reliability. Conducting non-functional regression tests presents additional challenges. For example, detecting changes in performance is often statistically complex,<ref>{{cite conference |last1=Reichelt |first1=D. G. |last2=Kühne |first2=S. |last3=Hasselbring |first3=W. |title=Automated identification of performance changes at code level |book-title=2022 IEEE 22nd International Conference on Software Quality, Reliability and Security (QRS) |pages=916–925 |year=2022 |publisher=IEEE|doi=10.1109/QRS57517.2022.00096|arxiv=2303.14256 }}</ref> while security issues typically arise from vulnerabilities in the software ecosystem rather than from individual code changes.<ref>{{cite journal |last1=Rajapakse |first1=R. N. |last2=Zahedi |first2=M. |last3=Babar |first3=M. A. |last4=Shen |first4=H. |title=Challenges and solutions when adopting DevSecOps: A systematic review |journal=Information and Software Technology |volume=141 |article-number=106700 |year=2022|doi=10.1016/j.infsof.2021.106700}}</ref>

==References==
{{reflist}}

{{Software testing}}

{{DEFAULTSORT:Regression Testing}}
[[Category:Software testing]]
[[Category:Extreme programming]]