When Tests Collide: The Impact of Order-Dependent Tests To ...

When Tests Collide: The Impact of Order-Dependent Tests To ...

Parameterized Unit Testing in the Open Source Wild Wing Lam (U. Illinois) In collaboration with Siwakorn Srisakaokul, Blake Bassett, Peyman Mahdian and Tao Xie (U. Illinois) Nikolai Tillman, Jonathan de Halleux (Microsoft Research) Motivation Behavioral specifications are useful in static or dynamic software verification Ex. Code Contracts, JML, or Design by Contract

Postconditions are not commonly written In 90 projects with 43823 contract clauses, 11355 are postconditions (26%) [Schiller et al. ICSE14] T. W. Schiller, K. Donohue, F. Coward, and M. D. Ernst. Case studies and tools for contract specifications. In Proceedings of the 36th International Conference on Software Engineering, pages 596607. ACM, 2014. What are Parameterized Unit Tests (PUTs)? Written in the same manner as traditional unit tests, but taking parameters Very similar to code that developers are used to writing,

while providing stronger test oracles Decompose specification into small scenario based pieces, asserting expected behavior under a certain scenario Example of Parameterized Unit Test Asserts properties about many Set instances, beyond just a few hard-coded instances as in Conventional Unit Tests (CUTs) [PexMethod] public static void testAddExisting1(Set s, int i) { PexAssume.IsNotNull(s);

PexAssume.IsTrue(s.Contains(i)); int oldSize = s.Size; s.Add(i); PexAssert.AreEqual(s.Size, oldSize); } Describes Sets size change after adding an element. Set s is not null and contains i int i can be any integer Motivation Push Example

Code under verification/test, Push of bounded stack storing set of integers Code Contract: Contract.Ensures((Contract.OldValue(this.IsFull()) || Contract.OldValue(this.IsMember(e))) || (this.IsMember(e) && Contract.OldValue(this.GetSize()) + 1 == this.GetSize())); Motivation Push Example Code under verification/test, Push of bounded stack storing set of integers Code Contract: Contract.Ensures((Contract.OldValue(this.IsFull()) || Contract.OldValue(this.IsMember(e))) ||

(this.IsMember(e) && Contract.OldValue(this.GetSize()) + 1 == this.GetSize())); PUT: Motivation Pop Example Code under verification/test, Pop of bounded stack storing set of integers Code Contract: PUT: Motivation Pop Example

Code under verification/test, Pop of bounded stack storing set of integers Code Contract: ??? PUT: Motivation Pop Example Code under verification/test, Pop of bounded stack storing set of integers Code Contract: ???

PUT: Parameterized Unit Testing Going Mainstream Parameterized Unit Tests (PUTs) commonly supported by various test frameworks .NET: Supported by .NET test frameworks http://www.mbunit.com/ http://www.nunit.org/

Java: Supported by JUnit 4.X http://www.junit.org/ Generating test inputs for PUTs supported by tools .NET: Supported by Microsoft Visual Studio 2015 IntelliTest Formerly Microsoft Research Pex: http://research.microsoft.com/pex/ Java: Supported by Agitar AgitarOne http://www.agitar.com/ Goal of Our Work To fill the gap that there exists no study to understand how developers use parameterized unit testing

Empirical study of the use of parameterized unit testing in the open source wild Suite of analysis tools for analyzing parameterized unit tests and associated software systems Study Setup Collect open source projects with PUTs written in C# (subjects) (1) Collect subjects from searchable online source code repositories (2) Deduplicate subjects obtained multiple times from repositories (3) Condense multi-project subjects (4) Filter developer-written PUTs (5) Categorize subjects into levels of parameterized test usage

Subject Statistics Subject name # of PexMethods # of UnitTestMethods PexFaultLocalization 168

1221 linqtoroot 133 344 QuickGraph0 112

1212 concolic-walk 87 1 stringextensions 67

466 isuka 54 300 utilities-net 44

299 henoch 27 170 bbcode 26 16

ConcurrentList 21 27 Identified 33

subjects that contained a high number of PUTs Total of 839 PUTs in them Stakeholders of Our Findings Developers who are writing PUTs Developers who have not yet written PUTs

Tool developers like ourselves for the analysis tool PUT framework developers Researchers Educators/trainers of software testing Anomaly Example in Our Subjects Developers can misunderstand/misuse PUTs as code contracts http://automatones.codeplex.com/ Research Questions How extensively are PUTs used?

How are CUTs used over PUTs? What motivations do developers have to use one over the other? How frequently are non-primitive parameters supplied to PUTs? What is the percentage of non-primitive types and parameters associated with factory methods? How extensively are PUTs used? On average 49.2% of public methods are tested with PUTs Methods not tested with PUTs are commonly helper methods or methods that are using

code contracts How extensively are PUTs used? On average 49.2% of public methods are tested with PUTs Methods not tested with PUTs are commonly helper methods or methods that are using code contracts How extensively are PUTs used? Statistics

How are CUTs used over PUTs? Average test method length PUTs: 12.9 lines CUTs: 13.9 lines CUTs are longer because developers write test code to produce a specific input How frequently are non-primitive parameters supplied to PUTs?

32.26% of non-primitive parameters in PUTs are uniquely typed Only 9.11% of non-primitive parameters in PUTs are associated with factory methods Developers rarely write factory methods Threats to Validity Construct Validity - use of our chosen metrics to model

our desired properties Internal Validity - complexity of software, faults in our analysis tools External Validity - our conclusions when applied to other subjects Conclusion Specification through code contracts and invariants are difficult Parameterized unit testing is a lightweight specification methodology Major findings: there exist cases of

Misusing/misunderstanding the use of PUTs Not fully leveraging PUTs when specific inputs are hardcoded Factory methods not sufficiently written to assist test generation tool More findings are in our paper draft under submission Thank you! Any questions? Conclusion Specification through code contracts and invariants are difficult

Parameterized unit testing is a lightweight specification methodology Major findings: there exist cases of Misusing/misunderstanding the use of PUTs Not fully leveraging PUTs when specific inputs are hardcoded Factory methods not sufficiently written to assist test generation tool More findings are in our paper draft under submission

Recently Viewed Presentations

  • Online II Moderations Growing the Community of Practice

    Online II Moderations Growing the Community of Practice

    iConnect is the Web Conference program that enables anyone, anywhere to join in the conversation. Parts of the Moderation Process Video Upload Involves selecting the 10 minute portion of video, converting the video to a smaller size without losing quality...
  • CAISOs Market and Reliability Operations, September 21, 2015

    CAISOs Market and Reliability Operations, September 21, 2015

    California ISO by the numbers. 65,225 MW of power plant capacity (net dependable capacity). 50,270. MW record peak demand (July 24, 2006) 27,076. market transactions per day. 26,024 . circuit-miles of transmission lines
  • WDCU EGM 17TH April 2019

    WDCU EGM 17TH April 2019

    law 42: the umpires shall act upon any unacceptable behaviour. four levels of offences and the corresponding actions by the umpires are indentified as level 1, level 2, level 3, and level 4. ... showing the umpire the bat indicating...
  •   :  . laila.altaweel@gmail.com Tishreen University Dr. Laila Al-Taweel

    : . [email protected] Tishreen University Dr. Laila Al-Taweel

    د. ليلى الطويل: النائب العلمي في كلية الاقتصاد - جامعة تشرين [email protected] * Tishreen University Dr. Laila Al-Taweel
  • Historie umělé inteligence

    Historie umělé inteligence

    Self-Organizing Map (SOM) (Teuvo Kohonen, 1982) Umí se učit bez dozoru. (unsupervised learning) Backpropagation (Arthur Bryson a Yu-Chi Ho, 1969) algoritmus učení vícevrstvé neuronové sítě Vyžaduje, aby neurony sítě neměly ostrý práh. Protože byl nepovšimnut, byl potom ...
  • From FIE to PLAAFPs to IEP's - Julious Morris Educational ...

    From FIE to PLAAFPs to IEP's - Julious Morris Educational ...

    Vineland Adaptive Behavior Scales (2nd ed.) (Vineland-II) Gray Oral Reading Test-Fourth Edition (GORT-4) Cognitive academic language proficiency (CALP) Basic interpersonal communicative skills (BICS) Cognitive Abilities Tests (CogAT) Multidimensional Aptitude Battery (MAB) Brigance Comprehensive Inventory of Basic Skills. More…
  • Measuring length- PowerPoint ELGs This powerpoint can be

    Measuring length- PowerPoint ELGs This powerpoint can be

    Measuring length- PowerPoint ELGs. This powerpoint can be used to support the teaching of 'Shape, space and measure' targeted towards the early learning goals. Teaching tips: Once children are familiar measuring using non standard units of measure, why not introduce...
  • Lesson 1 Secular Humanism  An Overview 1 Lesson

    Lesson 1 Secular Humanism An Overview 1 Lesson

    INTRODUCTION. A frequent message of the minor prophets (studied last quarter) was a call to the children of Israel to separate themselves from the sinful ways of neighboring nations, and return to keeping God's statutes and commandments.