BugStory
• Software Fault : A static defect in thesoftware (i.e., defect)，静态存在于软件中的缺陷，如code写错了
• Software Error : An incorrect internalstate that is the manifestation of some fault，软件运行时，运行到fault触发产生错误的中间状态。
• Software Failure : External, incorrectbehavior with respect to the requirements or other description of the expected behavior，Error传到软件外部，使得用户或测试人员观测到失效的行为。
Fault：
可能导致系统或功能失效的异常条件（Abnormal condition thatcan cause an element or an item to fail.），可译为“故障”。
Error：
计算、观察或测量值或条件，与真实、规定或理论上正确的值或条件之间的差异（Discrepancybetween a computed, observed or measured value or condition and the true,specified, or theoretically correct value or condition.），可译为“错误”。Error是能够导致系统出现Failure的系统内部状态。
Failure：
当一个系统不能执行所要求的功能时，即为Failure，可译为“失效”。（Termination ofthe ability of an element or an item to perform a function as required.）
PIEModel
1. Execution/Reachability : The location orlocations in the program that contain the fault must be reached，执行必须通过错误的代码
2. Infection : The state of the programmust be incorrect ，在执行错误代码时必须触发一个错误的中间状态
3. Propagation : The infected state mustpropagate to cause some output of the program to be incorrect，错误的中间状态必须传播到最后输出，使得观测到的输出结果和预期结果不一致，即失效。
产生fault的程序，可能在测试时不会触发错误的中间状态；同理，触发错误的中间状态可能不会使测试人员观察到失效的行为。
有没有一个fault，任何测试都不会将其测试出来？这样还能称之为fault吗？
有这样的faul
Terminology
测试是为了发现bug，通过实施测试和观察failure
debug是为了解决bug，通过定位，理解和修复fault
Validation
“The assurance that a product, service, orsystem meets the needs of the customer and other identified stakeholders. Itoften involves acceptance and suitability with external customers.”
Verification
“The evaluation of whether or not aproduct, service, or system complies with a regulation, requirement,specification, or imposed condition. It is often an
internal process.”
Verification是验证,是通过提供客观证据证明规定的要求是否得到满足,也就是说,输入与输出比较.
Validation是确认,是在验证好的基础上,对预期的使用和应用要求是否得到满足,也就是说,在确认时,应考虑使用和应用的条件范围要远远大于输入时确定的范围.一般是由客户或代表客户的人执行.
Verification也就是说要做正确、而Validation是看经过Verification是否是我们想要的。
而Validation首先前提是经过Verification，重要的是做的是否是customer需要的。
黑盒测试，通过测试来检测每个功能是否都能正常使用。在测试中，把程序看作一个不能打开的黑盒子，在完全不考虑程序内部结构和内部特性的情况下，在程序接口进行测试，它只检查程序功能是否按照需求规格说明书的规定正常使用，程序是否能适当地接收输入数据而产生正确的输出信息。黑盒测试着眼于程序外部结构，不考虑内部逻辑结构，主要针对软件界面和软件功能进行测试。
白盒测试，又称结构测试、透明盒测试、逻辑驱动测试或基于代码的测试。白盒测试是一种测试用例设计方法，盒子指的是被测试的软件，白盒指的是盒子是可视的，你清楚盒子内部的东西以及里面是如何运作的。"白盒"法全面了解程序内部逻辑结构、对所有逻辑路径进行测试。"白盒"法是穷举路径测试。在使用这一方案时，测试者必须检查程序的内部结构，从检查程序的逻辑着手，得出测试数据。贯穿程序的独立路径数是天文数字。
灰盒测试，是介于白盒测试与黑盒测试之间的一种测试，灰盒测试多用于集成测试阶段，不仅关注输出、输入的正确性，同时也关注程序内部的情况。灰盒测试不像白盒那样详细、完整，但又比黑盒测试更关注程序的内部逻辑，常常是通过一些表征性的现象、事件、标志来判断内部的运行状态。
测试等级：
系统测试,System testing
集成测试,Integration testing
模块测试,Module testing
单元测试,Unit testing
FaultRevisit
Fault is defined by testing! 故障是通过测试定义的
自动化软件测试 Automatic software testing
测试用例生成
测试数据生成
测试用例选择
•Regression Test Selection
•针对修改部分的测试
• Test Suite Reduction
•针对测试需求的用例集约简
• Test Case Prioritization
•针对测试需求的用例排序
软件行为学习
测试用例修复
测试用例演化
Junit
Junit测试时程序员测试，即所谓白盒测试
集成TestCase类即可利用junit测试
需要知道程序结构以及逻辑
设计不同的测试输入以及输出语言
尽可能覆盖多的语句以及分支
测试的技巧
1. 大量测试数据
2. 典型错误类型
3. 强制产生错误条件
CFG Generator 观测结构
CFG——Control FlowGraph 控制流图
观察程序的控制流走向
CodeCover
用来统计测试覆盖率，支持多种覆盖（包括语句覆盖，分支覆盖，MCDC覆盖等）
CFG
Test Path: A path that starts at an initialvertex and ends at a final vertex
Each test executes one and only one testpath
GraphCoverage Criteria 图覆盖准则
• Syntactic reach : A path exists in thegraph
• Semantic reach : A test exists that canexecute that path
语义可行的一定语法可行，但语法可行的不一定语义可行
Test Criteria
• Test Requirements (TR) :
Describe properties of test paths
• Test Criterion :
Rules that define test requirements
• Satisfaction:
Given a set TR of test requirements for acriterion C, a set of tests T satisfies C on a graph if and only if for everytest requirement in TR, there is a test path in path(T) that meets the testrequirement tr
StructuralCoverage 结构覆盖
Vertex Coverage (VC) 点覆盖
Test set T satisfies vertex coverage ongraph G if and only if for every syntactically reachable vertex v in V, thereis a path p in path(T) such that p covers v.
Edge Coverage (EC) 边覆盖 边覆盖一定满足点覆盖，反过来就不一定了
Test set T satisfies vertex coverage ongraph G if and only if for every syntactically reachable edge e in E, there isa path p in path(T) such that p covers e.
Edge-Pair Coverage (EPC) 边对覆盖
TR contains each reachable path of lengthto up 2, inclusive, in G.
Complete Path Coverage (CPC) : TR containsall paths in G.
n-Path Coverage (nPC) : TR contains eachreachable path of length up to n, inclusive, in G.
ControlFlow Graph
•A control flow graph (CFG) is a representation,using graph notation, of all paths that might be traversed through a programduring its execution.
if , if-return , while, do , for , breakand continue , switch
PrimePath Coverage
如果一个图含有一个循环，那么它就有无限条路径，这时，CPC是不可行的
Simple Path : A path from node ni to nj issimple if no node appears more than once, except possibly the first and lastnodes are the same 路径中没有相同的结点，除了第一个和最后一个可以相同
• No internal loops
• A loop is a simple path
Prime Path : A simple path that does notappear as a proper subpath of any other simple path
是简单路径，且不是其他简单路径的子路径
Prime Path Coverage (PPC) : TR containseach prime path in G.包括点覆盖，路径覆盖，但不包括边对覆盖
Round-Trip Path : A prime path that startsand ends at the same node
Simple Round Trip Coverage (SRTC) : TRcontains at least one round-trip path for each reachable vertex in G thatbegins and ends a round-trip path.
Complete Round Trip Coverage (CRTC) : TRcontains all roundtrip paths for each reachable vertex in G.
Prime path: the maximal-long simple path
BasicPath Coverage
介于分支覆盖和全路径覆盖中
Testing that fulfills the requirements ofbranch testing & also tests all of the independent paths that could be usedto construct any arbitrary path through the computer program.
Independent Path
•A path through the system is independentfrom other paths only if it includes some vertices or edges that are notcovered in the other path.
From linear algebra, it is known that eachmatrix has a unique rank (number of linearly independent rows) that is lessthan or equal to the number of columns.
•The number of linearly independent pathsis the rank of this matrix.
•The rank of this matrix is exactly the cyclomaticcomplexity of the graph. 圈复杂度
CC=E-V+2
McCabe’s Basic Path Testing
1. Generate control flow graph
2. Compute cyclomatic complexity
3. Select a set of basic paths
4. Generate tests for the basic paths
EventFlow Graph 事件流图
Definition: An Event Flow Graph (EFG) is a3-tuple M = ⟨V, I, E⟩, where:
1. V is a set of vertices representing allevents of objects.
2. I ⊆V is a set of initial vertices.
3. E ⊆V ×Vis a set of edges between vertices. (vi, vj) ∈E if vjmay be executed immediately after vi.
RandomTesting
Test cases are generated purely at random[SWEBOK v3.0]
–Input domain must be known
–Pick random points within input domain
–Automation
Problems in RT
• Define input domain
• Random mechanism 随机机制
• Randomness and IntegrityService(random.org) 随机性和完整性
ART Algorithm
randomly generate an input t, run t, add tto T
while (stop criteria not reached)
randomlygenerate k candidate input c1, … ck
for eachcandidate ci
compute min distance di to T
end for
select onecandidate t with min distance
run t, add t toT
end while
Problems in ART
• Distance
• Overhead 开销
• Curse of Dimensionality 维度灾难
Anti-Random Testing Process:
1 . 选择一条测试用例
2. 从所有可能的测试用例中，选择一条使其与已有测试用例的海明距离之和最大
3. 重复第二步，直至测试用例总量满足需求
DataFlow Coverage 数据流覆盖
Sets of Def and Use
• def (n) or def (e) :
The set of variables that are defined bynode n or edge e
• use (n) or use (e) :
The set of variables that are used by noden or edge e
DU pair :
A pair of locations (li, lj) such that avariable v is defined at li and used at lj
• Def-clear :
A path from li to lj is def-clear withrespect to variable v if v is not given another value on any of the nodes oredges in the path
• Reach :
If there is a def-clear path from li to ljwith respect to v, the def of v at li reaches the use at lj
• du-path :
A simple subpath that is def-clear withrespect to v from a defof v to a use of v
• du (ni, nj, v)
the set of du-paths from ni to nj
• du (ni, v)
the set of du-paths that start at ni
All-defs coverage (ADC) :
For each set of du-paths S = du (n, v), TRcontains at least one path d in S.
All-uses coverage (AUC) :
For each set of du-paths to uses S = du(ni, nj, v), TR contains at least one path d in S.
All-du-paths coverage (ADUPC) :
For each set S = du (ni, nj, v), TRcontains every path d in S.
EquivalencePartitioning 等价划分
• Can be equally applied at several levelsof testing
–Unit
– Integration
– System
• Relatively easy to apply with noautomation
• Easy to adjust the procedure to get moreor fewer tests
Input Domains
• The input domain for a program containsall the possible inputs to that program
• For even small programs, the input domainis so large that it might as well be infinite
• Testing is fundamentally about choosingfinite sets of values from the input domain
• Input parameters define the scope of theinput domain
– Parameters to a method
– Data read from a file
– Global variables
– User level inputs
• Domain for each input parameter ispartitioned into regions
• At least one value is chosen from eachregion
Partitioning Domains
• Domain D
• Partition scheme p of D
• The partition p defines a set of blocks,b1 , b2 , … bn
• The partition must satisfy two properties:
1. blocks must be pairwise disjoint (nooverlap) 划分的块之间不能有重叠
2. together the blocks cover the domain D(complete) 所有的块和在一起完全覆盖D
• 将不能穷举的测试过程进行分类
• 把输入域划分为若干子集，然后从每一个子集中选取少数具有代表性的数据作为测试用例
• 等价类即子集合，其中的数据在某种意义上等价
有效与无效等价类
(1) 有效等价类 – 有意义的、合理的输入数据
– 检验程序是否实现了规格说明中预先规定的功能和性能
(2) 无效等价类
– 是无意义的、不合理的输入数据所构成的集合
– 鉴别程序异常处理的情况，检查功能和性能的实现是否有不符合规格说明要求的地方
等价类的划分原则
• 完备性
–等价类的并集应涵盖整个输入域
• 无冗余
–等价类之间互不相交
Two Approaches
1. Interface-based approach 基于接口
– Develops characteristics directly fromindividual input parameters
– Simplest application
– Can be partially automated in somesituations
2. Functionality-based approach 基于功能
– Develops characteristics from abehavioral view of the program under test
– Harder to develop—requires more designeffort
– May result in better tests, or fewertests that are as effective
Interface-Based Approach
• Mechanically consider each parameter inisolation
• This is an easy modeling technique andrelies mostly on syntax
• Ignores relationships among parameters
Functionality-Based Approach
• Identify characteristics that correspondto the intended functionality
• Requires more design effort from tester
• Can incorporate domain and semanticknowledge
• Can use relationships among parameters
• Modeling can be based on requirements,not implementation
• The same parameter may appear in multiplecharacteristics, so it’s harder to translate values to test cases
等价类的划分方法
（1）按照区间划分 在输入条件规定了取值范围或值的个数的情况下，可以确定一个有效等价类和两个无效等价类。
（2）按照数值划分 在规定了一组输入数据 （假设包括n个输入值），并且程序要对每一 个输入值分别进行处理的情况下，可确定n个有效等价类（每个值确定一个有效等价类）和一个无效等价类（所有不允许的输入值的集合）。
（3）按照数值集合划分 在输入条件规定了输入值的集合或规定了“必须如何”的条件下，可以确定一个有效等价类和一个无效等价类（该集合有效值之外）
（4）按照限制条件或规则划分 在规定了输 入数据必须遵守的规则或限制条件的情况下可确定一个有效等价类（符合规则）和若干个无效等价类（从不同角度违反规则）。
（5）细分等价类 在确知已划分的等价类中 各元素在程序中的处理方式不同的情况下，则应再将该等价类进一步划分为更小的等价类，并建立等价类表。
LogicCoverage 逻辑覆盖
Decision Coverage (DC)： Executing true and false of decision.
Condition Coverage (CC): Executing true andfalse of each condition
Condition/Decision Coverage (C/DC): CombingDC and CC.
Multiple Condition Coverage
•Multiple condition coverage (MCC) reportswhether every possible combination of Boolean sub-expressions occurs.
•The test cases required for full multiplecondition coverage of a condition are essentially given by the logical operatortruth table for the condition.
Boundary-ValueAnalysis 边界值分析
An input variable with equivalent class[Min, Max]
Select
– Min
– Min+
– Nom
– Max-
– Max
Weak Robust-Boundary Analysis
Strong Robust-Boundary Analysis
MC/DC
•MC/DC or MCDC:
Executing the independent true and falseoutcomes of each condition.
•MC/DC ≥ C/DC
MC/DC: Discussion
• Modified condition/decision coverage wasdesigned for languages containing logical operators that do not short circuit. 短路
• The short circuit logical operators in C,C++ and Java only evaluate conditions when their result can affect theencompassing decision.
• MC/DC will be affected by the structuresof decisions in program.
CombinatorialTesting 组合测试
Weak Equivalence Class Testing 弱等价类测试，根据类别最多的输入来划分
Strong Equivalence Class Testing 强等价类测试，根据所有输入的类别来划分
Cost Effective Testing
Pair-wise Testing
t-wise/t-ways Combinatorial Testing
Variable strength combinatorial testing
MutationTesting 变异测试
• Faults are introduced into the program bycreating many versions of the program called mutants.
• Each mutant contains a single fault.
• Test cases are applied to the originalprogram and to the mutant program.
• The goal is to cause the mutant programto fail, thus demonstrating the effectiveness of the test case.
一种在细节方面改进程序源代码的软件测试方法。这些所谓的变异，是基于良好定义的变异操作，这些操作或者是模拟典型应用错误（例如：使用错误的操作符或者变量名字），或者是强制产生有效地测试（例如使得每个表达式都等于0）。目的是帮助测试者发现有效地测试，或者定位测试数据的弱点，或者是在执行中很少（或从不）使用的代码的弱点。
• Mutation Testing is a method of insertingfaults into programs to test whether the tests pick them up, thereby validatingor invalidating the tests.
• Mutation Testing is a testing techniquethat focuses on measuring the adequacy of test cases. 充分性
• Mutation Testing should be used inconjunction with traditional testing techniques, not instead of them.
• Mutation testing involves the creation ofa set of mutant programs of the program being tested.
• Each mutant differs from the originalprogram by one mutation.
• A mutation is a single syntactic changethat is made to a program statement.
Categories of Mutation Operators
Operand Mutation Operators: 操作数变异
• Replace a single operand with anotheroperand or constant.
Expression Mutation Operators: 表达式变异
• Replace an operator or insert newoperators.
Equivalent Mutant 等价变异体
• There may be surviving mutants thatcannot be killed, these are called Equivalent Mutants
• Although syntactically different, thesemutants are indistinguishable through testing.
• They therefore have to be checked ‘byhand’
一些不能被杀死的变异，即使语法上不通，通过测试是不能分辨的，只能手动检测
Mutation Score
• The mutation score for a set of testcases is the percentage of non-equivalent mutants killed by the test data.
• Mutation Score = 100 * K / (T - E)
• K= Number of killed mutants
• T= Number of total mutants
• E = Number of equivalent mutants
• A set of test cases is mutation adequateif its mutation score is 100%.
Fault-basedTesting 基于fault的测试
变异测试是运用最广泛的
• Operator Reference Fault (ORF)
• An occurrenceof a logical connective “&” is replaced by “|” or vice versa.
• Expression Negation Fault (ENF)
• Asubexpression (except conditions) is replaced by its negation.
• Variable Negation Fault (V NF)
• An occurrenceof a condition is replaced by its negation.
• Associative Shift Fault (ASF)
• ASF is causedby omission of the brackets because of themisunderstanding about operatorevaluation priorities.
• Missing Variable Fault (MVF)
• An occurrenceof a condition is omitted.
• Variable Reference Fault (VRF)
• An occurrenceof a condition is replaced by another possible condition .
• Clause Conjunction Fault (CCF)
• An occurrenceof condition c is replaced by c & c’
• Clause Disjunction Fault (CDF)
• Stuck-At-0 Fault (SA0)
• An occurrenceof a condition is replaced by 0
• Stuck-At-1 Fault (SA1)
AdvancedLogic Coverage 增强逻辑覆盖
DecisionTables 决策表
Decision tables represent logicalrelationships between conditions (roughly, inputs) and actions (roughly,outputs). Test cases are systematically derived … [SWEBOK v3.0]
• 决策表的优点：能够将复杂的问题按照各种可能的情况全部列举出来，简明并避免 遗漏。因此，利用决策表能够设计出完整的测试用例集合。
• 在一些数据处理问题当中，某些操作的实施依赖于多个逻辑条件的组合，即：针对不同逻辑条件的组合值，分别执行不同的操作。决策表很适合于处理这类问题。
决策表的组成
• 决策表通常由以下4部分组成：
– 条件桩—列出问题的所有条件
– 条件项—针对条件桩中条件列出所有可能的取值
– 动作桩—列出问题规定的可能采取的操作
– 动作项—指出条件项各取值情况下应采取的动作
条件桩 条件项
动作桩 动作项
任何一个条件组合的特定取值及相应要执行的动作为一条规则。在决策表中贯穿条件项和动作项的一列是一条规则。
决策表的生成
构造决策表的5个步骤：
（1）确定规则的个数（每个条件取真、假值）
（2）列出所有的条件桩和动作桩
（3）填入条件项
（4）填入动作项，得到初始决策表
（5）简化决策表，合并相似规则
若表中有两条以上规则具有相同的动作，并且在条件项之间存在极为相似的关系，便可以合并
合并后的条件项用符号“-”表示，说明执行的动作与该条件的取值无关，称为无关条件
Diversity
Graph-based Test Design-CFG
Graph-based Test Design-EFG
Adaptive Random Test Design
Equivalence Partition Test Design
Combinatorial Test Design
TestSuite Reduction 测试集简化
TestCase Prioritization 测试用例优先级技术
TestOracle
FaultLocalization
功能测试
功能测试 Functional Testing
根据产品特性和设 需求，验证一个产品的特性和行为是否满足设计需求
功能测试常用步骤
1. 根据需求来细分功能点
2. 根据功能点派生测试需求
3. 根据测试需求设计功能测试用例
4. 逐项执行功能测试用例验证产品
相关的测试类型
•正确性
•产品功能是否与需求和设计文档一致
•可靠性
•用户交互是否引发软件崩溃和其它异常
•易用性
•软件产品完成特定任务的难易程度
探索式测试
性能测试
•性能测试 Performance Testing
•验证产品的性能在特定负载和环境条件下使用是否满足性能指标
•进一步发现系统中存在的性能瓶颈，优化系统
性能测试度量方法
•不同的关注对象采用不同的性能的度量方法
•服务端性能采用CPU、内存等使用率来度量
•客户端性能通常根据系统处理特定用户请求的响应时间来度量
移动应用测试
