Jenkins+RobotFramework failed use case re-execution plan

The interface test case runs on the Jenkins node. In some cases, such as network fluctuations and other reasons, the test case will fail to run. At this time, an email and DingTalk alert will be triggered, and the responsible person will be notified. According to the existing policy, when this When the first build fails, the second build activity will be triggered immediately. If the second build still fails, the warning message will be triggered again. Under this strategy, the relevant responsible persons will receive some additional meaningless warning information (such as the first build timeout, but the second build is successful), so I wrote an extra script and used it as a Robotframework use case in Jenkins. When the execution of a use case fails, after all cases are executed, the cases that failed this time will be selected to try again, and then the two test report files will be merged.

The script content is very simple and highly scalable:

#!/usr/bin/env python

# -*- coding:utf8 -*-



import getopt

import os

import sys

from pathlib import Path

from robot.api import ExecutionResult



def parse_args() -> tuple:

    """解析命令行传入的参数"""
    opts, args = getopt.getopt(sys.argv[1:], '-i:-e:-F:-E:', ["includeTag=", "excludeTag=", "format=", "env="])
    try:
        target = args[0]
    except IndexError:
        target = "./"

    def _parse(option, default_value=None):
        if isinstance(option, tuple):
            temp = [opt_value for (opt_name, opt_value) in opts if opt_name in option]
        else:
            temp = [opt_value for (opt_name, opt_value) in opts if opt_name == option]

        return temp[0] if len(temp) > 0 else default_value

    include_tag = _parse(("-i", "--includeTag"))  # 包含用例标签
    exclude_tag = _parse(("-e", "--excludeTag"))  # 排除用例标签
    env = _parse(("-E", "--env"), 'm')  # 用例运行环境
    fm = _parse(("-F", "--format"), 'robot')  # 用例文件后缀名

    return include_tag, exclude_tag, env, fm, targetdef first_run(target, env, include_tag, exclude_tag, fm):    """首次运行用例
    项目的基本目录结构是固定的, 在命令行中写死了变量文件的相对路径.
    """
    if include_tag:
        cmd = f"robot -F {fm} -i {include_tag} --output output_origin.xml --log NONE --report NONE -V variables.py:{env} -V ../Common/Variables/commonVars.py:{env} {target}"
    elif exclude_tag is not None:
        cmd = f"robot -F {fm} -e {exclude_tag} --output output_origin.xml --log NONE --report NONE -V variables.py:{env} -V ../Common/Variables/commonVars.py:{env} {target}"
    else:
        cmd = f"robot -F {fm} --output output_origin.xml --log NONE --report NONE -V variables.py:{env} -V ../Common/Variables/commonVars.py:{env} {target}"

    print(f'First run cmd >>>> {cmd}')
    os.system(cmd)def parse_robot_result(xml_path) -> bool:
    """解析用例运行结果"""
    suite = ExecutionResult(xml_path).suite

    fail = {}
    for test in suite.tests:
        if test.status == "FAIL":
            fail.update({test.name: test.status})

    all_tests = suite.statistics.critical    print("*" * 50)
    print("当前运行目录为: ", os.getcwd())
    print("总测试条数:{0}, 初次运行时，通过的用例数: {1}, 失败的用例数: {2}".format(all_tests.total, all_tests.passed, all_tests.failed))
    if all_tests.failed > 0:
        print("其中失败的用例信息为: %s" % str(fail))
    print("*" * 50)

    return all_tests.failed > 0def rerun_fail_case(target, env, include_tag, exclude_tag, fm):    """ # TODO
    如果要重新运行整个套件，需要使用`rerunfailedsuites`, 如果只想重新运行失败的测试用例而不是套件中已通过的测试，则使用`rerunfailed`（必须保证case是独立的）
    -R, --rerunfailed <file>
        Selects failed tests from an earlier output file to be re-executed.
    -S, --rerunfailedsuites <file>
        Selects failed test suites from an earlier output file to be re-executed.
    """    if include_tag:
        cmd = f"robot -F {fm} -i {include_tag} -V variables.py:{env} -V ../Common/Variables/commonVars.py:{env} --rerunfailed output_origin.xml --output output_rerun.xml {target}"
    elif exclude_tag is not None:
        cmd = f"robot -F {fm} -e {exclude_tag} -V variables.py:{env} -V ../Common/Variables/commonVars.py:{env} --rerunfailed output_origin.xml --output output_rerun.xml {target}"
    else:
        cmd = f"robot -F {fm} -V variables.py:{env} -V ../Common/Variables/commonVars.py:{env} --rerunfailed output_origin.xml --output output_rerun.xml {target}"

    print(f'重复运行失败的用例: {cmd}')
    os.system(cmd)
    """再次运行失败的用例"""def merge_output():
    """合并xml文件，并生成测试报告
    注意集成到jenkins中时，需要指定 Output xml name为merge.xml
    """
    os.system("rebot --merge --output merge.xml *.xml")def main():
    include_tag, exclude_tag, env, fm, target = parse_args()    # 切换到output目录
    if Path(target).is_dir():
        os.chdir(Path(target))
    else:
        os.chdir(Path(target).parent)

    for xml in Path.cwd().glob("*.xml"):
        os.remove(xml)
    first_run(target, env, include_tag, exclude_tag, fm)
    failed = parse_robot_result("output_origin.xml")
    if failed:
        rerun_fail_case(target, env, include_tag, exclude_tag, fm)

    # 不论是否存在失败的用例, 都会合并测试报告
    merge_output()if __name__ == '__main__':
    main()

Except for the -E parameter, all others are command line parameters provided by robot. Variable files are used in the project to enable the use case to support switching the running environment. The -E parameter needs to be passed into the environment where the use case is running, and the -i or -e parameter Used to pass in labels to filter the test cases to be run this time. Multiple labels can be passed in, such as: H5ANDP1, H5ORMini, NotPaid, etc.

In the Jenkins project configuration, the Execute Windows Batch Cmd of the build operation configuration is as follows:

cd %WORKSPACE%/ParkTest/interface
python runrobot.py --env=%Env% -F robot -i %Tag% ./
exit 0

Env and Tag are both passed in during parameterized builds and have default values.

In the post-build operation, use the Robot Framework plug-in to collect the build results. Since the default output file name is modified in the script above, the corresponding configuration needs to be done here, as follows

When the first build of the project fails, the second build will only run the previously failed test cases and merge the two generated test reports, which are displayed in the test report as follows:

log.html

The email notification template I configured

This will reduce some invalid error emails.

Regarding the above scheme, one thing needs to be explained in particular, that is, the test cases in the project must be independent of each other before. I think it is necessary to maintain Case independence. Each Test Case should only test one scenario . Depending on the complexity of the case, different scenarios can be large or small, but they cannot affect each other. When we run one of the Cases randomly or run the Cases out of order, the test results should be accurate. Suite level and Directory level also need to pay attention to the issue of independence. Maintain the independence of Case. This should be used as a specification for writing automated use cases and strictly requires other members of the team.

Finally: The complete software testing video tutorial below has been compiled and uploaded. Friends who need it can get it by themselves [guaranteed 100% free]

Software Testing Interview Document

We must study to find a high-paying job. The following interview questions are the latest interview materials from first-tier Internet companies such as Alibaba, Tencent, Byte, etc., and some Byte bosses have given authoritative answers. After finishing this set I believe everyone can find a satisfactory job based on the interview information.