Hướng dẫn python mutation testing mutmut
Show Mutmut is a mutation testing system for Python, with a strong focus on ease of use. If you don’t know what mutation testing is try starting with this article. Some highlight features:
If you need to run mutmut on a python 2 code base use mutmut Install and run¶You can get started with a simple: pip install mutmut mutmut run This will by default run pytest (or unittest if pytest is unavailable) on tests in the “tests” or “test” folder and it will try to figure out where the code to mutate lies. Run for the available flags, to use other runners, etc. The
recommended way to use mutmut if the defaults aren’t working for you is to add a block in [mutmut] paths_to_mutate=src/ backup=False runner=python -m hammett -x tests_dir=tests/ dict_synonyms=Struct, NamedStruct To use multiple paths either in the [mutmut] paths_to_mutate=src/,src2/ tests_dir=tests/:tests2/ You can stop the mutation run at any time and mutmut will restart where you left off. It’s also smart enough to retest only the surviving mutants when the test suite changes. To print the results run You can also write a mutant to disk with Whitelisting¶You can mark lines like this: some_code_here() # pragma: no mutate to stop mutation on those lines. Some cases we’ve found where you need to whitelist lines are:
See also Advanced whitelisting and configuration Example mutations¶
In general the idea is that the mutations should be as subtle as possible. See Workflow¶This section describes how to work with mutmut to enhance your test suite.
Mutmut keeps a result cache in If you want to re-run all survivors after changing a lot of code or even the configuration, you can use for ID in $(mutmut result-ids survived); do mutmut run $ID; done (for bash). You can also tell mutmut to just check a single mutant: Advanced whitelisting and configuration¶mutmut
has an advanced configuration system. You create a file called def pre_mutation(context): context.config.test_command = 'python -m pytest -x ' + something_else or skip a mutant: def pre_mutation(context): if context.filename == 'foo.py': context.skip = True or skip logging: def pre_mutation(context): line = context.current_source_line.strip() if line.startswith('log.'): context.skip = True look at the code for the It is also possible to disable mutation of specific node types by passing the mutmut run --disable-mutation-types=string,decorator
Inversly, you can also only specify to only run specific mutations with Selecting tests to run¶If you have a large test suite or long running tests, it can be beneficial to narrow the set of tests to run for each mutant down to the tests that have a chance of killing it. Determining the relevant subset of tests depends on your project, its structure, and the metadata that you know about your tests. This section gives examples to show how this could be done for some concrete use cases. All examples use the default test runner ( Selection based on source and test layout¶If the location of the test module has a strict correlation with your source code layout, you can simply construct the path to the corresponding test file from mypackage ├── production_module.py ├── test_production_module.py └── subpackage ├── submodule.py └── test_submodule.py Your import os.path def pre_mutation(context): dirname, filename = os.path.split(context.filename) testfile = "test_" + filename context.config.test_command += ' ' + os.path.join(dirname, testfile) Selection based on imports¶If you can’t rely on the directory structure or naming of the test files, you may assume that the tests most likely to kill the mutant are located in test files that
directly import the module that is affected by the mutant. Using the import ast from pathlib import Path test_imports = {} class ImportVisitor(ast.NodeVisitor): """Visitor which records which modules are imported.""" def __init__(self) -> None: super().__init__() self.imports = [] def visit_Import(self, node: ast.Import) -> None: for alias in node.names: self.imports.append(alias.name) def visit_ImportFrom(self, node: ast.ImportFrom) -> None: self.imports.append(node.module) def init(): """Find all test files located under the 'tests' directory and create an abstract syntax tree for each. Let the ``ImportVisitor`` find out what modules they import and store the information in a global dictionary which can be accessed by ``pre_mutation(context)``.""" test_files = (Path(__file__).parent / "tests").rglob("test*.py") for fpath in test_files: visitor = ImportVisitor() visitor.visit(ast.parse(fpath.read_bytes())) test_imports[str(fpath)] = visitor.imports def pre_mutation(context): """Construct the module name from the filename and run all test files which import that module.""" tests_to_run = [] for testfile, imports in test_imports.items(): module_name = context.filename.rstrip(".py").replace("/", ".") if module_name in imports: tests_to_run.append(testfile) context.config.test_command += f"{' '.join(tests_to_run)}" Selection based on coverage contexts¶If you recorded coverage contexts and use the Let’s say you have used the built-in dynamic
context option of [run] dynamic_context = test_function
import os.path def pre_mutation(context): """Extract the coverage contexts if possible and only run the tests matching this data.""" if not context.config.coverage_data: # mutmut was run without ``--use-coverage`` return fname = os.path.abspath(context.filename) contexts_for_file = context.config.coverage_data.get(fname, {}) contexts_for_line = contexts_for_file.get(context.current_line_index, []) test_names = [ ctx.rsplit(".", 1)[-1] # extract only the final part after the last dot, which is the test function name for ctx in contexts_for_line if ctx # skip empty strings ] if not test_names: return context.config.test_command += f' -k "{" or ".join(test_names)}"' Pay attention that the format of the context name varies depending on the tool you use for creating the contexts. For example, the Making things more robust¶Despite your best efforts in picking the right subset of tests, it may happen that the mutant survives because the test which is able to kill it was not included in the test set. You can tell JUnit XML support¶In order to better integrate with CI/CD systems, mutmut junitxml --suspicious-policy=ignore --untested-policy=ignore
The possible values for these policies are:
If a failed mutant is included in the report, then the unified diff of the mutant will also be included for debugging purposes. |