Mutation testing is a powerful approach for assessing the fault-detection capability of a test suite. With SuMo, we propose a novel mutation testing tool for Ethereum Smart Contracts. SuMo implements a set of 44 mutation operators that were designed starting from the latest Solidity documentation, and from well-known mutation testing tools.
These allow to simulate a wide variety of faults that can be made by smart contract developers.
The set of operators was designed to limit the generation of stillborn mutants, which slow down the mutation testing process, and limit the usability of the tool.
________________________________________________________________________________________________
Usage of SuMo (Standalone version):
SuMo source code link
Once SuMo is installed, it is possible to launch a new mutation testing campaign.
However, you must make sure that:
- You have installed the ganache client (link), the truffle suite (link) and the solc compiler (link) on your pc
- all the contracts to be mutated are located in your
contractsdirectory. - the contracts to be mutated pass all the tests. If at least one of the tests fails, the mutation testing process cannot be run.
To exclude specific contracts from the mutation process, you must specify the contract name in the sumo\src\operator.config.json file.
Before starting the mutation testing, you can choose which mutation operators must be applied:
sumo listshows the currently enabled mutation operatorssumo enableenables all the mutation operatorssumo enable IDenables the mutation operator IDsumo disabledisables all the mutation operatorssumo disable IDdisables the mutation operator ID
Once everything is set up, you can use:
sumo preflightTo view all the available mutationssumo testTo launch the mutation testing process
Results
SuMo automatically creates a .sumo folder in the root directory of your project.
At the end of the mutation testing process the folder will contain:
report.txtTest report\aliveMutants that survived testing\killedMutants killed by tests
________________________________________________________________________________________________
Usage of SuMo (Online version):
It is possible to use SuMo also exploiting the user interface or calling directly the rest API.
User Interface: The user interface is available at this link.
To execute a specific test the user should:
- Select the folder containing the project. Please make sure to select the project root folder.
- Then make sure to upload a correct config.js file with the following information. Please substitute the part in bold with the name of your project folder, contract directory and the list of contacts that you would like to ignore. The directories should start with the name of the folder uploaded in the step above.
module.exports ={
projectDir: '../phoenix-orders-master',
contractsDir: '../phoenix-orders-master/contracts',
baselineDir: '.sumo/baseline', killedDir: '.sumo/killed',
aliveDir: '.sumo/alive', contractsGlob: '/**/*.sol',
ignore: [
'../phoenix-orders-master/contracts/Migrations.sol',
'../phoenix-orders-master/contracts/Interface.sol'
]
}- In the last step you should select the operators to apply and press the upload button. Please consider that the response could require also hours.
________________________________________________________________________________________________
Traditional Operators
| Operator | Description |
|---|---|
| ACM | Argument Change of overloaded Method call |
| AOR | Assignment Operator Replacement |
| BCRD | Break and Continue Replacement and Deletion |
| BLR | Boolean Literal Replacement |
| BOR | Binary Operator Insertion |
| CBD | Catch Block Deletion |
| CSC | Conditional Statement Change |
| ER | Enum Replacemet |
| ECS | Explicit Conversion to Smaller type |
| HLR | Hexadecimal Literal Replacement |
| ICM | Increments Mirror |
| ILR | Integer Literal Replacement |
| LCS | Loop Statement Change |
| OLFD | Overloaded Function Deletion |
| ORFD | Overridden Function Deletion |
| SKI | Super Keyword Insertion |
| SKD | Super Keyword Deletion |
| SLR | String Literal Replacement |
| UORD | Unary Operator Replacement and Deletion |
Solidity Operators
| Operator | Description |
|---|---|
| AVR | Address Value Replacement |
| CSC | Contract Constructor Deletion |
| DLR | Data Location Keyword Replacement |
| DOD | Delete Operator Deletion |
| ETR | Ether Transfer function Replacement |
| EED | Event Emission Deletion |
| EHC | Exception Handling Change |
| FVR | Function Visibility Replacement |
| GVR | Global Variable Replacement |
| MCR | Mathematical and Cryptographic function Replacement |
| MOD | Modifier Deletion |
| MOI | Modifier Insertion |
| MOC | Modifier Order Change |
| MOC | Modifier Order Change |
| MOR | Modifier Replacement |
| PKD | Payable Keyword Deletion |
| RSD | Return Statement Deletion |
| RVS | Return Values Swap |
| SFD | Selfdestruct Deletion |
| SFI | Selfdestruct Insertion |
| SFR | SafeMath Function Replacement |
| SCEC | Switch Call Expression Casting |
| TOR | Transaction Origin Replacement |
| VUR | Variable Unit Replacement |
| VVR | Variable Visibility Replacement |
