SuMo

Mutation testing is a powerful approach for assessing the fault-detection capability of a test suite. With SuMo, we propose a novel mutation testing tool for Ethereum Smart Contracts. SuMo implements a set of 44 mutation operators that were designed starting from the latest Solidity documentation, and from well-known mutation testing tools.
These allow to simulate a wide variety of faults that can be made by smart contract developers.
The set of operators was designed to limit the generation of stillborn mutants, which slow down the mutation testing process, and limit the usability of the tool.

________________________________________________________________________________________________

Usage of SuMo (Standalone version):

Once SuMo is installed, it is possible to launch a new mutation testing campaign.
However, you must make sure that:

  1. You have installed the ganache client (link), the truffle suite (link) and the solc compiler (link) on your pc
  2. all the contracts to be mutated are located in your contracts directory.
  3. the contracts to be mutated pass all the tests. If at least one of the tests fails, the mutation testing process cannot be run.

To exclude specific contracts from the mutation process, you must specify the contract name in the sumo\src\operator.config.json file.

Before starting the mutation testing, you can choose which mutation operators must be applied:

  • sumo list shows the currently enabled mutation operators
  • sumo enable enables all the mutation operators
  • sumo enable ID enables the mutation operator ID
  • sumo disable disables all the mutation operators
  • sumo disable ID disables the mutation operator ID

Once everything is set up, you can use:

  • sumo preflight To view all the available mutations
  • sumo test To launch the mutation testing process

Results

SuMo automatically creates a .sumo folder in the root directory of your project.
At the end of the mutation testing process the folder will contain:

  • report.txt Test report
  • \alive Mutants that survived testing
  • \killed Mutants killed by tests

________________________________________________________________________________________________

Usage of SuMo (Online version):

It is possible to use SuMo also exploiting the user interface or calling directly the rest API.

User Interface: The user interface is available at this link.

To execute a specific test the user should:

  • Select the folder containing the project. Please make sure to select the project root folder.
  • Then make sure to upload a correct config.js file with the following information. Please substitute the part in bold with the name of your project folder, contract directory and the list of contacts that you would like to ignore. The directories should start with the name of the folder uploaded in the step above.
module.exports ={
   projectDir: '../phoenix-orders-master', 
   contractsDir: '../phoenix-orders-master/contracts', 
   baselineDir: '.sumo/baseline', killedDir: '.sumo/killed', 
   aliveDir: '.sumo/alive', contractsGlob: '/**/*.sol', 
   ignore: [ 
   '../phoenix-orders-master/contracts/Migrations.sol', 
   '../phoenix-orders-master/contracts/Interface.sol' 
   ] 
}
  • In the last step you should select the operators to apply and press the upload button. Please consider that the response could require also hours.

________________________________________________________________________________________________

Traditional Operators

OperatorDescription
ACMArgument Change of overloaded Method call
AORAssignment Operator Replacement
BCRDBreak and Continue Replacement and Deletion
BLRBoolean Literal Replacement
BORBinary Operator Insertion
CBDCatch Block Deletion
CSCConditional Statement Change
EREnum Replacemet
ECSExplicit Conversion to Smaller type
HLRHexadecimal Literal Replacement
ICMIncrements Mirror
ILRInteger Literal Replacement
LCSLoop Statement Change
OLFDOverloaded Function Deletion
ORFDOverridden Function Deletion
SKISuper Keyword Insertion
SKDSuper Keyword Deletion
SLRString Literal Replacement
UORDUnary Operator Replacement and Deletion

Solidity Operators

OperatorDescription
AVRAddress Value Replacement
CSCContract Constructor Deletion
DLRData Location Keyword Replacement
DODDelete Operator Deletion
ETREther Transfer function Replacement
EEDEvent Emission Deletion
EHCException Handling Change
FVRFunction Visibility Replacement
GVRGlobal Variable Replacement
MCRMathematical and Cryptographic function Replacement
MODModifier Deletion
MOIModifier Insertion
MOCModifier Order Change
MOCModifier Order Change
MORModifier Replacement
PKDPayable Keyword Deletion
RSDReturn Statement Deletion
RVSReturn Values Swap
SFDSelfdestruct Deletion
SFISelfdestruct Insertion
SFRSafeMath Function Replacement
SCECSwitch Call Expression Casting
TORTransaction Origin Replacement
VURVariable Unit Replacement
VVRVariable Visibility Replacement