Fuzz-test the specified functions

The fuzzer calls each function in funs with the argument list provided in args (each of its elements in turn modified by each object in what) and records any errors or warnings that are thrown. If no error occurs within the first timeout seconds, the execution of the function being fuzzed is interrupted and the next one is started.

Usage

fuzz(
  funs,
  what = test_inputs(),
  args = NULL,
  package = NULL,
  listify_what = FALSE,
  ignore_patterns = "",
  ignore_warnings = FALSE,
  daemons = 2L,
  timeout = 2
)

Arguments

funs: A character vector of function names to test. If a "package" attribute is set and is no package argument is provided, functions are loaded from the namespace specified in the attribute.
what: A list of objects; each is used, in turn, to modify the list of arguments in args before calling each of the functions in funs. If no inputs are provided, a default set of inputs generated by test_inputs will be used. If set to NULL, then args must be specified, and all functions will be called with that exact list of arguments with no fuzzing occurring.
args: A list of default values for function arguments. Each argument in the list is in turn replaced by each element of what, then each modified argument list is used to fuzz the functions in funs. If named arguments are present, their names are used as argument names in the fuzzed functions. If NULL (default), only the first argument of each function is fuzzed.
package: A character string specifying the name of the package to search for functions. If NULL (default), the function will first check the "package" attribute of funs, and if that is not set, names will be searched in the global namespace.
listify_what: Whether each input in what should also be tested in its listified version (FALSE by default). When set to TRUE, if what is list(x = x), the function will operate as if it were list(x = x, "list(x)" = list(x)), for any input object x.
ignore_patterns: One or more strings containing regular expressions to match the errors to ignore. The strings "unused argument" and "is missing, with no default" are always ignored.
ignore_warnings: Whether warnings should be ignored (FALSE by default).
daemons: Number of daemons to use (2 by default). As many mirai daemons as specified will be started when entering the function and closed at the end, unless active daemons are already available, in which case the argument is ignored and the active daemons are used.
timeout: Number of seconds (2 by default) after which the function being fuzzed is interrupted with result status set to "OK".

Value

An object of class cbtf that stores the results obtained for each of the functions tested. This contains the following fields:

runs: a list of data frames, each containing the results of fuzzing all the functions in funs with one of the inputs in what. The data frame contains the following columns and attributes:
- res: The result of the fuzz test, see below for the possible values.
- msg: The error or warning message returned by the function, if any.
- attr(*, "what"): The character representation of the input tested.
funs: a vector of names of the functions tested.
args: a named list of arguments, with names generated by deparsing the args argument if not already provided.
package: a character string specifying the package name where function names were searched, or NA if none was provided.
ignore_patterns: The value of the ignore_patterns argument.
ignore_warnings: The value of the ignore_warnings argument.

The res column in each of the data frames in the $runs field can contain the following values:

OK: either no error or warning was produced (in which case, the msg entry is left blank), or it was whitelisted (in which case, the message received is stored in msg), or it was timed out (in which case, msg records that a timeout was applied).
SKIP: no test was run, either because the given name cannot be found, or it doesn't correspond to a function, or the function accepts no arguments, or more arguments were provided than the function accepts; the exact reason is given in msg.
WARN: a warning was thrown for which no whitelisting occurred and ignore_warnings = FALSE; its message is stored in msg.
FAIL: an error was thrown for which no whitelisting occurred; its message is stored in msg.

Details

Multiple arguments

An list of arguments to be passed to the functions being fuzzed can be provided via the args argument. Each element in that list is modified in turn by each object in what and the resulting list of arguments is then passed to each function via do.call(). If more arguments are given than the number formal arguments accepted by a function, that function will produce a "SKIP" result.

If arguments are named, they will be passed with their name to the fuzzed functions. If a function doesn't have a formal argument of that name and doesn't accept ..., then the standard R behaviour is to return an "unused argument" error. This is whitelisted by default in fuzz(), and the corresponding result status is set to "OK".

It is possible to define arguments that should remain unchanged while fuzzing by prefixing their name with ... These arguments will use the values assigned in args without modification. For example, to ensure that the argument na.rm is always set to TRUE, it should be specified as ..na.rm = TRUE in args. If all elements in args are fixed, what is ignored and all functions in funs will be called with the provided args list.

Parallel execution

The implementation uses mirai as a backend to execute tasks asynchronously in parallel worker processes. The function can start a pool of persistent background processes (daemons) of size given by the daemons argument (note that starting more daemons than available cores yields no benefit). Alternatively, the function can also make use of already active daemons started with the mirai::daemons function: this allows to control in greater detail the number of processes to use, which can also be remote.

Whitelisting

In order to reduce the number of false positive results produced, this function applies the following set rules, to establish if an error or warning condition should ignored (whitelisting):

If the name of the function appears in the error or warning message, as it is considered that the condition has been handled by the developer.
If the error or warning message contains the text "is missing, with no default", which is produced when a missing argument is used without a value being assigned to it.
If the error or warning message contains any of the patterns specified in ignore_patterns.
If a warning is thrown but ignore_warnings = TRUE is set.

In all whitelisted cases, the result is "OK", and the message that was received is stored in the $msg field (see the Value section).

Note: Whitelisting can also be applied post-hoc on the results of a fuzz run using the whitelist function.

Examples

## set up persistent background processes
mirai::daemons(2L)

## this should produce no errors
res <- fuzz(funs = c("list", "matrix", "mean"),
            what = test_inputs(c("numeric", "raw")))
#> ℹ Fuzzing 3 functions with 9 inputs (using 2 daemons)
#> ℹ Functions will be searched in the global namespace as `package` was not specified
#> ℹ 27 tests run  [23ms]
summary(res)
#> Fuzzed 3 functions on 9 inputs:
#>         
#>          FAIL WARN SKIP OK
#>   list      0    0    0  9
#>   matrix    0    0    0  9
#>   mean      0    3    0  6
#> 
#> [ FAIL 0 | WARN 3 | SKIP 0 | OK 24 ]

## display all results even for successful tests
print(res, show_all = TRUE)
#> ✖  🚨   CAUGHT BY THE FUZZ!   🚨
#> 
#> ── Test input [[7]]: charToRaw("0") 
#>    mean  WARN  argument is not numeric or logical: returning NA
#> 
#> ── Test input [[8]]: charToRaw("abc") 
#>    mean  WARN  argument is not numeric or logical: returning NA
#> 
#> ── Test input [[9]]: raw() 
#>    mean  WARN  argument is not numeric or logical: returning NA
#> 
#>  [ FAIL 0 | WARN 3 | SKIP 0 | OK 24 ] 

## this will catch an error (false positive)
fuzz(funs = "matrix",  what = test_inputs("scalar"))
#> ℹ Fuzzing 1 function with 10 inputs (using 2 daemons)
#> ℹ Functions will be searched in the global namespace as `package` was not specified
#> ℹ 10 tests run  [7ms]
#> ✖  🚨   CAUGHT BY THE FUZZ!   🚨
#> 
#> ── Test input [[10]]: NULL 
#>  matrix  FAIL  'data' must be of a vector type, was 'NULL'
#> 
#>  [ FAIL 1 | WARN 0 | SKIP 0 | OK 9 ] 

## apply a whitelist pattern to remove the false positive
fuzz(funs = "matrix",  what = test_inputs("scalar"),
     ignore_patterns = "'data' must be of a vector type")
#> ℹ Fuzzing 1 function with 10 inputs (using 2 daemons)
#> ℹ Functions will be searched in the global namespace as `package` was not specified
#> ℹ 10 tests run  [7ms]
#> ✔  🏃 You didn't get caught by the fuzz!
#> 
#>  [ FAIL 0 | WARN 0 | SKIP 0 | OK 10 ] 

## close the background processes
mirai::daemons(0L)