This function calls each of the functions in funs
with each of the
objects specified in what
, recording if any errors or warnings are
thrown in the process.
Usage
fuzz(
funs,
what = test_inputs(),
package = NULL,
listify_what = FALSE,
ignore_patterns = "",
ignore_warnings = FALSE
)
Arguments
- funs
A character vector of function names to test. If a
"package"
attribute is set and is nopackage
argument is provided, functions are loaded from the namespace specified in the attribute.- what
A list of objects to be passed, one at a time, as the first argument to each function in
funs
. Ideally, the list should be named, so that each input can be pretty-printed with its corresponding name; function namify provides an automatic way to create a named list. For unnamed lists, a deparsed representation of the inputs will be used, which may appear unwieldy in some cases. If no inputs are provided, a default set of inputs generated by test_inputs will be used.- package
A character string specifying the name of the package to search for functions. If
NULL
(default), the function will first check the"package"
attribute offuns
, and if that is not set, names will be searched in the global namespace.- listify_what
Whether each input in
what
should also be tested in its listified version (FALSE
by default). When set toTRUE
, ifwhat
islist(x = x)
, the function will operate as if it werelist(x = x, "list(x)" = list(x))
, for any input objectx
.- ignore_patterns
One or more strings containing regular expressions to match the errors to ignore. The string "is missing, with no default" is always ignored.
- ignore_warnings
Whether warnings should be ignored (
FALSE
by default).
Value
An object of class cbtf
that stores the results obtained for each of the
functions tested. This contains the following fields:
- runs
a list of data frames, each containing the results of fuzzing all the functions in
funs
with one of the inputs inwhat
. The data frame contains the following columns and attributes:
-res
: The result of the fuzz test, see below for the possible values.
-msg
: The error or warning message returned by the function, if any.
-attr(*, "what")
: The character representation of the input tested.- funs
a vector of names of the functions tested.
- package
a character string specifying the package name where function names were searched, or
NA
if none was provided.- ignore_patterns
The value of the
ignore_patterns
argument.- ignore_warnings
The value of the
ignore_warnings
argument.
The res
column in each of the data frames in the $runs
field can
contain the following values:
OK: either no error or warning was produced (in which case, the
msg
entry is left blank), or it was whitelisted (in which case, the message received is stored inmsg
).SKIP: no test was run, either because the given name cannot be found, or it doesn't correspond to a function, or the function accepts no arguments, or the function contains a call to readline; the exact reason is given in
msg
.WARN: a warning was thrown for which no whitelisting occurred and
ignore_warnings = FALSE
; its message is stored inmsg
.FAIL: an error was thrown for which no whitelisting occurred; its message is stored in
msg
.
Details
In order to reduce the number of false positive results produced, this function applies the following set rules, to establish if an error or warning condition should ignored (whitelisting):
If the name of the function appears in the error or warning message, as it is considered that the condition has been handled by the developer.
If the error or warning message contains the text "is missing, with no default", which is produced when a missing argument is used without a value being assigned to it.
If the error or warning message contains any of the patterns specified in
ignore_patterns
.If a warning is thrown but
ignore_warnings = TRUE
is set.
In all whitelisted cases, the result is "OK", and the message that
was received is stored in the $msg
field (see the Value section).
Examples
## this should produce no errors
res <- fuzz(funs = c("list", "matrix", "mean"),
what = test_inputs(c("numeric", "raw")))
#> ℹ Fuzzing 3 functions on 8 inputs
#> ℹ Functions will be searched in the global namespace as 'package' was not specified
summary(res)
#> Fuzzed 3 functions on 8 inputs:
#>
#> FAIL WARN SKIP OK
#> list 0 0 0 8
#> matrix 0 0 0 8
#> mean 0 0 0 8
#>
#> [ FAIL 0 | WARN 0 | SKIP 0 | OK 24 ]
## display all results even for successful tests
print(res, show_all = TRUE)
#> ✔ 🏃 You didn't get caught by the fuzz!
#>
#> ── Test input: c(1.309605, 0.585381, -0.461072)
#> list OK
#> matrix OK
#> mean OK
#>
#> ── Test input: c(-1, 0, NaN, 10000)
#> list OK
#> matrix OK
#> mean OK
#>
#> ── Test input: c(Inf, -0.5, 1234)
#> list OK
#> matrix OK
#> mean OK
#>
#> ── Test input: c(0, NA)
#> list OK
#> matrix OK
#> mean OK
#>
#> ── Test input: numeric()
#> list OK
#> matrix OK
#> mean OK
#>
#> ── Test input: charToRaw("0")
#> list OK
#> matrix OK
#> mean OK argument is not numeric or logical: returning NA
#>
#> ── Test input: charToRaw("abc")
#> list OK
#> matrix OK
#> mean OK argument is not numeric or logical: returning NA
#>
#> ── Test input: raw()
#> list OK
#> matrix OK
#> mean OK argument is not numeric or logical: returning NA
#>
#> [ FAIL 0 | WARN 0 | SKIP 0 | OK 24 ]
## this will catch an error (false positive)
fuzz(funs = "matrix", what = test_inputs("scalar"))
#> ℹ Fuzzing 1 function on 7 inputs
#> ℹ Functions will be searched in the global namespace as 'package' was not specified
#> ✖ 🚨 CAUGHT BY THE FUZZ! 🚨
#>
#> ── Test input: NULL
#> matrix FAIL 'data' must be of a vector type, was 'NULL'
#>
#> [ FAIL 1 | WARN 0 | SKIP 0 | OK 6 ]