
Run every scenario of a project version in new R sessions
Source:R/run_scenarios.R
run_scenarios.Rdrun_scenarios() runs <home>/<version>_run.R once for each simulation of
each scenario, each time in a new R session. It then stacks the
simulations of each scenario and checks the assertions of the project.
Arguments
- version
The version, for example
"v01".- scenario
NULLfor every scenario, or a character vector of the scenarios to run.- home
The project folder.
- workers
The number of simulations that run at the same time, one whole number of 1 or more. Each simulation runs in its own new R session.
- resume
FALSEruns every simulation of the chosen scenarios.TRUEkeeps each simulation with results from the same code, and runs the others. See the section Resume.
Value
Invisibly, a data.frame of every assertion, with the columns
scenario, assertion, expected, observed and pass.
Details
run_scenarios() does these steps in this order:
It reads
CFG$scenariosin a new R session that sources<version>_functions/withorg::initialize_project(). The R session of the caller does not change. It stops when a scenario other than"real"has no functionexpect_<scenario>(). It checks every chosen scenario before it runs any of them.With
resume = TRUE, it chooses the simulations to keep, and prints one line for each of them.It deletes these files, and it stops before any run when one of them stays:
the
check.csvof each chosen scenario.the stacked
results.qs2of each chosen scenario with more than 1 simulation.the
results.qs2and therun.okof each simulation that it runs.
Files of the scenarios that it does not run stay as they are.
It runs the script once for each simulation that it does not keep. Each run is a new R session with the arguments
scenario=<name> sim=<k>, and withhomeas the working directory. At mostworkersruns go at the same time. The stdout and the stderr of each run go torun.login the results folder of the simulation. Each run writesrun.logagain, also a run that succeeds.A run succeeds when all of these hold:
It quits with status 0, so
end_if_fake_scenario()can end the run of a fake scenario.Its
results.qs2exists and passescheck_results().manifest$scenario,manifest$simandmanifest$simsare those of the simulation.
After a run that succeeds,
run_scenarios()writesrun.okbesiderun.log.run.okholds one line, the md5 ofresults.qs2. A run that stops after it writesresults.qs2thus has norun.ok. This is also the case when the caller is interrupted.run.logandrun.okare run artefacts. List both in the.gitignoreof the project.Each other run fails. For a run that fails,
run_scenarios()prints the wholerun.logwithcat(). After the first failure, it starts no new run, and the runs that started go on to their end. It then stops with an error that names the scenario, the simulation and therun.logof each failed run. It stacks and checks no scenario.For each scenario with more than 1 simulation, it stacks the files
<version>_results/<scenario>/sim_<kkkk>/results.qs2into<version>_results/<scenario>/results.qs2. It stops when the files are not exactly the simulations 1 tosims. AftersimsinCFG$scenariosbecomes smaller, the folders of the higher simulations stay. Stacking then stops, and the error names their files. Delete those folders to stack again. The section Stacking gives the rules.For each scenario that has a function
expect_<scenario>()in<version>_functions/, it calls that function on the results of the scenario in a new R session.expect_real()is optional. The function MUST return a data.frame with at least one row and the columnsassertion,expected,observedandpass.run_scenarios()writes it to<version>_results/<scenario>/check.csvand prints it.
Each of these counts as one failed assertion, and run_scenarios() writes
it to check.csv too:
expect_<scenario>()stops with an error. The assertion holds the error message.expect_<scenario>()returns zero rows.expect_<scenario>()returns something other than a data.frame with the four columns.
After every scenario is done, run_scenarios() stops when a value of
pass is not TRUE. The error lists the failed assertions.
Stacking
The stacked results hold every element of the results of simulation 1, in the same order. These rules apply at each depth:
A data.frame gets a first column
sim, and the rows of the simulations in order. It keeps the class of simulation 1, so a data.table stays a data.table.A list is stacked element by element, and pbn matches the elements by name. So each element of a list MUST have its own name. An unnamed or partly named list stops stacking. A scenario with 1 simulation MAY hold such a list.
Each other element becomes a list with one entry for each simulation, named
sim_0001,sim_0002and so on. This includes an atomic vector, a matrix, a factor, a date andNULL.
Stacking stops when two simulations differ in the structure at one path. The error names the path and the two simulations. Examples:
a list with other names or another length.
a data.frame with other columns, a column of another class, or a factor column with other levels.
a list in the place of a data.frame, or
NULLin the place of a value.
Stacking also stops on a data.frame with a column sim, because pbn adds
that column.
The manifest is that of simulation 1. In it, sim and n_rows_data
are NA, sims is the number of simulations, and stacked is TRUE.
For a fake scenario, stacking stops when a simulation has no valid
manifest$fake_md5, or when the simulations do not have one value of it.
It also stops when the simulations do not have one value of
manifest$settings.
A scenario with 1 simulation is not stacked. Its results.qs2 is the file
that the run wrote.
Workers
Each simulation depends only on its scenario and its number, so workers
does not change the results files or check.csv. Only the order of the
lines on the console can change. pbn does not limit the threads that R or
the BLAS library starts in each R session.
Resume
With resume = TRUE, run_scenarios() keeps a simulation when all of
these hold:
Its
results.qs2exists and passescheck_results().Its
run.okexists and holds the md5 ofresults.qs2now. So the run completed, and nothing changedresults.qs2after it.manifest$scenario,manifest$simandmanifest$simsare those of the simulation.manifest$source_md5is the md5 of the files of<version>_run.Rand<version>_functions/now.For a fake scenario,
manifest$fake_md5is the md5 of the code offake_<scenario>()now.
Each other simulation runs again, and the console line of its start gives
the reason. The stacking and the expect_ assertions always run again.
Resume does not find a change in the real data, or in the version of R or
of a package. It also does not find a change in a file outside
<version>_run.R and <version>_functions/. After such a change, use
resume = FALSE.
See also
Other project setup:
end_if_fake_scenario(),
initialize_project(),
use_scenario()
Examples
# \donttest{
home <- file.path(tempdir(), "pbn_run_scenarios_example")
dir.create(file.path(home, "v01_functions"), recursive = TRUE)
dir.create(file.path(home, "data"))
utils::write.csv(data.frame(x = 1:4), file.path(home, "data", "d.csv"))
writeLines(
c(
"CFG <- list(",
" scenarios = data.frame(scenario = c('real', 'null'), sims = c(1, 2))",
")",
"fake_null <- function(dir) {",
" utils::write.csv(data.frame(x = rnorm(4)), file.path(dir, 'd.csv'))",
"}",
"expect_null <- function(results) {",
" n <- length(unique(results$estimates$sim))",
" data.frame(assertion = 'two simulations', expected = 2,",
" observed = n, pass = n == 2)",
"}"
),
file.path(home, "v01_functions", "_cfg.R")
)
writeLines(
c(
'p <- pbn::initialize_project("v01", home = ".", data_raw = "data")',
"d <- utils::read.csv(file.path(p$data_raw, 'd.csv'))",
"pbn::save_results(",
" estimates = data.frame(term = 'mean_x', estimate = mean(d$x)),",
" n_rows = nrow(d),",
" d = d",
")"
),
file.path(home, "v01_run.R")
)
pbn::run_scenarios("v01", home = home)
#> pbn: scenario real, simulation 1: started
#> pbn: scenario real, simulation 1: ran
#> pbn: scenario null, simulation 1: started
#> pbn: scenario null, simulation 1: ran
#> pbn: scenario null, simulation 2: started
#> pbn: scenario null, simulation 2: ran
#> Assertions of the scenario "null":
#> assertion expected observed pass
#> two simulations 2 2 TRUE
stacked <- pbn::read_results("v01", scenario = "null", home = home)
stacked$estimates
#> sim term estimate
#> 1 1 mean_x 0.07921043
#> 2 2 mean_x -0.06364893
stacked$n_rows
#> $sim_0001
#> [1] 4
#>
#> $sim_0002
#> [1] 4
#>
unlink(home, recursive = TRUE)
# }