Model Selection with PEtab Select

Sometimes we have competing hypotheses (model structures) that we want to compare to ultimately select the best model/hypothesis. There are various approaches for model selection, such as forward search, backward search, and exhaustive search, where models are compared based on information criteria like AIC or BIC. Additionally, there are efficient algorithms that combine both backward and forward search, such as Famos [11]. All these model selection methods are supported by the Python package PEtab Select, for which PEtab.jl provides an interface.

This advanced documentation page assumes that you know how to import and crate PEtab problems in the standard format (a tutorial can be found here) as well as the basics of multi-start parameter estimation with PEtab.jl (a tutorial can be found here). Additionally, since PEtab Select is a Python package, to run this code you need to have PyCall.jl installed, and you must build PyCall with a Python environment that has petab_select installed:

using PyCall
# Path to Python executable with PEtab Select installed
path_python_exe = "path_python"
ENV["PYTHON"] = path_python_exe
# Build PyCall with the PEtab Select Python environment
import Pkg
Pkg.build("PyCall")

Note

Model selection is currently only possible for problem in the PEtab Select standard format. We plan to add a Julia interface.

Model Selection Example

PEtab.jl provides support for PEtab Select through the petab_select function:

PEtab.petab_select — Function

petab_select(path_yaml, alg; nmultistarts = 100, kwargs...) -> path_res

For a PEtab-select problem, perform model selection with the method specified in the PEtab select problem files. Returns the path (path_res) to a YAML-file with model selection results.

The general model selection (e.g. to use forward-search) options are specified in the PEtab-select files. For details on how to set this up, see the PEtab-select documentation.

For each round of model selection, the candidate models are parameter estimated using multi-start parameter estimation, with nmultistarts performed for each model. The objective values obtained from parameter estimation are then used for the next round of model evaluation.

A list of available and recommended optimization algorithms (alg) can be found in the package documentation and calibrate documentation.

See also calibrate_multistart.

Keyword Arguments

kwargs: The same keywords accepted by PEtabODEProblem and calibrate.

source

As an example, for a simple signaling model (files can be downloaded from here), you can run PEtab Select with the IPNewton() algorithm:

using Optim, PEtab, PyCall
path_yaml = joinpath(@__DIR__, "assets", "petab_select", "petab_select_problem.yaml")
path_res = petab_select(path_yaml, IPNewton(); nmultistarts=10)

┌ Info: PEtab select problem info
│ Method: brute_force
└ Criterion: AIC
[ Info: Model selection round 1 with 1 candidates - as the code compiles in this round it takes extra long time https://xkcd.com/303/
[ Info: Callibrating model M1_1
[ Info: Saving results for best model at /home/sebpe/.julia/dev/PEtab/docs/build/assets/petab_select/PEtab_select_brute_force_AIC.yaml

Where the YAML file storing the model selection results is saved at path_res.

[11]: M. Gabel, T. Hohl, A. Imle, O. T. Fackler and F. Graw. FAMoS: A Flexible and dynamic Algorithm for Model Selection to analyse complex systems dynamics. PLOS Computational Biology 15, e1007230 (2019).