mvmetrics — Joint Performance Metrics for Multivariate Responses

Why mvmetrics

Standard metrics can't see dependence

When your model predicts multiple variables simultaneously, evaluating each one independently misses the point. A model can achieve identical per-variable RMSE to an independent baseline while capturing — or ignoring — the physical coupling between variables. mvmetrics quantifies that difference directly.

🎯

Detect dependence recovery

Metric D's S_dep component measures the Frobenius distance between observed and predicted PIT correlation matrices — directly quantifying how well a model recovers cross-variable coupling.

⚖️

CV-weighted importance

The novel E-CVWMD metric assigns higher weight to variables with greater relative variability (CV), reflecting that a 5% error on solar radiation and on temperature are not equally consequential.

📊

Bootstrap uncertainty

Non-parametric bootstrap CIs for all metrics distinguish real performance differences from sampling noise. Non-overlapping intervals give informal statistical evidence without strong distributional assumptions.

🤖

Automatic recommendation

The built-in decision tree detects scale heterogeneity, Metric A flatness, and dependence signal to recommend the most informative metric family for your specific data.

Quick Start

One function, complete report

Supply observed data, a list of prediction matrices, and draw arrays. evaluate_models() handles the rest.

Basic Usage

Installation

Individual Metrics

── Setup ─────────────────────────────────────────
library(mvmetrics)

# obs      : [N x K] observed matrix  (4 continuous + 1 binary)
# pred_list: named list of [N x K] prediction matrices
# draws_list: named list of [N x K x B] draw arrays
── Evaluate ──────────────────────────────────────
report <- evaluate_models(
  obs        = obs,
  pred_list  = list(M1 = pred_m1, M3 = pred_m3),
  draws_list = list(M1 = draws_m1, M3 = draws_m3),
  mu_train   = mu_real,   # training means  → CV weights for Metric E
  sd_train   = sd_real,   # training SDs
  K_c        = 4,         # number of continuous variables
  bootstrap  = TRUE,
  B_boot     = 500
)
── Output ────────────────────────────────────────
print(report)           # scores + rankings + recommendation
summary(report)         # compact summary
plot_metrics(report, type = "forest")  # bootstrap CI forest plot
plot_metrics(report, type = "decomp")  # S_marg vs S_dep stacked bars

# Option 1 — from GitHub (recommended)
install.packages("devtools")
devtools::install_github("darango/mvmetrics")

# Option 2 — from local source
devtools::document("path/to/mvmetrics")
devtools::install("path/to/mvmetrics")

# Suggested dependencies for full functionality
install.packages(c("energy", "ggplot2", "MASS"))

# Marginal metrics per variable
marginal_rmse(obs[, 1:4], pred[, 1:4])
marginal_logloss(obs[, 5], pred[, 5])

# Normalise before Metrics B and C
obs_norm <- normalise_matrix(obs[, 1:4], mu_train, sd_train)

# Metric B — Energy Score (strictly proper)
metric_B_energy_score(obs_norm, draws_norm)

# Metric D — MDD decomposition, alpha sensitivity
for (a in c(0.3, 0.5, 0.7)) {
  res <- metric_D_mdd(obs, pred, draws, K_c=4, alpha=a)
  cat(res$S_dep, "\n")
}

# Metric E — E-CVWMD (novel, CV-weighted)
metric_E_cvwmd(obs, pred, draws,
               mu_train = mu_real, sd_train = sd_real,
               K_c = 4, run_dcor = TRUE)

Reference

Exported functions

Function	Type	Description
evaluate_models()	function	Main wrapper. Computes all metrics A–E, rankings, recommendation, and optional bootstrap CIs.
metric_A()	function	Metric A: Weighted sum of marginal scores (RMSE + Log-Loss). Marginal baseline.
metric_B_energy_score()	function	Metric B: Multivariate Energy Score. Strictly proper; sensitive to joint distribution.
metric_C_variogram_score()	function	Metric C: Variogram Score (order p=0.5). Proper; directly penalises covariance errors.
metric_D_mdd()	function	Metric D: Marginal-Dependence Decomposition. Separates S_marg from S_dep via PIT correlations.
metric_E_cvwmd()	function	Novel. Metric E: E-CVWMD with CV-derived weights and adaptive α* via distance correlation test.
bootstrap_metrics()	function	Non-parametric bootstrap for 95% percentile CIs and pairwise CI overlap test.
plot_metrics()	function	Four plot types: bars, forest, heatmap, decomp. Requires ggplot2.
normalise_matrix()	function	Column-wise z-score normalisation. Required before Metrics B and C.
marginal_rmse()	function	RMSE per variable (named vector).
marginal_logloss()	function	Binary cross-entropy Log-Loss.
recommend_metric()	function	Rule-based metric recommendation engine. Called automatically by evaluate_models().

Citation

How to cite

BibTeX

@phdthesis{arangolondono2026,
  author  = {Arango Londo{\~{n}}o, David},
  title   = {Joint Modeling of Precipitation and Temperature in Colombia
             for Daily Series Reconstruction and Subseasonal Forecasting
             Using Generalized Mixed Models},
  school  = {Universidad Nacional de Colombia},
  year    = {2026},
  note    = {R package mvmetrics, v0.1.0}
}

R Citation citation("mvmetrics")

News

Changelog

mvmetrics 0.1.0

April 2026 — Initial release

Five metric families (A–E) with full roxygen2 documentation.
Novel E-CVWMD (Metric E) with CV-derived weights and adaptive α*.
evaluate_models() wrapper with automatic ranking and recommendation.
Non-parametric bootstrap CIs via bootstrap_metrics().
Four ggplot2 plot types: bars, forest, heatmap, decomp.
20+ unit tests via testthat.