There are at least three different packages implementing the Model
Confidence Set (MCS) procedure of Hansen, Lunde and Nason (2011). These
three packages, however, produce very different MCS p-values and rather
different superior model sets.
cl <- makeForkCluster(nnodes = 8) # Running Linux, so can fork.
data(Loss) # Loss data from the MCS package
loss_mat <- as.matrix(Loss[, 1:10]) # mcsTest requires a "plain" matrix
model_names <- colnames(loss_mat)
# Parameters for functions...
alpha <- 0.2
B <- 50000 # Make this big, in order to reduce the variability.
block_len <- 33 # Block length calculated as in MCS package.
# Calculate MCS using various packages. Takes a while...
rugarch_pkg <- mcsTest(losses = loss_mat, alpha = alpha, nboot = B,
nblock = block_len, boot = "block")
mcs_pkg <- MCSprocedure(Loss = loss_mat, alpha = alpha, B = B,
statistic = 'TR', k = block_len, cl = cl)
modelconf_pkg <- estMCS(loss = loss_mat, test = "t.range", B = B,
l = block_len)
# Extract and format results for comparison.
rugarch_results <- cbind(Model = model_names[c(rugarch_pkg$excludedR,
pval = rugarch_pkg$pvalsR)
In all cases, p-values are based on the range statistic T_R.
The superior set of models (SSM) produced by the MCS and modelconf packages
are the same, but the MCS p-values are quite different. The rank of models
in the SSM are also different. The SSM produced by the rugarch package
differs drastically, and contains only two models.
Differences in p-values are to be expected, since these procedures are
based on the bootstrap. However, given the large number of bootstrap
replications, surely the differences should be small? Also, notice the
repetition or ties in p-values with the results from mcsTest and estMCS.
Am I doing something wrong or misinterpreting the documentation? Does any
one else use any of these procedures? Which do you prefer and recommend? I
was, initially, planning on using the MCS package. But I don't like that it
only returns the SSM. That's when I started looking at alternatives, and
now I'm totally confused!
Any help, whatsoever, would be greatly appreciated.
Stéfan Janse van Rensburg
Ph.D. Candidate, Mathematical Statistics, NMU