There are at least three different packages implementing the Model

Confidence Set (MCS) procedure of Hansen, Lunde and Nason (2011). These

three packages, however, produce very different MCS p-values and rather

different superior model sets.

To illustrate, I'm including the following minimal working example. Code is

also available at the following link:

https://gist.github.com/sjvrensburg/57c325f62c52b9df56c82ca41f81de28#####################################################

require(MCS)

require(rugarch)

require(modelconf)

require(parallel)

require(dplyr)

cl <- makeForkCluster(nnodes = 8) # Running Linux, so can fork.

data(Loss) # Loss data from the MCS package

loss_mat <- as.matrix(Loss[, 1:10]) # mcsTest requires a "plain" matrix

model_names <- colnames(loss_mat)

# Parameters for functions...

alpha <- 0.2

B <- 50000 # Make this big, in order to reduce the variability.

block_len <- 33 # Block length calculated as in MCS package.

# Calculate MCS using various packages. Takes a while...

set.seed(123456)

rugarch_pkg <- mcsTest(losses = loss_mat, alpha = alpha, nboot = B,

nblock = block_len, boot = "block")

set.seed(123456)

mcs_pkg <- MCSprocedure(Loss = loss_mat, alpha = alpha, B = B,

statistic = 'TR', k = block_len, cl = cl)

set.seed(123456)

modelconf_pkg <- estMCS(loss = loss_mat, test = "t.range", B = B,

l = block_len)

stopCluster()

# Extract and format results for comparison.

rugarch_results <- cbind(Model = model_names[c(rugarch_pkg$excludedR,

rugarch_pkg$includedR)],

pval = rugarch_pkg$pvalsR)

mcs_results <- mcs_pkg@show %>%

as.data.frame %>%

cbind(Model = row.names(mcs_pkg@show), .) %>%

select(Model, MCS_R) %>%

arrange(MCS_R)

modelconf_results <- modelconf_pkg %>%

as.data.frame %>%

cbind(Model = row.names(modelconf_pkg), .) %>%

select(Model, `MCS p-val`) %>%

arrange(`MCS p-val`)

# Inspect results

# rugarch: Only sGARCH-ghyp & sGARCH-sged in SSM

print(tail(rugarch_results, 5))

# Model

# [6,] "sGARCH-sstd" "0.06714"

# [7,] "sGARCH-snorm" "0.06714"

# [8,] "sGARCH-jsu" "0.06714"

# [9,] "sGARCH-ghyp" "0.33244"

# [10,] "sGARCH-sged" "1"

# MCS: only returns SSM, which includes 7 models.

print(tail(mcs_results, 5))

# 3 sGARCH-ghyp 0.82144

# 4 sGARCH-ged 0.96256

# 5 eGARCH-std 0.96704

# 6 sGARCH-snorm 0.99584

# 7 sGARCH-sged 1.00000

# modelconf: SSM includes 7 models.

print(tail(modelconf_results, 5))

# Model MCS p-val

# 6 sGARCH-ghyp 0.70332

# 7 sGARCH-ged 0.87124

# 8 sGARCH-snorm 0.87124

# 9 eGARCH-std 0.87124

# 10 sGARCH-sged 1.00000

#######################################################

In all cases, p-values are based on the range statistic T_R.

The superior set of models (SSM) produced by the MCS and modelconf packages

are the same, but the MCS p-values are quite different. The rank of models

in the SSM are also different. The SSM produced by the rugarch package

differs drastically, and contains only two models.

Differences in p-values are to be expected, since these procedures are

based on the bootstrap. However, given the large number of bootstrap

replications, surely the differences should be small? Also, notice the

repetition or ties in p-values with the results from mcsTest and estMCS.

Am I doing something wrong or misinterpreting the documentation? Does any

one else use any of these procedures? Which do you prefer and recommend? I

was, initially, planning on using the MCS package. But I don't like that it

only returns the SSM. That's when I started looking at alternatives, and

now I'm totally confused!

Any help, whatsoever, would be greatly appreciated.

Kind regards,

StÃ©fan Janse van Rensburg

-----

Ph.D. Candidate, Mathematical Statistics, NMU

LinkedIn:

http://www.linkedin.com/profile/public-profile-settings?trk=prof-edit-edit-public_profileMy photography website:

http://www.sjvrensburg.com/ [[alternative HTML version deleted]]

_______________________________________________

[hidden email] mailing list

https://stat.ethz.ch/mailman/listinfo/r-sig-finance-- Subscriber-posting only. If you want to post, subscribe first.

-- Also note that this is not the r-help list where general R questions should go.