cor.mtest produces wrong p-values for Spearman with pairwise.complete.obs and missing data

[jukop]

Hello,

I encountered the issue in the headline. I was able to produce correct p-values using psych::corr.test. My first quess is that denominator is n which is taken from dimension of the matrix but not as number of pairs which were acceptable due to missing values.

[cejourci]

My problem as well. I can't find any spesification of test statistics underlying cor.mtest (pearson, spearman etc.) and missing-method (pairwise complete obs vs complete obs) so I can't trust corrplot to produce correct p-values for the correlation matrix coefficients. Will try the psych corr.test, but hope that mtest can function with more detail!

[jeffzemla]

I am able to replicate this issue.

Both packages use cor.test under the hood to calculate p-values. cor.test defaults to exact=TRUE which uses a permutation test to calculate p-values. corr.test from psych overrides this and specifies exact=FALSE

You can replicate this behavior by passing the argument to cor.mtest.

Minimum reproducible example:

library(psych)
library(corrplot)

dat <- matrix(rnorm(10), ncol=2)

# same p-values using parametric approximation
corr.test(dat[,1],dat[,2], method="spearman")
cor.mtest(dat, method="spearman", exact=FALSE)
cor.test(dat[,1], dat[,2], method="spearman", exact=FALSE)

# same p-values using exact test
cor.mtest(dat, method="spearman", exact=TRUE)
cor.test(dat[,1], dat[,2], method="spearman", exact=TRUE)
cor.mtest(dat, method="spearman")
cor.test(dat[,1], dat[,2], method="spearman")

You will probably only notice a difference with very small samples. The methods should converge with large sample sizes.

psych documentation says this:

exact is set to be FALSE which means that exact p values for small samples are not found given the problem of ties.

However ties are not really an issue because cor.test will automatically switch to exact=FALSE if it encounters a tie.

Regardless, I think the behavior of cor.mtest makes more sense than that of corr.test, since cor.mtest matches the built-in cor.test while corr.test does not. It's also more flexible, since (as far as I can tell) corr.test does not allow you to override the exact flag.

Perhaps the documentation could be improved for transparency, but I don't think there is anything to fix in the code.