[For those that have worked with/thought about ISC]: We have a dataset where the same group of subjects watched different videos, and we’d like to know if intersubject correlation (ISC) in a given region is higher in one video than another. Because of the dependence structure of the data, we want to do this nonparametrically using bootstrapping and/or permutation testing. Some options that have occurred to us are:

Subjectwise bootstrapping: for each bootstrap, recompute the ISC matrix for each video and find the difference in medians to build up a bootstrap distribution, then do a bootstrap hypothesis test (i.e., compare the observed difference in medians to the bootstrapped distribution centered by the observed difference). Pros: straightforward/defensible; cons: may be too conservative?

Elementwise (i.e., subjectpairwise) permutation: given the two ISC matrices (one for each video), randomly flip some elements from one video to the other (such that each subject pair still has one value in each “video”), recompute difference in medians between the matrices to build up null distribution. Pros: takes advantage of the paired nature of the data; cons: may be too liberal?
We’ve seen lots of papers comparing ISC between different populations for the same video, but not many comparing ISC between different videos for the same population (and the statistics for the latter are rather unconvincing, at least for the ones we’ve seen). Thoughts appreciated – thanks in advance!
2 Likes
It seems like (1) might take better advantage of the paired nature of the data if you:
i) “Aligned” the bootstrapping across the two videos. That is, on each bootstrap, take a single random draw of participants (with replacement) and apply this draw to determine the ISC matrix for both videos. This might be what you had already planned, but I wasn’t completely sure from the description.
ii) Rather than take the difference in medians as your test statistic, take the median of differences. That is, elementwise substract the bootstrapped ISC matrices from each other (perhaps after ztransformation), and take the median of these differences. Together with (i), this would mean that paired values are being subtracted from one another. You can then build up a distribution of these medians of differences and compare it to observed test stat as you suggest.
I did some quick simulations (admittedly without all of the complications of ISC), and the median of differences looks like it should have a narrower sampling distribution (hence be more powerful) than the difference of medians, in line with my reasoning above. The degree of improvement will depend on how much subjectlevel dependency there is in your data. The simulation Rcode is below.
# simulation parameters
set.seed(1) # set random seed
nsim < 100 # number of simulations
n < 100 # sample size
nboot < 1000 # number of bootstrap samples
dep < 2 # subject dependency parameter (higher = stronger effect of subject)
# loop through simulations
sdmd < rep(NA,nsim) # SD of sampling distribution of medians of differences
sddm < rep(NA,nsim) # SD of sampling distribution of differences of medians
for (k in 1:nsim){
s < rnorm(n,0,5) # subject vector  creates dependency
x1 < rnorm(n) + s + .5 # condition 1
x2 < rnorm(n) + s # condition 2
medofdiff < rep(NA,nboot)
diffofmed < rep(NA,nboot)
# loop through bootstrap
for (i in 1:nboot){
sel < sample(n,n,T) # single bootstrap index for both conditions
bx1 < x1[sel]
bx2 < x2[sel]
medofdiff[i]<median(bx1bx2)
diffofmed[i]<median(bx1)median(bx2)
}
sdmd[k] < sd(medofdiff)
sddm[k] < sd(diffofmed)
print(k)
}
mean(sdmd) # mean width of one sampling distribution
mean(sddm) # mean width of the other
mean(sddm>sdmd) # proportion of wider sampling distributions
2 Likes
I agree with Mark that using the same resampling scheme across both matrices and subtracting the differences and then taking the median might give you more power akin to a withinsubjects analysis.
One additional method for assessing within subject ISC differences would be to permute your time series before creating the ISC matrix using circle shifting or phase randomization and then compute an ISC for each condition. Then similar to Mark’s suggestion, you would subtract the matrices (probably after ztransformation) and compute the mean/median ISC of differences. This would give a null distribution to compare to the observed ISC difference.
1 Like
Thanks Mark, this is super helpful. Yes, we had been using the same random draw for both videos on each bootstrap, but we hadn’t thought to take the median of differences rather than difference of medians – makes total sense why that should be more powerful, though (especially since we chose these videos specifically to provoke different reactions in different people, so hopefully the subject factor does come into play to a certain extent). Will give it a shot empirically and report back. Thanks again!
1 Like
Thanks Luke! We may give the temporal permutation approach a shot, too.