Software: FDR

False Discovery Rate

False Discovery Rate (FDR) is a new approach to the multiple comparisons problem. Instead of controlling the chance of any false positives (as Bonferroni or random field methods do), FDR controls the expected proportion of false positives among suprathreshold voxels. A FDR threshold is determined from the observed p-value distribution, and hence is adaptive to the amount of signal in your data.

FDR is more sensitive than traditional methods simply because it is using a more lenient metric for false positives. However, if there is truly no signal anywhere in the brain, a FDR-controlling method has the same control as standard methods. That is, if the null hypothesis is true everywhere, a FDR procedure will control the chance of a false positive anywhere in the brain at the specified level. (In technical terms, FDR methods have weak control of Familywise Type I error; for more on the standard Familywise approach, see Nichols & Hayasaka, Controlling the Familywise Error Rate in Functional Neuroimaging: A Comparative Review, Stat. Meth. in Med. Research, 12(5): 419-446, 2003.)

Below is software, in the form of an extension to SPM, and an article introducing FDR to the neuroimaging community. Please feel free to contact me with any questions.

 

New: Slides showing impact of signal extent.


Software: FDR inference in Matlab


The following function will supply p-value thresholds which control the expected FDR at a specified rate.

  • FDR.m 
    This function takes a vector of p-values and a FDR rate. It returns two p-value thresholds, one based on an assumption of independence or positive dependence, and one that makes no assumptions about how the tests are correlated. For imaging data, an assumption of positive dependence is reasonable, so it should be OK to use the first (more sensitive) threshold.

    Here is one example of how to use this function with a T image.

     
          Timg     = % read t image, with df degrees of freedom
          Timg(~isfinite(Timg(:))) = [];
     
          P        = 1-cdf('T',Timg(:),df);
     
          [pID pN] = FDR(P,0.05);
     
          tID      = icdf('T',1-pID,df)     % T threshold, indep or pos. correl.
          tN       = icdf('T',1-pN,df)      % T threshold, no correl. assumptions
          


  • Software: FDR for multiple images


    Use this Matlab function, multFDR.m, to find a common FDR threshold for a set of images. It will return a single common t threshold as well as a set of individual t thresholds for comparison.

    It requires SPM99 or SPM2, but can be used with any Analyze-format T image.



    Software: Illustrating the FDR threshold


    Use this Matlab function, FDRill.m, to evaluate an image's t distribution and observe the PP-plot graph that determines the FDR threshold.

    It requires SPM99 or SPM2, but can be used with any Analyze-format T image.



    Software: Extending SPM99 for FDR inference


    (Note: SPM2 already includes FDR, so no patches are needed.)

    SPM provides corrected p-values based on Familywise (Type I) Error control; a Familywise Error (FWE) is one or more false positive anywhere in the brain. The following new and modified SPM m-files will add a column of FDR-corrected p-values to SPM output and allow for the specification of FDR-corrected thresholds. Otherwise it does not alter SPM's behavior*.

    The compressed tar file below contains the 11 m-files needed to implement FDR within SPM. The files will expand into a separate spm_fdr directory. Include this directory before your spm99 directory, e.g.

     
    	path('/my/path/spm_fdr',path)
                  

    Please note changes to spm_VOI.m as of September 30, 2002. See this email. Thanks to Ahmed Toosy for reporting this problem.

    Download: spm_fdr.tar.gz (21KB)

    Files:

  • spm_uc_FDR.m 
    This new function is the core of the FDR calculations. Based on a statistic image, it finds the FDR threshold for a given q; q is the upper bound on the expected FDR, and can be thought of as 'alpha'.
  • spm_P_FDR.m
    This new function provides the inversion of spm_uc_FDR, providing a p-value for a given statistic value. Note, since FDR is based on the empirical distribution of the p-values, the natural "corrected p-values" are not necessarily monotonic; monotonicity is enforced, but sometimes this results in odd p-values (e.g. there might be many voxels with different statistic values that share the same p-value).

     

  • spm_list.m
    This SPM function required the most substantial changes, as it generates the tabular output of statistic values and corrected and uncorrected p-values. There is now an additional column for FDR-corrected p-values, and the footer now reflects the FDR of the threshold and an upper bound on expected number of false positives (among the suprathreshold voxels, conditional on the number of suprathreshold voxels).
  • spm_VOI.m
    This function required minor changes to pass on information about the mask used.
  • spm_getSPM.m
    This function required minor changes to pass on information about the (mapped) statistic image and the mask used. Also, it now asks for what type of corrected p-value (FWE or FDR) instead of just yes-no.

     

  • spm_P.m, spm_P_Bonf.m, spm_P_RF.m, spm_uc.m, spm_uc_Bonf.m, spm_uc_RF.m
    This functions are a part of recent changes to the SPM code, and, as such, can be installed in your master SPM99 directory. These functions are not specifically FDR-related, but the FDR code assumes the presence of these updated files.
  • See this note on a bug fix to Matlab core function betainc/ betacore. The symptom is that SPM hangs when it's determing FDR p-values.

    *This tar file includes new modifications to SPM that were released June 28, 2001, in the form of the last six functions listed above. These modifications modify the (FWE) corrected p-value procedures to always take the minimum of random field and Bonferroni results. As released, SPM99 only gives random field p-values and thresholds even if Bonferroni results are more powerful. This is a minor change will, at most, reduce corrected p-values and thresholds in certain situations (typically, low degrees of freedom analyses).

     

    Changes to spm_P.m as of October 14, 2001. See this email.

    Changes as of August 7, 2001. See this email.



    Article


    Thresholding of Statistical Maps in Functional Neuroimaging Using the False Discovery Rate.
    Christopher R. Genovese, Nicole A. Lazar, Thomas E. Nichols (2002).
    NeuroImage 15:870-878.

    Download: Published Version | Preprint

    Note on the preprint: The preprint reflected the current results at the time of submission. Since then, the independence FDR result of 1995 (BH 1995; in the above paper, "c(V)=1" on pg 9) has been shown to be more general (BY 2001). In particular, a technical condition known as positive regression dependency is sufficient to use the 1995 result. Examples of this condition include multivariate normal data where the covariance matrix has all positive elements; this seems a reasonable assumption for imaging, and hence the code above uses the c(V)=1 result.

    BH 1995: Y. Benjamini and Y Hochberg. Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. J. Roy Stat Soc, Ser B. 57:289-300.

    BY 2001: Y. Benjamini and D. Yekutieli. The Control of the false discovery rate in multiple testing under dependency. The Annals of Statistics, 29(4):1165-1188, 2001. Preprint verison available at http://www.math.tau.ac.il/~ybenja/ under "Papers"



    Presentation Slides


    These PowerPoint slides are from a talk presented Feburary 6, 2004, at the Department of Statistical Science at Southern Methodist University, Dallas, TX. They illustrate how the FDR threshold will be lower with greater extent of activation.



    FDR Citations in Neuroimaging


     

  • Ellis SP, Underwood MD, Arango V, & Mann JJ.
    Mixed models and multiple comparisons in analysis of human neurochemical maps.
    Psychiatry Research: Neuroimaging 99(2):111-119, AUG 28 2000.
  • Rombouts S A R B, Barkhof F, van Meel C S, Scheltens P.
    Alterations in brain activation during cholinergic enhancement with rivastigmine in Alzheimer's disease.
    Journal of Neurology, Neurosurgery & Psychiatry, 2002 73: 665-671.
  • Konishi S, Uchida I, Okuaki T, Machida T, Shirouzu I, Miyashita Y
    Neural correlates of recency judgment
    Journal of Neuroscience, 22 (21): 9549-9555, 2002
  • Tregellas JR, Tanabe JL, Miller DE, Freedman R
    Monitoring eye movements during fMRI tasks with echo planar images.
    HUMAN BRAIN MAPPING 17 (4): 237-243 DEC 2002.
  • Wilke M, Schmithorst VJ, Holland SK.
    Assessment of spatial normalization of whole-brain magnetic resonance images in children.
    Human Brain Mapping 17 (1): 48-60 SEP 2002
  • Bor D, Duncan J, Wiseman RJ, et al.
    Encoding strategies dissociate prefrontal activity from working memory demand.
    NEURON 37 (2): 361-367 JAN 23 2003
  • Stanley DA, Rubin N
    fMRI activation in response to illusory contours and salient regions in the human Lateral Occipital Complex.
    NEURON 37 (2): 323-331 JAN 23 2003
  • Haline E. Schendan, Meghan M. Searl, Rebecca J. Melrose, and Chantal E. Stern
    An fMRI Study of the Role of the Medial Temporal Lobe in Implicit and Explicit Sequence Learning
    Neuron, Vol. 37, 1013 1025, March 27, 2003
  • CY Sylvester and TD Wager and SC Lacey and L Hernandez and TE Nichols and EE Smith and J Jonides
    Switching attention and resolving interference: fMRI measures of executive functions.
    Neuropsychologia, 2003 41(3):357-370.
  • Davis, M. H. & Johnsrude, I. S.
    "Hierarchical processing in spoken language comprehension"
    Journal of Neuroscience, 23(8), 3423-3431, 2003.
  • Alle Meije Wink & Jos B. T. M. Roerdink
    Denoising functional MR images: a comparison of wavelet denoising and Gaussian smoothing
    IEEE Transactions on Medical Imaging 23(3): 374-387, March 2004.
  • Eun Yeon Joo, Woo Suk Tae, Jee Hyun Kim, Byung Tae Kim & Seung Bong Hong
    Glucose hypometabolism of hypothalamus and thalamus in narcolepsy
    Ann Neurology 56(3):437-440, September 2004.
  • Davis, M. H., Meunier, F., and Marslen-Wilson, W.D.
    "Neural responses to morphological, syntactic and semantic properties of single words: an fMRI study."
    Brain and Language, 89, 439-449, 2004.
  • Woo Suk Tae, Eun Yeon Joo, Jee Hyun Kim, Sun Jung Han, et al.
    Cerebral perfusion changes in mesial temporal lobe epilepsy: SPM analysis of ictal and interictal SPECT
    NeuroImage 24: 101-110, January 2005.
  • Daselaar SM, Rombouts SARB, Veltman DJ, Raaijmakers JGW, Jonker C.
    Similar network activated by young and old adults during the acquisition of a motor sequence.
    Neurobiology of Aging, 24, 1013-1019, 2003.
  • Rhodri Cusack, Matthew Brett & Katja Osswald.
    An Evaluation of the Use of Magnetic Field Maps to Undistort Echo-Planar Images
    NeuroImage 18(1):127-142, 2003
  • Jacobs, J., Hwang-Grodzins, G., Curran, T., and Kahana, M.J. (2006). EEG oscillations and recognition memory: Theta correlates of memory retrieval and decision making.
    NeuroImage, 32, 978-987.
  • Langers, D.R.M., Jansen, J.F.A. & Backes, W.H.
    Enhanced signal detection in neuroimaging by means of regional control of the global false discovery rate.
    NeuroImage 38, 43-56 (2007).
  • If you know of other papers which have used FDR thresholds, please send me the reference !