Remove highly correlated variables

前端 未结 3 1315
遥遥无期
遥遥无期 2021-01-30 05:21

I have a huge dataframe 5600 X 6592 and I want to remove any variables that are correlated to each other more than 0.99 I do know how to do this the long way, step by step i.e.

相关标签:
3条回答
  • 2021-01-30 05:53

    @David A small change in your code make it more robust to negative correlation , by providing

    abs(x) > 0.99 
    

    instead of only

    x > 0.99
    

    data.new <- data[,!apply(tmp,2,function(x) any(abs(x) > 0.99))]

    cheers..!!!

    0 讨论(0)
  • 2021-01-30 06:04

    I'm sure there are many ways to do this and certainly some better than this, but this should work. I basically just set the upper triangle to be zero and then remove any rows that have values over 0.99.

    > tmp <- cor(data)
    > tmp[upper.tri(tmp)] <- 0
    > diag(tmp) <- 0
    # Above two commands can be replaced with 
    # tmp[!lower.tri(tmp)] <- 0
    #
    > 
    > data.new <- data[,!apply(tmp,2,function(x) any(x > 0.99))]
    > head(data.new)
       V2 V3 V5
    1   2 10  4
    2   2 20 10
    3   5 10 31
    4   4 20  2
    5 366 10  2
    6  65 20  5
    
    0 讨论(0)
  • 2021-01-30 06:11

    This is my R code this would be helpfull for you

    library('caret')
    
    df1 = read.csv("stack.csv")
    
    print (df1)
    
         GA     PN     PC   MBP    GR    AP
    1 0.033  6.652  6.681 0.194 0.874 3.177
    2 0.034  9.039  6.224 0.194 1.137 3.400
    3 0.035 10.936 10.304 1.015 0.911 4.900
    4 0.022 10.110  9.603 1.374 0.848 4.566
    5 0.035  2.963 17.156 0.599 0.823 9.406
    6 0.033 10.872 10.244 1.015 0.574 4.871
    7 0.035 21.694 22.389 1.015 0.859 9.259
    8 0.035 10.936 10.304 1.015 0.911 4.500
    
    
    df2 = cor(df1)
    hc = findCorrelation(df2, cutoff=0.3) # putt any value as a "cutoff" 
    hc = sort(hc)
    reduced_Data = df1[,-c(hc)]
    print (reduced_Data)
    
         GA     PN    GR    AP
    1 0.033  6.652 0.874 3.177
    2 0.034  9.039 1.137 3.400
    3 0.035 10.936 0.911 4.900
    4 0.022 10.110 0.848 4.566
    5 0.035  2.963 0.823 9.406
    6 0.033 10.872 0.574 4.871
    7 0.035 21.694 0.859 9.259
    8 0.035 10.936 0.911 4.500
    

    and to write down a reduced data into new csv just use:

    write.csv(reduced_Data, file = "outfile.csv", row.names = FALSE)
    
    0 讨论(0)
提交回复
热议问题