How to correlate scalar values of two pandas dataframes

断了今生、忘了曾经 提交于 2020-12-12 06:16:19

问题


How do I correlate two pandas dataframes, find a single r value for all values? I don't want to correlate columns or rows, but all scalar values. One dataframe is the x axis, and the other dataframe is the y axis.

I downloaded identically structured csv files here: https://www.gapminder.org/data/ The tables have years for columns, countries for rows, with numerical values for the indicator that each table reports.

For instance, I want to see how the Political Participation Indicator (gapminder calls it an index, but I don't want to confuse it with a dataframe index) correlates overall with the Government Functioning Indicator, by year and country.

pol_partix_idx_EIU_df = pd.read_csv('polpartix_eiu.csv',index_col=0)
govt_idx_EIU_df = pd.read_csv('gvtx_eiu.csv',index_col=0)

pol_partix_idx_EIU_df.head()

    2006    2007    2008    2009    2010    2011    2012    2013    2014    2015    2016    2017    2018
country                                                 
Afghanistan 0.222   0.222   0.222   0.250   0.278   0.278   0.278   0.278   0.389   0.389   0.278   0.278   0.444
Albania 0.444   0.444   0.444   0.444   0.444   0.500   0.500   0.500   0.500   0.556   0.556   0.556   0.556
Algeria 0.222   0.194   0.167   0.223   0.278   0.278   0.389   0.389   0.389   0.389   0.389   0.389   0.389
Angola  0.111   0.250   0.389   0.416   0.444   0.444   0.500   0.500   0.500   0.500   0.556   0.556   0.556
Argentina   0.556   0.556   0.556   0.556   0.556   0.556   0.556   0.556   0.556   0.611   0.611   0.611   0.611

You can correlate by column or row:

pol_partix_idx_EIU_df.corrwith(govt_idx_EIU_df, axis=0)

2006    0.738297

2007    0.745321

2008    0.731913

...

2018    0.718520

dtype: float64


pol_partix_idx_EIU_df.corrwith(govt_idx_EIU_df, axis=1)

country

Afghanistan    6.790123e-01

Albania       -5.664265e-01

...

Zimbabwe       4.456537e-01

Length: 164, dtype: float64

But, I want a single r value that compares every field in one table with every corresponding field in the other table. Essentially, I want the r value of this scatterplot:

plt.scatter(pol_cultx_idx_EIU_df,govt_idx_EIU_df)
plt.xlabel('Political participation')
plt.ylabel('Government functioning')

(The example code won't color the plot like this, but plots the same points.)

The second part of the question would be how to do this with tables that aren't exactly identical in structure. Every table (dataframe) I want to compare has country records and year columns, but not all of them have the same countries or years. In the example above, they do. How do I get a single r value for only the shared rows and columns of the dataframes?


回答1:


I've simulated a setup that I think mimics yours--three dataframes with countries across rows and years across columns. I then concatenate the different sets of data into a single dataframe. And show how to compute the correlation between them. Let me know if something about this example doesn't match your setup.

import pandas as pd

set1 = pd.DataFrame({1980:[4, 11, 0], 1981:[5, 10, 2], 1982:[0, 3, 1]},
    index=pd.Index(['USA', 'UK', 'Iran'], name='country'))
set1.columns.name = 'year'
set1
year     1980  1981  1982
country                  
USA         4     5     0
UK         11    10     3
Iran        0     2     1
set2 = pd.DataFrame({1981:[2, 1, 10], 1982:[15, 1, 12], 1983:[10, 13, 1]},
    index=pd.Index(['USA', 'UK', 'Turkey'], name='country'))
set2.columns.name = 'year'
set2
year     1981  1982  1983
country                  
USA         2    15    10
UK          1     1    13
Turkey     10    12     1

Notice that, like your setup, some countries/years are not present in different datasets.

set3 = pd.DataFrame({1980:[12, 11, 4], 1982:[9, 8, 11]},
    index=pd.Index(['USA', 'UK', 'Turkey'], name='country'))
set3.columns.name = 'year'

We can turns these into multi-indexed series by stacking along year and then concatenate these across columns using pd.concat.

df = pd.concat([set1.stack('year'), set2.stack('year'), set3.stack('year')],
    keys=['set1', 'set2', 'set3'], names=['set'], axis=1)
df
set           set1  set2  set3
country year                  
Iran    1980   0.0   NaN   NaN
        1981   2.0   NaN   NaN
        1982   1.0   NaN   NaN
Turkey  1980   NaN   NaN   4.0
        1981   NaN  10.0   NaN
        1982   NaN  12.0  11.0
        1983   NaN   1.0   NaN
UK      1980  11.0   NaN  11.0
        1981  10.0   1.0   NaN
        1982   3.0   1.0   8.0
        1983   NaN  13.0   NaN
USA     1980   4.0   NaN  12.0
        1981   5.0   2.0   NaN
        1982   0.0  15.0   9.0
        1983   NaN  10.0   NaN

And we can compute a 3x3 correlation matrix across the three different sets.

df.corr()
set       set1      set2      set3
set                               
set1  1.000000 -0.723632  0.509902
set2 -0.723632  1.000000  0.606891
set3  0.509902  0.606891  1.000000



回答2:


Here's what I did, but it's still not as slick as if I had found a built-in pandas feature or package.

Because I ultimately wanted to do this with more than two tables, I put the tables (dataframes) into a dictionary.

Then, I changed each table into a one-column table that has a MultiIndex representing the original column names and index values. The field values are the original column values strung end to end.

Then, I merged these new tables into one full outer join on the MultiIndex. Now I can correlate any two of the original tables by correlating their respective columns in the final table.

import pandas as pd

gvtx_eiu_df = pd.read_csv('gvtx_eiu.csv',index_col=0,
                          skip_blank_lines=False)
gvtx_eiu_df.columns.name = 'year'
polpartix_eiu_df = pd.read_csv('polpartix_eiu.csv',index_col=0,
                               skip_blank_lines=False)
polpartix_eiu_df.columns.name = 'year'
clean_elec_idea_df = pd.read_csv('clean_elec_idea.csv', index_col=0,
                                 skip_blank_lines=False)
clean_elec_idea_df.columns.name = 'year'

test_table_dict = {'gvtx_eiu': gvtx_eiu_df,
                   'polpartix_eiu': polpartix_eiu_df,
                   'clean_elec_idea': clean_elec_idea_df}
'''
# Updated to not use this anymore. Using stack now, thanks to @jtorca. So it
# fits more neatly into one function.

# Serialize df columns into MultiIndex df, index=(year, country), one column
def df_to_multidx_df(df: pd.DataFrame, cols_idx1_name: str = 'Previous Columns',
                     idx_idx2_name: str = 'Previous Index',
                     val_col_name: str = 'Values') -> pd.DataFrame:
    #Takes 2d dataframe (df) with a single-level index and one or more
    #single-level columns. All df values must be the same type.
    #Parameters:
    #    df: 2d dataframe with single-level index and one or more
    #        single-level columns. All df values must be the same type.
    #    cols_idx1_name: 1st index title for returned dataframe; index is df
    #        column names.
    #    idx_idx2_name: 2nd index title for returned dataframe; index is df
    #        index.
    #Returns:
    #    a 2d dataframe with a MultiIndex constructed of table_df column
    #    names and index values. Has a single column with field values that are
    #    all df columns strung end to end.

    # Create MultiIndex from product of index values and column names.
    mult_idx = pd.MultiIndex.from_product([df.columns, df.index],
                                          names=[cols_idx1_name, idx_idx2_name])
    # 1D list of table values in same order as MultiIndex.
    val_list = [val for col in df for val in df[col]]
    
    return pd.DataFrame(val_list, index=mult_idx, columns=[val_col_name])
'''

def df_dict_to_multidx_df(df_dict: dict) -> pd.DataFrame:
#     , cols_idx1_name: str = 'idx1',
#     idx_idx2_name: str = 'idx2') -> pd.DataFrame:
    '''Converts a dictionary (df_dict) of 2d dataframes, each with single-level
    indices and columns, into a 2d dataframe (multidx_df) with each column
    containing the the values of one of df_dict's dataframes. The index of
    multidx_df is a MultiIndex of the input dataframes' column names and index
    values. Dataframes are joined in full outer join on the MultiIndex.
        NOTE: each input dataframe's index and columns row must be named
        beforehand in order to name the columns in the multiindex and join on it.
    Parameters:
        df_dict: dictionary of 2d dataframes, each with single-level
            indices and columns.
    Returns:
        multidx_df = MultiIndex dataframe.'''
    
    df_dict_copy = df_dict.copy()
        
    # Full outer join each table to multidx_df on MultiIndex.
        # Start with first indicator to have a left df to merge.
    first_key = next(iter(df_dict_copy))
    multidx_df = pd.DataFrame(df_dict_copy.pop(first_key).stack(),
                                     columns=[first_key])
    for key, df in df_dict_copy.items():
        df = pd.DataFrame(df.stack(), columns=[key])
        multidx_df = multidx_df.merge(right=df, how='outer',
                                     on=multidx_df.index.names[:2])

        # concat twice as fast as merge
#         multidx_df = pd.concat([multidx_df, df], names=['indicator'], axis=1)
    
    return multidx_df

###Test Code

print(gvtx_eiu_df)

#               2006    2007   2008   2009   2010   2011   2012   2013   2014  \
# country                                                                       
# Afghanistan    NaN  0.0395  0.079  0.079  0.079  0.079  0.079  0.079  0.114   
# Albania      0.507  0.5070  0.507  0.507  0.507  0.471  0.400  0.400  0.400   
# Algeria      0.221  0.2210  0.221  0.221  0.221  0.221  0.221  0.221  0.221   
# Angola       0.214  0.2680  0.321  0.321  0.321  0.321  0.321  0.321  0.321   
# Argentina    0.500  0.5000  0.500  0.535  0.571  0.571  0.571  0.571  0.571   
# ...            ...     ...    ...    ...    ...    ...    ...    ...    ...   
# Venezuela    0.364  0.3960  0.429  0.411  0.393  0.393  0.429  0.429  0.429   
# Vietnam      0.429  0.4290  0.429  0.429  0.429  0.429  0.393  0.393  0.393   
# Yemen        0.271  0.2610  0.250  0.214  0.179  0.036  0.143  0.143  0.143   
# Zambia       0.464  0.4640  0.464  0.500  0.536  0.500  0.536  0.536  0.536   
# Zimbabwe     0.079  0.0790  0.079  0.104  0.129  0.129  0.129  0.129  0.129   

#               2015   2016   2017   2018  
# country                                  
# Afghanistan  0.114  0.114  0.114  0.114  
# Albania      0.436  0.436  0.471  0.471  
# Algeria      0.221  0.221  0.221  0.221  
# Angola       0.321  0.321  0.286  0.286  
# Argentina    0.500  0.500  0.500  0.536  
# ...            ...    ...    ...    ...  
# Venezuela    0.393  0.250  0.286  0.179  
# Vietnam      0.393  0.321  0.321  0.321  
# Yemen        0.036    NaN    NaN    NaN  
# Zambia       0.536  0.536  0.500  0.464  
# Zimbabwe     0.200  0.200  0.200  0.200  

# [164 rows x 13 columns]


test_serialized = df_to_multidx_df(df=gvtx_eiu_df, cols_idx1_name='Year',
                                   idx_idx2_name='Country',
                                   val_col_name='gvtx_eiu')
print(test_serialized)

#                       gvtx_eiu
# Year Country                  
# 2006 Afghanistan           NaN
#      Albania             0.507
#      Algeria             0.221
#      Angola              0.214
#      Argentina           0.500
# ...                        ...
# 2018 Venezuela           0.179
#      Vietnam             0.321
#      Yemen                 NaN
#      Zambia              0.464
#      Zimbabwe            0.200

# [2132 rows x 1 columns]


test_multidx_df = table_dict_to_multidx_df(test_table_dict, 'Year', 'Country')

print(test_multidx_df)

#                       gvtx_eiu       polpartix_eiu  clean_elec_idea
# Year Country                                                       
# 2006 Afghanistan           NaN               0.222            0.475
#      Albania             0.507               0.444            0.541
#      Algeria             0.221               0.222            0.399
#      Angola              0.214               0.111              NaN
#      Argentina           0.500               0.556            0.778
# ...                        ...                 ...              ...
# 2017 Somalia               NaN                 NaN            0.394
#      South Sudan           NaN                 NaN              NaN
# 2018 Georgia               NaN                 NaN            0.605
#      Somalia               NaN                 NaN              NaN
#      South Sudan           NaN                 NaN              NaN

# [6976 rows x 3 columns]

test_multidx_profile = ProfileReport(test_multidx_df, title='Test MultIdx Profile')

The output is exactly what I was going for, but in addition to wishing for a one- or two-statement solution, I'm not completely happy with iterating through an input dictionary of dataframes. I tried to make the input a dataframe of dataframes so I could apply(lambda) to save some memory I think, but no dice getting apply() to work right, and it's time to move on.



来源:https://stackoverflow.com/questions/64216845/how-to-correlate-scalar-values-of-two-pandas-dataframes

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!