Dataframe Rows are matching with each other in TF-IDF Cosine similarity i

问题

I am trying to learn data science and found this great article online.

https://bergvca.github.io/2017/10/14/super-fast-string-matching.html

I have this database full of company names, but am finding that the results where the similarity is equal to 1, they are in fact literally the same exact row. I obviously want to catch duplicates, but I do not want the same row to match itself.

On a side note, this has opened my eyes to pandas and NLP. Super fascinating field - Hopefully, somebody can help me out here.

import pandas as pd
import re
from sklearn.feature_extraction.text import TfidfVectorizer
import numpy as np
from scipy.sparse import csr_matrix
import sparse_dot_topn.sparse_dot_topn as ct

pd.set_option('display.max_colwidth', -1)
df = pd.read_csv('CSV/Contacts.csv',  dtype=str)
print(df.shape)
df.head(2)

Shape: (72489, 3)

    Id  Name    Email
0   0031J00001bvXFTQA2 FRESHPOINT ATLANTA, INC  dotcomp@sysco.com
1   0031J00001aJtFaQAK  VIRGIL  dotcom@corp.sysco.com

Then I clean the data

# Clean the data
df.dropna()
# df['Email'] = df['Email'].str.replace('[^a-zA-Z]', '')
# df['Email'] = df['Email'].str.replace(r'[^\w\s]+', '')

contact_emails = df['Email']

Then I implement the N-Grams function

def ngrams(string, n=3):
    string = re.sub(r'[,-./]|\sBD',r'', string)
    ngrams = zip(*[string[i:] for i in range(n)])
    return [''.join(ngram) for ngram in ngrams]

Then I get the TF-IDF Matrix

# get Tf-IDF Matrix
vectorizer = TfidfVectorizer(min_df=1, analyzer=ngrams)
tf_idf_matrix = vectorizer.fit_transform(contact_emails.apply(lambda x: np.str_(x)))

Then I implement the Cosine Similarity function - Which I am still not quite sure what each parameter does.

def awesome_cossim_top(A, B, ntop, lower_bound=0):
    # force A and B as a CSR matrix.
    # If they have already been CSR, there is no overhead
    A = A.tocsr()
    B = B.tocsr()
    M, _ = A.shape
    _, N = B.shape

    idx_dtype = np.int32

    nnz_max = M*ntop

    indptr = np.zeros(M+1, dtype=idx_dtype)
    indices = np.zeros(nnz_max, dtype=idx_dtype)
    data = np.zeros(nnz_max, dtype=A.dtype)

    ct.sparse_dot_topn(
        M, N, np.asarray(A.indptr, dtype=idx_dtype),
        np.asarray(A.indices, dtype=idx_dtype),
        A.data,
        np.asarray(B.indptr, dtype=idx_dtype),
        np.asarray(B.indices, dtype=idx_dtype),
        B.data,
        ntop,
        lower_bound,
        indptr, indices, data)

    return csr_matrix((data,indices,indptr),shape=(M,N))

Then we actually find the matches. I am not sure what the transpose does in this case and how that finds matches.

matches = awesome_cossim_top(tf_idf_matrix, tf_idf_matrix.transpose(), 10, 0.7)

Then here is the function for extracting the matches.

def get_matches_df(sparse_matrix, email_vector,email_ids, top=5):
    non_zeros = sparse_matrix.nonzero()

    sparserows = non_zeros[0]
    sparsecols = non_zeros[1]


    if top:
        nr_matches = top
    else:
        nr_matches = sparsecols.size
    left_name_Ids = np.empty([nr_matches], dtype=object)
    right_name_Ids = np.empty([nr_matches], dtype=object)

    left_side = np.empty([nr_matches], dtype=object)
    right_side = np.empty([nr_matches], dtype=object)
    similairity = np.zeros(nr_matches)

    for index in range(nr_matches):        
        left_name_Ids[index] = email_ids[sparserows[index]]
        left_side[index] = email_vector[sparserows[index]]

        right_name_Ids[index] = email_ids[sparsecols[index]]
        right_side[index] = email_vector[sparsecols[index]]
        similairity[index] = sparse_matrix.data[index]

    return pd.DataFrame({
                        'SFDC_ID':  left_name_Ids,
                        'left_side': left_side,
                        'right_SFDC_ID':right_name_Ids,
                          'right_side': right_side,
                           'similairity': similairity})

Then I call the function and pass in the params

name_Ids = df['Id']
matches_df = get_matches_df(matches, contact_emails,name_Ids, top=72489)

Now I only want to extract matches that are 90% similar or more.

matches_df = matches_df[matches_df['similairity'] > 0.9]

Then I sort the values by similarity

matches_df.sort_values('similairity' )

So what I am finding is that the same rows are being matched with each other. I know this because the SFDC ids are exactly the same - Why is this happening? How can I avoid this in the future? I obviously do not want the row to asses itself when finding similarities.

来源：https://stackoverflow.com/questions/60467185/dataframe-rows-are-matching-with-each-other-in-tf-idf-cosine-similarity-i

标签

python

pandas

nlp

tf-idf

cosine-similarity