Setting explicit rules for matching records using Python Dedupe library

问题

I'm using the Dedupe library to match person records to each other. My data includes name, date of birth, address, phone number and other personally identifying information.

Here is my question: I always want to match two records with 100% confidence if they have a matching name and phone number (for example).

Here is an example of some of my code:

fields = [
    {'field' : 'LAST_NM', 'variable name' : 'last_nm', 'type': 'String'},
    {'field' : 'FRST_NM', 'variable name' : 'frst_nm', 'type': 'String'},
    {'field' : 'FULL_NM', 'variable name' : 'full_nm', 'type': 'Name'},
    {'field' : 'BRTH_DT', 'variable name' : 'brth_dt', 'type': 'String'},
    {'field' : 'SEX_CD', 'type': 'Exact'},
    {'field' : 'FULL_US_ADDRESS', 'variable name' : 'us_address', 'type': 'Address'},
    {'field' : 'APT_NUM', 'type': 'Exact'},
    {'field' : 'CITY', 'type': 'ShortString'},
    {'field' : 'STATE', 'type': 'ShortString'},
    {'field' : 'ZIP_CD', 'type': 'ShortString'},
    {'field' : 'HOME_PHONE', 'variable name' : 'home_phone', 'type': 'Exact'},
    {'type': 'Interaction', 'interaction variables' : ['full_nm', 'home_phone']},

In the Dedupe library, is there any way for me to explicitly match two or more fields? According to the docs, "An interaction field multiplies the values of the multiple variables." (https://dedupe.readthedocs.org/en/latest/Variable-definition.html#interaction). I want to implement a strict rule that it matches with 100% confidence - not merely multiplying the values of the variables. The reason I ask is that I have found that occasionally Dedupe misses some matches on these two criteria (likely a result of me not training long enough, but regardless, I just want to hard code these matches into my script).

Any suggestions?

回答1:

Dedupe does not have this feature and probably never will (I'm one of the main authors). If it's truly a rule that exact matches on these fields means that records are co-referent, you can write some code to explicitly match these before sending the rest of the records into Dedupe.

exact_matches = defaultdict(list)
for record_id, record in records.items():
    match_key = (record['name'], record['phone'])
    exact_matches[match_key].append(record_id)

partially_deduplicated = []
exact_lookup = {}
for match_group in exact_matches.values():
     head_id = match_group.pop()
     partially_deduplicated.append((head_id, records[head_id]))
     for dupe_id in match_group :
         exact_lookup[dupe_id] = head_id

回答2:

Set all the fields you want to match exactly to type 'exact' - for example:

{'field' : 'FULL_NM', 'variable name' : 'full_nm', 'type': 'Exact'},

来源：https://stackoverflow.com/questions/32550441/setting-explicit-rules-for-matching-records-using-python-dedupe-library

标签

python

duplicates

record-linkage

python-dedupe