How to filter on pandas dataframe when column data type is a list

Question

I am having some trouble filtering a pandas dataframe on a column (let\'s call it column_1) whose data type is a list. Specifically, I want to return only rows such that co

Answer 1

consider the pd.Series s

s = pd.Series([[1, 2, 3], list('abcd'), [9, 8, 3], ['a', 4]])
print(s)

0       [1, 2, 3]
1    [a, b, c, d]
2       [9, 8, 3]
3          [a, 4]
dtype: object

And a testing list test

test = ['b', 3, 4]

Apply a lambda function that converts each element of s to a set and intersection with test

print(s.apply(lambda x: list(set(x).intersection(test))))

0    [3]
1    [b]
2    [3]
3    [4]
dtype: object

To use it as a mask, use bool instead of list

s.apply(lambda x: bool(set(x).intersection(test)))

0    True
1    True
2    True
3    True
dtype: bool

Answer 2

Hi for long term use you can wrap the whole work flow in functions and apply the functions where you need. As you did not put any example dataset. I am taking an example data set and resolving it. Considering I have text database. First I will find the #tags into a list then I will search the only #tags I want and filter the data.

# find all the tags in the message
def find_hashtags(post_msg):
    combo = r'#\w+'
    rx = re.compile(combo)
    hash_tags = rx.findall(post_msg)
    return hash_tags


# find the requered match according to a tag list and return true or false
def match_tags(tag_list, htag_list):
    matched_items = bool(set(tag_list).intersection(htag_list))
    return matched_items


test_data = [{'text': 'Head nipid mõnusateks sõitudeks kitsastel tänavatel. #TipStop'},
 {'text': 'Homses Rooli Võimus uus #Peugeot208!\nVaata kindlasti.'},
 {'text': 'Soovitame ennast tulevikuks ette valmistada, electric car sest uus #PeugeotE208 on peagi kohal!  ⚡️⚡️\n#UnboringTheFuture'},
 {'text': "Aeg on täiesti uueks roadtrip'i kogemuseks! \nLase ennast üllatada - #Peugeot5008!"},
 {'text': 'Tõeline ikoon, mille stiil avaldab muljet läbi eco car, electric cars generatsioonide #Peugeot504!'}
]

test_df = pd.DataFrame(test_data)

# find all the hashtags
test_df["hashtags"] = test_df["text"].apply(lambda x: find_hashtags(x))

# the only hashtags we are interested
tag_search = ["#TipStop", "#Peugeot208"]

# match the tags in our list
test_df["tag_exist"] = test_df["hashtags"].apply(lambda x: match_tags(x, tag_search))

# filter the data
main_df = test_df[test_df.tag_exist]