Remove uni-grams from a list of bi-grams

Question

I have managed to create 2 lists from text documents. The first is my bi-gram list:

keywords = ['nike shoes','nike clothing', 'nike black', 'nike white']

and a list of stop words:

stops = ['clothing','black','white']

I want to remove the Stops from my Keywords list. Using the above example, the output I am after should look like this:

new_keywords = ['nike shoes','nike', 'nike', 'nike'] --> eventually I'd like to remove those dupes. 

This is what I've done so far:

keywords = open("keywords.txt", "r")
new_keywords = keywords.read().split(",")
stops = open("stops.txt","r")
new_stops = stops.read().split(",")
[i for i in new_keywords if i not in new_stops]

The problem I am having is that it is looking for the 2 words combos rather than the single word stops....

Answer 1

You can do it in steps. First define a helper function:

def removeStop(bigram, stops):
    return ' '.join(w for w in bigram.split() if not w in stops)

And then:

[removeStop(i,new_stops) for i in new_keywords] 

Answer 2

assuming you have the 2 lists this will do what you want:

new_keywords = []

for k in keywords:
    temp = False

    for s in stops:
        if s in k:
           new_keywords.append(k.replace(s,""))
           temp = True

    if temp == False:
        new_keywords.append(k)

This will create a list like you posted:

['nike shoes', 'nike ', 'nike ', 'nike ']

To eliminate the doubles do this:

new_keywords = list(set(new_keywords))

So the final list looks like this:

['nike shoes', 'nike ']