Python lists, csv, duplication removal

问题

Even though I'm very new with python, I can't understand how I haven't been able to solve this issue / take a right approach. So any help, link to a helpful tutorial is appreciated highly as I have to do this kind of stuff from time to time.

I have a CSV file that I need to reformat / modify a bit.

I need to store the amount of samples that the gene is in.

input file:

AHCTF1: Sample1, Sample2, Sample4
AHCTF1: Sample2, Sample7, Sample12
AHCTF1: Sample5, Sample6, Sample7

result:

 AHCTF1 in 7 samples (Sample1, Sample2, Sample4, Sample5, Sample6, Sample7, Sample12)

code:

f = open("/CSV-sorted.csv")
gene_prev = ""

hit_list = []

csv_f = csv.reader(f)

for lines in csv_f:

    #time.sleep(0.1)
    gene = lines[0]
    sample = lines[11].split(",")
    repeat = lines[8]

    for samples in sample:
        hit_list.append(samples)

    if gene == gene_prev:

        for samples in sample:

            hit_list.append(samples)

        print gene
        print hit_list
        print set(hit_list)
        print "samples:", len(set(hit_list))


    hit_list = []

    gene_prev = gene

So in a nutshell I'd like to combine the hits for every gene and make a set from them to remove duplications.

Maybe dictionary would be the way to do it:s ave gene as a key and add samples as values?

Found this - Similar / useful: How can I combine dictionaries with the same keys in python?

回答1:

The standard way to remove duplicates is to convert to a set.

However I think there's some stuff wrong with the way you're reading the file. First problem: it isn't a csv file (you have a colon between the first two fields). Second what is

gene = lines[0]
sample = lines[11].split(",")
repeat = lines[8]

supposed to do?

If I was writing this I would replace the ":" with another ",". So with this modification and using a dictionary of sets your code would look something like:

# Read in csv file and convert to list of list of entries. Use with so that 
# the file is automatically closed when we are done with it
csvlines = []
with open("CSV-sorted.csv") as f:
    for line in f:
        # Use strip() to clean up trailing whitespace, use split() to split
        # on commas.
        a = [entry.strip() for entry in line.split(',')]
        csvlines.append(a)

# I'll print it here so you can see what it looks like:
print(csvlines)



# Next up: converting our list of lists to a dict of sets.

# Create empty dict
sample_dict = {}

# Fill in the dict
for line in csvlines:
    gene = line[0] # gene is first entry
    samples = set(line[1:]) # rest of the entries are samples

    # If this gene is in the dict already then join the two sets of samples
    if gene in sample_dict:
        sample_dict[gene] = sample_dict[gene].union(samples)

    # otherwise just put it in
    else:
        sample_dict[gene] = samples


# Now you can print the dictionary:
print(sample_dict)

The output is:

[['AHCTF1', 'Sample1', 'Sample2', 'Sample4'], ['AHCTF1', 'Sample2', 'Sample7', 'Sample12'], ['AHCTF1', 'Sample5', 'Sample6', 'Sample7']]
{'AHCTF1': {'Sample12', 'Sample1', 'Sample2', 'Sample5', 'Sample4', 'Sample7', 'Sample6'}}

where the second line is your dictionary.

来源：https://stackoverflow.com/questions/25486554/python-lists-csv-duplication-removal

标签

python

python-2.7