How to extract chunks from BIO chunked sentences? - python

限于喜欢 提交于 2019-12-05 03:58:10
def extract_chunks(tagged_sent, chunk_type):
    grp1, grp2, chunk_type = [], [], "-" + chunk_type
    for ind, (s, tp) in enumerate(tagged_sent):
        if tp.endswith(chunk_type):
            if not tp.startswith("B"):
                grp2.append(str(ind))
                grp1.append(s)
            else:
                if grp1:
                    yield " ".join(grp1), "-".join(grp2)
                grp1, grp2 = [s], [str(ind)]
    yield " ".join(grp1), "-".join(grp2)

Output:

In [2]: l = [('The', 'B-NP'), ('Mitsubishi', 'I-NP'), ('Electric', 'I-NP'), ('Company', 'I-NP'), ('Managing', 'B-NP'),
   ...:                ('Director', 'I-NP'), ('ate', 'B-VP'), ('ramen', 'B-NP')]

In [3]: list(extract_chunks(l, "NP"))
Out[3]: 
[('The Mitsubishi Electric Company', '0-1-2-3'),
 ('Managing Director', '4-5'),
 ('ramen', '7')]

In [4]: l = [('What', 'B-NP'), ('is', 'B-VP'), ('the', 'B-NP'), ('airspeed', 'I-NP'), ('of', 'B-PP'), ('an', 'B-NP'), ('unladen', 'I-NP'), ('swallow', 'I-NP'), ('?', 'O')]

In [5]: list(extract_chunks(l, "NP"))
Out[5]: [('What', '0'), ('the airspeed', '2-3'), ('an unladen swallow', '5-6-7')]

Try this, it will extract all types of chunks with the indices of their respective words.

def extract_chunks(tagged_sent, chunk_type='NP'):
    out_sen = []
    for idx, word_pos in enumerate(tagged_sent):
        word,bio = word_pos
        boundary,tag = bio.split("-") if "-" in bio else ('','O')
        if tag != chunk_type:continue
        if boundary == "B":
            out_sen.append([word, str(idx)])
        elif boundary == "I":
            out_sen[-1][0] += " "+ word
            out_sen[-1][-1] += "-"+ str(idx)
        else:
            out_sen.append([word, str(idx)])
    return out_sen

Demo:

>>> tagged_sent = [('The', 'B-NP'), ('Mitsubishi', 'I-NP'),  ('Electric', 'I-NP'), ('Company', 'I-NP'), ('Managing', 'B-NP'), ('Director', 'I-NP'), ('ate', 'B-VP'), ('ramen', 'B-NP')]
>>> output_sent = extract_chunks(tagged_sent)
>>> print map(tuple, output_sent)
[('The Mitsubishi Electric Company', '0-1-2-3'), ('Managing Director', '4-5'), ('ramen', '7')]

I would do it like this:

import re
def extract_chunks(tagged_sent, chunk_type):
    # compiles the expression we want to match
    regex = re.compile(chunk_type)

    # filters matched items in a dictionary whose keys are the matched indexes
    first_step = {index_:tag[0] for index_, tag in enumerate(tagged_sent) if regex.findall(tag[1])}

    # builds list of lists following output format
    second_step = []
    for key_ in sorted(first_step.keys()):
        if second_step and int(second_step [len(second_step )-1][1].split('-')[-1]) == key_ -1:           
            second_step[len(second_step)-1][0] += ' {0}'.format(first_step[key_])
            second_step[len(second_step)-1][1] += '-{0}'.format(str(key_))
        else:
            second_step.append([first_step[key_], str(key_)])

    # builds output in final format
    return [tuple(item) for item in second_step]

You can adapt it to use generators instead of building the whole output in memory like I am doing and refactory it for better performance (I'm in a hurry so the code is far from optimal).

Hope it helps!

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!