Spacy's BERT model doesn't learn

问题

I've been trying to use spaCy's pretrained BERT model de_trf_bertbasecased_lg to increase accuracy in my classification project. I used to build a model from scratch using de_core_news_sm and everything worked fine: I had an accuracy around 70%. But now I am using BERT pretrained model instead and I'm getting 0% accuracy. I don't believe that it's working so bad, so I'm assuming that there is just a problem with my code. I might have missed something important but I can't figure out what. I used the code in this article as an example.

Here is my code:

import spacy
from spacy.util import minibatch
from random import shuffle

spacy.require_gpu()
nlp = spacy.load('de_trf_bertbasecased_lg')

data = get_data()  # get_data() function returns a list with train data (I'll explain later how it looks)

textcat = nlp.create_pipe("trf_textcat", config={"exclusive_classes": False})

for category in categories:  # categories - a list of 21 different categories used for classification
    textcat.add_label(category)
nlp.add_pipe(textcat)

num = 0  # number used for counting batches
optimizer = nlp.resume_training()
for i in range(2):
    shuffle(data)
    losses = {}
    for batch in minibatch(data):
        texts, cats = zip(*batch)
        nlp.update(texts, cats, sgd=optimizer, losses=losses)
        num += 1

        if num % 10000 == 0:  # test model's performance every 10000 batches
            acc = test(nlp)  # function test() will be explained later
            print(f'Accuracy: {acc}')

nlp.to_disk('model/')

Function get_data() opens files with different categories, creates a tuple like this one (text, {'cats' : {'category1': 0, 'category2':1, ...}}), gathers all these tuples into one array, which is then being returned to the main function.

Function test(nlp) opens the file with test data, predicts categories for each line in the file and checks, whether the prediction was correct.

Again, everything worked just fine with de_core_news_sm, so I'm pretty sure that functions get_data() and test(nlp) are working fine. Code above looks like in example but still 0% accuracy.I don't understand what I'm doing wrong.

Thanks in advance for any help!

UPDATE

Trying to understand the above problem I decided to try the model with only a few examples (just like it is advised here). Here is the code:

import spacy
from spacy.util import minibatch
import random
import torch

train_data = [
    ("It is realy cool", {"cats": {"POSITIVE": 1.0, "NEGATIVE": 0.0}}),
    ("I hate it", {"cats": {"POSITIVE": 0.0, "NEGATIVE": 1.0}})
]

is_using_gpu = spacy.prefer_gpu()
if is_using_gpu:
    torch.set_default_tensor_type("torch.cuda.FloatTensor")

nlp = spacy.load("en_trf_bertbaseuncased_lg")
textcat = nlp.create_pipe("trf_textcat", config={"exclusive_classes": True})
for label in ("POSITIVE", "NEGATIVE"):
    textcat.add_label(label)
nlp.add_pipe(textcat)

optimizer = nlp.resume_training()
for i in range(10):
    random.shuffle(train_data)
    losses = {}
    for batch in minibatch(train_data):
        texts, cats = zip(*batch)
        nlp.update(texts, cats, sgd=optimizer, losses=losses)
    print(i, losses)
print()

test_data = [
    "It is really cool",
    "I hate it",
    "Great!",
    "I do not think this is cool"
]

for line in test_data:
    print(line)
    print(nlp(line).cats)

And the output was:

0 {'trf_textcat': 0.125}
1 {'trf_textcat': 0.12423406541347504}
2 {'trf_textcat': 0.12188033014535904}
3 {'trf_textcat': 0.12363225221633911}
4 {'trf_textcat': 0.11996611207723618}
5 {'trf_textcat': 0.14696261286735535}
6 {'trf_textcat': 0.12320466339588165}
7 {'trf_textcat': 0.12096124142408371}
8 {'trf_textcat': 0.15916231274604797}
9 {'trf_textcat': 0.1238454058766365}

It is really cool
{'POSITIVE': 0.47827497124671936, 'NEGATIVE': 0.5217249989509583}
I hate it
{'POSITIVE': 0.47827598452568054, 'NEGATIVE': 0.5217240452766418}
Great!
{'POSITIVE': 0.4782750606536865, 'NEGATIVE': 0.5217249393463135}
I do not think this is cool
{'POSITIVE': 0.478275328874588, 'NEGATIVE': 0.5217246413230896}

Not only the model performs bad, the loss is not getting smaller and scores for all the test sentences are almost the same. And most importantly: it didn't even get those questions correct, that happened to be in the train data. So my question is: does the model even learn? And what am I doing wrong?

Any thoughts?

来源：https://stackoverflow.com/questions/61943409/spacys-bert-model-doesnt-learn