LUIS - understand any person name

Question

we are building a product on LUIS / Microsoft Bot framework and one of the doubt we have is Person Name understanding. The product is set to use by anyone by just signing up to our website. Which means any company who is signing up can have any number of employees with any name obviously.

What we understood is the user entity is not able to recognize all names. We have created a phrase list but as per we know there is a limit to phrase list (10K or even if its 100K) and names in the world can never have a limit. The other way we are thinking is to not train the entity with utterances. However if we have 100s of customers with 1000s of users each, the utterances will not be a good idea in that case.

I do not see any other way of handling this situation. Probably I am missing something here? Anyone faced similar problem and how it is handled?

The worst case would be to create a separate LUIS instance for each customer but that's really a big task to do only because we cant handle names.

Answer 1

As you might already know, a person's name could literally be anything: e.g. an animal, car, month, or color. So, there isn't any definitive way to identify something as a name. The closest you can come is via text analysis parts of speech and either taking a guess or comparing to an existing list. LUIS or any other NLP tool is unlikely to help with this. Here's one approach that might work out better. Try something like Microsoft's Text Analytics cognitive service, with a POST to the Key Phrases endpoint, like this:

https://westus.api.cognitive.microsoft.com/text/analytics/v2.0/keyPhrases

and the body:

{
  "documents": [
    {
      "language": "en-us",
      "id": "myid",
      "text": "Please book a flight for John Smith at 2:30pm on Wednesday."
    }
  ]
}

That returns:

{
  "languageDetection": {
    "documents": [
      {
        "id": "e4263091-2d54-4ab7-b660-d2b393c4a889",
        "detectedLanguages": [
          {
            "name": "English",
            "iso6391Name": "en",
            "score": 1.0
          }
        ]
      }
    ],
    "errors": []
  },
  "keyPhrases": {
    "documents": [
      {
        "id": "e4263091-2d54-4ab7-b660-d2b393c4a889",
        "keyPhrases": [
          "John Smith",
          "flight"
        ]
      }
    ],
    "errors": []
  },
  "sentiment": {
    "documents": [
      {
        "id": "e4263091-2d54-4ab7-b660-d2b393c4a889",
        "score": 0.5
      }
    ],
    "errors": []
  }
}

Notice that you get "John Smith" and "flight" back as key phrases. "flight" is definitely not a name, but "John Smith" might be, giving you a better idea of what the name is. Additionally, if you have a database of customer names, you can compare the value to a customer name, either exact or soundex, to increase your confidence in the name.

Sometimes, the services don't give you an 100% answer and you have to be creative with work-arounds. Please see the Text Analytics API docs for more info.

Answer 2

Have asked this question to few MS guys in my local region however it seems there is no way LUIS at moment can identify names.

Its not good as being NLP, it is not able to handle such things :(

I found wit.ai (best so far) in identifying names and IBM Watson is also good upto some level. Lets see how they turn out in future but for now I switched to https://wit.ai