Any ideas to find the similarity of a list of short phrases ?
I want to build a program to to compare the similarity of a input phrase with a array of short phrases.
And find the most similar one.
Should I start from extract the keyword from the input first then compare with the array ?
One of the characteristic of the input and data array is all of them are noun phrases like animal name, household item and place name etc. Would it help me to exploit ?
Posted Apr 22, 2020, 5:18 PM UTC
11
If you are trying to find a list of phrases whose meaning is similar, this is a non-trivial problem, and while you can accomplish it in python (to the degree that a computer can accomplish this at all), there are many ways to approach it that are too diverse to cover in a reddit comment. If you're looking to compare how similar they are in terms of their lexical content (same letters, same words, same typetc.), your best bet is probably something that does levenshtein distance, such as
fuzzywuzzy. You could come to some sort of "hybrid" approach in a couple of ways, first by extracting semantically meaningful tokens using, e.g.,nltkor replace all synonyms of the same word with that word using either your own wordlist or something likewordnet, and thereafter usingfuzzywuzzy.