Patent attributes
Methods, systems, and computer-readable media for similarity detection based on token distinctiveness are disclosed. A similarity detection system determines candidate items for a seed item based on a comparison of tokens in textual descriptions of the candidate items to tokens in a textual description of the seed item. The system uses machine learning to determine importance scores for the tokens of the seed item. An importance score is determined based on the frequency of the individual token and the frequency of the most commonly occurring token in the candidate items. Importance scores for the same token differ from the seed item to another seed item. Based on the importance scores, the system determines similarity scores for the candidate items to the seed item. The system selects, from the candidate items, a set of similar items to the seed item based (at least in part) on the similarity scores.
