使用Word2Vec进行文本语义相似度计算

本示例展示利用PaddleHub“端到端地”完成文本相似度计算。

一、准备文本数据

如

  1. 驾驶违章一次扣12分用两个驾驶证处理可以吗    一次性扣12分的违章,能用不满十二分的驾驶证扣分吗
  2. 水果放冰箱里储存好吗    中国银行纪念币网上怎么预约
  3. 电脑反应很慢怎么办    反应速度慢,电脑总是卡是怎么回事

二、分词

利用PaddleHub Module LAC对文本数据进行分词。

  1. # coding:utf-8
  2. #  Copyright (c) 2019  PaddlePaddle Authors. All Rights Reserved.
  3. #
  4. # Licensed under the Apache License, Version 2.0 (the "License"
  5. # you may not use this file except in compliance with the License.
  6. # You may obtain a copy of the License at
  7. #
  8. #     http://www.apache.org/licenses/LICENSE-2.0
  9. #
  10. # Unless required by applicable law or agreed to in writing, software
  11. # distributed under the License is distributed on an "AS IS" BASIS,
  12. # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  13. # See the License for the specific language governing permissions and
  14. # limitations under the License.
  15. """similarity between two sentences"""
  17. import numpy as np
  18. import scipy
  19. from scipy.spatial import distance
  21. from paddlehub.reader.tokenization import load_vocab
  22. import paddle.fluid as fluid
  23. import paddlehub as hub
  25. raw_data = [
  26.     ["驾驶违章一次扣12分用两个驾驶证处理可以吗", "一次性扣12分的违章,能用不满十二分的驾驶证扣分吗"],
  27.     ["水果放冰箱里储存好吗", "中国银行纪念币网上怎么预约"],
  28.     ["电脑反应很慢怎么办", "反应速度慢,电脑总是卡是怎么回事"]
  29. ]
  31. lac = hub.Module(name="lac")
  33. processed_data = []
  34. for text_pair in raw_data:
  35.     inputs = {"text" : text_pair}
  36.     results = lac.lexical_analysis(data=inputs, use_gpu=True, batch_size=2)
  37.     data = []
  38.     for result in results:
  39.         data.append(" ".join(result["word"]))
  40.     processed_data.append(data)

三、计算文本语义相似度

将分词文本中的单词相应替换为wordid，之后输入wor2vec module中计算两个文本语义相似度。

  1. def convert_tokens_to_ids(vocab, text):
  2.     wids = []
  3.     tokens = text.split(" ")
  4.     for token in tokens:
  5.         wid = vocab.get(token, None)
  6.         if not wid:
  7.             wid = vocab["unknown"]
  8.         wids.append(wid)
  9.     return wids
  11. module = hub.Module(name="word2vec_skipgram")
  12. inputs, outputs, program = module.context(trainable=False)
  13. vocab = load_vocab(module.get_vocab_path())
  15. word_ids = inputs["word_ids"]
  16. embedding = outputs["word_embs"]
  18. place = fluid.CPUPlace()
  19. exe = fluid.Executor(place)
  20. feeder = fluid.DataFeeder(feed_list=[word_ids], place=place)
  22. for item in processed_data:
  23.     text_a = convert_tokens_to_ids(vocab, item[0])
  24.     text_b = convert_tokens_to_ids(vocab, item[1])
  26.     vecs_a, = exe.run(
  27.         program,
  28.         feed=feeder.feed([[text_a]]),
  29.         fetch_list=[embedding.name],
  30.         return_numpy=False)
  31.     vecs_a = np.array(vecs_a)
  32.     vecs_b, = exe.run(
  33.         program,
  34.         feed=feeder.feed([[text_b]]),
  35.         fetch_list=[embedding.name],
  36.         return_numpy=False)
  37.     vecs_b = np.array(vecs_b)
  39.     sent_emb_a = np.sum(vecs_a, axis=0)
  40.     sent_emb_b = np.sum(vecs_b, axis=0)
  41.     cos_sim = 1 - distance.cosine(sent_emb_a, sent_emb_b)
  43.     print("text_a: %s; text_b: %s; cosine_similarity: %.5f" %
  44.           (item[0], item[1], cos_sim))