Mysql query of inverted index data

六眼飞鱼酱① 提交于 2019-12-12 01:44:39

问题


I have thousand of pages in website which I parsed and stored it as Inverted Index viz

document

  • docid (PK,FK)
  • url
  • charactercount
  • wordcount

Charactercount and wordcount helps me determine long document from short which I may use later.

word

  • wordid (PK,FK)
  • word
  • doc_freq
  • inverse_doc_freq

For inverse_doc_freq calculation I use fictional high number (100000000) to prevent total document recalculation.

loc

  • wordid
  • docid
  • word_freq
  • weight

(wordid & docid combined unique)

The weight is a score calculated on simple basis like word in title + word in url + word frquency etc.

I am having problem framing my sql query for search words. For 3 word search I am doing like

  1. Break query into each word
  2. Check inverse_doc_freq for each word and remove low idf word (removal of stop word)
  3. stem the remaining word (assume still 3 words remain)
  4. Query for each word

It is at stage 4 that I am getting stuck ! My SQL query is like

SELECT d.docid,url,inverse_doc_freq,word_freq,weight from document d,word w,loc l WHERE d.docid=l.docid AND w.wordid=l.wordid AND (word='word1' OR word='word2' OR word='word3') ORDER BY weight DESC

The returned documents are not correct though. Trust I might have to Search thrice to find documents for each word and then try to find the common documents, but how ? Is it possible to use only 1 MySQL query for it ? Also is it possible to use TF-IDF and how ?


回答1:


You need to aggregate at the document level.

select d.docid, d.url, sum(weight) as weight
from document d join
     loc l
     on d.docid = l.docid join
     word w
     on w.wordid = l.wordid
where w.word in ('word1', 'word2', 'word3')
group by d.docid
order by weight DESC;


来源:https://stackoverflow.com/questions/23539380/mysql-query-of-inverted-index-data

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!