Solr search for hashtag or mentions

Question

We are using solr version 3.5 to search though Tweets, I am using WordDelimiterFactory with the following setting, to be able to search for @username

Answer 1

OK, so reading through the SOLR-2059 patch that you mentioned, it looks like they have replaced the handleAsChar attribute on the WordDelimiterFactory with the types attribute. Here is the specification for that attribute from the Analyzers, Tokenizers and Token Filters Solr Wiki page:

types="wdfftypes.txt" allows customized tokenization for this filter. The file should exist in the solr/conf directory, and entries are of the form (without quotes) "% => ALPHA" or "\u002C => DIGIT". Allowable types are: LOWER, UPPER, ALPHA, DIGIT, ALPHANUM, SUBWORD_DELIM.

So then if we take this documentation, plus the example of the file from SOLR-2059, I would recommend the following:

<filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0" splitOnCaseChange="0" splitOnNumerics="0" preserveOriginal="1" types="twittertypes.txt"/>

Then define the twittertypes.txt file as follows and place it in the same folder as your schema.xml file in your Solr instance (probably the conf folder).

 # A customized type mapping for WordDelimiterFilterFactory
 # the allowable types are: LOWER, UPPER, ALPHA, DIGIT, ALPHANUM, SUBWORD_DELIM
 #    
 # the default for any character without a mapping is always computed from 
 # Unicode character properties

 # Map the $, %, '.', and ',' characters to DIGIT 
 # This might be useful for financial data.
 @ => ALPHA
 \u0023 => ALPHA

Notice that you need to use the Unicode character (UTF-8) for the hash symbol, since it is treated as comment in the text file.

According to all of the documentation, this should fix your issue and treat the # and @ symbols as alpha characters, which will provide the behavior you are looking for.

Answer 2

You can also build a custom tokenizer, which will parse usernames and hashtags as special tokens natively. You can then add a custom filter which normalizes the case of these usernames and hashtags (given that they are not case-sensitive), while leaving the other tokens untouched:

<fieldType name="text_twitter" class="solr.TextField" positionIncrementGap="100" multiValued="true">
  <analyzer type="index">
    <tokenizer class="org.opentapioca.analysis.twitter.TwitterTokenizerFactory" />
    <filter class="org.opentapioca.analysis.twitter.TwitterLowercaseFilterFactory" />
  </analyzer>
  <analyzer type="query">
     <tokenizer class="org.opentapioca.analysis.twitter.TwitterTokenizerFactory" />
     <filter class="org.opentapioca.analysis.twitter.TwitterLowercaseFilterFactory" />
  </analyzer>
</fieldType>