DelimitedPayloadFilter in PyLucene?

问题

I am trying to implement a python version of the java from http://searchhub.org/2010/04/18/refresh-getting-started-with-payloads/ using pylucene. My analyzer is producing an lucene.InvalidArgsError on the init call to the DelimitedTokenFilter

The class is below, and any help is greatly appreciated. The java version compiled with the JAR files from the pylucene 3.6 build works fine.

import lucene
class PayloadAnalyzer(lucene.PythonAnalyzer):
    encoder = None
    def __init__(self, encoder): 
        lucene.PythonAnalyzer.__init__(self) 
        self.encoder = encoder

    def tokenStream(self, fieldName, reader):
        result = lucene.WhitespaceTokenizer( lucene.Version.LUCENE_CURRENT, reader )
        result = lucene.LowerCaseFilter( lucene.Version.LUCENE_CURRENT, result )
        result = lucene.DelimitedPayloadTokenFilter( result, '|', self.encoder )
        return result

回答1:

The doc of jcc says:

When JCC sees these special extension java classes it generates the C++ code implementing the native methods they declare. These native methods call the corresponding Python method implementations passing in parameters and returning the result to the Java VM caller.

So you should edit the file java/org/apache/pylucene/search/similarities/PythonDefaultSimilarity.java in pylucene.

Add some code like this:

import org.apache.lucene.util.BytesRef;
public native float scorePayload(int docId, int start, int end, BytesRef payload);

After this, your code can override the method scorePayload.

class PayloadSimilarity(PythonDefaultSimilarity):

    def scorePayload(self, docId, start, end, payload):
        return PayloadHelper.decodeFloat(payload.bytes, end)


class PayloadAnalyzer(PythonAnalyzer):
    encoder = None

    def __init__(self, encoder):
        super(PayloadAnalyzer, self).__init__()
        self.encoder = encoder

    def createComponents(self, fieldName, reader):
        source = WhitespaceTokenizer(Version.LUCENE_44, reader)
        result = LowerCaseFilter(Version.LUCENE_44, source)
        result = DelimitedPayloadTokenFilter(result, u'|', self.encoder)
        return self.TokenStreamComponents(source, result)

I test the code above under pylucene4.8. It works fine.

来源：https://stackoverflow.com/questions/13426157/delimitedpayloadfilter-in-pylucene

标签

analyzer

payload

pylucene