问题
I'm using the elasticsearch-py client within PySpark using Python 3 and I'm running into a problem using the analyze() function with ES in conjunction with an RDD. In particular, each record in my RDD is a string of text and I'm trying to analyze it to get out the token information, but I'm getting an error when trying to use it within a map function in Spark.
For example, this works perfectly fine:
from elasticsearch import Elasticsearch
es = Elasticsearch()
t = 'the quick brown fox'
es.indices.analyze(text=t)['tokens'][0]
{'end_offset': 3,
'position': 1,
'start_offset': 0,
'token': 'the',
'type': '<ALPHANUM>'}
However, when I try this:
trdd = sc.parallelize(['the quick brown fox'])
trdd.map(lambda x: es.indices.analyze(text=x)['tokens'][0]).collect()
I get a really really long error message related to pickling (Here's the end of it):
(self, obj) 109if'recursion'in.[0]: 110="""Could not pickle object as excessively deep recursion required."""--> 111 picklePicklingErrormsg
save_memoryviewself obj
: Could not pickle object as excessively deep recursion required.
raise.() 112 113def(,):PicklingError
I'm not sure what the error means. Am I doing something wrong? Is there a way to map the ES analyze function onto records of an RDD?
Edit: I'm also getting this behavior when applying other functions from elasticsearch-py as well (for example, es.termvector()).
回答1:
Essentially the Elasticsearch client is not serializable. So what you need to do is create an instance of the client for each partition, and process them:
def get_tokens(part):
es = Elasticsearch()
yield [es.indices.analyze(text=x)['tokens'][0] for x in part]
rdd = sc.parallelize([['the quick brown fox'], ['brown quick dog']], numSlices=2)
rdd.mapPartitions(lambda p: get_tokens(p)).collect()
Should give the following result:
Out[17]:
[[{u'end_offset': 3,
u'position': 1,
u'start_offset': 0,
u'token': u'the',
u'type': u'<ALPHANUM>'}],
[{u'end_offset': 5,
u'position': 1,
u'start_offset': 0,
u'token': u'brown',
u'type': u'<ALPHANUM>'}]]
Note that for large data sets, this is going to be very inefficient as it involves a REST call to ES for each element in the dataset.
来源：https://stackoverflow.com/questions/32161865/elasticsearch-analyze-not-compatible-with-spark-in-python