How to classify images using Spark and Caffe

Question

I am using Caffe to do image classification, can I am using MAC OS X, Pyhton.

Right now I know how to classify a list of images using Caffe with Spark python, but if

Answer 1

When you work with complex, non-native objects initialization has to moved directly to the workers for example with singleton module:

net_builder.py:

import cafe 

net = None

def build_net(*args, **kwargs):
     ...  # Initialize net here
     return net       

def get_net(*args, **kwargs):
    global net
    if net is None:
        net = build_net(*args, **kwargs)
    return net

main.py:

import net_builder

sc.addPyFile("net_builder.py")

def classify_image(image_path, transformer, *args, **kwargs):
    net = net_builder.get_net(*args, **kwargs)

It means you'll have to distribute all required files as well. It can be done either manually or using SparkFiles mechanism.

On a side note you should take a look at the SparkNet package.