Deploy Apache Spark application from another application in Java, best practice

Question

I am a new user of Spark. I have a web service that allows a user to request the server to perform a complex data analysis by reading from a database and pushing the results bac

Answer 1

I've had a similar requirement. Here's what I did:

1. To submit apps, I use the hidden Spark REST Submission API: http://arturmkrtchyan.com/apache-spark-hidden-rest-api

2. Using this same API you can query status for a Driver or you can Kill your Job later

3. There's also another hidden UI Json API: http://[master-node]:[master-ui-port]/json/ which exposes all information available on the master UI in JSON format.

Using "Submission API" I submit a driver and using the "Master UI API" I wait until my Driver and App state are RUNNING