I'm exploring the the SCDF for orchestrating the existing batch task for my system. I noticed that the SC Stream App Starters already provides some OOTB ready to use streaming applications, which suits my use case of:
- Extracting xml file from FTP -> the ftp source
- Customize transformation
- Upload the transformed xml file to another FTP -> the ftp sink
But, my system only requires the above mentioned batch job to be executed at every 10 minutes interval. Implementing it using streams app is fine but it requires the stream app to be running 24-7 (using SCDF with Kubernetes). My concern is that I have 20+ of the above batch tasks, and each of the stream app will needs to be "running" to get triggered at every 10 minutes interval.
After reading on the Spring Cloud Task, I noticed that it better suits my usage, which does not require the stream apps to be constantly running (short-lived). But, it does not have the OOTB ready to use task apps that I need: ftp-source and ftp-sink.
Besides that, using Spring Cloud Task does not allow me to pass the XML file from previous task app to the next one. It seems like it is a code smell to pass data from task to task according to comment in this question.
Then, in order to implement the above use case, task-app-1 will download the xml file and output to some storage, then task-app-2 pick up the xml file from the storage and perform customize transformation and upload to another storage. Then, task-app-3 picks up the xml file and upload to ftp. Will this be appropriate??
Besides that, I also have the idea of registering multiple streams with the above use case and trigger the deployment of the stream through external source (SCDF REST API)? Will this defeat the purpose of existing task schedule feature?
Anyone can advise on this?
For your case, as you learnt you need to create a set of batch jobs that perform your requirement. You can check this example as a reference. Also, you can potentially use Composed Task Runner to drive this specific case. For more information on Composed Tasks, you can refer to the documentation and an example here
来源：https://stackoverflow.com/questions/54124680/spring-cloud-stream-app-vs-spring-cloud-task-app