How to design an execution engine for a sequence of tasks

Question

I am trying to code a problem in Java where I have to execute a bunch of tasks.

Problem

Execute a job which consists of multiple tasks and thos

Answer 1

This question is very loaded. I can see at least three different subsystems in your design:

• task DAG (partial completion to me just means you actually have 2 nodes instead of 1), which can be persisted in a database for example;
• work queue, where the next task to execute must be enqueued (this could be some kind of message queue, again for persistence/transactionality);
• the actual execution framework, which might involve accessing external resources like services/databases and can potentially be distributed.

Your description is very high level so you could start designing the abstractions you need for these three pieces and how they interact with each other (in terms of interfaces).

Once you have all of that, you can start providing some simple implementations. I would start from a "local mode", using a simple in memory DAG, a blocking queue and some type of java executor.

Your question does not provide details on SLA, length of jobs, failure/retry policies, transactions, etc., so it's hard to say how your modules should be implemented. But I suggest to think in terms of high level abstractions and iterate on the implementations. Great code will never fix bad design.

You could stop there, or start replacing each implementation with third party products if you need to.