task-queue

I am currently designing a web application that will allow users to schedule tasks which will be executed against an HTTP API (on behalf of them). The tasks can be recurring and the minimal time resolution that can be used for scheduling will be one minute. Because of the nature of the tasks I think it makes sense to execute them asynchronously. However, how should the architecture of this part look like? I thought about using a task queue to create tasks by the web application and let them be executed by a worker. In this case, I have several questions: How do I handle recurring tasks? How do

is there a way to determine when a set of Google App Engine tasks (and child tasks they spawn) have all completed? Let's say that I have 100 tasks to execute and 10 of those spawn 10 child tasks each. That's 200 tasks. Let's also say that those child tasks might spawn more tasks, recursively, etc... Is there a way to determine when all tasks have completed? I tried using the app engine pipeline API, but it doesn't look like it's going to work out for my particular use case, even though it is a great API. My use case is that I want to make a whole bunch of rate limited URL fetch calls while

I'm putting several tasks into a task queue and would like to know when the specific tasks are done. I haven't found anything in the API about call backs, or checking the status of a task, so I thought I'd see what other people do, or if there's a work around (or official) way to check. I don't care about individual tasks, if it helps, I'm putting 6 different tasks in, and want to know when all 6 are complete. Thanks! The new REST/JSON task queue API will let you do this. http://code.google.com/appengine/docs/python/taskqueue/rest.html This does not scale well to thousands of tasks... I do

I have several actions that I want to execute in the background, but they have to be executed synchronously one after the other. I was wondering if it's a good idea to use the Task.ContinueWith method to achieve this. Do you foresee any problems with this? My code looks something like this: private object syncRoot =new object(); private Task latestTask; public void EnqueueAction(System.Action action) { lock (syncRoot) { if (latestTask == null) latestTask = Task.Factory.StartNew(action); else latestTask = latestTask.ContinueWith(tsk => action()); } } This should work as designed (using the fact

What is a good aproach to handle background processes in a NodeJS application? Scenario : After a user posts something to an app I want to crunch the data, request additional data from external resources, etc. All of this is quite time consuming, so I want it out of the req/res loop. Ideal would be to just have a queue of jobs where you can quickly dump a job on and a daemon or task runner will always take the oldest one and process it. In RoR I would have done it with something like Delayed Job. What is the Node equivalent of this API? If you want something lightweight, that runs in the same

I would like to build a priority queue in python in which the queue contains different dictionaries with their priority numbers. So when a "get function" is called, the dictionary with the highest priority(lowest number) will be pulled out of the queue and when "add function" is called, the new dictionary will be added to the queue and sorted based on its priority number. Please do help out... Thanks in advance! Use the heapq module in the standard library. You don't specify how you wanted to associate priorities with dictionaries, but here's a simple implementation: import heapq class