I have an external API that I want to call for multiple requests. The API has set up limits of service and there could be done only 10 queries per minute. But in my app there are situations that I need to create more than 50 requests at the same time. I mean I know that I need to call that API but I don’t need to call it immediately.
What I currently have
I created Work Queue for API requests. The queue ensures error handling which I cannot achieve through classic workflow and in addition, it works asynchronously so the user can still use application without need to wait for the results.
The WorkQueue is modelled in the most simple way - I create entity ApiCall and the queue reacts on RecordCreated event to add new query into queue and automatically start processing.
What I would need
I would need to adjust model that there would be api calls added into queue based on certain schedule. For example schedule each call of the same API for every 6 seconds.
I am new into WorkQueues, so I don’t know how the state management works and what needs to be done to achieve this. I might also add Wait task into Workflow but I am not sure if it would not block the app capacity. Thank you for your advices.
In essence each work queue item is independent and is to be processed independently (queue tasks always react on each queue entry). What you ask for is some kind of aggregation of events, which the queue does not support in any built in way.
Without additional development I would suggest to use a scheduler scenario, so even without queues you would just process all unprocessed records every 6 minutes.
Another option would be to include the aggregation to the queues and that would go to queue development ideas which we have some and we want to work on queues quite a lot in the next builds (e.g. see Work Queues - Support for Retry or Work Queue Groups).
One way would be to support batch queue auto processing so instead of 1 entry the workflow would get multiple queue entries to process. It would also need some scheduling mechanism (6x per minute), now it gets executed as soon as the queue entry appears. Of course the question is what would happen if one of the items would fail processing, then the whole batch has to fail and all the queue items have to move to the error queue for example.
thank you for your response. What I was actually thinking was that I could generate new “ApiCalls” that would contain some kind of scheduling such as field that would contain expected time to schedule. And when the condition would be applicable, the entry would be scheduled into the queue. So instead of adding it as RecordCreated, add the new record based on the condition with comparison with current date. But I don’t know if this is realizable or it would be just a dead end.
Scheduler scenario - does the scheduler already work in html5 version. So can I configure scheduler that would run workflow? And when scheduler runs workflow, how it is with error handling? I started to use WorkQueue because I could not read error message from API in workflow. Here in the community I have found this post End of iteration in for-each block due to error - #2 by urbanekv where I got the inspiration to use Work Queues for handling every API request. It was some additional work but it works great regarding error handling because I can see what result I received from API especially in case of API’s error message. Otherwise (without queue) I could only continue in workflow on “Finished” transition but I could not see and work with the error message.
So regarding the scheduler scenario, before I give it a try, I would like to know that scheduler is now working and how can I treat with exceptions? Can I use only scheduler scenario or should I combine the two approaches?
Scheduler would create ApiCall entity based on some other entity or condition (or would change the state of ApiCall entity).
WorkQueue would add ApiCall entity based on state condition or as RecordCreated if I would not be able to use states. (I don’t have the experience yet )
BTW: Regarding Work Queues - Support for Retry - In our .NET projects which are not based on Origam, we use Hangfire scheduling component which in code works very well including scheduling and enqueing. It also has retry implemented and it could be configurable as it states in the requirement. If you are interested (we already talked about it), I can show you how we use it on our projects.
I thought you can only call the API once in 6 minutes? That means after a call you have to wait 6 minutes before firing another one? Then your suggestion would not help as you would only delay calling each queue entry. Or maybe I don’t fully understand the scenario.
Scheduler – we currently did not convert the scheduler.exe as we have been experimenting with external schedulers (e.g. Azure Logic Apps or just cron). These would call an API we would expose. We are still thinking about the best approach.
As for Hangfire we looked it already and first of all it is not fully open sourced and so that’s the first problem. Also we are not sure if we should include such a huge functionality that many parts of ORIGAM would depend on. You know we plan for 10-20 years ahead and so we have to be cautious with including huge chunks of someone else’s code.
No, that is missunderstanding, probably my description was not clear. The API allows me “10 requests in a rolling 60 second window” so they suggest to call API once in 6 seconds interval which satisfies the limit. So there is no aggregation needed, only 6 seconds delay between each call.
I generate ± 50 requests at once but I would like to ensure that they would be “consumed” by queue in 6 seconds intervals. Or that the call would wait …
Queue than starts workflow that calls API and process results. I might put Wait task for 6 seconds here … But … Would the Wait task mean that it would stop the whole queue or would the queue continue with another entry? I currently use one queue for multiple different API Calls so I would rather not block the queue. But … I can create another queue for blocking tasks, obviously.
But I thought it could be more efficient to set schedule time on creation and load ApiCall into WorkQueue based on certain condition (state?)
Ok, so scheduler scenario is not usable at this moment.
Great. I understand and good to hear about 10 - 20 years plan :-). I know there is also similar component https://www.quartz-scheduler.net which is licensed under Apache License v2.0. May be have a look at it as well. In our solutions I quite like when the scheduling functionality is embedded in app because it is easy for deployment and updates. You don’t need to configure anything somewhere outside the application and you depend only on its own functionality and you can deploy it anywhere wihtout any other configuration (except to ensure that the app is always working and don’t go asleep).
Yes and I think in your case that is desirable. That is the only way how to make the queue making 6 seconds delay between the entries. But currently it actually stops all the queues as we only have one thread that processes all the queues as you can see in the request Workqueue - implement round-robin processing. So not a big win for you anyway.
We have many thoughts on queue improvements and we should really start a topic on this. I will do it.
EDIT: Actually there is one: Work Queue Improvements. We should start putting the ideas there.
)