I have an external API that I want to call for multiple requests. The API has set up limits of service and there could be done only 10 queries per minute. But in my app there are situations that I need to create more than 50 requests at the same time. I mean I know that I need to call that API but I don’t need to call it immediately.
What I currently have
- I created Work Queue for API requests. The queue ensures error handling which I cannot achieve through classic workflow and in addition, it works asynchronously so the user can still use application without need to wait for the results.
- The WorkQueue is modelled in the most simple way - I create entity ApiCall and the queue reacts on RecordCreated event to add new query into queue and automatically start processing.
What I would need
I would need to adjust model that there would be api calls added into queue based on certain schedule. For example schedule each call of the same API for every 6 seconds.
I am new into WorkQueues, so I don’t know how the state management works and what needs to be done to achieve this. I might also add Wait task into Workflow but I am not sure if it would not block the app capacity. Thank you for your advices.
In essence each work queue item is independent and is to be processed independently (queue tasks always react on each queue entry). What you ask for is some kind of aggregation of events, which the queue does not support in any built in way.
Without additional development I would suggest to use a scheduler scenario, so even without queues you would just process all unprocessed records every 6 minutes.
Another option would be to include the aggregation to the queues and that would go to queue development ideas which we have some and we want to work on queues quite a lot in the next builds (e.g. see Work Queues - Support for Retry or Work Queue Groups).
One way would be to support batch queue auto processing so instead of 1 entry the workflow would get multiple queue entries to process. It would also need some scheduling mechanism (6x per minute), now it gets executed as soon as the queue entry appears. Of course the question is what would happen if one of the items would fail processing, then the whole batch has to fail and all the queue items have to move to the error queue for example.
thank you for your response. What I was actually thinking was that I could generate new “ApiCalls” that would contain some kind of scheduling such as field that would contain expected time to schedule. And when the condition would be applicable, the entry would be scheduled into the queue. So instead of adding it as RecordCreated, add the new record based on the condition with comparison with current date. But I don’t know if this is realizable or it would be just a dead end.
Scheduler scenario - does the scheduler already work in html5 version. So can I configure scheduler that would run workflow? And when scheduler runs workflow, how it is with error handling? I started to use WorkQueue because I could not read error message from API in workflow. Here in the community I have found this post End of iteration in for-each block due to error - #2 by urbanekv where I got the inspiration to use Work Queues for handling every API request. It was some additional work but it works great regarding error handling because I can see what result I received from API especially in case of API’s error message. Otherwise (without queue) I could only continue in workflow on “Finished” transition but I could not see and work with the error message.
So regarding the scheduler scenario, before I give it a try, I would like to know that scheduler is now working and how can I treat with exceptions? Can I use only scheduler scenario or should I combine the two approaches?
- Scheduler would create ApiCall entity based on some other entity or condition (or would change the state of ApiCall entity).
- WorkQueue would add ApiCall entity based on state condition or as RecordCreated if I would not be able to use states. (I don’t have the experience yet )
BTW: Regarding Work Queues - Support for Retry - In our .NET projects which are not based on Origam, we use Hangfire scheduling component which in code works very well including scheduling and enqueing. It also has retry implemented and it could be configurable as it states in the requirement. If you are interested (we already talked about it), I can show you how we use it on our projects.
Yes and I think in your case that is desirable. That is the only way how to make the queue making 6 seconds delay between the entries. But currently it actually stops all the queues as we only have one thread that processes all the queues as you can see in the request Workqueue - implement round-robin processing. So not a big win for you anyway.
We have many thoughts on queue improvements and we should really start a topic on this. I will do it.
EDIT: Actually there is one: Work Queue Improvements. We should start putting the ideas there.