throttling webhooks

[pricerc]

we're dealing with what I suspect is "fair use" throttling from a hosting provider whenever there are bulk updates being pushed out through web hooks to a WooCommerce site.

we've just got subscriptions on inventory.updated, inventory.stocklevel, and inventory.created, but it seems if we have more than about a dozen at once, say after importing a new range of products,, the performance drops off rapidly on the hosted REST API.

We're angling to move the customer to a better provider, but in the mean time, is there a way to throttle outbound webhook responses, so that we only send, say, one every 5 seconds? At the very least, I'd like to validate my hunch.

[SBarnes]

Hi Ryan,

There is a system setting on the REST api called WebhooksRetryInterval which you could try increasing although I don't think this is going to help you much as the first try doesn't take the interval into consideration.

It will be the stock level hook that is causing the problem due to every time stock levels in Jiwa change a message is sent out, I have seen this before but with Magento not handling it as well.

I have discussed this with Mike and I believe there is a dev ticket to offer an alterative solution, I did something with triggers on certain tables and a scheduled plugin to get around it but I have since thought of a simpler solution which is as follows, if you don't think you could do this yourself you could always have Jiwa quote you to do it.

1. Set up a table with an identity field as primary key as well as having the inventory id, logical warehouse field and an integer field to use as the status
2. Write a custom route to receive the web hook inventory change and have a subscription point at that and in that route write the data to the table setting the status to zero (unprocessed) and simply return http status ok so the hook is finished
3. In the same plugin set a timer and when the timer goes off(this might be every ten minutes say ) call a stored procedure that would set all unprocessed records to status processing (status to 1) and return a record set of the distinct inventory id and logical id where status equals processing
4. For each record in the record set send out the hook and after processing each record update the table for any records with that inventory id and logical id and status processing(value 1) to status value 2 (processed)

If you wanted to you could even work out the actual SOH value and send that as well to save WooCommerce needing to call in and get the SOH value but what I have described above would say for a certain combination of inventory id and logical id where there might be twenty records in the table only send it once, and setting a timer inside the rest api should work and be fine as that's already in use by web hooks itself for example.

Mike or Scott might be able to offer you a better solution but I don't believe a solution exists at the moment.

[Mike.Sheen]

updates being pushed out through web hooks to a WooCommerce site.

Maybe don't use webhooks at all?

The aim no doubt is to keep the web store up to date with new products, changes to products and stock levels.

I'd change that to be a regular update once a day in the dead of the night instead. Is it really that important for new products to appear immediately on the web store? Employing tactics like Stuart mentioned, you can somewhat limit the volume of the regular update to just known changes.

As for stock levels, if sales orders are being pushed into Jiwa via the Jiwa REST API from the web store, then after each order is sent to Jiwa, at that point you can query the Jiwa API for the new stock levels of just the items on the order - so the web store has a resonable degree of accuracy as to the stock levels at all times. I'd still also have a daily SOH level update of all products, however.

[SBarnes]

Along a similar line to what Mike is advising it all comes down to granularity of the information, in other words how up to date or accurate do you need things to be, how fine grained, for instance it can depend on the product range, are these products that when sold out are never repeated or are they things that more can be ordered of in that case SOH isn't that relevant in the former it is.

Quite often I found the answer is a combination of web hooks and Mike's in the dead of the night approach, you have to consider accuracy vs volume of information that's going to need to be processed.

[pricerc]

... Is it really that important for new products to appear immediately on the web store?

Apparently, yes. That's why I got a phone call last night.

A certificate that needed renewing meant that updates had stalled for a few days.

They only noticed after advertising their cool new product on Facebook, only to be informed by potential customers that it wasn't there....

After sorting out the certificate problem, there was naturally a significant backlog...

We've tried a few different approaches, and this one mostly works well. Except when we get what appears to be throttling when this kind of thing happens.

We've got a few longer-term plans to improve things, I was just hoping for a) an interim measure and b) a troubleshooting step that could rule out a possible cause.

In particular, I'd rather not waste days working on a solution to the wrong problem.

[pricerc]

Hi Ryan,

There is a system setting on the REST api called WebhooksRetryInterval which you could try increasing although I don't think this is going to help you much as the first try doesn't take the interval into consideration.

This doesn't change the interval between individual responses - only the interval between retries of the same message.

If you have 200 messages sitting at retry #2, they all get streamed out en masse. At least that's what seems to be happening.

[pricerc]

1. Set up a table with an identity field as primary key as well as having the inventory id, logical warehouse field and an integer field to use as the status
2. Write a custom route to receive the web hook inventory change and have a subscription point at that and in that route write the data to the table setting the status to zero (unprocessed) and simply return http status ok so the hook is finished
3. In the same plugin set a timer and when the timer goes off(this might be every ten minutes say ) call a stored procedure that would set all unprocessed records to status processing (status to 1) and return a record set of the distinct inventory id and logical id where status equals processing
4. For each record in the record set send out the hook and after processing each record update the table for any records with that inventory id and logical id and status processing(value 1) to status value 2 (processed)

we actually have a .NET 6 based 'proxy' running on a separate server between Jiwa and WooCommerce (rather than having the REST API exposed directly through a dodgy ISP-provided router) and we're planning to do something a bit like this in it. Except we'll probably keep the webhooks as the triggers.

But's it's not a 10 minute job. And the developer (not me) who has been (and will be) working on it has a full-time job somewhere else and recently became a first-time dad....

[SBarnes]

There is a system setting on the REST api called WebhooksRetryInterval which you could try increasing although I don't think this is going to help you much as the first try doesn't take the interval into consideration.

if you don't think you could do this yourself you could always have Jiwa quote you to do it

[pricerc]

There is a system setting on the REST api called WebhooksRetryInterval which you could try increasing although I don't think this is going to help you much as the first try doesn't take the interval into consideration.

I was agreeing with you

if you don't think you could do this yourself you could always have Jiwa quote you to do it

There are 'budgetry' constraints... it's a family business and the developer is part of the family...

And it's not urgent.

[Mike.Sheen]

Apparently, yes. That's why I got a phone call last night.

I was just hoping for a) an interim measure

You could look into employing something like Zapier to act as an intermediary and apply some kind of queueing to give you the load levelling you want.

I don't know if Zapier specifically does or does not have a queueing mechanism, but it might. There are heaps of other similar services.

Best case scenario is it might be a 10 minute job to direct all the webhooks to Zapier instead and have it relay it to the web store.

Worst case scenario is it will take you 3 days to work out that Zapier doesn't quite suit, and an unknown number of more days exploring similar services, so you may as well adapt the existing proxy you have between the Jiwa API and the webstore to relay webooks as well via a queuing mechanism.