Enterprise-level deployment of LLM inference gateways: We reduced the 429 error rate from 13% to 0, and also saved 28% in costs.

Last month during the Black Friday promotion, three members of our cross-border e-commerce technology team in Singapore stayed in the monitoring room for 36 hours non-stop. They were closely watching the curve showing the request success rate, which was so high that it almost seemed to jump off the screen. Last year, during the same major promotional period, we faced issues due to the rate limiting imposed by a single LLM (Large Language Model) service provider.13% of the product description generation requests result in a 429 error.The merchant's backend has been down for almost 4 hours, and they have received more than 2,000 complaints from customers alone.

First, let's understand what an enterprise-level deployment of a reasoning gateway actually is.

It's not some fancy new framework; it's essentially a traffic routing layer that sits between your business services and the various LLM interfaces. The core parameters and anchors are quite simple:Under normal load conditions, the scheduling delay must not exceed 10 milliseconds.If it exceeds the limit, it will only hold back the business.

Three tangible benefits we've summarized after encountering various challenges:

• Firstly, we completely eliminated the risk of throttling: We are now connecting to three major LLM service providers simultaneously, and the gateway automatically distributes requests to the node with the current sufficient quota and the fastest response time. This year, during the peak period of Black Friday, the QPS increased by 2.3 times compared to last year, and there was not a single instance of a batch 429 error throughout the process.
• Secondly, there is a visible reduction in costs: We automatically route requests for generating product labels with low priority to smaller models that offer better cost-performance ratios, without having to modify the business code. As a result, we have saved 28% on token costs in just 7 days.
• Finally, it saves the backend from having to repeat the same work: before, every time we switched models or added a new service provider, we had to individually modify the interfaces of three business modules. Now, all the configurations are done at the gateway level, and it can be done in just half a day.

Don't just look at the benefits; we've definitely stumbled into these 3 pitfalls.

We had an accident just in the first week of going live: After enabling the automatic fallback feature, the gateway directed a batch of high-priority customized copywriting requests that were supposed to use GPT-4 to a less effective smaller model. As a result, more than 200 merchant advertising copywriting pieces were deemed unsuitable, and we ended up compensating with promotional vouchers worth just under ten thousand yuan. It was only later that we realized that not all requests are suitable for automatic downgrading; in highly sensitive scenarios, additional fallback validation rules must be implemented.

There are also many cost-related issues that haven't been mentioned: if your QPS (Queries Per Second) remains below 10 for an extended period, the server costs and maintenance expenses for the gateway itself will be higher than the cost of purchasing a higher-tier quota package from an LLM (Large Language Model) service provider. In that case, it's simply not necessary to proceed with this approach.

Also, don’t believe the claims of “full functionality out of the box.” We tried three open-source gateways, and the default traffic distribution rules didn’t suit the e-commerce scenario at all. It took us a whole two days just to adjust the weight rules.

Who should go? Who really doesn’t need to waste time at all?

Provide the criteria directly without getting entangled in details:

• The following criteria can be considered for consideration: daily LLM request volume exceeds 10,000 times, using more than 2 models simultaneously, and requiring service availability to be higher than 99.9%;
• If your team is a small startup with fewer than 5 people, your core business is not heavily dependent on large models, and the monthly cost of using LLMs is not even as high as the salary of a backend engineer, it's not worth considering enterprise-level deployment. It's more cost-effective to use the native interfaces provided by service providers directly.

2 Practical Tips for First-Time Users

Don't make a full switch right away; start with 10% of the low-priority traffic for a week of grayscale testing. Focus on two key indicators: whether there are any instances of duplicate requests, and whether the delays caused by scheduling are truly within an acceptable range. We did the same thing by testing the traffic for automatic after-sales responses for three days to ensure there were no issues before switching to the core business.

Question: Should I build the gateway from scratch? Answer: Unless there are people in your team who are particularly idle, it’s much quicker to use a mature open-source version and modify it – it will save you at least two months of work, and the stability will also be higher.