Creating Orders

Between 14:00 - 15:15 UTC, users were experiencing delays in uploading orders to the system and accepting online orders. This was caused by a database migration script that exceeded the maintenance window. As a result, the database was under heavy maintenance work while the traffic started to increase. This meant uploading a single order was taking more than usual. The increased uploading times affected the rate at which our system can handle processing concurrent orders. Which lead to partner systems & integrations receiving 429 (Retry request) responses. The maintenance was expected to be concluded between 11:00 and 12:00 UTC, before peak traffic. Our engineering team underestimated one maintenance task for when it runs on accounts with a large number of orders. We have identified the task and resolved the issue for future runs. We've also adjusted our maintenance window to account for similar underestimations if they happen in the future, and they shouldn't. At 15:15 UTC, our engineering team confirmed the issue was resolved and response times went back to normal. We apologize for any inconvenience this caused.

Created over 1 year ago