Incidents reported on status page for Latitude https://status.latitude.so/ https://d1lppblt9t2x15.cloudfront.net/logos/c7afe3c41f6722fca005cc9c9be14f07.png https://status.latitude.so/ en https://status.latitude.so/ Fri, 12 Sep 2025 21:43:38 +0000 https://status.latitude.so/#c5cf52acda37cd58e6c0197af186e3b079a09ab9e1813b266edd8b1b26ecaf4c app.latitude.so/api/health recovered https://status.latitude.so/ Fri, 12 Sep 2025 21:34:37 +0000 https://status.latitude.so/#c5cf52acda37cd58e6c0197af186e3b079a09ab9e1813b266edd8b1b26ecaf4c app.latitude.so/api/health went down https://status.latitude.so/ Fri, 12 Sep 2025 21:27:58 +0000 https://status.latitude.so/#0ac5b2d11d2964c2c5ac02b20b93ba870f45f213c6fa5960c05b75f15dcd749b workers.latitude.so/health recovered https://status.latitude.so/ Fri, 12 Sep 2025 21:03:46 +0000 https://status.latitude.so/#0ac5b2d11d2964c2c5ac02b20b93ba870f45f213c6fa5960c05b75f15dcd749b workers.latitude.so/health went down https://status.latitude.so/ Thu, 21 Aug 2025 17:12:21 +0000 https://status.latitude.so/#20ca9cb9a1cfa7e4bc88963b1249fdbfc063c952f4ef7fe539937043e1930ee9 gateway.latitude.so/health recovered https://status.latitude.so/ Thu, 21 Aug 2025 17:11:20 +0000 https://status.latitude.so/#20ca9cb9a1cfa7e4bc88963b1249fdbfc063c952f4ef7fe539937043e1930ee9 gateway.latitude.so/health went down https://status.latitude.so/ Thu, 21 Aug 2025 16:58:17 +0000 https://status.latitude.so/#8096cffda1efceb2e029a870f98eca7256dca629003113657b7a0b43774c21f6 gateway.latitude.so/health recovered https://status.latitude.so/ Thu, 21 Aug 2025 16:56:12 +0000 https://status.latitude.so/#8096cffda1efceb2e029a870f98eca7256dca629003113657b7a0b43774c21f6 gateway.latitude.so/health went down https://status.latitude.so/ Thu, 21 Aug 2025 14:19:16 +0000 https://status.latitude.so/#1cb9d1e2d2aec92bc7366465127535b4e0208175be4233b1dbe65fc4f9af8bec gateway.latitude.so/health recovered https://status.latitude.so/ Thu, 21 Aug 2025 14:17:13 +0000 https://status.latitude.so/#1cb9d1e2d2aec92bc7366465127535b4e0208175be4233b1dbe65fc4f9af8bec gateway.latitude.so/health went down https://status.latitude.so/ Thu, 21 Aug 2025 13:38:21 +0000 https://status.latitude.so/#ec6f210f6813492b533b2d257865af1ad422c48c195cf6d4480430709d8e43b7 gateway.latitude.so/health recovered https://status.latitude.so/ Thu, 21 Aug 2025 13:35:03 +0000 https://status.latitude.so/#ec6f210f6813492b533b2d257865af1ad422c48c195cf6d4480430709d8e43b7 gateway.latitude.so/health went down https://status.latitude.so/ Thu, 14 Aug 2025 15:05:51 +0000 https://status.latitude.so/#25889ee0e37a8b9dcb7680e7771832562e3f3702815c275775c5203405b329af workers.latitude.so/health recovered https://status.latitude.so/ Thu, 14 Aug 2025 15:03:51 +0000 https://status.latitude.so/#25889ee0e37a8b9dcb7680e7771832562e3f3702815c275775c5203405b329af workers.latitude.so/health went down https://status.latitude.so/ Fri, 08 Aug 2025 09:32:49 +0000 https://status.latitude.so/#71f2e6ffccbf89dfbaa57d70621748a5cb30b117704a2ad3fd3301bb66d52209 workers.latitude.so/health recovered https://status.latitude.so/ Fri, 08 Aug 2025 09:30:46 +0000 https://status.latitude.so/#71f2e6ffccbf89dfbaa57d70621748a5cb30b117704a2ad3fd3301bb66d52209 workers.latitude.so/health went down https://status.latitude.so/ Tue, 29 Jul 2025 09:01:17 +0000 https://status.latitude.so/#33ef1e5bec360b1f266b661070ca8c28c6dd307ecddefc1ef5e730e500e422cf workers.latitude.so/health recovered https://status.latitude.so/ Tue, 29 Jul 2025 08:58:26 +0000 https://status.latitude.so/#33ef1e5bec360b1f266b661070ca8c28c6dd307ecddefc1ef5e730e500e422cf workers.latitude.so/health went down https://status.latitude.so/ Tue, 29 Jul 2025 07:19:05 +0000 https://status.latitude.so/#ef040acd075f519ae6efc630ff1c91ee749aa90bac36905dc764d0ced5bfdb49 workers.latitude.so/health recovered https://status.latitude.so/ Tue, 29 Jul 2025 07:13:05 +0000 https://status.latitude.so/#ef040acd075f519ae6efc630ff1c91ee749aa90bac36905dc764d0ced5bfdb49 workers.latitude.so/health went down https://status.latitude.so/ Sun, 22 Jun 2025 10:52:03 +0000 https://status.latitude.so/#aeaf1c673ea7844e38e7e736490f62242d90cd49223a1155cfbcabe0cef83174 workers.latitude.so/health recovered https://status.latitude.so/ Sun, 22 Jun 2025 10:49:10 +0000 https://status.latitude.so/#aeaf1c673ea7844e38e7e736490f62242d90cd49223a1155cfbcabe0cef83174 workers.latitude.so/health went down https://status.latitude.so/incident/594759 Sat, 31 May 2025 18:06:00 -0000 https://status.latitude.so/incident/594759#4cb588adf7d9330510f4921f14adc35d7781bacc4981ea0117cb773eb763ffda ## Incident Summary On 30 May 2025 at 18:22 CEST, the Gateway proxy experienced an outage that lasted two hours and twenty-two minutes. During this interval all customer traffic routed through the proxy failed, resulting in a complete loss of service. ## Root Cause The incident was triggered by an unexpected surge in incoming requests that drove a corresponding spike in outbound calls to external LLM providers, primarily OpenAI. Although the autoscaler immediately began launching additional Gateway instances, the increased egress traffic exhausted the outbound bandwidth of the NAT Gateway that serves the cluster. Each new instance had to download its Docker image through the same congested link, causing image pulls that normally complete in under a minute to stretch beyond twenty minutes. Because capacity scaled up so slowly, the existing Gateway instances became overwhelmed and unhealthy, which led to a total shutdown of the service. ## Resolution Once engineers confirmed that NAT egress saturation was the bottleneck, they first detached the Gateway service from incoming traffic to prevent further failures. They then replaced the overloaded NAT Gateway with a higher-capacity unit and started a minimal set of fresh Gateway instances behind the new network layer. After each instance passed its health checks, traffic was gradually reintroduced. Full functionality returned two hours and twenty-two minutes after the initial failure. ## Preventive Measures The NAT Gateway is now part of an autoscaling group so network capacity can expand automatically with demand. New monitoring thresholds generate alerts whenever NAT egress approaches saturation, giving operators early warning before user impact occurs. In addition, the baseline number of Gateway instances has been increased to provide greater headroom during sudden traffic spikes. We apologise for the disruption and remain committed to strengthening our infrastructure to prevent similar incidents in the future. https://status.latitude.so/incident/582904 Sat, 31 May 2025 09:15:00 -0000 https://status.latitude.so/incident/582904#acd736369ad51ed96139f510bde5f8df47275db7900010720f9ebc63a14a03d2 Maintenance completed https://status.latitude.so/incident/582904 Sat, 31 May 2025 09:15:00 -0000 https://status.latitude.so/incident/582904#acd736369ad51ed96139f510bde5f8df47275db7900010720f9ebc63a14a03d2 Maintenance completed https://status.latitude.so/incident/582904 Sat, 31 May 2025 09:15:00 -0000 https://status.latitude.so/incident/582904#acd736369ad51ed96139f510bde5f8df47275db7900010720f9ebc63a14a03d2 Maintenance completed https://status.latitude.so/incident/582904 Sat, 31 May 2025 08:00:00 -0000 https://status.latitude.so/incident/582904#6e30c55ac28bbe3d282542cd50bc2a759d109994710915bea5be4091650e66fa We are going to upgrade our main database and cache cluster instances with higher cpu and memory in order to accomodate capacity from new customers. This change will cause downtime of some minutes. https://status.latitude.so/incident/582904 Sat, 31 May 2025 08:00:00 -0000 https://status.latitude.so/incident/582904#6e30c55ac28bbe3d282542cd50bc2a759d109994710915bea5be4091650e66fa We are going to upgrade our main database and cache cluster instances with higher cpu and memory in order to accomodate capacity from new customers. This change will cause downtime of some minutes. https://status.latitude.so/incident/582904 Sat, 31 May 2025 08:00:00 -0000 https://status.latitude.so/incident/582904#6e30c55ac28bbe3d282542cd50bc2a759d109994710915bea5be4091650e66fa We are going to upgrade our main database and cache cluster instances with higher cpu and memory in order to accomodate capacity from new customers. This change will cause downtime of some minutes.