Incidents reported on status page for Latitude https://status.latitude.so/ https://d1lppblt9t2x15.cloudfront.net/logos/c7afe3c41f6722fca005cc9c9be14f07.png https://status.latitude.so/ en https://status.latitude.so/ Tue, 09 Dec 2025 16:38:52 +0000 https://status.latitude.so/#f5760fd0e9ff67aa59e1151c79ac3f610a4a5a1e028c6aafe7ad6a0115d0c8f7 app.latitude.so/api/health recovered https://status.latitude.so/ Tue, 09 Dec 2025 16:27:55 +0000 https://status.latitude.so/#f5760fd0e9ff67aa59e1151c79ac3f610a4a5a1e028c6aafe7ad6a0115d0c8f7 app.latitude.so/api/health went down https://status.latitude.so/ Tue, 02 Dec 2025 13:35:16 +0000 https://status.latitude.so/#abbf7fae1488a1c49827c88c1b2dbdac4013142add031b45e864222dad593784 app.latitude.so/api/health recovered https://status.latitude.so/ Tue, 02 Dec 2025 13:32:17 +0000 https://status.latitude.so/#abbf7fae1488a1c49827c88c1b2dbdac4013142add031b45e864222dad593784 app.latitude.so/api/health went down https://status.latitude.so/ Mon, 01 Dec 2025 14:20:57 +0000 https://status.latitude.so/#babc7db3200520d1ede88d705f4990e7d63fcafa21c0c56ccd0b226ee6041952 workers.latitude.so/health recovered https://status.latitude.so/ Mon, 01 Dec 2025 14:18:40 +0000 https://status.latitude.so/#babc7db3200520d1ede88d705f4990e7d63fcafa21c0c56ccd0b226ee6041952 workers.latitude.so/health went down https://status.latitude.so/ Thu, 27 Nov 2025 11:17:00 +0000 https://status.latitude.so/#ec5e2e619ad2df310c3e23370f07dcd4c83e0a8c19ba9a51123046f9d05a5c4a workers.latitude.so/health recovered https://status.latitude.so/ Thu, 27 Nov 2025 11:14:17 +0000 https://status.latitude.so/#ec5e2e619ad2df310c3e23370f07dcd4c83e0a8c19ba9a51123046f9d05a5c4a workers.latitude.so/health went down https://status.latitude.so/ Tue, 25 Nov 2025 14:42:45 +0000 https://status.latitude.so/#9963e6f25c80ad91e647bd45c07514d51ce062d06b07f4e6cb79d6cd47e65872 app.latitude.so/api/health recovered https://status.latitude.so/ Tue, 25 Nov 2025 14:39:27 +0000 https://status.latitude.so/#9963e6f25c80ad91e647bd45c07514d51ce062d06b07f4e6cb79d6cd47e65872 app.latitude.so/api/health went down https://status.latitude.so/ Thu, 20 Nov 2025 20:44:22 +0000 https://status.latitude.so/#2ac461441cddc297b9345cde4f6ca989c29bc6d049542e4e7e99f72ab93338e4 app.latitude.so/api/health recovered https://status.latitude.so/ Thu, 20 Nov 2025 20:41:41 +0000 https://status.latitude.so/#2ac461441cddc297b9345cde4f6ca989c29bc6d049542e4e7e99f72ab93338e4 app.latitude.so/api/health went down https://status.latitude.so/ Thu, 20 Nov 2025 20:26:22 +0000 https://status.latitude.so/#277f83849f88991922351297d8d5231b873110c61cb6792a2f08a00fda473f8d app.latitude.so/api/health recovered https://status.latitude.so/ Thu, 20 Nov 2025 20:23:22 +0000 https://status.latitude.so/#277f83849f88991922351297d8d5231b873110c61cb6792a2f08a00fda473f8d app.latitude.so/api/health went down https://status.latitude.so/ Thu, 20 Nov 2025 19:26:20 +0000 https://status.latitude.so/#851eb9338fd5b0458bdcc886d5195543994b098289b8c51b9658c44b494e09c1 app.latitude.so/api/health recovered https://status.latitude.so/ Thu, 20 Nov 2025 19:17:22 +0000 https://status.latitude.so/#851eb9338fd5b0458bdcc886d5195543994b098289b8c51b9658c44b494e09c1 app.latitude.so/api/health went down https://status.latitude.so/ Thu, 20 Nov 2025 17:18:23 +0000 https://status.latitude.so/#ed78049d540c66bf043ceafc491f7de0b110d74147f31d6f50e83438d81963a6 app.latitude.so/api/health recovered https://status.latitude.so/ Thu, 20 Nov 2025 17:14:20 +0000 https://status.latitude.so/#ed78049d540c66bf043ceafc491f7de0b110d74147f31d6f50e83438d81963a6 app.latitude.so/api/health went down https://status.latitude.so/ Thu, 20 Nov 2025 13:41:47 +0000 https://status.latitude.so/#d224f07d58bdf4e9caef5b8b6fc78c78513b1210457d38a813d207ef842278b5 app.latitude.so/api/health recovered https://status.latitude.so/ Thu, 20 Nov 2025 13:37:06 +0000 https://status.latitude.so/#d224f07d58bdf4e9caef5b8b6fc78c78513b1210457d38a813d207ef842278b5 app.latitude.so/api/health went down https://status.latitude.so/ Mon, 17 Nov 2025 14:38:05 +0000 https://status.latitude.so/#864c9991c59f33e10ec0e6f5acb34a061003c8d79cecbc9cf1e221c6a7ab61d9 app.latitude.so/api/health recovered https://status.latitude.so/ Mon, 17 Nov 2025 14:32:05 +0000 https://status.latitude.so/#864c9991c59f33e10ec0e6f5acb34a061003c8d79cecbc9cf1e221c6a7ab61d9 app.latitude.so/api/health went down https://status.latitude.so/ Wed, 12 Nov 2025 10:27:27 +0000 https://status.latitude.so/#58e0c2fe7807cfd72edb9d40db6968d48f599a2b631254702d5bdb9ad8e3b8aa workers.latitude.so/health recovered https://status.latitude.so/ Wed, 12 Nov 2025 10:24:41 +0000 https://status.latitude.so/#58e0c2fe7807cfd72edb9d40db6968d48f599a2b631254702d5bdb9ad8e3b8aa workers.latitude.so/health went down https://status.latitude.so/ Wed, 05 Nov 2025 02:37:15 +0000 https://status.latitude.so/#e65ff13413bc9b7afad90bc2f29dee53d1478b88dd8ac6c3065c2a6ef2206a06 workers.latitude.so/health recovered https://status.latitude.so/ Wed, 05 Nov 2025 02:31:15 +0000 https://status.latitude.so/#e65ff13413bc9b7afad90bc2f29dee53d1478b88dd8ac6c3065c2a6ef2206a06 workers.latitude.so/health went down https://status.latitude.so/ Mon, 03 Nov 2025 21:18:36 +0000 https://status.latitude.so/#29be6cbae06ce8e93d2d0d3596f51c5b90115873ba2d0b50824ff2b628ba44be workers.latitude.so/health recovered https://status.latitude.so/ Mon, 03 Nov 2025 21:12:46 +0000 https://status.latitude.so/#29be6cbae06ce8e93d2d0d3596f51c5b90115873ba2d0b50824ff2b628ba44be workers.latitude.so/health went down https://status.latitude.so/ Wed, 24 Sep 2025 06:20:20 +0000 https://status.latitude.so/#972389ec43c7b52d0399255c910080ed59b1c3b795aa4849e3a4aa3102c22f68 workers.latitude.so/health recovered https://status.latitude.so/ Wed, 24 Sep 2025 06:17:21 +0000 https://status.latitude.so/#972389ec43c7b52d0399255c910080ed59b1c3b795aa4849e3a4aa3102c22f68 workers.latitude.so/health went down https://status.latitude.so/ Tue, 23 Sep 2025 06:18:16 +0000 https://status.latitude.so/#f042102ec0465a7f44626381e661b8fc1c099d5c5d6b1fc2523b47cdd57f35a9 workers.latitude.so/health recovered https://status.latitude.so/ Tue, 23 Sep 2025 06:16:42 +0000 https://status.latitude.so/#f042102ec0465a7f44626381e661b8fc1c099d5c5d6b1fc2523b47cdd57f35a9 workers.latitude.so/health went down https://status.latitude.so/incident/594759 Sat, 31 May 2025 18:06:00 -0000 https://status.latitude.so/incident/594759#4cb588adf7d9330510f4921f14adc35d7781bacc4981ea0117cb773eb763ffda ## Incident Summary On 30 May 2025 at 18:22 CEST, the Gateway proxy experienced an outage that lasted two hours and twenty-two minutes. During this interval all customer traffic routed through the proxy failed, resulting in a complete loss of service. ## Root Cause The incident was triggered by an unexpected surge in incoming requests that drove a corresponding spike in outbound calls to external LLM providers, primarily OpenAI. Although the autoscaler immediately began launching additional Gateway instances, the increased egress traffic exhausted the outbound bandwidth of the NAT Gateway that serves the cluster. Each new instance had to download its Docker image through the same congested link, causing image pulls that normally complete in under a minute to stretch beyond twenty minutes. Because capacity scaled up so slowly, the existing Gateway instances became overwhelmed and unhealthy, which led to a total shutdown of the service. ## Resolution Once engineers confirmed that NAT egress saturation was the bottleneck, they first detached the Gateway service from incoming traffic to prevent further failures. They then replaced the overloaded NAT Gateway with a higher-capacity unit and started a minimal set of fresh Gateway instances behind the new network layer. After each instance passed its health checks, traffic was gradually reintroduced. Full functionality returned two hours and twenty-two minutes after the initial failure. ## Preventive Measures The NAT Gateway is now part of an autoscaling group so network capacity can expand automatically with demand. New monitoring thresholds generate alerts whenever NAT egress approaches saturation, giving operators early warning before user impact occurs. In addition, the baseline number of Gateway instances has been increased to provide greater headroom during sudden traffic spikes. We apologise for the disruption and remain committed to strengthening our infrastructure to prevent similar incidents in the future. https://status.latitude.so/incident/582904 Sat, 31 May 2025 09:15:00 -0000 https://status.latitude.so/incident/582904#acd736369ad51ed96139f510bde5f8df47275db7900010720f9ebc63a14a03d2 Maintenance completed https://status.latitude.so/incident/582904 Sat, 31 May 2025 09:15:00 -0000 https://status.latitude.so/incident/582904#acd736369ad51ed96139f510bde5f8df47275db7900010720f9ebc63a14a03d2 Maintenance completed https://status.latitude.so/incident/582904 Sat, 31 May 2025 09:15:00 -0000 https://status.latitude.so/incident/582904#acd736369ad51ed96139f510bde5f8df47275db7900010720f9ebc63a14a03d2 Maintenance completed https://status.latitude.so/incident/582904 Sat, 31 May 2025 08:00:00 -0000 https://status.latitude.so/incident/582904#6e30c55ac28bbe3d282542cd50bc2a759d109994710915bea5be4091650e66fa We are going to upgrade our main database and cache cluster instances with higher cpu and memory in order to accomodate capacity from new customers. This change will cause downtime of some minutes. https://status.latitude.so/incident/582904 Sat, 31 May 2025 08:00:00 -0000 https://status.latitude.so/incident/582904#6e30c55ac28bbe3d282542cd50bc2a759d109994710915bea5be4091650e66fa We are going to upgrade our main database and cache cluster instances with higher cpu and memory in order to accomodate capacity from new customers. This change will cause downtime of some minutes. https://status.latitude.so/incident/582904 Sat, 31 May 2025 08:00:00 -0000 https://status.latitude.so/incident/582904#6e30c55ac28bbe3d282542cd50bc2a759d109994710915bea5be4091650e66fa We are going to upgrade our main database and cache cluster instances with higher cpu and memory in order to accomodate capacity from new customers. This change will cause downtime of some minutes.