Incident Postmortem - Some customers are unable to interact with the admin console

Date of Incident: 2025-09-24

Time of Incident (UTC): 02:27 - 17:16

Service(s) Affected: Admin console, Sign in

Impact Duration: 36:49

Summary

Some customers with certain account configurations were placed on a blocklist and presented with a 403 error page after accessing the admin console.

Impact on Customers

• Admin console: Affected customers were presented with a 403 error page whenever they tried to interact with any of the admin console pages.
• Log in: Affected customers were also unable to log in to the application.
• Number of Affected Customers (approximate): 515
• Geographic Regions Affected (if applicable): All regions

What Happened?

• Timeline of Events (UTC):

  • Sep 24th 2:27am: Spike in application monitoring alerted engineers to increased rates of IP blocking
  • Sep 24th 3:00am: Cause identified as a change to requests in the application, which had been partially rolled out via a feature flag.
  • Sep 24th 4:03am: The feature flag was enabled to all customers which reduced the spike, but IP blocks continued throughout the day.
  • Sep 24th 10:03pm: Merged an application change to revert the change to prevent the issue reoccurring.
  • Sep 25th 5:03pm: The change was deployed with scheduled application release, error rate dropped off shortly after.

• Root Cause Analysis: The issue was caused by GET requests to the Users API exceeding the URL length limit due to a recent change to append a list of UUIDs to the request parameters to resolve customer reported performance issues.

• Contributing factors:

  • Requests were switched from GET to POST to prevent requests from exceeding the URL limit, however an issue with the feature flag configuration was causing UUIDs to be sent with the GET endpoint.
  • An underlying issue with the feature flag not resolving as expected in the application.

How Was It Resolved?

• Mitigation Steps: Customers were manually removed from the blocklist at multiple points in time as we evaluated the root cause and worked to patch the root issue.
• Resolution Steps: The issue was mitigated by removing UUIDs at the API level if a GET request is used. Additional logging has been added to identify the root cause of the feature flag configuration issue.
• Verification of Resolution: We monitored our server logs to ensure that we did not observe any additional GET requests to the affected URL.

What We Are Doing to Prevent Future Incidents

• Audit additional admin console API requests: We’re performing a sweep of admin console API requests to ensure the utilization of POST requests with highly parameterized URLs.
• Remove the feature flag misconfiguration: We’re correcting the way the feature flag is configured to ensure consistent outcomes.

Next Steps and Communication

No action is required from our customers at this time.

‌

We are committed to providing a reliable and stable service, and we are taking the necessary steps to learn from this event and prevent it from happening again. Thank you for your understanding.

Sincerely,

The 1Password Team