Provisioning Invites could not be accepted

Date of Incident: 2025-09-23
Time of Incident (UTC): 17:18 - 00:46
Service(s) Affected: Sign Up
Impact Duration: 7h 28m

Summary

For 7 hours and 28 minutes, 1Password Provisioning invites could not be accepted, presenting to the user as an invite expiry. Invites could not be accepted due to a web browser routing defect that was not caught during development, review, or release. First identified by customer reports approximately two and a half hours after release, the issue was escalated to development teams and an incident was immediately called. The root cause was identified as a defect introduced by a web client modification, and a fix was created, tested, and released. By 00:46 UTC, the fix was deployed to all environments and service was fully restored.

Impact on Customers

• Sign-up: Provisioning invites could not be accepted.
• Number of Affected Customers (approximate): 100% of provisioning invites could not be accepted
• Customer-facing impact: Users clicking their invite links encountered a misleading ‘Invite Expired’ message.
• Geographic Regions Affected: 1Password USA/Canada/EU/Enterprise

What Happened?

A change to the web client contained a router defect that incorrectly rendered provisioning invites as expired. Users were presented with an error message that erroneously stated the invite was expired. The change responsible for introducing the defect was able to be released because it was not captured under automatic change notification rules, was lacking automated test coverage, and was not included in the set of manual tests.

• Timeline of Events (UTC):

  • 17:18: 1Password Release containing defect
  • 19:53 (2 hours, 35 minutes later) First customer report
  • 20:49 (56 minutes later) Escalation to developer teams
  • 21:01: (12 minutes later) Incident called
  • 22:02: (1 hour, 1 minute later) Root cause identified
  • 22:29: (27 minutes later) Fix created and testing initiated
  • 23:49: (1 hour, 30 minutes later) Fix merged
  • 00:46: (57 minutes later) Fix released and service fully restored

• Root Cause Analysis: A change modified the order in which key provisioning web routes were rendered. As a result, the route handling provisioning invitations failed to use the correct query parameters and the invite rendered as expired.

• Contributing Factors: Automated tests on this endpoint do not exist. Manual testing missed testing the Provisioning routes. The modified code was not covered by automatic change notification rules to notify the Provisioning team. An existing bug that can fail the resending of invites was an initial red herring during the investigation.

How Was It Resolved?

• Resolution Steps: The defect in the 1Password web client was corrected so provisioning invites would render correctly.
• Verification of Resolution: 1Password engineering tested the changes and validated that the functionality was restored, as well as verifying that requests for the affected endpoints were successful after the fix was deployed.

What We Are Doing to Prevent Future Incidents

• Improve automated tests: We are enhancing our automated tests for the Provisioning Invite routes.
• Expand automatic change notifications: Expanding coverage of automatic change notification rules for areas of code owned by the Provisioning team.

Next Steps and Communication

• No action is required from our customers at this time. Existing invites do not need to be resent and may be accepted.
• If you are still experiencing issues, please contact our support team at support@1password.com.
  

We are committed to providing a reliable and stable service, and we are taking the necessary steps to learn from this event and prevent it from happening again. Thank you for your understanding.

Sincerely,

The 1Password Team