Incident began at 2021-09-22 14:10 and ended at 2021-09-23 00:30 (all times are US/Pacific).
We are posting this retroactively to provide details of a recent service incident.
We apologize for the inconvenience this issue may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support
(All Times US/Pacific)
Incident Start: 22 September 2021 14:10
Incident End: 23 September 2021 00:30
Duration: 10 hours, 20 minutes
Affected Services and Features:
- Google Cloud Console: Cloud console user interface received a warning notification for US customers with a self-serve billing account.
Google Cloud customers received a warning message on the Google Cloud console user interface and an email regarding an issue with processing their account payment. Impacted customers were redirected to update payment information. Attempts to update the primary payment method were unsuccessful, and the issues persisted. Initial investigation of this issue revealed the impact was triggered after an update to the payment configuration was released during the impact time frame.
Google Cloud customers received a warning message on the Google Cloud console user interface and an email: “Your payment information could not be processed. Visit the payment overview page to make sure your payment information is up to date and to pay any outstanding charges.”
- After customers clicked on the “Fix It” option, a pop-up message was displayed, prompting them to update their primary payment method.
- Customers were unable to fix the account after updating the primary payment method and the warning message reoccurred post the customer action.
After resolution, the account warning message disappeared, and the account state was set back to the proper Payments setting.
Our engineering team has mitigated the customer impact after complete rollback of the changes that triggered the issue.
Affected products: Google Cloud Console