Sep 29, 11:19 EDT
Postmortem - **Summary**
The Pardot application experienced delays in email sending on Wednesday, September 21 from 15:00pm UTC until 19:30pm UTC. Pardot uses the Marketing Cloud infrastructure to send emails. The root cause of the delays was that a Marketing Cloud control database, which queues calls for different systems, hit a file size limit. While file size growth is normally handled automatically, the growth command was blocked by a checkpoint process. After a restart of the server, the system quickly recovered. No Pardot emails were lost, but the temporary inaccessibility of the control database caused Pardot messages to be queued within Marketing Cloud. Once the database functionality was restored, all Pardot emails that had been queued were sent on a delayed basis.
**Timeline**
On Friday 9/21 at 14:30pm UTC, Marketing Cloud email team received alerts of blocking on the database, and began investigation. At 15:10pm UTC, Pardot engineers were notified of the delays affecting Pardot emails. The Marketing Cloud team continued their incident response process, and at 17:20pm UTC determined that a reboot of the database server was the best course of action. That server reset completed at 17:29pm UTC. Pardot emails began sending at 17:39pm UTC, and our teams continued to monitor. At 19:30pm, the backlog of Pardot emails completed sending, and an “all clear” was given.
**Remedy & Future Prevention**
To prevent similar situations in the future, our Marketing Cloud database administrators will be working with the storage team to investigate and optimize slow SAN (Storage Area Network) operations. The database, performance, development teams are working to determine if performance improvements can be made in relation to the control database. We will also be improving the process around proactive alerting for Marketing Cloud issues that impact Pardot email deliverability.
Thank you for bearing with us during this issue and for your continued trust in us.
Zach Bailey, Sr. Director of Software Engineering
Sep 21, 16:09 EDT
Resolved - Email sending has returned to normal. Please contact our Support Team at http://help.pardot.com if you encounter any further issues.
Sep 21, 14:10 EDT
Identified - Our developers have restarted one of our backend systems and are seeing improvements. Delays may continue as the system works through the backlog.
Sep 21, 11:30 EDT
Investigating - We’re currently experiencing delays with email sending. Our Engineers are currently investigating as top priority. Updates will be posted here as we learn more.