Nov 23, 18:03 EST
Postmortem - On 11/20, around 3:35pm EST, our automated monitoring systems alerted us to an increase in errors when loading pages within the app. This behavior resulted in the application intermittently
loading slower than normal, and in some cases not at all. Additionally, loading Pardot-hosted assets such as forms and landing pages may have resulted in intermittent errors during this time.
We immediately initiated our incident response process and determined the primary root cause to be a configuration change that had recently been applied to many of our front-end web servers. This configuration change inadvertently resulted in a file not being accessible, causing the increased error rate.
Once the root cause was identified, we began rolling out a fix at approximately 4:00pm EST. Full service was restored at approximately 4:30pm EST.
To prevent issues like this in the future, we are investigating more automated testing options to proactively detect similar issues before configuration changes are applied. We are also augmenting our code release process with additional safeguards and monitoring, enabling us to detect these errors more quickly and allowing our release automation to respond to them automatically. Finally, we will be isolating future configuration changes in such a way that adjustments such as these will affect a much smaller part of our application.
We appreciate your patience and continued trust as we worked to resolve this situation.
Zach Bailey, Sr. Director of Engineering
Nov 20, 16:46 EST
Resolved - Application performance has returned to normal. If you have any questions related to this matter, please contact our Support Team at help.pardot.com. A full postmortem will be posted as soon as next steps have been identified.
Nov 20, 16:28 EST
Monitoring - The fix has been deployed and we're continuing to monitor. Another update will be posted once performance has been fully restored.
Nov 20, 16:11 EST
Identified - We're actively working to deploy a fix. Access to the Pardot app may be intermittent while we work.
Nov 20, 15:59 EST
Investigating - At approximately 3:53 PM EST the Pardot web application began experiencing a service interruption. Our engineers are actively investigating. Updates will be posted here as we learn more.