All Avero Services Offfline
Incident Report for Avero
Postmortem
  • 1:08AM ET First alert received

    • Confirmed data center not accessible
    • Performed initial troubleshooting but was unable to connect to any systems
  • 1:24AM ET Ticket from last week reopened with Verizon

  • 1:25AM ET Ticket opened with Equinex

  • 1:31AM ET Data center tech acknowledged and headed to cabinet

  • 1:33AM ET Traceroute details added to Verizon ticket, first escalation request

  • 1:40AM ET Data center tech arrived at cabinet and confirmed power on, no link light on Internet port on switch

    • Mike continued to troubleshoot connectivity with data center tech on the phone.
    • This included moving the internet handoff to our backup switch.
  • 1:44AM ET Second escalation request to Verizon

  • 1:46AM ET Third escalation request to Verizon

  • 1:53AM ET Fourth escalation request to Verizon

  • 2:04AM ET Fifth escalation request to Verizon

  • 2:06AM ET Verizon ticket picked up

  • 2:09AM ET Verizon ticket update - NOC has been called for this critical ticket

  • 2:10AM ET Connection restored and all services back online

Posted Oct 12, 2018 - 02:35 EDT

Resolved
Services are running normally
Posted Oct 12, 2018 - 02:25 EDT
Monitoring
At this time all services are back online and we are continuing to monitor
Posted Oct 12, 2018 - 02:10 EDT
Identified
The issue appears to be with our internet connection in the datacenter. We are working with our provider to resolve
Posted Oct 12, 2018 - 02:09 EDT
Update
We are working with datacenter personnel to troubleshoot but have not identified the root cause yet.
Posted Oct 12, 2018 - 01:47 EDT
Investigating
Currently all Avero services are offline. We are investigating
Posted Oct 12, 2018 - 01:18 EDT
This incident affected: Avero (Avero (Web), Avero Mobile (App)), Avero FCM (Avero FCM Mobile (App)), Data Acquisition and Load (Data Load, Data Acquisition), Notifications and Email Reports (Flash Reports, Avero Alerts and Notifications, Enhanced Notifications), and Data Warehouse.