18:45, Got prometheus alarm phone call, mongo-all-1, service down
18:46, The EC2 Status check failed, it should be hardward problem on aws side.
18:50, Tried to reboot instance, nothing happend
18:51, Tried to stop instance
18:55, instance was still stuck at “Stopping” status, tried to “Force Stop”
19:57, Create AWS support case 51170******, with online web chat support.
AWS Eng told me there was a hardware problem behind this instance, and fix it with their background operation.
Service recovered at 19:15 PST time.
Chat Notes:
