cloud-test: Connection Refused!

A fellow Technical Solutions Engineer recently found their Google Cloud Platform project in an interesting state. They could create Compute Engine VM instances that would boot, but could not remotely connect via SSH into any of them. While this problem is often due to a misconfigured firewall rule, a quick check of the rules showed this was not the case, as an SSH rule existed and its SRC_RANGES value was non-discriminatory:

$ gcloud compute firewall-rules list -r .*ssh.*
NAME              NETWORK SRC_RANGES RULES SRC_TAGS TARGET_TAGS
default-allow-ssh default 0.0.0.0/0 tcp:22

We ruled out a system-level firewall misconfiguration, as new systems from default images would not share that issue. As a sanity check, we used tcptraceroute to ensure traffic was reaching the instance:

$ sudo tcptraceroute -P 22 130.211.181.201
Selected device en0, address 172.31.130.174, port 22 for outgoing packets
Tracing the path to 130.211.181.201 on TCP port 80 (http), 30 hops max
1 172.31.131.252 1.247 ms 0.256 ms 0.250 ms
2 * * *
...
10 * * *
11 201.181.211.130.bc.googleusercontent.com (130.211.181.201) [closed] 38.175 ms 38.918 ms 38.072 ms

We would expect the last hop to report open, not closed. Typically, this value means that the instance has responded but the port wasn't open for communication. With no firewall interference, we knew it had to be something else. The next step was to grep through the serial port output to see if sshd had started:

$ gcloud compute instances get-serial-port-output gcp-rge0-blog --zone us-central1-a | grep Starting.*sshd
[....] Starting OpenBSD Secure Shell server: sshd
Jan 14 23:19:19 gcp-rge0-blog sshd[1911]: Server listening on 0.0.0.0 port 22.
[ ok ] Starting OpenBSD Secure Shell server: sshd.

Okay, that looked fine. With the most obvious points of interference ruled out, the network routes were the next best bet:

$ gcloud compute routes list
NAME                            NETWORK     DEST_RANGE      NEXT_HOP                 PRIORITY
default-route-31a84e4cfff40b29 default     10.240.0.0/16                            1000

Now we’ve found the root cause. The default route for non-local traffic (0.0.0.0/0) had been inadvertently deleted, which caused all external traffic to be lost on the return path. Recreating the missing route solved the issue:

$ gcloud compute routes create default-internet --destination-range 0.0.0.0/0 --next-hop-gateway default-internet-gateway
Created [https://www.googleapis.com/compute/v1/projects/PROJECTID/global/routes/default-internet].

$ gcloud compute routes list
NAME                            NETWORK     DEST_RANGE      NEXT_HOP                 PRIORITY
default-route-31a84e4cfff40b29 default     10.240.0.0/16                            1000
default-internet     default     0.0.0.0/0       default-internet-gateway 1000

Now, the instances are once again reachable by SSH and any other external method. Case closed!

You can find a lot of help and information in the Google Cloud Platform documentation and more information on troubleshooting Compute Engine specifically here.
- Posted by Josh Moore, Technical Solutions Engineer