Goes To Show
Often when we get used to workarounds, tooling to assist, or pretty much anything in life that allows us to be removed from the original task, we do lose some retention of knowledge and/or ability to perform that original task. Whether it’s having to write in cursive, setting up a webserver, or writing a check; Once we have created a workaround or the ability to abstract and move away from managing that “Thing” we can sometimes become so far removed or abstracted from it, that… we forget about those details and layers of abstraction we created. I call this the “Goes To Show” effect.
This isn’t to say that it’s necessarily a bad thing. I think it’s safe to safe few people lament not having to write checks anymore, or having Google and Wikipedia at their fingertips. But in some cases I do believe it does have an unintended side effect that we tend to forget those details whilst still thinking we have retained them.
As an AWS employee, one of the things I get incredibly used to is all of the tooling and abstractions that get built around using the service for us… the employees. From how we go about logging in, to small, seemingly inconsequential scripts, a lot of the boiler plate work and best practices are done for us.
So imagine my chagrin, when I attempted to do something in my very own personal account (one for which I’ve not used in the last 4 years since working at AWS) as simple, and as easy as set-up a template for launching instances.
Create a Launch Template.
Launch an instance from that template.
Log into the instance.
Imagine my surprise as when I went to login to my instance I saw the following:
ssh homebase
<script> echo “Hello World” </script>
ssh: connect to host <ip_address> port 22: Network is unreachable
First thing I did was to ensure that on my network I was actually able to resolve the Public IP for that instance:
nslookup ec2-<ip_address>.us-east-2.compute.amazonaws.com
Server: 8.8.8.8
Address: 8.8.8.8#53Non-authoritative answer:
Name: ec2-<ip_address>.us-east-2.compute.amazonaws.com
Address: <ip_address>
Well that meant that at the very least:
My host had a public facing IP and network connection
The fully qualified domain name (FQDN) did resolve
Let’s try to see if we can ping it (not a failproof test, as some servers turn off ICMP response by default, but a start).
ping ec2-<ip_address>.us-east-2.compute.amazonaws.com
ping <ip_address> (<ip_address>) 56(84) bytes of data.
^C
--- <ip_address> ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4081ms
As surmised, no dice.
We created, deleted, recreated new SSH keypairs, and launched new instances to no avail. Checked the VPC settings and couldn’t see what might be blocking the connection, but I felt it had to be a something, though I was unfamiliar with the “resource not available” error. Normally I’m used to seeing no message (unable to resolve or timing out), or messages relating to authentication or incorrect ssh keys, but this… was new, or at least it had been so long since I had seen it that I was unaware of it.
Going down the path of attempting to traceroute or use tracepath I realized that the issue was most likely related to a firewall. Potentially on my router or mesh network preventing egress SSH access? Had I set that up. I checked my setting and didn’t see any issue there so it must be relegated to my AWS account.
And like a bolt of lightning, I suddenly realized the culprit. My security group settings. Something as simple and a nominal change that is always taken care of at work, had to be adjusted here. The fix of course was simple, add a ingress rule to my Security Group that the instance had attached, and specify allowing SSH access for my network ranges, thus keeping things secure but allowing access to my VPC.
Small, simple challenges like this… just “Goes to Show” that you never are above checking the obvious.