If you work in tech, you use SSH every day. But for a lot of developers, it's just a black box. Letโs pull back the curtain and trace a real-world hands-on sequence on an Ubuntu server to see exactly how the network handoff and Linux PAM (Pluggable Authentication Modules) pipeline work under the hood.
1. The Core Concept: Public vs. Private Keys
Instead of using passwords, secure modern SSH relies on asymmetric cryptography:
- The Public Key: Think of this like a padlock. You can hand it out to anyone or leave it on a public server. It doesn't matter who sees it.
- The Private Key: This is the physical key that opens the padlock. It stays safely on your laptop, and you never share it.
2. Hands-On: Solving "Permission Denied" in Real-Time ๐ ๏ธ
Step 1: The Error
Connecting to a cloud instance without an authorized key fails immediately:
mhasifkamal@Mhs-MacBook-Air work % ssh mhasifkamal@34.131.76.231
mhasifkamal@34.131.76.231: Permission denied (publickey).
Step 2: Generating the Match
Run ssh-keygen. On modern operating systems, it automatically defaults to the highly secure ED25519 algorithm:
mhasifkamal@Mhs-MacBook-Air work % ssh-keygen
Generating public/private ed25519 key pair.
Enter file in which to save the key (/Users/mhasifkamal/.ssh/id_ed25519): ssh_key
Your identification has been saved in ssh_key
Your public key has been saved in ssh_key.pub
Step 3: Verifying the Files
Your system sets strict, private read/write privileges (-rw-------) for the private key, while leaving the .pub lock file readable:
mhasifkamal@Mhs-MacBook-Air work % ls -l
-rw------- 1 mhasifkamal staff 432 Jun 25 21:42 ssh_key
-rw-r--r-- 1 mhasifkamal staff 115 Jun 25 21:42 ssh_key.pub
mhasifkamal@Mhs-MacBook-Air work % cat ssh_key.pub
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFHgEWeyk6UwHRs4VknwryjptBQcG/dYufjCJu8oq+Qe...
Step 4: The Payoff
Point directly to your private key file using the -i flag to unlock entry:
mhasifkamal@Mhs-MacBook-Air work % ssh -i ssh_key mhasifkamal@34.131.76.231
Welcome to Ubuntu 24.04.4 LTS (GNU/Linux 6.17.0-1018-gcp x86_64)
mhasifkamal@instance-20260620-113027:~$
3. Where the Lock Lives ๐
Once you log in, navigate to your home directory's hidden .ssh folder. All authorized locks are stored in a single text file:
mhasifkamal@instance-20260620-113027:~$ cd .ssh/
mhasifkamal@instance-20260620-113027:~/.ssh$ ls
authorized_keys
mhasifkamal@instance-20260620-113027:~/.ssh$ sudo vi authorized_keys
Cloud setup panels and commands like ssh-copy-id simply append your public key string into this authorized_keys file.
4. Toggling Passwords & The Cloud Override Trap ๐ธ๏ธ
Let's create a traditional password user named test-user on the server:
mhasifkamal@instance-20260620-113027:~/.ssh$ sudo adduser test-user
New password:
passwd: password updated successfully
Trying to log in immediately from your laptop will still fail:
mhasifkamal@Mhs-MacBook-Air work % ssh test-user@34.131.76.231
test-user@34.131.76.231: Permission denied (publickey).
The Fix
To allow temporary password logins, you must modify the incoming SSH daemon settings. Avoid ssh_config (which handles outbound client connections) and focus on the drop-in cloud override files inside /etc/ssh/sshd_config.d/:
mhasifkamal@instance-20260620-113027:/etc/ssh$ ls /etc/ssh/sshd_config.d/
50-cloudimg-settings.conf 60-cloudimg-settings.conf
Open these overrides, change the parameter to yes, and restart the daemon:
sudo vi /etc/ssh/sshd_config.d/*
# Set: PasswordAuthentication yes
sudo systemctl restart ssh
Now, the server prompts for the password and allows authentication cleanly:
mhasifkamal@Mhs-MacBook-Air work % ssh test-user@34.131.76.231
test-user@34.131.76.231's password:
Welcome to Ubuntu 24.04.4 LTS!
5. Under the Hood: The Complete Architecture Flow ๐
Here is exactly how the network handoff transitions directly into Linux's internal PAM (Pluggable Authentication Modules) pipeline:
[ LAPTOP ] [ UBUNTU SERVER ]
โ โ
โโโโ 1. Protocol Handshake (Agree on SSH versions) โโโโโโโโโ>โ
โ โ
<โโโ 2. Diffie-Hellman Key Exchange (Secure Tunnel Built) โโ>โ
โ โ
<โโโ 3. Asymmetric Key Challenge (Puzzle sent to client) โโโโโค
โ โ
โโโโ 4. Challenge Solved (Proof verified by sshd) โโโโโโโโโโ>โ
โ โ
โ [ Hand-off to PAM Engine ]
โ โ
โ โโโโโโโโโโดโโโโโโโโโ
โ โ /etc/pam.d/ โ
โ โโโโโโโโโโฌโโโโโโโโโ
โ โ
โ ๐ 1. common-auth
โ (Checks credentials via pam_unix.so)
โ โ
โ ๐ 2. common-account
โ (Checks password expiration/locks)
โ โ
โ ๐ ๏ธ 3. common-session
โ (Sets env, umask, & log limits)
โ โ
<โโโ 5. Shell Opened (PTY Allocated to user) โโโโโโโโโโโโโโโโโ
The PAM Blueprint
Modern Linux environments manage application access dynamically via files in /etc/pam.d/. (The old master file /etc/pam.conf is ignored).
When we look inside the system authentication rules (cat /etc/pam.d/common-auth), we see the literal logical circuit board:
auth [success=1 default=ignore] pam_unix.so nullok
auth requisite pam_deny.so
auth required pam_permit.so
-
pam_unix.so: Checks credentials against system hashes. If successful,[success=1]jumps past the next line. -
pam_deny.so: The trap line. If your credentials fail, PAM stops here, breaks the circuit, and kicks you out with aPermission deniederror. -
pam_permit.so: The green light. Passes a success flag back to the environment, allowingcommon-sessionto spin up your environment variables and launch yourbashorzshshell.
Wrap Up ๐ฏ
SSH builds the secure tunnel across the web, but PAM controls your access rights once you cross the threshold. Hardening your server by setting your drop-in configs back to PasswordAuthentication no keeps the automated brute-force bots completely out of your architecture.
What does your local ~/.ssh/config file look like? Let's chat in the comments! ๐
#sre #ssh #linux #devops #security #sysadmin #tutorial