If you work in tech, you use SSH every day. But for a lot of developers, it's just a black box. Let’s pull back the curtain and trace a real-world hands-on sequence on an Ubuntu server to see exactly how the network handoff and Linux PAM (Pluggable Authentication Modules) pipeline work under the hood.
1. The Core Concept: Public vs. Private Keys
Instead of using passwords, secure modern SSH relies on asymmetric cryptography:
- The Public Key: Think of this like a padlock. You can hand it out to anyone or leave it on a public server. It doesn't matter who sees it.
- The Private Key: This is the physical key that opens the padlock. It stays safely on your laptop, and you never share it.
2. Hands-On: Solving "Permission Denied" in Real-Time 🛠️
Step 1: The Error
Connecting to a cloud instance without an authorized key fails immediately:
mhasifkamal@Mhs-MacBook-Air work % ssh mhasifkamal@34.131.76.231
mhasifkamal@34.131.76.231: Permission denied (publickey).
Step 2: Generating the Match
Run ssh-keygen. On modern operating systems, it automatically defaults to the highly secure ED25519 algorithm:
mhasifkamal@Mhs-MacBook-Air work % ssh-keygen
Generating public/private ed25519 key pair.
Enter file in which to save the key (/Users/mhasifkamal/.ssh/id_ed25519): ssh_key
Your identification has been saved in ssh_key
Your public key has been saved in ssh_key.pub
Step 3: Verifying the Files
Your system sets strict, private read/write privileges (-rw-------) for the private key, while leaving the .pub lock file readable:
mhasifkamal@Mhs-MacBook-Air work % ls -l
-rw------- 1 mhasifkamal staff 432 Jun 25 21:42 ssh_key
-rw-r--r-- 1 mhasifkamal staff 115 Jun 25 21:42 ssh_key.pub
mhasifkamal@Mhs-MacBook-Air work % cat ssh_key.pub
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFHgEWeyk6UwHRs4VknwryjptBQcG/dYufjCJu8oq+Qe...
Step 4: The Payoff
Point directly to your private key file using the -i flag to unlock entry:
mhasifkamal@Mhs-MacBook-Air work % ssh -i ssh_key mhasifkamal@34.131.76.231
Welcome to Ubuntu 24.04.4 LTS (GNU/Linux 6.17.0-1018-gcp x86_64)
mhasifkamal@instance-20260620-113027:~$
3. Where the Lock Lives 📂
Once you log in, navigate to your home directory's hidden .ssh folder. All authorized locks are stored in a single text file:
mhasifkamal@instance-20260620-113027:~$ cd .ssh/
mhasifkamal@instance-20260620-113027:~/.ssh$ ls
authorized_keys
mhasifkamal@instance-20260620-113027:~/.ssh$ sudo vi authorized_keys
Cloud setup panels and commands like ssh-copy-id simply append your public key string into this authorized_keys file.
4. Toggling Passwords & The Cloud Override Trap 🕸️
Let's create a traditional password user named test-user on the server:
mhasifkamal@instance-20260620-113027:~/.ssh$ sudo adduser test-user
New password:
passwd: password updated successfully
Trying to log in immediately from your laptop will still fail:
mhasifkamal@Mhs-MacBook-Air work % ssh test-user@34.131.76.231
test-user@34.131.76.231: Permission denied (publickey).
The Fix
To allow temporary password logins, you must modify the incoming SSH daemon settings. Avoid ssh_config (which handles outbound client connections) and focus on the drop-in cloud override files inside /etc/ssh/sshd_config.d/:
mhasifkamal@instance-20260620-113027:/etc/ssh$ ls /etc/ssh/sshd_config.d/
50-cloudimg-settings.conf 60-cloudimg-settings.conf
Open these overrides, change the parameter to yes, and restart the daemon:
sudo vi /etc/ssh/sshd_config.d/*
# Set: PasswordAuthentication yes
sudo systemctl restart ssh
Now, the server prompts for the password and allows authentication cleanly:
mhasifkamal@Mhs-MacBook-Air work % ssh test-user@34.131.76.231
test-user@34.131.76.231's password:
Welcome to Ubuntu 24.04.4 LTS!
5. Under the Hood: The Complete Architecture Flow 🔄
Here is exactly how the network handoff transitions directly into Linux's internal PAM (Pluggable Authentication Modules) pipeline:
[ LAPTOP ] [ UBUNTU SERVER ]
│ │
├─── 1. Protocol Handshake (Agree on SSH versions) ─────────>│
│ │
<─── 2. Diffie-Hellman Key Exchange (Secure Tunnel Built) ──>│
│ │
<─── 3. Asymmetric Key Challenge (Puzzle sent to client) ────┤
│ │
├─── 4. Challenge Solved (Proof verified by sshd) ──────────>│
│ │
│ [ Hand-off to PAM Engine ]
│ │
│ ┌────────┴────────┐
│ │ /etc/pam.d/ │
│ └────────┬────────┘
│ │
│ 🔒 1. common-auth
│ (Checks credentials via pam_unix.so)
│ │
│ 📋 2. common-account
│ (Checks password expiration/locks)
│ │
│ 🛠️ 3. common-session
│ (Sets env, umask, & log limits)
│ │
<─── 5. Shell Opened (PTY Allocated to user) ────────────────┘
The PAM Blueprint
Modern Linux environments manage application access dynamically via files in /etc/pam.d/. (The old master file /etc/pam.conf is ignored).
When we look inside the system authentication rules (cat /etc/pam.d/common-auth), we see the literal logical circuit board:
auth [success=1 default=ignore] pam_unix.so nullok
auth requisite pam_deny.so
auth required pam_permit.so
-
pam_unix.so: Checks credentials against system hashes. If successful,[success=1]jumps past the next line. -
pam_deny.so: The trap line. If your credentials fail, PAM stops here, breaks the circuit, and kicks you out with aPermission deniederror. -
pam_permit.so: The green light. Passes a success flag back to the environment, allowingcommon-sessionto spin up your environment variables and launch yourbashorzshshell.
Wrap Up 🎯
SSH builds the secure tunnel across the web, but PAM controls your access rights once you cross the threshold. Hardening your server by setting your drop-in configs back to PasswordAuthentication no keeps the automated brute-force bots completely out of your architecture.
What does your local ~/.ssh/config file look like? Let's chat in the comments! 👇
#sre #ssh #linux #devops #security #sysadmin #tutorial