Every public server becomes part of the internet’s background noise very quickly.

That was not obvious to me in the same way until I started watching production traffic closely. I was not only seeing normal users, crawlers, and health checks. I was also seeing bots probing predictable paths:

/.env
/.env.production
/backup/.env
/wp/.env
/magento/.env
/api/v2/.env
/gateway/.env
/vendor/.env
/storage/.env
/.git/config
/credentials.json
/service-account.json
/__env.js
/actuator/env
/admin/phpinfo.php
/wp-admin/install.php

These were not random requests. They were patterns.

The same categories of paths appeared again and again, usually from new IP addresses, often through CDN ranges, and usually expecting one mistake: a leaked environment file, a forgotten backup, an exposed Git directory, a debug endpoint, a WordPress installer, a Spring actuator route, or a service account JSON file accidentally placed under the web root.

That experience changed how I think about server hardening.

I do not treat security as one tool anymore. I treat it as layers:

• firewall first
• SSH exposure reduction
• key-only authentication
• non-root users
• Fail2Ban for behavior-based blocking
• Nginx deny rules and allow lists
• Docker isolation
• process and resource monitoring
• CDN and WAF in front
• forensic habits when something looks wrong
• secret isolation, especially around .env

This article is a practical write-up of how I harden Linux servers based on the real traffic I monitored and handled.

I also built a small Go project for this workflow: WatchTower-Sentinel.

GitHub: https://github.com/amirsefati/WatchTower-Sentinel

It tails Nginx access logs, tracks first-seen client IPs, watches CPU/RAM pressure, inspects suspicious processes, and sends concise Telegram alerts. In my case, it helped me identify real bot behavior and extract request patterns from production-like traffic instead of guessing from theory.

The first rule: assume your server is already being scanned

A fresh public IP is not invisible.

Once a service is reachable from the internet, it will eventually receive probes. Some are harmless crawlers. Some are noisy automated scanners. Some are looking for one specific mistake.

The most common mistake I saw in logs was not a complex exploit. It was a simple file exposure attempt:

GET /.env
GET /.env.production
GET /backup/.env
GET /wp/.env
GET /storage/.env
GET /credentials.json
GET /service-account.json
GET /.git/config

The attacker does not need a zero-day if the application serves secrets as static files.

That is why my hardening starts with boring basics. Boring security is usually the security that actually works.

Step 1: create a normal user and stop working as root

The first thing I do on a server is create a non-root user.

adduser deploy
usermod -aG sudo deploy

Then I copy my SSH key:

mkdir -p /home/deploy/.ssh
nano /home/deploy/.ssh/authorized_keys

chown -R deploy:deploy /home/deploy/.ssh
chmod 700 /home/deploy/.ssh
chmod 600 /home/deploy/.ssh/authorized_keys

After that, I test login in a second terminal before touching root SSH access:

ssh deploy@SERVER_IP

Only after I confirm that the normal user works, I reduce root exposure.

Security is not only about blocking attackers. It is also about avoiding self-inflicted downtime. Never close the old door before testing the new one.

Step 2: harden SSH before enabling aggressive firewall rules

I usually edit:

sudo nano /etc/ssh/sshd_config

These are the important settings:

Port 2222
PermitRootLogin no
PasswordAuthentication no
PubkeyAuthentication yes
KbdInteractiveAuthentication no
ChallengeResponseAuthentication no
X11Forwarding no
AllowUsers deploy

Then I validate the config:

sudo sshd -t

If validation passes:

sudo systemctl reload ssh

Then I test the new port from another terminal:

ssh -p 2222 deploy@SERVER_IP

Only after that do I close the old SSH port at the firewall level.

Changing the SSH port is not real authentication security by itself. It does not replace keys. But it reduces the volume of automated noise hitting port 22, and that matters because clean logs are easier to investigate.

The real security improvement is:

root login disabled
password login disabled
only specific users allowed
key-based authentication required

Step 3: enable UFW carefully

The easiest way to lock yourself out of a server is to enable a firewall before allowing SSH.

So I always allow the new SSH port first:

sudo ufw default deny incoming
sudo ufw default allow outgoing

sudo ufw allow 2222/tcp
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp

Then enable:

sudo ufw enable
sudo ufw status verbose

If I know my own static IP, I prefer to restrict SSH even more:

sudo ufw delete allow 2222/tcp
sudo ufw allow from YOUR_PUBLIC_IP to any port 2222 proto tcp

This is much better than exposing SSH to the entire internet.

For production servers, I do not like leaving management ports open globally. SSH should be reachable only from trusted IPs, a VPN, a bastion host, or a private network whenever possible.

Step 4: install Fail2Ban for SSH and Nginx behavior

Firewall rules are static. Fail2Ban adds behavior.

Install it:

sudo apt update
sudo apt install fail2ban -y
sudo systemctl enable --now fail2ban

Create a local jail file:

sudo nano /etc/fail2ban/jail.local

For SSH:

[sshd]
enabled = true
port = 2222
filter = sshd
logpath = /var/log/auth.log
maxretry = 3
findtime = 10m
bantime = 1h
backend = systemd

Then restart:

sudo systemctl restart fail2ban
sudo fail2ban-client status
sudo fail2ban-client status sshd

Fail2Ban is not only for SSH. It becomes more useful when I add Nginx patterns for real traffic I see.

For example, I saw repeated sensitive-path probes like:

/.env
/.env.production
/.git/config
/credentials.json
/service-account.json
/actuator/env
/admin/phpinfo.php

So I can create a filter:

sudo nano /etc/fail2ban/filter.d/nginx-sensitive-paths.conf

Example pattern:

[Definition]
failregex = ^<HOST> - .* "(GET|POST|HEAD) /(.*)?(\.env|\.git/config|credentials\.json|service-account\.json|__env\.js|actuator/env|phpinfo\.php|wp-admin/install\.php).*" (403|404|444) .*
ignoreregex =

Then add a jail:

[nginx-sensitive-paths]
enabled = true
port = http,https
filter = nginx-sensitive-paths
logpath = /var/log/nginx/access.log
maxretry = 3
findtime = 10m
bantime = 6h

Before trusting a custom filter, I test it:

sudo fail2ban-regex /var/log/nginx/access.log /etc/fail2ban/filter.d/nginx-sensitive-paths.conf

This part is important. A bad regex can either miss real attacks or ban normal users. I prefer starting strict, observing, and then tuning.

My rule is simple: Fail2Ban should block behavior, not curiosity. One weird request may be noise. Repeated sensitive path probing is a pattern.

Step 5: make Nginx reject sensitive files before the application sees them

The application should not be responsible for every bad request.

If a request is obviously targeting secrets, Git metadata, backups, or internal files, Nginx can reject it immediately.

A basic hardening snippet:

# Block hidden files such as .env, .git, .htaccess
location ~ /\.(?!well-known) {
    deny all;
    access_log off;
    log_not_found off;
}

# Block common secret and config file names
location ~* ^/(.*)?(\.env|\.env\..*|credentials\.json|service-account\.json|__env\.js|composer\.(json|lock)|package-lock\.json|yarn\.lock)$ {
    deny all;
    access_log /var/log/nginx/security-access.log;
}

# Block backup/archive/database dump files
location ~* \.(bak|backup|old|orig|save|swp|sql|sqlite|db|tar|gz|zip|7z|rar)$ {
    deny all;
    access_log /var/log/nginx/security-access.log;
}

# Block obvious PHP probing on non-PHP apps
location ~* /(phpinfo\.php|wp-admin/install\.php|xmlrpc\.php)$ {
    return 404;
}

For some routes, I use allow lists.

For example, if an admin panel must only be accessible from office/VPN IPs:

location /admin/ {
    allow YOUR_TRUSTED_IP;
    deny all;

    proxy_pass http://127.0.0.1:3050;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
}

If the application is behind Cloudflare or another CDN, the real client IP must be restored correctly. Otherwise Nginx and Fail2Ban may only see the CDN proxy IP. That makes banning dangerous because you might ban a proxy instead of the attacker.

In that case, configure the real IP module with trusted CDN ranges and use the correct header, for example:

real_ip_header CF-Connecting-IP;
set_real_ip_from CLOUDFLARE_IP_RANGE;

The exact ranges must be kept updated from the CDN provider.

The .env file deserves its own section

The .env file is one of the most targeted files on the internet.

That is because it often contains exactly what attackers want:

DATABASE_URL
REDIS_URL
JWT_SECRET
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
DIGITALOCEAN_SPACES_KEY
STRIPE_SECRET_KEY
SMTP_PASSWORD
TELEGRAM_BOT_TOKEN
GOOGLE_SERVICE_ACCOUNT_JSON
SENTRY_DSN

A leaked .env can turn a simple HTTP misconfiguration into a full infrastructure incident.

The biggest problem is that .env files are convenient during development, so teams sometimes treat them casually. But in production, .env is not just a config file. It is a secret boundary.

Here is how I handle it.

1. Never place .env under the public web root

This is the most important rule.

Bad idea:

/var/www/app/public/.env
/var/www/html/.env
/usr/share/nginx/html/.env

Better:

/opt/myapp/.env
/etc/myapp/myapp.env
/home/deploy/apps/myapp/.env

The file should exist outside any directory that Nginx can serve as static content.

2. Use strict permissions

For a single application user:

sudo chown deploy:deploy /opt/myapp/.env
sudo chmod 600 /opt/myapp/.env

That means only the owner can read and write it.

For a systemd service, I prefer an environment file:

[Service]
User=deploy
Group=deploy
EnvironmentFile=/etc/myapp/myapp.env
ExecStart=/usr/bin/node /opt/myapp/server.js

Then:

sudo chown root:deploy /etc/myapp/myapp.env
sudo chmod 640 /etc/myapp/myapp.env

This allows the service group to read it while preventing random users from reading secrets.

3. Never commit .env

My .gitignore always includes:

.env
.env.*
!.env.example

And .env.example must contain only safe placeholders:

DATABASE_URL=postgres://user:password@localhost:5432/app
JWT_SECRET=change-me
TELEGRAM_BOT_TOKEN=change-me

The example file documents required variables without leaking real values.

4. Do not bake secrets into Docker images

This is a common mistake.

Bad:

ENV DATABASE_URL=postgres://real-secret
COPY .env /app/.env

Better:

services:
  api:
    image: my-api:latest
    env_file:
      - /etc/myapp/myapp.env

Even better in orchestrated environments: use secret managers, platform secrets, Docker secrets, Kubernetes Secrets, or a cloud secret manager.

The image should be portable. Secrets should be injected at runtime.

5. Rotate secrets after exposure

If .env was exposed, removing the file is not enough.

I rotate:

database passwords
API keys
cloud access keys
JWT secrets
SMTP credentials
bot tokens
webhook secrets
object storage keys
third-party service tokens

Then I check logs for suspicious access during the exposure window.

A leaked secret must be treated as used, not just viewed.

Docker: do not casually run everything as root

Docker is not a magic sandbox.

If a process runs as root inside a container, it is still a risk. The level of risk depends on the runtime, capabilities, mounts, namespaces, and daemon configuration, but I avoid unnecessary root containers.

In Dockerfiles, I prefer:

FROM node:22-alpine

WORKDIR /app

COPY package*.json ./
RUN npm ci --omit=dev

COPY . .

RUN addgroup -S appgroup && adduser -S appuser -G appgroup
USER appuser

CMD ["node", "server.js"]

In Compose:

services:
  api:
    build: .
    user: "10001:10001"
    read_only: true
    cap_drop:
      - ALL
    security_opt:
      - no-new-privileges:true
    tmpfs:
      - /tmp
    ports:
      - "127.0.0.1:3000:3000"

Important habits:

do not mount / unnecessarily
do not mount docker.sock into random containers
drop Linux capabilities when possible
use read-only filesystems where possible
bind services to 127.0.0.1 behind Nginx
avoid privileged: true unless there is a very strong reason

If I need stronger isolation, I look at rootless Docker, user namespaces, seccomp, AppArmor, SELinux, or moving the workload to a more controlled orchestrated environment.

The main idea is simple: if the app is compromised, the attacker should hit walls immediately.

Process limits and resource protection

Security is also availability.

A compromised app, a miner, or a broken process can consume CPU, RAM, file descriptors, or process slots.

For systemd services, I use limits like:

[Service]
User=deploy
Group=deploy
Restart=always
RestartSec=5

NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=full
ProtectHome=true

MemoryMax=500M
CPUQuota=80%
TasksMax=200
LimitNOFILE=65535

For Docker Compose:

services:
  api:
    deploy:
      resources:
        limits:
          cpus: "1.0"
          memory: 512M

Depending on the environment, Compose resource limits may behave differently, so I always verify on the target host.

I also monitor:

top
htop
ps aux --sort=-%cpu | head
ps aux --sort=-%mem | head
systemctl status SERVICE_NAME
journalctl -u SERVICE_NAME -f

And for network activity:

ss -tunap
sudo lsof -i -P -n

This is where WatchTower-Sentinel helped me. Instead of manually checking all the time, I wanted a small sentinel that could detect first-seen IPs, suspicious request paths, high CPU/RAM pressure, and risky process activity, then send compact Telegram alerts.

Detecting miner-like infections and suspicious processes

When I suspect a miner or unwanted process, I do not start by deleting random files.

I first preserve enough information to understand what happened.

My quick triage flow:

uptime
top
ps aux --sort=-%cpu | head -30
ps aux --sort=-%mem | head -30

Then I inspect suspicious processes:

readlink -f /proc/PID/exe
tr '\0' ' ' < /proc/PID/cmdline
ls -la /proc/PID/fd
cat /proc/PID/environ 2>/dev/null | tr '\0' '\n'

Network connections:

ss -tunap
sudo lsof -i -P -n

Recently changed files:

sudo find /tmp /var/tmp /dev/shm -type f -mtime -2 -ls 2>/dev/null
sudo find /etc/systemd /etc/cron* /var/spool/cron -type f -mtime -7 -ls 2>/dev/null

Persistence checks:

crontab -l
sudo ls -la /etc/cron.d /etc/cron.hourly /etc/cron.daily
systemctl list-timers
systemctl list-units --type=service --state=running

Logs:

sudo journalctl --since "24 hours ago"
sudo grep -i "failed password" /var/log/auth.log
sudo grep -i "accepted" /var/log/auth.log

Common miner red flags:

high CPU with unknown binary
process running from /tmp, /var/tmp, or /dev/shm
weird random process names
outbound connections to unknown IPs
cron jobs that download shell scripts
systemd services with suspicious ExecStart
unexpected SSH keys added to authorized_keys

When I handled suspicious cases, I treated it like forensics first and cleanup second.

The professional approach is:

identify the process
identify how it started
identify persistence
identify network connections
identify modified files
rotate secrets
patch the entry point
rebuild if trust is lost

If the server is seriously compromised, I do not pretend that deleting one process is enough. I rebuild from a clean image, restore trusted data, rotate credentials, and close the original entry point.

That is the difference between “killing a miner” and actually fixing the incident.

CDN and WAF: why I prefer putting apps behind a protective layer

A CDN is not just for performance.

For public apps, I prefer having a CDN or reverse proxy layer in front because it gives me:

TLS termination
DDoS absorption
bot filtering
WAF managed rules
rate limiting
country/IP rules
header normalization
origin hiding

A WAF is especially useful for common attack classes:

path traversal
SQL injection patterns
XSS probes
known CMS exploit paths
suspicious user agents
automated scanners

But I do not rely on WAF alone.

The origin server must still be hardened.

If the origin IP is exposed and accepts traffic directly, attackers can bypass the CDN. So I restrict the origin to CDN IP ranges where possible, or I put the app behind private networking and only expose the proxy.

A good pattern is:

Internet
  -> CDN / WAF
    -> Nginx
      -> local app on 127.0.0.1
        -> private database/cache

Not:

Internet
  -> Node.js app directly
  -> database accidentally exposed

Nginx security habits that helped me

Here are Nginx patterns I use often.

Hide server tokens

server_tokens off;

Limit request body size

client_max_body_size 10m;

Basic rate limiting

limit_req_zone $binary_remote_addr zone=api_limit:10m rate=10r/s;

server {
    location /api/ {
        limit_req zone=api_limit burst=20 nodelay;
        proxy_pass http://127.0.0.1:3000;
    }
}

Security headers

add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-Content-Type-Options "nosniff" always;
add_header Referrer-Policy "strict-origin-when-cross-origin" always;
add_header Permissions-Policy "geolocation=(), microphone=(), camera=()" always;

For HSTS, I only enable it when I am sure HTTPS is fully correct:

add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;

Deny direct access to internal paths

location ~* ^/(internal|private|backup|storage|vendor)/ {
    deny all;
}

Return 444 for obvious garbage

Sometimes I use:

return 444;

for abusive traffic. It closes the connection without a response. I use it carefully because normal debugging becomes harder if overused.

How WatchTower-Sentinel helped me see patterns

I built WatchTower-Sentinel because I wanted lightweight visibility without deploying a heavy SIEM for every small server.

The idea is simple:

tail Nginx access logs
detect new client IPs
detect request bursts
detect sensitive path scans
watch CPU/RAM pressure
inspect suspicious processes
send compact Telegram alerts

In my Telegram reports, I could see events like:

SENSITIVE_PATH_SCAN
path=/.env
status=404

or:

NEW_IP
path=/credentials.json
status=404

or:

NEW_IP
path=/actuator/env
status=404

This changed the conversation from “maybe bots are scanning us” to “these are the exact paths they are probing.”

That is a big difference.

Once I had the patterns, I could turn them into:

Nginx deny rules
Fail2Ban filters
WAF rules
alert categories
incident review notes

That feedback loop is the most valuable part:

observe -> classify -> block -> monitor -> tune

Security improves when the server teaches you what is happening.

My practical hardening checklist

This is the checklist I like to apply before I trust a server:

Create non-root sudo user
Install SSH key
Disable root SSH login
Disable password SSH login
Move SSH to a non-default port
Allow SSH only from trusted IPs if possible
Enable UFW with default deny incoming
Allow only required ports
Install and configure Fail2Ban
Add custom Nginx filters for sensitive path scans
Block hidden files and secret files in Nginx
Keep .env outside public web roots
Set .env permissions to 600 or 640
Never commit .env
Never bake secrets into Docker images
Run app containers as non-root
Drop Docker capabilities
Avoid privileged containers
Bind internal services to 127.0.0.1
Put production apps behind CDN/WAF
Restrict origin access where possible
Monitor CPU/RAM/process/network behavior
Check cron/systemd persistence during incidents
Rotate secrets after any suspected exposure
Rebuild compromised servers when trust is lost

None of these steps are exotic. But together, they make a huge difference.

Final thoughts

The internet constantly tests basic mistakes.

Most of the traffic I observed was not sophisticated. It was automated, repetitive, and opportunistic.

But that is exactly why basic hardening matters.

If a bot requests /.env and receives 404 or 403, that is good.

If Nginx blocks it before the app sees it, better.

If Fail2Ban detects repeated probes and bans the source, better.

If the origin is behind a CDN/WAF and SSH is restricted, better.

If secrets are outside the web root, permissioned correctly, never committed, and rotated after exposure, much better.

My biggest lesson from running and monitoring real servers is this:

Security is not one big tool. It is a set of small decisions that reduce blast radius.

That is how I approach Linux hardening now.

I start with the boring layers, I watch real traffic, I convert patterns into controls, and I keep improving the system based on what the internet is actually doing to it.

That is also why I built WatchTower-Sentinel.

Not because alerts are cool, but because visibility changes how you defend a server.

Repository:

https://github.com/amirsefati/WatchTower-Sentinel

References

• Ubuntu UFW documentation: https://ubuntu.com/server/docs/how-to/security/firewalls/
• Nginx access module: https://nginx.org/en/docs/http/ngx_http_access_module.html
• Nginx documentation: https://nginx.org/en/docs/
• Fail2Ban filters documentation: https://fail2ban.readthedocs.io/en/latest/filters.html
• Docker rootless mode: https://docs.docker.com/engine/security/rootless/