Why is so important for a Linux admin to master the sos command?

If you make a living by troubleshooting or diagnosing Linux systems whether in large production environments, or in small business with desktop computers, I think that you will find this article extremely useful.

In this article, I will provide a comprehensive overview of the sos command and its many features and at the end I will tell you why is so important to have it in your list of tools. To keep the article concise and easy to read, I will limit the depth of each topic and include only brief examples where appropriate and will provide links to other articles if you’d like to dive deeper on a specific feature.

That said, the sos tool is remarkably powerful and versatile. Its capabilities extend far beyond what can reasonably be covered in a single article. In fact, there is enough material to fill an entire book dedicated to sos, its architecture, its plugins, and the many ways it can be used to collect, preserve, and analyze Linux system diagnostic data.

During Linux troubleshooting there is only one command to rule them all.

1. For those who don’t know what the sos command is and what is used for:

sos is the ultimate Linux troubleshooting command. It collects system configuration, logs, and diagnostic data in one go, giving sysadmin a comprehensive snapshot of a system’s state at a given moment. Whether you’re diagnosing performance issues, investigating a security incident, acquiring hardware and software inventory, or working with vendor support, a sosreport speeds up diagnostic, preserves evidence, improves collaboration, and simplifies troubleshooting.

The sos command (formerly known as sosreport), is an open source extensible tool designed to perform a complete scanning and data-gathering, deep into the heart of your Linux system. It is a tool included in the sos package on most Linux distributions. It collects system configuration, logs, and diagnostic data in one go

sos is also an open-source Python project that has been actively developed since 2009. Originally created by Red Hat Inc., the project has since seen contributions from Canonical Ltd., as well as companies such as:

Rackspace US, Inc.,
EMC Corporation (now Dell),
IBM Corporation,
Hewlett-Packard Development Company, L.P.,
Oracle Corporation,

and organisations like The Linux Foundation and the Samba project.

The project is actively maintained on GitHub, with regular updates that include new plugins, enhanced functionality, and bug fixes. As of the date of this article, the latest version is sos-4.11.2, which includes support for more than 400 different plugins.

If you want to learn more about the project visit its GitHub page.

To give you an idea of its power, after executing this command:

sudo sos report

in 53 seconds it generates a compressed and encrypted tar file of less than 15MB containing over 10,000 text files, including logs, output from more than 500 diagnostic commands, and over 1,800 configuration files. These numbers where obtained from a small lab VM; in big busy server it can take up to 5 minutes and file size can reach 400MB.

2. For those who already know and have used the sos command:
There two main misconception about the sos command spread all over the Internet:

sos command is only available in RedHat. False. sos is included in the sos package on most Linux distributions.
sos execution time and file size are so big that its use becomes impractical. False. The sos command is highly configurable and the collection speed and the file size can be fine tuned to met any requirement. This is achieved by selecting specific plugins.

3. Plugins
The most powerful characteristics of the sos command is that it is extensible. The report functionality of sos operates through a system of plugins, each plugin is designed to collect specific types of information like a system component or software product. Each plugin focuses on a different aspect of the Linux system. Plugins are typically implemented as Python scripts or modules that are executed during the sos report generation process. These plugins can:

Parse configuration files.

Collect log files.
Gather statistics about running processes or system resources.
Create dumps of specific application data.
Collect data in predefined formats to ensure consistency.

You can even create your own plugins. Creating a custom plugin requires knowledge of Python and familiarity with the sos framework. For more technical details, refer to the official Sos Report Plugin Wiki.

By default sos execution will first detects what plugins can be executed in the system and try to run most of them, however, this can be controlled to only execute one plugin if needed or just a list of specific plugins.

Let’s talk about of one common plugin that will always be executed (unless sos is instructed not to). The memory plugin. The memory plugin collects memory usage statistics, NUMA topology, huge pages configuration, and /proc/meminfo details.

when executed it collects these files:

/proc/pci
/proc/meminfo
/proc/vmstat
/proc/swaps
/proc/slabinfo
/proc/pagetypeinfo
/proc/vmallocinfo
/sys/kernel/mm/ksm
/sys/kernel/mm/transparent_hugepage
/sys/kernel/mm/hugepages
/sys/kernel/mm/lru_gen/enabled
/sys/kernel/mm/lru_gen/min_ttl_ms

and will executed these commands:

free
free -m
swapon - bytes - show
swapon - summary - verbose
lsmem -a -o RANGE SIZE STATE REMOVABLE ZONES NODE BLOCK
slabtop -o

another example is the process plugin; it will gather running process list, process tree, and resource usage statistics from ps and /proc. When executed it collects these files:

/proc/sched_debug
/proc/stat
/sys/kernel/debug/sched/debug
/sys/kernel/debug/sched/features
/sys/kernel/sched_ext/
/proc/[0–9]*/smaps

and will execute these commands:

ps auxwwwm
pstree -lp
ps alxwww
ps auxfwww
ps -elfL
pidstat -p ALL -rudvwsRU - human -h
pidstat -tl
lsof +M -n -l -c ''
lsof +M -n -l`

These are only two plugins out of 408 now supported. You can imagine how much data a sosreport can contain just by extrapolating from these two. It is important to mention that as a project rule, no plugin will ever collect the same files or execute the same commands already included by another plugin.

To control the amount of data collected, the sos command supports several options:

The —only-plugins option to control exactly what to retrieve:

sudo sos report --only-plugins=system,memory,process,block,docker

The — skip-plugins option prevent the execution of specific plugins:

sudo sos report --skip-plugins=mysql,apt,cron

The --add-plugin option to add specific plugins to the final archive:

sudo sos report --add-plugins=memory,block

The — log-size=10 option is used to limit the size of the collected logs to 10 MB instead of the default 25MB

The — journal-size=15 option is used to limit the size of the collected journal to 15MB instead of the default 25MB

If you need to know what plugins exists, or which plugin is the one that retrieves an specific command output file, a particular log, a configuration file or what are the existing profiles you can look into the plugin table included in this article

4. Profiles
sos command comes with predefined profiles. A profile is just a name for a group of plugins. In version 4.11.1 there are 40 profiles supported.

For example, the ansible profile will include these plugins: aap_containerized, aap_controller, aap_eda, aap_gateway, aap_hub, aap_receptor, ansible, ceph_ansible, and openstack_ansible.

The desktop profile will include these plugins: soundcard, wireless, x11, lightdm, opencl, opengl, unity, vulkan

If you are troubleshooting a Linux desktop machine you can execute the sos command like this:

sudo sos report --profile=desktop --add-plugin=system,memory,process,block

and the final sosreport will include soundcard, wireless, x11, lightdm, opencl, opengl, unity, vulkan, system, memory, process and block with all the files and command output for each plugin.

If you want to know more about plugins this article describes them in great detail.

5. Including your own log files and commands
sos report command includes a plugin called sos_extras that collects extra data defined in one or more configuration files inside /etc/sos/extras.d directory.

This plugin is designed to collect additional custom or user-specified information during the generation of an sos report. It extends the default functionality of sos report by allowing administrators to tailor the output to their specific troubleshooting or analysis needs.

If you want to know how to include your own logs this article describes them in great detail.

6. Encryption and obfuscation
The sos command includes features to help ensure that sensitive information is obfuscated before data is shared. For example, hostnames, IP addresses, and UUIDs can be anonymized, providing peace of mind when reports need to be sent to third-party vendors or remote teams. This ensures that while the report provides a detailed view of the system, it does not expose critical information.

Additionally, encryption can be applied to reports by using GPG which adds an extra layer of security when transmitting diagnostic data across insecure channels making the sos report a trusted ally in the quest for system issues and root cause.

The obfuscation can be activated with the — clean flag, and encryption is summoned with the — encrypt, — encrypt-pass or — encrypt-key flags.

7. layout of a sosreport
The final sosreport as mentioned before is a compressed and possible encrypted tar file that when extracted, it generates a directory structure like this:

.
├── boot/
├── etc/
├── lib/          -> usr/lib
├── proc/
├── root/
├── run/
├── sos_commands/
├── sos_logs/
├── sos_reports/
├── sos_strings/
├── sys/
├── usr/
├── var/
├── version.txt
├── environment
├── date           -> sos_commands/systemd/timedatectl
├── df             -> sos_commands/filesys/df_-al_-x_autofs
├── dmidecode      -> sos_commands/hardware/dmidecode
├── free           -> sos_commands/memory/free
├── hostname       -> sos_commands/host/hostname
├── installed-rpms -> sos_commands/rpm/sh_-c_rpm_-nodigest_-qa_-qf_-59_NVRA_INSTALLTIME_date_sort_-V
├── ip_addr        -> sos_commands/networking/ip_-o_addr
├── ip_route       -> sos_commands/networking/ip_route_show_table_all
├── last           -> sos_commands/login/last
├── lsmod          -> sos_commands/kernel/lsmod
├── lsof           -> sos_commands/process/lsof_M_-n_-l_-c
├── lspci          -> sos_commands/pci/lspci_-nnvv
├── mount          -> sos_commands/filesys/mount_-l
├── netstat        -> sos_commands/networking/netstat_-W_-neopa
├── ps             -> sos_commands/process/ps_auxwwwm
├── pstree         -> sos_commands/process/pstree_-lp
├── root-symlinks  -> sos_commands/host/find_._-maxdepth_2_-type_l_-ls
├── uname          -> sos_commands/kernel/uname_-a
├── uptime         -> sos_commands/host/uptime
└── vgdisplay      -> sos_commands/lvm2/vgdisplay_-vv_-

14 directories, 22 files

All these directories with the exception of sos_commands, sos_logs, sos_reports and sos_strings are partial copies of a regular Linux system and the contents of these directories will be mostly configurations files.

At the root level, there are some symbolic links that point to to frequently used commands output files inside sos_commands directory. Most notably uptime, df, ps, netstat, mount and others depending of the OS and the version of the system.

The sos_strings directory contains a dump of the Linux journal an can be quite large. Depending of the Linux distro and version this directory is where you will find the logs; for older OS versions the main logs will be under /var/log. Always check both directories to find the log file that you want.

The sos_reports directory contains a full listing of the files and commands executed that are included in this sos-report in text format, json format and html format:

The sos.json is the index of the sos-report in json format. It is used to find any file or command output in the sos directory structure. Each json object listed in this file has an “href” field that specifies the file location in the sos-report directory structure.
The manifest.json file in the sos_reports directory describes how the execution of the sos report went, it includes execution timestamps, commands executed with parameters, files copied for every plugin and obfuscation process data done during the sos report activity.
The sos_logs directory contains the log of the sos execution and any error found during it execution can be seen in the sos.log file.

The sos_commands directory contains a large set of subdirectories each one corresponding to each of the different sos plugins that were trigered during the sos command execution. Inside each plugin-subdirectory are stored the output of the commands related to that specific plugin. The sos_commands directory structure for the two plugins memory and process looks like this:

sos_commands/memory
├── free
├── free_-m
├── lsmem_-a_-o_RANGE_SIZE_STATE_REMOVABLE_ZONES_NODE_BLOCK
├── slabtop_-o
├── swapon_ - bytes_ - show
└── swapon_ - summary_ - verbose


sos_commands/process
├── lsof_M_-n_-l_-c
├── pids_to_packages.json
├── ps_alxwww
├── ps_auxwwwm
├── ps_axo_flags_state_uid_pid_ppid_pgid_sid_cls_pri_addr_sz_wchan_20_lstart_tty_time_cmd
├── ps_axo_pid_ppid_user_group_lwp_nlwp_start_time_comm_cgroup
├── ps_-elfL
└── pstree_-lp

Note that the names of the files under a plugin subdirectory are the actual Linux command with the options used during execution to produce its contents.

-rw-r - r - 1 1000 1000 793019 Sep 2 2024 lsof_M_-n_-l_-c
-rw-r - r - 1 1000 1000 47 Sep 2 2024 pids_to_packages.json
-rw-r - r - 1 1000 1000 28230 Sep 2 2024 ps_alxwww
-rw-r - r - 1 1000 1000 79800 Sep 2 2024 ps_auxwwwm
-rw-r - r - 1 1000 1000 43118 Sep 2 2024 ps_axo_flags_state_uid_pid_ppid_pgid_sid_cls_pri_addr_sz_wchan_20_lstart_tty_time_cmd
-rw-r - r - 1 1000 1000 24603 Sep 2 2024 ps_axo_pid_ppid_user_group_lwp_nlwp_start_time_comm_cgroup
-rw-r - r - 1 1000 1000 108266 Sep 2 2024 ps_-elfL
-rw-r - r - 1 1000 1000 43999 Sep 2 2024 pstree_-lp

When analysing a sosreport, this directory is where you are going to spend most of the time.

8. Auto upload and global conf
The sos command’s integration with protocols like HTTPS, Amazon S3, SFTP, and FTP simplifies direct upload, reducing the manual steps involved in diagnostics.

It is possible to upload the resulting sos report archive-file directly from the command line to a sos report diagnostic and analysis service or to the server from your technical support team. The file can be transferred using either SFTP or an HTTPS upload request.

sudo sos report - upload-url "https://sos-vault.com/api/upload" - upload-user "USER@example.com" - upload-pass "***UPLOAD PASSWORD***" - upload-method post

If you want to learn more about upload features of sos command, this article contains detailed information.

sos command-line options can be overwhelming, to say the least. A single sos report execution can involve dozens of flags, plugin toggles, size limits, and behavioural switches. Remembering the correct combination — especially under pressure during a production incident — is impractical, even for experienced administrators.

Fortunately, there are two structured and reliable mechanisms to overcome this complexity: the global /etc/sos/sos.conf configuration file and presets.

This is an example of a sos.conf file :

Full Configuration (with Encryption, Obfuscation, and Upload)

[global]
# Enable batch mode to skip interactive prompts
batch = true
# Set the number of threads for data collection
threads = 4
verbosity = 1
log-size = 100
# xz is often the default, but you can explicitly ensure it here
compression-type = xz
tmp-dir = /var/tmp/sos
skip-plugins = logs, debug
[report]
# - - ENCRYPTION - -
# Encryption Key to encrypt the resulting archive. 
# Provide the same Decrypt Key configured in your sos-vault account ("Settings → Keys")
# for automatic unpack to work.
encrypt-pass = ***ENCRYPTION PASSWORD***
# - - OBFUSCATION - -
# Enables the 'clean' functionality during the report run.
# Masking sensitive data like IPs, hostnames, etc. 
clean = true
# Optional: Path to a custom keyword file for extra obfuscation
# keywords = /path/to/my_keywords.txt
# - - UPLOAD SETTINGS - -
# The destination URL for the report
upload-url = https://sos-vault.com/api/upload
# Credentials for the sos report management system (SRMS)
# Your email address
upload-user = email@example.com
# The Upload Key generated in your sos-vault account ("Settings → Keys") 
upload-password = ***UPLOAD PASSWORD***
upload-method = post
[plugin_options]
networking.traceroute = yes
networking.ping_count = 3

Practical Behaviour Example

Without sos.conf:

sudo sos report -q - clean - batch - case-id "CASE_ID" - encrypt-pass "***ENCRYPTION PASSWORD***" - upload-url "https://sos-vault.com/api/upload" - upload-user "email@example.com" - upload-pass "***UPLOAD PASSWORD***" - upload-method post

With sos.conf:

sudo sos report - case-id "CASE_ID"

Same behaviour, because defaults are pre-configured.

If you want to know how to configure sos the command this article describes it in great detail.

9. Automation with Presets
A preset is a named, reusable configuration that instructs sos report to behave in a precisely defined way — limiting which plugins run, which options are applied, and how the resulting archive is handled — all without the operator ever having to type a complex command.

When paired with an incident management pipeline, presets become something far more powerful: they turn sos report into a targeted, automated diagnostic probe that can be fired in response to a specific alert, collect only the data relevant to that alert, and ship a compact, encrypted report to an analysis platform — all within seconds.

Presets are defined in JSON files stored under the /etc/sos/presets.d/ directory. Each file can contain one or more named presets, each specifying a set of options that will be applied when sos report is invoked with the —preset flag. A preset for a disk alert may look like this looks like this:

{
 "diskProblem": {
 "desc": "Disk and filesystem diagnostic for storage saturation alerts",
 "note": "Triggered by Grafana: filesystem_usage > 85% or disk I/O latency spike.",
 "args": {
 "batch": true,
 "clean: true,
 "log-size": 15,
 "journal-size": 15,
 "only-plugins": "host,kernel,memory,process,release,date,filesystem,block,cifs,scsi,logs,lvm2,zfs",
 "encrypt-pass": "ENCRYPTION_PASSWORD",
 "upload-url": "https://sos-vault.com/api/upload",
 "upload-user": "email@example.com",
 "upload-pass": "UPLOAD_PASSWORD",
 "upload-method": "post"
 }
 }
}

When a disk alert is received, the pipeline can trigger this command in the affected node:

sudo sos report --preset diskProblem --case-id="{{ case_id }}"

An encrypted, obfuscated sosreport will be generated and automatically uploaded for analysis in just a few seconds.

A great article describing how to use presets to integrate sos command to a Grafana->Ansible incident management pipeline can be found in this article.

10. sos collect
sos is in deed a very versatile and powerful tool that can collect data from whole swarm of Linux systems with a single command.

To gather information across multiple nodes in a high-availability cluster or distributed system, sos command provides the collect option. sos can collect data from all nodes in the cluster. Collect is used to capture reports on multiple systems simultaneously. These systems can either be defined by the user at the command line and/or defined by clustering software that exists either on the local system or on a “primary” system that is able to inform about other nodes in the cluster.

sos collect can be run either on a central server that has SSH key authentication setup for the nodes in a given cluster, or from a “primary” node in a cluster that has SSH keys configured for the other nodes.

sudo sos collect - nodes 192.168.1.10,192.168.1.11,192.168.1.12

You can find more details about sos collect in this article.

11. So, why is so important then?
There are many reasons why it is important for Linux administrators to master the sos command. Not only does sos significantly reduces diagnostic and troubleshooting time, but, more importantly, it allows you to establish your own structured troubleshooting framework.

It gives you the opportunity to set a customized Linux diagnostic methodology for you and your team, that will make the troubleshooting process autonomous (i.e. independant of each individual Linux skills level).

By simply listing a set of steps like: first look at this file, second look at this other file, then look at this other file, etc. you can easily define your best diagnostic path. It is like a runbook but without executing any command.

Once this diagnostic path has been established, the logic consequence is that it can be automated, and then the next logic step is that sos can be integrated into your automatic incident management pipeline and more over, it can also be used for safely feed to an LLM for automatic root cause analysis with out needing to give access to your production environments.

By collecting diagnostic data in a consistent and repeatable manner, you can preserve evidence of system conditions at the time an incident occurs. This creates a valuable troubleshooting history for your server fleet and enables trend analysis, knowledge sharing, and post-incident reviews and even training for new staff. In some highly regulated environments it can even be used to meet compliance requirements and asses STIG and CIS system compliance through time.

Another major advantage is that the analysis can be performed away from the affected system. There is no need to repeatedly log into a production server that is already experiencing problems, consume additional resources, or risk making mistakes at 4:00 AM while working under pressure on a critical production system. (That has certainly happened to me more than once.)

By securely storing sosreports in a central location, teams can maintain data consistency throughout the troubleshooting process. Everyone analyzes the same data set, eliminating discrepancies caused by changing system conditions. This also opens the door to partial or even fully automated analysis.

To fully exploit the benefits of the sos command, administrators should understand how plugins, profiles, and presets work, and how to select the most appropriate ones for each situation. They should also be familiar with the options available to control the amount of data collected, particularly log file sizes, as well as the sosreport directory structure and the techniques required to navigate and analyze it efficiently.

Because production server data often contains sensitive and confidential information, administrators must also understand the security features provided by sos. These capabilities help ensure the integrity, confidentiality, and privacy of sosreports during storage, transfer, and collaboration.

Yep, the sos command is pure power.

推荐订阅源

DEV Community