eComscan scan times before and after Yargo

YARA is the industry standard for pattern matching in malware detection. Maintained by VirusTotal, it powers threat detection at nearly every security vendor. At Sansec, we rely on YARA for eComscan and our global threat monitor, scanning hundreds of thousands of stores daily.

But YARA was primarily designed for binary malware analysis, and wrapping its C library in Go was painful. We scan text files: PHP, JavaScript, HTML templates. So we built Yargo, a pure Go YARA engine optimized for source code.

How YARA works

A YARA rule defines string patterns and a condition that determines when the rule matches:

rule php_backdoor {
    strings:
        $assert = "assert"
        $serialize = "serialize"
        $session = "session"
    condition:
        all of them
}

Under the hood, YARA extracts short byte sequences ("atoms") from each pattern and loads them into an Aho-Corasick automaton, a state machine that can match thousands of patterns in a single pass over the input. When an atom matches, YARA verifies the full pattern at that position.

The structure starts as a trie: a tree where each edge represents one byte, and paths from root to node spell out the patterns. To search, you walk the trie byte by byte through the input.

Aho-Corasick automaton for the 4-byte atoms asse, seri, and sess. Teal nodes are match states, dashed arrows are failure links.

The failure links (dashed arrows) are what make it fast: when a path doesn't match, instead of restarting from the root, the automaton jumps to the longest suffix that is also a prefix of another pattern. This means no byte is ever read twice.

This two-phase approach (cheap pre-filter, expensive verification) is what makes YARA fast enough to scan thousands of files against thousands of rules.

Low-hanging fruit

Before deciding to build a new engine, we first looked at optimizing our malware signature database.

We maintain a large list of burner domains (disposable domains used by attackers to host malware or exfiltrate sensitive data), currently over 12,000 entries. These were originally written using word boundary regexes (\b[domain]\b) instead of YARA's more efficient fullword modifier.

Switching to fullword string matches eliminated thousands of expensive regex verifications per scan. The performance improvement was significant, but with the signatures optimized, the bottleneck shifted to the engine itself.

Outgrowing go-yara

We use Go for most of our projects, so we relied on go-yara, the C bindings for libyara, for years.

The biggest pain point was CGo. It requires a C compiler, pkg-config, and a pre-installed libyara. Cross-compilation is tedious enough that the go-yara docs need a dedicated guide just to explain it. CGo also prevents fully static binaries, one of Go's biggest advantages. We even maintained an ancient build server just to keep compatibility with merchants running older kernels.

On top of that, YARA's internals are optimized for binary malware analysis, not source code scanning.

Introducing Yargo

Yargo is a pure Go implementation of the YARA features we actually need. It follows the same architecture:

1. Parse: goyacc-based LALR(1) parser turns YARA rules into an AST
2. Compile: extract atoms from patterns, build the AC automaton
3. Scan: AC pre-filter, regex verification, condition evaluation

The key changes target how Aho-Corasick handles text files.

Full string literals in the automaton

YARA truncates every atom to 4 bytes. A 19-byte pattern like eval(base64_decode( gets reduced to a single 4-byte substring, whichever scores highest. But any 4-byte sequence will match far more often than the full string.

Yargo puts entire string literals into the AC automaton, so that same pattern enters as all 19 bytes and only matches when the actual obfuscated code is present. This comes at the cost of a larger automaton, but the extra memory usage is in the order of megabytes.

Smarter regex atoms

YARA can use regex atoms as short as 1 byte. Depending on the rules and the input, this can lead to a lot of unnecessary verifications.

Yargo requires a minimum atom length of 3 bytes. With 256^3 (16.7 million) possible values, the chance of a false atom match drops dramatically. We had to adjust a small number of signatures to accommodate this, but the performance gain was significant.

Scoring atoms for source code

YARA scores atom quality generically: common bytes like 0x00 and 0x20 get penalized.

Yargo's quality function is tuned for web source code:

• Common PHP/JS tokens (return, function, var, ();) are banned from atom selection entirely
• Alphabetic bytes (very common in source code) score lower than non-ASCII bytes
• The heuristic picks atoms that discriminate well in the kind of files we actually scan

Tuned byte frequencies

The AC pre-filter uses byte frequency data generated from real ecommerce stores, so the scanner knows which bytes are actually rare and can skip ahead more effectively.

Performance

eComscan runs over 57,000 scans per day. Yargo has been processing all of them since early February with no regressions.

The signature optimization alone cut median scan times nearly in half. Deploying Yargo cut them by another 6.8x: average scan time went from 12.5 minutes to under 2 minutes, and median scans now complete in under 1 minute.

In its first two weeks of production, Yargo has saved over 116,000 CPU-hours compared to the old engine.

Future work

Yargo currently implements the subset of YARA that we need for our use cases. We're considering making it more backward compatible with YARA, so it can serve as a drop-in replacement for go-yara in other projects.

Yargo is available at github.com/sansecio/yargo under the MIT license.

Read more

• Sansec adds support for Sylius 1 & 2