Hunt or be Hunted: Every service was unique, and that was the problem

The stack that came back flat

Stacking is usually the first thing we reach for on a persistence hunt. Pull every service, autorun or scheduled task across the environment, count how many endpoints have each one, and sort ascending. The entries at the bottom, the ones that show up on one or two machines, are where you start digging.

In one Tanium Customer's environment, that didn't work at all. Each endpoint returned somewhere between a hundred and five hundred rows, and almost every one of them was unique to that endpoint. When we stacked the results, nearly everything had a count of one. There was no tail to read, because the whole result set was the tail.

The data was fine. The strings were the problem. Windows tacks a unique suffix onto the name of every per-user service, so the same service on two machines shows up under two different names. Autoruns and scheduled tasks that live under a user profile carry the username in their path. Some task and autorun entry names carry their own unique IDs on top of that. Each of those makes something common look rare, and a stack has no way to tell the difference.

Once we rewrote those strings, so the same thing looked the same on every machine, the result set dropped to about one percent of its original size. We didn't drop a single row to get there.

Why we didn't just filter it

Filtering is the obvious move, and it's where most people start: exclude the noisy paths, exclude the services you already trust, stack whatever's left.

The trouble is that filtering throws rows away, and what we needed was to merge them. Exclude every autorun under a user profile, and you get a shorter list, but you've also lost the ability to see that four hundred machines share one autorun, and a single machine has something nobody else does. That comparison is the reason you stack in the first place.

It's also miserable in practice. Past a handful of exclusions, managing them in the console gets cumbersome fast, and every exclusion becomes one more thing somebody must keep current.

We had two other options and neither helped. There's a different services sensor that stacks cleanly, but it doesn't return the same level of detail, and the detail was why we were running the hunt. Shipping the raw data to the SIEM and doing the work there was off the table too. The volume ruled it out, so we never tried it.

Rewrite the string, keep the row

The fix is to normalize the strings on the endpoint before they come back, so the same thing produces the same string everywhere. It comes down to four rules:

1. Find the part of the string that changes per user or per machine but tells you nothing about what the thing actually is. That's usually a profile path, a GUID, or a suffix added at install time.
2. Replace it with a fixed placeholder. An autorun path under C:\Users\jsmith\ becomes C:\Users\USER\. While stacking, we didn't care whose profile it lived in. We cared what was in it.
3. Strip the unique suffix or ID off the name.
4. Leave everything else exactly as it was and never drop a row.

The last rule is the important one. The number of rows coming out must match the number going in. If the normalization only ever rewrites text, nothing can go missing from the results. The worst that can happen is a row gets grouped with others that now look identical to it, and if it looks identical to four hundred other rows, it wasn't the outlier you were after anyway.

None of this depends on our platform. If whatever you stack with lets you rewrite a string before you count it, you can do the same thing.

How it played out on the three surfaces

• Autoruns: user directory paths replaced with a placeholder, plus the unique IDs that some entry names carry stripped off.
• Scheduled tasks: the same treatment. User paths go to a placeholder, and unique IDs come off the task names.
• Services: most per-user service names end in a unique string marking the per-user instance. Removing it folds what's really one service back into one row.

All three were built the same way. Each one calls the native sensor through Tanium's own sensor library, loops over the results, and rewrites the strings before handing them back. That design is also why performance ended up being the hardest part, which comes up again below.

How we know it didn't hide anything

The best evidence comes from how it's built. Rows get rewritten, never removed, so a malicious entry can end up grouped with others but it can't drop out of the output.

Testing is the weaker half. The sensors ran in several dev and test environments while we built them, and we believe the known-bad results in those environments were used to confirm the sensors still surfaced them. A year on, we don't remember the specifics, we didn't keep before-and-after counts, and nobody outside the build reviewed the rules.

If you build this yourself, do the validation properly: run a planted test artifact through the normalized output, confirm it still stands out, and write down the distinct-string counts before and after.

Where it breaks

Normalization throws information away on purpose, and that has consequences you should know before you build it.

• An attacker could hide in the suffix. If someone named a malicious service so it seemed to end in the per-user suffix we strip, part of the name would get cut off. It would still show up, because we never drop rows, and a mangled service name that only exists on one machine is going to look weird to whoever's reading the stack.
• Mass deployment beats it. If an attacker pushes the same service or task to every endpoint, it's common, and stacking won't flag it. Stacking is built to find outliers, and something installed on every box doesn't qualify.
• Performance was the biggest problem. Calling the native sensor and then looping over every result on the endpoint costs more than running the native sensor alone. On constrained machines, that overhead can decide whether you run it at all.
• The rules need an owner. Every rule bakes in an assumption about how a piece of software names things and where it installs. Vendors change both, and when they do, the rule quietly stops matching.
• Skip it when the identifier is what you're hunting. If the specific per-user instance is the thing you care about, collapsing it removes the signal you came for.

ATT&CK coverage

These are the three persistence surfaces where high cardinality hurt the most, so they're the ones we built first. The approach isn't limited to them.

Check whether this is your problem

There isn't a single query that gives you a yes or no. The quickest check we know is a ratio. Stack one of your native autorun or service properties across the environment and compare the number of rows you get back to your endpoint count. If you're getting three times as many rows as endpoints or more, the property is carrying per-endpoint uniqueness, and your stack is mostly noise. That's roughly where normalizing starts to pay for itself.

Why we're not publishing the sensors

The sensors aren't ready to be handed out. The performance cost needs to be closely evaluated and considered for each and every deployment. HuntIQ Threat Hunters work in their customers’ environment to ensure the Tanium Sensors don’t impact the customer’s business. The logic above is the part that carries over to other platforms allowing all our customers to benefit regardless of what platform they use.

Learn how Tanium Security Operations and HuntIQ help you hunt ahead of the next threat. See a demo of how Tanium Atlas puts that hunting to work across your fleet.