Lately I've been spending a lot of time with document malware and exploring techniques for detection. Malicious documents pose interesting challenges and have become the typical first vector for adversaries to achieve a foothold. Despite this, document malware seems largely overlooked by academics compared to their executable counterparts. In short, it's an area worth exploring.

As I compared the current detection techniques, I noticed the pros and cons are nuanced. Static analysis is quick and accurate, but very vulnerable to evasion via adversarial perturbation. Dynamic analysis is slower and error prone because samples can fail to trigger, but produce richer data. These characteristics are further amplified by the use of machine learning. Adding junk bytes between the elements of a PDF can evade a static feature learner in seconds. Unfortunately, dynamic features don't fare much better.

In time I've begun to formulate a theory that there are currently three kinds of document malware. By applying this insight, I find I can substantially influence the results of my experiments. While not as conclusive as something like a double disassociation, I want to share what I've observed in hopes that it'll inspire others when designing their detection systems and provoke feedback.

So without further ado, here's my theory. Currently, there are three kinds of document malware: exploit-based, abuse-based, and phishing-based. While I propose categories, note that they are not mutually exclusive. Malware authors can always chain and blend techniques to achieve their goals. Therefore, these categories should be seen as points on a spectrum. That said, complexity invites failure, so I do not expect real world authors to stray far from these focal points. The following paragraphs elaborate on each category in reverse order.

Phishing-based

This is the trickiest category for a computer scientist because humans are complicated and confusing entities. The exemplar of this category is a document that tries to convince the user to take some compromising action. There may be a link to a fake website or steps that lead to disabling a security feature. Admittedly, this is a category I currently steer clear of. The best solutions are better education, stronger polices, and a clear strategy for recovery when a human mistake in inevitably made. Also note that this category is the most likely to be chained with others. For example, a document may need to lure the victim into clicking a button before a payload can be used. In rare cases, user interaction may even be leveraged to thwart automated analysis. Thankfully, I have yet to see a malware author embed a CAPTCHA in their document.

Abuse-based

This category does not consider the human user, but distinguishes itself in how it uses the target application. The exemplar here is a PDF containing JavaScript or a Word document with macros that uses provided APIs to download and execute additional malware. The key point to emphasize here is these documents do not violate the specification of the application. The program is functioning as intended. This is why I label these instances as abuse rather than exploitation. This distinction is critical to consider when designing a detection framework. On the one hand, they're tricky because they blend in with benign behavior. This is why, for example, looking at system call sequences is a bad idea for this category. On the other hand, since the program's integrity isn't compromised, it's safe to make strong assumptions in these situations. This is where static analysis is best suited. Designed correctly, such frameworks can leverage strong models of the target program to accurately and statically detect abuse. This saves resources and minimizes exposure to noise.

Exploit-based

Which brings us to the last category. Here's were we see the stuff 0-days and CVEs are made from. It's also where all corners of the security community like to flaunt their technical knowhow with intricate ROP chains and compact shellcode. Jargon aside, this category differs from abuse-based in that it does violate the program's specification. Memory corruption, underflows, overflows, and more are all fair game for achieving arbitrary code execution. This means the author's attacks can take unusual forms, like malformed images, and detection frameworks have to cope. For this reason, this is where dynamic analysis overtakes static. Because exploits by definition break models and assumptions, the only way towards proactive detection is to actually trigger them.

I hope readers find this categorization useful. Note that this is not the only way document malware can be divided. For example, payloads can execute inside the target application or in a separate process, creating a spectrum of intrinsic verses extrinsic behaviors. Regardless, my three category theory has helped me design novel systems and interpret the evaluation results, so I wanted to share it.

Thanks for reading.