Introduction
In our previous article (Securing software against reverse-engineering in the age of AI and LLMs), we've explained the basic challenges of protecting applications against modern reverse engineering approaches and tooling.
In this one, we will dive into basic technical examples demonstrating what obfuscated code looks like and what we're building to make the reverse engineering process significantly more expensive.
We will look at it from the perspective of a human reverse engineer. But we'll keep in mind that computers are much better and faster at processing and reasoning about that same data a human sees.
The common approach
Most protection solutions provide an SDK with "marker" functions that allow users to tell the obfuscator to obfuscate/virtualize code between the "start" and "end" marker. The code (native instructions) found inside these markers is transformed into a series of virtual instructions understood by a randomly generated virtual machine that implements runtime handling of the virtual instructions.
void important_function()
{
protect_start();
// implementation
protect_end();
}
In addition to that, usually there is some form of a string-obfuscation function intended to prevent important strings from being encoded in the final binary in plaintext form. The string is only decrypted on the first use (this is not how license checks are actually performed, the following code is just an example of obfuscation integration and its results).
void important_function(std::string const& user, std::string const& session, std::string const& license)
{
protect_start();
// implementation
// ...
char const* decrypted = protected_string("9f86d081884c7d65");
if (license == decrypted)
{
// implementation
}
// rest of implementation
// ...
protect_end();
}
The common result
After running the protected binary in a debugger, the reverse engineer is eventually presented with something like this during its runtime:
Illustration of obfuscated code in a debugger
The protected code is executed in a randomly-generated virtual machine while most of the data sits in memory in plaintext (strings, integer constants, etc).
Issues with traditional obfuscators
Research papers about deobfuscation of virtualized code followed by various tools and libraries developed over the past decades have usually exploited the following properties of widely used virtualization-based protection solutions:
1. Virtual machine architecture
While there is no "correct" VM architecture that a software protection should utilize, some traditional solutions have kept the same architecture for years, in some cases even up to 20 years. When it comes to software protection on systems where you cannot avoid the possibility of reverse engineering, it's important to understand that people will eventually understand how a certain VM operates internally. Keeping it the same for years or decades eliminates large amount of work a reverse engineer would normally have to go through.
2. Weak obfuscation of VM runtime code
There is a huge difference between generating code that confuses humans and generating code that makes computers struggle. For a human, simply a large number of instructions poses a time-consuming problem. Over the years, reverse engineers have developed deobfuscation tools that produce highly reduced and deobfuscated code without implementing logic targeting a specific virtual machine.
The main reason for this are simple obfuscation primitives which are trivial to eliminate for a tool based on technologies used by modern reverse engineering platforms. A million instructions of any kind may have been effective 20 years ago -- today it's not about just the amount, but about the type of operations performed.
3. Obfuscated code implies importance
When reverse engineering an application, one does not attempt to understand every function and block of code found within the binary. It's always about locating code of importance, which is usually tiny compared to the entire application.
Majority of code found in most compiled binaries is code that handles programming language runtime, third-party libraries, UI components, handling of various network protocols and so on. What developers usually obfuscate are strictly parts of high importance to them:
- Initialization
- License checks
or simply whatever is "secret sauce" of your product (e.g. core part of an anti-cheat engine, an algorithm you came up with that performs a certain operation in a clever way, etc).
Although you do have a possibility of inserting protection markers inside "irrelevant" functions in an attempt to confuse reverse engineers, we come to the next point:
4. Plaintext data hints
When reverse engineering an application, one of the priorities is eliminating code you don't want to waste time on. Code that does not matter, even if it's obfuscated/virtualized.
Debuggers, disassemblers and other reverse engineering tools provide insight into data referenced by code, regardless of whether you're disassembling a binary statically or you're performing live debugging. Strings, instances of C/C++ classes, integer constants, error codes, etc. Being aware of the data that the code you're looking at works with is very often a huge hint on whether you should waste further time on it or you should simply skip it.
Many virtualization-based solutions have focused primarily on protection of code, with the exception of commonly provided "string encryption" functionality.
Our approach
What we're building at PROTEKKT is aimed at solving shortcomings of traditional obfuscations. Although the basic implementation principle from the perspective of software developers does not change significantly, we will explain what our solution does differently and why.
As we mentioned in the previous article, code obfuscation and protection against reverse engineering is primarily about complexity. It's about creating a "mess" that is hard to reason about for both humans and computers.
Our solution to significantly increasing the complexity of protected code can be viewed through three major approaches. Approaches that we don't apply separately, but as a combination resulting in highly obfuscated code and data that is not easily simplified by modern tools.
1. Data obfuscation
As mentioned earlier in the article, we believe that many software obfuscation solutions have overlooked and completely disregarded the huge importance of data obfuscation. Importance of data that plays a huge role in the amount of time required to reverse engineer an application or its internal algorithms.
While most do offer a simple "string encryption" interface, we believe that a modern solution needs to provide something more robust, an interface suitable for all types of data. From strings, boolean and integer constants to complex data structures.
For this reason, we've built an SDK that provides ability to easily obfuscate any type of data. Data types and data containers that are easy to integrate into your code, in many cases working as drop-in replacements for traditional ones. What our SDK offers is not just data obfuscation that is "obfuscated until used" -- it provides obfuscated storage and access at all times. It allows arithmetic operations on obfuscated data and comparisons against runtime inputs without restoring protected data into plaintext form.
In the next article, we will be talking more about our data obfuscation SDK and how we utilize modern programming language features to implement easy-to-use interfaces that provide highly obfuscated storage and access to data. We will demonstrate in more details what our library named POL (PROTEKKT Obfuscated Library) is capable of and how easily instances of STL containers are replaced by our variants - obfuscated variants of both primitive types and complex data containers (lists, ordered and unordered maps and sets, vectors, strings, tuples, etc). Containers that keep obfuscated forms of both data they hold and their internal structures.
2. Complex code obfuscation primitives
Effectiveness of code obfuscation depends primarily on the type of obfuscation primitives and operations performed rather than just the amount of them. We've built a code generation engine that heavily utilizes primitives that are known to cause deobfuscation and optimization/simplification algorithms to give up or explode in state.
One of our primary objectives was ability of our engine to generate code performing operations similar to those found in cryptography or performing operations using floating point math rather than just basic arithmetic and binary operations, as is commonly done in many code obfuscators. The reason behind this is that algorithms and tools have become extremely fast at understanding and simplifying millions of basic operations (arithmetic addition/subtraction and comparison between them, bitwise operations such as xor, or, shifts, rotates, etc).
Floating point math is hard for both humans and computers. It is the nature of how it's performed and implemented that makes it hard to reason about. With the exception of a limited number of operations and algorithms usually found in 3D game engines, video rendering and graphics in general, compilers and optimization tools are primarily built around simplifying integer math. On the other hand, floating point math is very often left as it is in its original form.
3. Gluing everything together
Data obfuscation and complex code obfuscation primitives are an effective way to make modern reverse engineering tools explode in terms of internal state and resources required. We decided to go a step further and combine both.
Our engine keeps track of all code and data that needs to be protected and inserts various dependencies between them even if they don't originally depend one on another. This allows us to introduce complexity in a clever way without actually changing the result of program output.
The primary objective of our obfuscation solutions is implementing it in a way that makes use of any feature implicitly improve complexity of all other components, without the developer having to do much work. Code obfuscation, data obfuscation, anti-dump and so on -- everything is automatically glued together and complexity of every component adds up with every feature used.
Read the next part: How we use modern C++ to enable seamless data obfuscation