Introduction to IDAPython for Vulnerability Hunting - Part 2

Overview

In our last post we reviewed some basic techniques for hunting vulnerabilities in binaries using IDAPython. In this post we will expand upon that work and extend our IDAPython script to help detect a real Microsoft Office vulnerability that was recently found in the wild. The vulnerability that will be discussed is a remote code execution vulnerability that existed in the Microsoft Office EQNEDT32.exe component which was also known as the Microsoft Office Equation Editor. This program was in the news in January when, due to a number of discovered vulnerabilities. Microsoft completely removed the application (and all of its functionality) from Microsoft Office in a security update. The vulnerabilities that ended up resulting in Equation Editor being killed are of the exact type that we attempted to identify with the script written in the previous blog post. However, calls to strcpy in the Equation Editor application were optimized out and inlined by the compiler, rendering the script that we wrote previously ineffective at locating the vulnerable strcpy usages.

While the techniques that we reviewed in the previous post are useful in finding a wide range of dangerous function calls, there are certain situations where the script as written in the previous blog post will not detect the dangerous programming constructs that we intend and would expect it to detect. This most commonly occurs when compiler optimizations are used which replace function calls to string manipulation functions (such as strcpy and strcat) with inline assembly to improve program performance. Since this optimization removes the call instruction that we relied upon in our previous blog post, our previous detection method does not work in this scenario. In this post we will cover how to identify dangerous function calls even when the function call itself has been optimized out of the program and inlined.

Understanding the Inlined strcpy()

Before we are able to find and detect inlined calls to strcpy, we first need to understand what an inlined strcpy would look like. Let us take a look at an IDA Pro screenshot below that shows the disassembly view side-by-side with the HexRays decompiler output of an instance where a call to strcpy is inlined.

Example of inlined strcpy() - Disassembly on left, HexRays decompiled version on right

In the above screenshot we can observe that the decompiler output on the right side shows that a call to strcpy in made, but when we look to the left side disassembly there is not a corresponding call to strcpy. When looking for inlined string functions, a common feature of the inlined assembly to watch for is the use of the “repeat” assembly instructions (rep, repnz, repz) in performing the string operations. With this in mind, let’s dig into the disassembly to see what the compiler has used to replace strcpy in the disassembly above.

First, we observe the instruction at 0x411646: `repne scasb`. This instruction is commonly used to get the length of a string (and is often used when strlen() is inlined by the compiler). Looking at the arguments used to set up the `repne scasb` call we can see that it is getting the length of the string “arg_0”. Since performing a strcpy requires knowing the length of the source string (and consequently the number of bytes/characters to copy), this is typically the first step in performing a strcpy.

Next we continue down and see two similar looking instructions in `rep movsd` and `rep movsb`. These instructions copy a string located in the esi register into the edi register. The difference between these two instructions being the `rep movsd` instruction moves DWORDs from esi to edi, while `rep movsb` copies bytes. Both instructions repeat the copy instruction a number of times based upon the value in the ecx register.

Viewing the above code we can observe that the code uses the string length found by `repne scasb` instruction in order to determine the size of the string that is being copied. We can observe this by viewing seeing that following instruction 0x41164C the length of the string is stored in both eax and ecx. Prior to executing the `rep movsd` instruction we can see that ecx is shifted right by two. This results in only full DWORDs from the source string being copied to the destination. Next we observe that at instruction 0x41165A that the stored string length is moved back into ecx and then bitwise-AND’d with 3. This results in any remaining bytes that were not copied in the `rep movsd` instruction to be copied to the destination.  

Automating Vuln Hunting with the Inlined strcpy()

Now that we understand how the compiler optimized strcpy function calls we are able to enhance our vulnerability hunting scripts to allow us to find instances where they an inlined strcpy occurs. In order to help us do this, we will use the IDAPython API and the search functionality that it provides us with. Looking at the above section the primary instructions that are more or less unique to strcpy are the `rep movsd` with the `rep movsb` instruction following shortly thereafter to copy any remaining uncopied bytes.

So, using the IDAPython API to search for all instances of `rep movsd` followed by `rep movsb` 7 bytes after it gives us the following code snippet:

ea = 0
while ea != BADADDR:
   addr = FindText(ea+2,SEARCH_DOWN|SEARCH_NEXT, 0, 0, "rep movsd");
   ea = addr
   if "movsb" in GetDisasm(addr+7):
       print "strcpy found at 0x%X"%addr

If we run this against the EQNEDT32.exe, then we get the following output in IDA Pro:

Output of Script Against EQNEDT32.exe

If we begin to look at all of the instances where the script reports detected instances of inlined strcpy we find several instances where this script picks up other functions that are not strcpy but are similar. The most common function (other than strcpy) that this script detects is strcat(). Once we think about the similarities in functionality between those functions, it makes a lot of sense. Both strcat and strcpy are dangerous string copying functions that copy the entire length of a source string into a destination string regardless of the size of the destination buffer. Additionally, strcat introduces the same dangers as strcpy to an application, finding both with the same script is a way to kill two birds with one stone.

Now that we have code to find inlined strcpy and strcat in the code, we can add that together with our previous code in order to search specifically for inline strcpy and strcat that copy the data into stack buffers. This gives us the code snippet shown below:

# Check inline functions
info = idaapi.get_inf_structure()
ea = 0

while ea != BADADDR:
   addr = FindText(ea+2,SEARCH_DOWN|SEARCH_NEXT, 0, 0, "rep movsd");
   ea = addr
   _addr = ea

   if "movsb" in GetDisasm(addr+7):
       opnd = "edi" # Make variable based on architecture
       if info.is_64bit():
           opnd = "rdi"

      
       val = None
       function_head = GetFunctionAttr(_addr, idc.FUNCATTR_START)
       while True:
           _addr = idc.PrevHead(_addr)
           _op = GetMnem(_addr).lower()

           if _op in ("ret", "retn", "jmp", "b") or _addr < function_head:
               break

           elif _op == "lea" and GetOpnd(_addr, 0) == opnd:
               # We found the origin of the destination, check to see if it is in the stack
               if is_stack_buffer(_addr, 1):
                   print "0x%X"%_addr
                   break
               else: break

           elif _op == "mov" and GetOpnd(_addr, 0) == opnd:
               op_type = GetOpType(_addr, 1)

               if op_type == o_reg:
                   opnd = GetOpnd(_addr, 1)
                   addr = _addr
               else:
                   break

Running the above script and analyzing the results gives us a list of 32 locations in the code where a strcpy() or strcat() call was inlined by the compiler and used to copy a string into a stack buffer.

Improving Upon Previous Stack Buffer Check

Additionally, now that we have some additional experience with IDA Python, let us improve our previous scripts in order to write scripts that are compatible with all recent versions of IDA Pro. Having scripts that function on varying versions of the IDA API can be extremely useful as currently many IDA Pro users are still using IDA 6 API while many others have upgraded to the newer IDA 7.

When IDA 7 was released, it came with a large number of changes to the API that were not backwards compatible. As a result, we need to perform a bit of a hack in order to make our is_stack_buffer() function compatible with both IDA 6 and IDA 7 versions of the IDA Python API. To make matters worse, not only has IDA modified the get_stkvar() function signature between IDA 6 and IDA 7, it also appears that they introduced a bug (or removed functionality) so that the get_stkvar() function no longer automatically handles stack variables with negative offsets.

As a refresher, I have included the is_stack_buffer() function from the previous blog post below:

def is_stack_buffer(addr, idx):
   inst = DecodeInstruction(addr)
   return get_stkvar(inst[idx], inst[idx].addr) != None

First, we begin to introduce this functionality by adding a try-catch to surround our call to get_stkvar() as well as introduce a variable to hold the returned value from get_stkvar(). Since our previous blog post worked for the IDA 6 family API our “try” block will handle IDA 6 and will throw an exception causing us to handle the IDA 7 API within our “catch” block.

Now in order to properly handle the IDA 7 API, in the catch block for the we must pass an additional “instruction” argument to the get_stkvar() call as well as perform a check on the value of inst[idx].addr. We can think of “inst[idx].addr” as a signed integer that has been cast into an unsigned integer. Unfortunately, due to a bug in the IDA 7 API, get_stkvar() no longer performs the needed conversion of this value and, as a result, does not function properly on negative values of “inst[idx].addr” out of the box. This bug has been reported to the Hex-Rays team but, as of when this was written, has not been patched and requires us to convert negative numbers into the correct Python representation prior to passing them to the function. To do this we check to see if the signed bit of the value is set and, if it is, then we convert it to the proper negative representation using two’s complement.

def twos_compl(val, bits=32):
   """compute the 2's complement of int value val"""
   
   # if sign bit is set e.g., 8bit: 128-255 
   if (val & (1 << (bits - 1))) != 0: 
       val = val - (1 << bits)        # compute negative value

   return val                             # return positive value as is

   
def is_stack_buffer(addr, idx):
   inst = DecodeInstruction(addr)

   # IDA < 7.0
   try:
       ret = get_stkvar(inst[idx], inst[idx].addr) != None

   # IDA >= 7.0
   except:
       from ida_frame import *
       v = twos_compl(inst[idx].addr)
       ret = get_stkvar(inst, inst[idx], v)

   return ret

Microsoft Office Vulnerability

The Equation Editor application makes a great example program since it was until very recently, a widely distributed real-world application that we are able to use to test our IDAPython scripts. This application was an extremely attractive target for attackers since, in addition to it being widely distributed, it also lacks common exploit mitigations including DEP, ASLR, and stack cookies.

Running the IDAPython script that we have just written finds and flags a number of addresses including the address, 0x411658. Performing some further analysis reveals that this is the exact piece in the code that caused CVE-2017-11882, the initial remote code execution vulnerability found in Equation Editor.

Additionally, in the time that followed the public release of CVE-2017-11882, security researchers began to turn their focus to EQNEDT32.exe due to the creative work done by Microsoft to manually patch the assembly (which prompted rumors that Microsoft had somehow lost the source code to EQNEDT32.exe). This increased interest within the security community led to a number of additional vulnerabilities being subsequently found in EQNEDT32.exe (with most of them being stack-buffer overflows). These vulnerabilities include: CVE-2018-0802, CVE-2018-0804, CVE-2018-0805, CVE-2018-0806, CVE-2018-0807, CVE-2018-0845, and CVE-2018-0862. While there are relatively scarce details surrounding most of those vulnerabilities, given the results of the IDAPython scripting that we have performed, we should not be surprised that a number of additional vulnerabilities were found in this application.

Introduction to IDAPython for Vulnerability Hunting

Overview

IDAPython is a powerful tool that can be used to automate tedious or complicated reverse engineering tasks. While much has been written about using IDAPython to simplify basic reversing tasks, little has been written about using IDAPython to assist in auditing binaries for vulnerabilities. Since this is not a new idea (Halvar Flake presented on automating vulnerability research with IDA scripting in 2001), it is a bit surprising that there is not more written on this topic. This may be partially a result of the increasing complexity required to perform exploitation on modern operating systems. However, there is still a lot of value in being able to automate portions of the vulnerability research process.

In this post we will begin to describe using basic IDAPython techniques to detect dangerous programming constructs which often result in stack-buffer overflows. Throughout this blog post, I will be walking through automating the detection of a basic stack-buffer overflow using the “ascii_easy” binary from http://pwnable.kr. While this binary is small enough to manually reverse in its entirety, it serves as a good educational example whereby the same IDAPython techniques can be applied to much larger and more complex binaries.

Getting Started

Before we start writing any IDAPython, we must first determine what we would like our scripts to look for. In this case, I have selected a binary with one of the most simple types of vulnerabilities, a stack-buffer overflow caused by using `strcpy` to copy a user-controlled string into a stack-buffer. Now that we know what we are looking for, we can begin to think about how to automate finding these types of vulnerabilities.

For our purposes here, we will break this down into two steps:

1. Locating all function calls that may cause the stack-buffer overflow (in this case `strcpy`)
2. Analyzing usages of function calls to determine whether a usage is “interesting” (likely to cause an exploitable overflow)

Locating Function Calls

In order to find all calls to the `strcpy` function, we must first locate the `strcpy` function itself. This is easy to do with the functionality provided by the IDAPython API. Using the code snippet below we can print all function names in the binary:

for functionAddr in Functions():    
   print(GetFunctionName(functionAddr))

Running this IDAPython script on the ascii_easy binary gives us the following output. We can see that all of the function names were printed in the output window of IDA Pro.  

Next, we add code to filter through the list of functions in order to find the `strcpy` function that is of interest to us. Using simple string comparisons will do the trick here. Since we oftentimes deal with functions that are similar, but slightly differing names (such as `strcpy` vs `_strcpy` in the example program) due to how imported functions are named, it is best to check for substrings rather than exact strings.

Building upon our previous snippet, we now have the following code:

for functionAddr in Functions():    
    if “strcpy” in GetFunctionName(functionAddr):        
        print hex(functionAddr)

Now that we have the function that we are interested in, we have to identify all locations where it is called. This involves a couple of steps. First we get all cross-references to `strcpy` and then we check each cross-reference to find which cross references are actual `strcpy` function calls. Putting this all together gives us the piece of code below:

for functionAddr in Functions():    
    # Check each function to look for strcpy        
    if "strcpy" in GetFunctionName(functionAddr): 
        xrefs = CodeRefsTo(functionAddr, False)                
        # Iterate over each cross-reference
        for xref in xrefs:                            
            # Check to see if this cross-reference is a function call                            
            if GetMnem(xref).lower() == "call":           
                print hex(xref)

Running this against the ascii_easy binary yields all calls of `strcpy` in the binary. The result is shown below:

Analysis of Function Calls

Now, with the above code, we know how to get the addresses of all calls to `strcpy` in a program. While in the case of the ascii_easy application there is only a single call to `strcpy` (which also happens to be vulnerable), many applications will have a large number of calls to `strcpy` (with a large number not being vulnerable) so we need some way to analyze calls to `strcpy` in order to prioritize function calls that are more likely to be vulnerable.

One common feature of exploitable buffers overflows is that they oftentimes involve stack buffers. While exploiting buffer overflows in the heap and elsewhere is possible, stack-buffer overflows represent a simpler exploitation path.

This involves a bit of analysis of the destination argument to the strcpy function. We know that the destination argument is the first argument to the strcpy function and we are able to find this argument by going backwards through the disassembly from the function call. The disassembly of the call to strcpy is included below.

In analyzing the above code, there are two ways that one might find the destination argument to the _strcpy function. The first method would be to rely on the automatic IDA Pro analysis which automatically annotates known function arguments. As we can see in the above screenshot, IDA Pro has automatically detected the “dest” argument to the _strcpy function and has marked it as such with a comment at the instruction where the argument is pushed onto the stack.

Another simple way to detect arguments to the function would be to move backwards through the assembly, starting at the function call looking for “push” instructions. Each time we find an instruction, we can increment a counter until we locate the index of the argument that we are looking for. In this case, since we are looking for the “dest” argument that happens to be the first argument, this method would halt at the first instance of a “push” instruction prior to the function call.

In both of these cases, while we are traversing backwards through the code, we are forced to be careful to identify certain instructions that break sequential code flow. Instructions such as “ret” and “jmp” cause changes in the code flow that make it difficult to accurately identify the arguments. Additionally, we must also make sure that we don’t traverse backwards through the code past the start of the function that we are currently in. For now, we will simply work to identify instances of non-sequential code flow while searching for the arguments and halt the search if any instances of non-sequential code flow is found.

We will use the second method of finding arguments (looking for arguments being pushed to the stack). In order to assist us in finding arguments in this way, we should create a helper function. This function will work backwards from the address of a function call, tracking the arguments pushed to the stack and return the operand corresponding to our specified argument.

So for the above example of the call to _strcpy in  ascii_easy, our helper function will return the value “eax” since the “eax” register stores the destination argument of strcpy when it is pushed to the stack as an argument to _strcpy. Using some basic python in conjunction with the IDAPython API, we are able to build a function that does that as shown below.

def find_arg(addr, arg_num):
   # Get the start address of the function that we are in
   function_head = GetFunctionAttr(addr, idc.FUNCATTR_START)    
   steps = 0
   arg_count = 0
   # It is unlikely the arguments are 100 instructions away, include this as a safety check
   while steps < 100:    
       steps = steps + 1
       # Get the previous instruction
       addr = idc.PrevHead(addr)  
       # Get the name of the previous instruction
       op = GetMnem(addr).lower()         
       # Check to ensure that we haven’t reached anything that breaks sequential code flow        
       if op in ("ret", "retn", "jmp", "b") or addr < function_head:
           return
       if op == "push":
           arg_count = arg_count + 1
           if arg_count == arg_num:
               # Return the operand that was pushed to the stack
               return GetOpnd(addr, 0) 

Using this helper function we are able to determine that the “eax” register was used to store the destination argument prior to calling _strcpy. In order to determine whether eax is pointing to a stack buffer when it is pushed to the stack we must now continue to try to track where the value in “eax” came from. In order to do this, we use a similar search loop to that which we used in our previous helper function:

# Assume _addr is the address of the call to _strcpy 
# Assume opnd is “eax” 
# Find the start address of the function that we are searching in
function_head = GetFunctionAttr(_addr, idc.FUNCATTR_START)
addr = _addr 
while True:
   _addr = idc.PrevHead(_addr)
   _op = GetMnem(_addr).lower()    
   if _op in ("ret", "retn", "jmp", "b") or _addr < function_head:
       break
   elif _op == "lea" and GetOpnd(_addr, 0) == opnd:
       # We found the destination buffer, check to see if it is in the stack
       if is_stack_buffer(_addr, 1):
           print "STACK BUFFER STRCOPY FOUND at 0x%X" % addr
           break
   # If we detect that the register that we are trying to locate comes from some other register
   # then we update our loop to begin looking for the source of the data in that other register
   elif _op == "mov" and GetOpnd(_addr, 0) == opnd:
       op_type = GetOpType(_addr, 1)
       if op_type == o_reg:
           opnd = GetOpnd(_addr, 1)
           addr = _addr
       else:
           break

In the above code we perform a backwards search through the assembly looking for instructions where the register that holds the destination buffer gets its value. The code also performs a number of other checks such as checking to ensure that we haven’t searched past the start of the function or hit any instructions that would cause a change in the code flow. The code also attempts to trace back the value of any other registers that may have been the source of the register that we were originally searching for. For example, this code attempts to account for the situation demonstrated below.

... 
lea ebx [ebp-0x24] 
... 
mov eax, ebx
...
push eax
...

Additionally, in the above code, we reference the function is_stack_buffer(). This function is one of the last pieces of this script and something that is not defined in the IDA API. This is an additional helper function that we will write in order to assist us with our bug hunting. The purpose of this function is quite simple: given the address of an instruction and an index of an operand, report whether the variable is a stack buffer. While the IDA API doesn’t provide us with this functionality directly, it does provide us with the ability to check this through other means. Using the get_stkvar function and checking whether the result is None or an object, we are able to effectively check whether an operand is a stack variable. We can see our helper function in the code below:

def is_stack_buffer(addr, idx):
   inst = DecodeInstruction(addr)
   return get_stkvar(inst[idx], inst[idx].addr) != None

Note that the above helper function is not compatible with the IDA 7 API. In our next blog post we will present a new method of checking whether an argument is a stack buffer while maintaining compatibility with all recent versions of the IDA API.

So now we can put all of this together into a nice script as shown below in order to find all of the instances of strcpy being used in order to copy data into a stack buffer. With these skills it is possible for us to extend these capabilities beyond just strcpy but also to similar functions such as strcat, sprintf, etc. (see the Microsoft Banned Functions List for inspiration) as well as to adding additional analysis to our script. The script is included  in its entirety at the bottom of the post. Running the script results in our successfully finding the vulnerable strcpy as shown below.

Script

def is_stack_buffer(addr, idx):
   inst = DecodeInstruction(addr)
   return get_stkvar(inst[idx], inst[idx].addr) != None 

def find_arg(addr, arg_num):
   # Get the start address of the function that we are in
   function_head = GetFunctionAttr(addr, idc.FUNCATTR_START)    
   steps = 0
   arg_count = 0
   # It is unlikely the arguments are 100 instructions away, include this as a safety check
   while steps < 100:    
       steps = steps + 1
       # Get the previous instruction
       addr = idc.PrevHead(addr)  
       # Get the name of the previous instruction        
       op = GetMnem(addr).lower() 
       # Check to ensure that we havent reached anything that breaks sequential code flow        
       if op in ("ret", "retn", "jmp", "b") or addr < function_head:            
           return
       if op == "push":
           arg_count = arg_count + 1
           if arg_count == arg_num:
               #Return the operand that was pushed to the stack 
               return GetOpnd(addr, 0) 

for functionAddr in Functions():
   # Check each function to look for strcpy
   if "strcpy" in GetFunctionName(functionAddr): 
       xrefs = CodeRefsTo(functionAddr, False)
       # Iterate over each cross-reference
       for xref in xrefs:
           # Check to see if this cross-reference is a function call
           if GetMnem(xref).lower() == "call":
               # Since the dest is the first argument of strcpy
               opnd = find_arg(xref, 1) 
               function_head = GetFunctionAttr(xref, idc.FUNCATTR_START)
               addr = xref
               _addr = xref                
               while True:
                   _addr = idc.PrevHead(_addr)
                   _op = GetMnem(_addr).lower()                    
                   if _op in ("ret", "retn", "jmp", "b") or _addr < function_head:
                       break
                   elif _op == "lea" and GetOpnd(_addr, 0) == opnd:
                       # We found the destination buffer, check to see if it is in the stack
                       if is_stack_buffer(_addr, 1):
                           print "STACK BUFFER STRCOPY FOUND at 0x%X" % addr                            break
                   # If we detect that the register that we are trying to locate comes from some other register
                   # then we update our loop to begin looking for the source of the data in that other register
                   elif _op == "mov" and GetOpnd(_addr, 0) == opnd:
                       op_type = GetOpType(_addr, 1)
                       if op_type == o_reg:
                           opnd = GetOpnd(_addr, 1)
                           addr = _addr
                       else:
                           break

Full Script Posted at: https://github.com/Somerset-Recon/blog/blob/master/into_vr_script.py