NOTE: This blog post is based on our DEF CON talk with the same title. If you would like to view the slides from DEF CON, they can be viewed here. Demonstration videos will be posted soon.
Overview
As IoT devices continue to become more and more commonplace, new threats and attack vectors are introduced that must be considered. Embedded devices contain a variety of distinct surfaces that a determined attacker could target. One such attack vector that must be considered is the development supply chain. In developing an IoT device, a development team requires a variety of special components, tools, and debuggers. Any of these products could be targeted by an attacker to compromise the integrity of the device that is being developed. With that in mind, we analyze the security of the Segger J-Link Debug Probes. Hardware debuggers, such as the J-Link, are critical tools in assisting developers with building embedded devices. Segger claims that their J-Link devices are “the most widely used debug probes in the world."
J-Link Attack Surface
The Segger J-Link debug probes come with a variety of supporting software packages that are used in order to interact with the debug probes. Included in this software is:
Many user-mode applications
USB driver
Full Integrated Development Environment (IDE)
Analyzing the user-mode applications that were distributed with the J-Link revealed that many of the applications were missing binary protections which can assist in preventing the successful exploitation of vulnerabilities. The analysis of the binary protections revealed:
DEP/NX was enabled
ASLR was enabled
PIE was not enabled
Stack canaries were NOT present in *nix binaries, stack canaries were present in Windows
SafeSEH was used in Windows binaries
As we began to analyze the applications included with J-Link, we quickly identified a number of input vectors that these applications accepted. These input vectors included command line arguments, files, and network interfaces. With this information in mind, we began to examine the applications’ security.
Vulnerability Research
After identifying input vectors and getting a feel for the applications, we determined to move forward with security analysis through a combination of fuzzing and reverse engineering. As we began to further analyze these applications and began to compare the Linux and Windows versions of these packages we found that the majority of the code was cross-compiled. This made our lives easier as we knew that the functionality was nearly identical between the Windows and Linux versions of the application.
Additionally, we realized that much of the interesting application logic appeared to require traversing deep, complicated code paths to reach. As a result of this, we decided to use a generational fuzzing approach in order to attempt to achieve better fuzzing coverage of these hard to reach code paths. This method involved using knowledge of the binary gathered from reverse engineering in order to determine the structure of data that each respective application expects to receive and leads to the “interesting” code sections and then recreating that data structure within the context of our fuzzer’s data specification format.
Since we were planning to generationally fuzz both network and file formats we decided to use the Peach fuzzer. Peach allows us to define our data formats in a simple XML file format and includes support for all of the desired input vectors (networking, files, command line) out of the box.
We then developed several data format specifications (known in Peach as pit files) and began fuzzing various J-Link applications. We started seeing crashes right away, but we also began to have issues as the J-Link debug probes entered a bad state and were disconnected from the VM that we were using for fuzzing. This caused our fuzzing to halt as the applications that we were fuzzing require that a J-Link device be present as the applications require the device in order to properly execute.
In order to keep our J-Link attached to our VM we developed a custom crash monitor in order to ensure that the device was attached prior to executing any fuzzing iteration. The crash monitor is triggered on any crash that occurs while fuzzing and executes a user-specified set of actions. We wrote a custom script for the crash monitor to execute that utilized libvirt to check if the J-Link device was still attached to the VM and, if it was not attached, then reattach it. This allowed us to continue fuzzing the applications without issue.
Soon we were forced to halt our fuzzing efforts since we had observed so many crashes that we were running out of disk space due to the crash data stored by Peach. While triaging the crashes we noticed some interesting things about the crashes. First, we observed that a huge number of crashes were identical and were being automatically flagged as exploitable (thanks to !exploitable). These crashes made up such a large portion of our crashes that we received less coverage overall from our fuzzers than we had initially hoped for. Further analysis of our crashes revealed that, while we had built the data models to reach deep and interesting code paths, easy to trigger bugs that were located early in the execution path were causing crashes prior to reaching the code paths that we had initially targeted for security analysis.
Even though we received less coverage than we had hoped, we were still left with a variety of distinct crashes that appeared to be exploitable after some initial triage. With this information, we then attempted to see if we could fully exploit any of the issues that we had discovered.
Vulnerabilities Discovered
CVE-2018-9094 - Format String Vulnerability
One of the first vulnerabilities that we found was a format string vulnerability. This vulnerability can be demonstrated with a command line argument. Simply passing in a format specifier (such as “%p”) as part of the file name on the command line results in the format specifier to be formatted as such. We show this below with “fileName%p” in the command line being converted to “fileName00922C0F” in the output.
Format String Vulnerability Demonstration
In the vulnerable application, JFlashSPI_CL.exe, we have a user controlled string which inserted into a larger log message string via sprintf. This log message string is then passed as the first argument into a “printf-style” function which accepts format specifiers. Hex-Rays decompiler output showing this vulnerable code snippet is shown below and clearly shows the vulnerability.
Hex-Rays Output of the Source of the Vulnerability
Further reversing reveals that the “custom_printf” function shown above is a logging function that is included in the J-Link applications. This function accepts a subset of the format specifiers that are accepted by printf and does not accept the “%n” family of specifiers that allow one to generate arbitrary writes via a format string vulnerability. With that being said, it is still possible to generate an arbitrary read with this code.
We can demonstrate this with the following command:
JFlashSPI_CL.exe -open xAAAA%X%X%X%X%X%X%X%X%X%X%X%X%X%X%X%X%X%X%X%X%X%X%X%X%X%X%X%s
Running this command results in an arbitrary read, in this case of the address 0x41414141 (or “AAAA”) and causes the following exception to be thrown:
Exception Thrown when Reading Memory "AAAA"
CVE-2018-9095 - Command File Stack Buffer Overflow
Next, we analyzed the individual crash that made up a huge majority of our total crashes and caused us to eat up most of our VM’s disk space. This vulnerability is a traditional stack buffer overflow. In this vulnerability, we can overflow a stack buffer by including a line with 511 characters or more in a command file that is parsed by the JLinkExe application.
Since this application is compiled with NX protections in addition to ASLR, we are forced to bypass these in order to exploit this vulnerability and gain arbitrary code execution. In order to bypass the NX protections, we utilized ROP in order to perform all operations needed to gain execution. Using Ropper, we searched through the ROP gadgets that were present in JLinkExe in order to bypass ASLR and then gain execution. Since ASLR was enabled, we need to determine the address of libc and, subsequently, system(). Fortunately, we have enough gadgets present in JLinkExe in order to leak the address of libc.
We are able to leak the address of libc using a traditional GOT deference technique in which we dereference the GOT entry of some libc function and then utilize that dereferenced address in order to calculate the address of the desired function - in our case system() - by performing arithmetic to add or subtract the static offset of our desired function from the dereferenced function.
This left us needing to pass an argument to system() in order to gain code execution. Since this vulnerability is triggered by a file that is local to the exploited system, we focused on just getting shell via this system rather than passing arbitrary, user-controlled payloads to it. This made finishing our exploitation somewhat simpler since we now just had to find a string to pass to the call to system in order to get our shell. However, since we were unable to send null bytes, we were limited with regards to which addresses we could pass as an argument to system. For example, our first thought was to use the “/bin/sh” string in the libc library, but the address of that string contains a null byte, forcing us to use a different address. As a result of that, we started focusing on strings in the valid address space and realized that we had a number of options to work with.
$ strings JLinkExe | grep "sh$" fflush SWOFlush .gnu.hash
By pointing our argument to system() into the middle of a string that ended with “sh” we were able to execute the “sh” command as our argument to system(). This allows us to bypass the limitations on addresses and launches a shell for us, giving us the execution that we desired.
CVE-2018-9097 - Settings File Buffer Overflow
In the course of our fuzzing we found another buffer overrun caused by an input file JLinkExe. This vulnerability was caused by a buffer overrun in the BSS segment of a shared library that was used by the main executable. At this point, we could write an arbitrary number of non-null bytes into the BSS segment of the libjlinkarm.so.6.30.2, but we still needed some way to gain control of execution in the application in order to execute our code. Since we had the ability to overwrite arbitrary data into the BSS segment, we began to look into ways that we could turn that into code execution.
Using the IDA Sploiter plugin in IDA Pro, we searched for writable function pointers that we could overwrite. Fortunately, we were able to identify a number of overwritable function pointers following our overflowable buffer. After a little bit more research into those function pointers we were able to identify a function pointer that we could overwrite without causing the application to crash and would also get called soon after overwriting. With that, we were able to consistently redirect execution to a user-controlled location. At this point, we decided not to finish full exploitation on this vulnerability as and to instead focus more on the remote vulnerabilities that we had found. The reason for this was the similarity in the attack vector and impact of exploitation between this vulnerability and CVE-2018-9095.
CVE-2018-9096 - Telnet Stack Buffer Overflow
One interesting observation that we made when initially analyzing the applications that are included with the J-Link was that an application, JLinkRemoteServer, listens on a number of ports. Of particular interest to us was that the application listens on port 23.
$ sudo netstat -tulpn | grep JLinkRemote tcp 0 0 0.0.0.0:24 0.0.0.0:* LISTEN 31417/./JLinkRemote tcp 0 0 127.0.0.1:19080 0.0.0.0:* LISTEN 31417/./JLinkRemote tcp 0 0 0.0.0.0:19020 0.0.0.0:* LISTEN 31417/./JLinkRemote tcp 0 0 127.0.0.1:19021 0.0.0.0:* LISTEN 31417/./JLinkRemote tcp 0 0 127.0.0.1:19030 0.0.0.0:* LISTEN 31417/./JLinkRemote tcp 0 0 0.0.0.0:23 0.0.0.0:* LISTEN 31417/./JLinkRemote
Since port 23 is commonly used by telnet we decided to focus some of our reverse engineering efforts on this executable to determine what the purpose of this open port is. After beginning to look into this portion of the application in IDA Pro, we were quickly able to identify that this was, in fact, a Telnet server within the application.
Hex-Rays Output Showing the Creation of a Telnet Server Thread
With this information, we configured our fuzzer to focus on the open ports with a special focus on the Telnet server as it appeared to have a large attack surface. Fuzzing the Telnet server revealed one interesting crash that appeared to be exploitable but was difficult for our team to reproduce. Additional analysis led us to discover that this was the result of a stack-buffer overflow. However, we were still unable to consistently trigger that crash. After further reversing of this application we were able to identify a race condition caused the application to either crash or continue running.
While we were unable to cause this crash to happen consistently, we were able to write an exploit which exploited this vulnerability in the instances where the race condition triggers the crash. In order to do this we followed a similar technique to what we used in the local stack buffer overflow exploit where we utilize a ROP chain in order to leak an address inside of libc through a GOT dereference and then use that leaked address to calculate the address of system().
Once we had the address of system() we wanted to develop a method for the exploit to execute an arbitrary, user-controlled command. While looking at the state of program memory during exploitation, we found that user-controlled data was also being stored in static locations in program memory. Due to the multithreaded nature of the JLinkRemoteServer application, the exact location where this data was stored in memory varied between two locations. Due to these locations being somewhat close to each other in memory, we attempted to develop a solution in order to allow our exploit to work consistently, regardless of which memory location the data was stored at.
While brainstorming potential solutions to this problem, we devised a solution using a clever trick. This trick is very similar to using a nop-sled. A nop-sled is when a shell-code payload is prepended with many nops or no-op instructions in order to increase the likelihood of executing the shellcode payload by allowing the application’s execution to be redirected anywhere into the nop-sled that was prepended to the payload and always execute the payload since the nop instructions are valid instructions which do not change the state of the application.
As we thought more about this technique, we began thinking about whether there was anything similar that we could prepend to our text-based command payload which would have the same effect. We immediately decided to try using spaces to prepend to our payload in order to try what we termed as a “space-sled”. Using this space-sled we prepend spaces to our command so that regardless of which location the user-controlled data was copied to, we would be able to point to the latter location in memory and land in a usable portion of the command string.
CVE-2018-9093 - Tunnel Server Backdoor
Lastly, we have the J-Link tunnel server which effectively is a backdoor to J-Link devices via a Segger proxy server. The purpose of the tunnel server is to enable remote debugging of embedded devices, but, given that the tunnel server does not implement even the most basic of security measures, in doing so opens any J-Link device using this feature vulnerable to attack.
When the remote server runs with a J-Link device attached, the JLinkRemoteServer application registers the J-Link device serial number with the Segger tunnel server. In order to remotely access this remote device, a client must connect to the tunnel server and provide a serial number of the device that the client wishes to connect to.
Since serial numbers are 9 decimal digits, this means that there are 10 billion possible serial numbers. Assuming that valid serial numbers are randomly distributed throughout this space and the rate that we can check to see if a serial number is connected is 10 serial numbers/sec (this is about what we’ve seen in our testing) it would take over 31 years to check the entire serial number space. This seemed large enough that it would prove generally unfeasible for an attacker to brute force serial numbers on a large scale.
However, we realized that if we could shrink the space of serial numbers then we could potentially reduce the amount of time required to brute-force the space of serial numbers. In order to attempt to reduce the number of serial numbers that we needed to brute force, we began by trying to gather as much information on device models and serial numbers as we could. We did this by searching online for images of J-Link devices where we were able to read the device serial numbers from the images as well as gather the serial numbers from all of the devices that we had access to. Between these two methods we were able to gather around 30 J-Link serial numbers to analyze.
After reviewing these serial numbers we were able to detect several patterns in the way that Segger assigned J-Link serial numbers. The serial numbers seemed to be divided into three distinct sections: device model, device version, and an incremented device number.
Segger Serial Number Divided into Sections
With this information, we were able to reduce the serial number space required in order to get good coverage of J-Link serial numbers from the initial 10 billion to around 100,000. Using the same rate as above, that effectively reduces the time to brute from from over 31 years to less than 3 hours. This reduction in address space suddenly makes an attack on the Segger J-Link server much more appealing to an attacker.
Impact of Vulnerabilities
The vulnerabilities discovered are listed above in increasing level of severity. Below we discuss the impact of each of these vulnerabilities and explain the potential impact of an attacker exploiting each vulnerability.
The first vulnerability, CVE-2018-9094, is not able to be used to gain code execution due to the custom format string functions that are in use and, even if it were, would be a lower severity simply due to the input vector (the command line) that is used to exploit that vulnerability.
Next up, we have CVE-2018-9095 and CVE-2018-9097. These two vulnerabilities involve malicious files and result in gaining code execution when a malicious file is opened by a J-Link application. These are more severe since malicious files are a common attack vector and are able to be spread via email or any other file transfer mechanism. An attacker could use one of these vulnerabilities to attempt to gain access to the a system by getting an unwitting user to open a malicious file in a vulnerable application. As a result, these vulnerabilities are a high severity, but exploitation still relies on a user opening an untrusted file.
CVE-2018-9096 is our first remote exploit. This exploit allows a user to gain code execution on a system remotely and without any interaction from the victim. This is a critical vulnerability as it allows an attacker to spread throughout a network to any machine that is running the vulnerable application.
Lastly, we have CVE-2018-9093. This vulnerability is perhaps the most severe of these vulnerabilities. This vulnerability allows attackers to gain access into a network, bypassing firewalls and other security systems to gain access inside a network. Once inside the network with this vulnerability an attacker could read or write the firmware of any devices attached to the remote J-Link. To make matters worse, this vulnerability simply requires knowledge of the Segger J-Link serial number scheme and a small bit of reverse engineering to discover the hardcoded “magic numbers” that Segger uses in order to make a connection to via their tunnel server.
Disclosure
Following this research, we disclosed these vulnerabilities to Segger on April 4, 2018. They were acknowledged by Segger shortly later and we have seen many of these issues patched in several of the subsequent releases of the J-Link software.
Conclusion/Takeaways
As we have previously demonstrated with our security research into Hello Barbie and Election Safe Locks, many embedded devices have severe security flaws. With that being said, if you are a developer or device manufacturer who is attempting to build a secure device it is important to consider your supply chain. As we have shown here, important elements of the device development process oftentimes contain vulnerabilities that could compromise the integrity of a device that is being developed as well as the entire network.