On the 9th June, MWR InfoSecurity were at Singapore University of Technology and Design (SUTD) for a competition based around the cyber security of Industrial Control Systems (ICSs). This event, called S317, was a Capture the Flag (CTF) style event, designed to pit experts in ICS security from industry and academia against each other.
The S317 CTF comprised two components: an online qualifier and a live final. The online qualifier was open to any interested parties and required teams to solve a multitude of ICS-focused challenges. These challenges covered areas such as PLC programming, network forensics and reverse engineering. The top 6 qualifying teams were invited to Singapore for a live phase, which involved conducting practical attacks against large-scale, realistic ICS water treatment and distribution testbeds.
For this event, two MWR Security Consultants teamed up with two academics from Lancaster University’s Security Lancaster Institute. From MWR, William Jardine and William Knowles; from Lancaster University, Rob Antrobus and Kiran Joshi.
This post will introduce SUTD’s testbeds, the challenges presented to participating teams during the live phase, and the attacks conducted by the joint MWR and Lancaster University team. The post will be heavily oriented towards a discussion of the attacks conducted, and will cover, in particular:
General advice on measures that could be used to detect and defend against these attacks will then be discussed.
The testbeds which were used to host the CTF are called SWaT (Secure Water Treatment) and WADI (Water Distribution), and together encompass the intake of raw water, the treatment and cleansing of that water, and then its distribution.
The architectures of SUTD’s ICS testbeds have been described in great detail elsewhere; however, as a means of demonstrating the scale of these testbeds, and framing the discussion of the attacks conducted, SWaT is discussed in brief below. The architecture of SWaT is as follows:
This system consists of two main levels:
SWaT is made up of Allen Bradley ControlLogix PLCs, while WADI uses a mix of National Instruments Programmable Automation Controllers (PACs) and Schneider Remote Terminal Units (RTUs). Therefore, some of the attacks described below required consideration of the protocols used by these components: EtherNet/IP for Allen Bradley PLCs and Modbus over TCP for the Schneider RTUs. National Instruments PACs were not targeted during this CTF.
Competing teams were tasked with conducting a series of attacks that targeted particular sub-processes and components of the testbeds. These attacks were broadly split into two areas: attacks that involved physical manipulation of the processes (e.g., turning on pumps), and attacks that involved hindering visibility of the processes (e.g., manipulating sensor data). The points awarded to each challenge were also affected by two modifiers:
Due to the manner in which point scoring was awarded through the attacker profiles, MWR and Lancaster University’s goal was to complete the challenges through the outsider profile using covert mechanisms where possible. The first section below describes how the environment was initially compromised and a foothold gained, and subsequent sections will then discuss how this was leveraged to complete the challenges.
The SWaT environment had been implemented such that certain Level 1 devices were connected to the Internet. A mock phishing scenario was devised which targeted a user with access to one of these devices – namely, a SCADA/engineering workstation (a Windows PC used by ICS engineers to configure and observe devices on the network, and often used for tasks such as programming PLCs and HMIs). This scenario involved a Microsoft Word document which contained a malicious VBA macro.
This malicious macro was embedded within a legitimate advisory from ICS-CERT relevant to the components used in the testbed to add legitimacy to the attack. Through social engineering methods, the user was then convinced to click “Enable Content”, which silently executed a PowerShell command to establish a connection to an external command and control (C2) server. Communication between the compromised SCADA workstation and the C2 server was forwarded through other servers hosting seemingly legitimate websites. The communication was done over HTTP beacons to mask it as legitimate web browsing traffic, which aids in evading outbound filtering and detective security controls.
Once within the ICS network, with an implant running with the standard privileges of a user on the SCADA workstation, MWR then escalated to a high-integrity process and then to SYSTEM-level access. Mimikatz was then used to retrieve all credentials from memory, which revealed passwords for multiple user accounts and an administrator. Further enumeration identified that the SCADA workstation was on the same subnet as both an Historian (used to store historical process data) and an HMI (used to observe and control processes, therefore, affecting the physical state of the testbed).
The obtained credentials were used with psexec to pivot to the Historian database server and gain an implant on it. C2 traffic between the SCADA workstation and the Historian was then sent over SMB beacons; due to the legitimate nature of SMB within Windows environments, SMB beaconing is less likely to be detected as malicious behaviour. MWR were then able to issue commands on the Historian via the SCADA workstation over SMB, which in turn communicated back to the external C2 server using HTTP beacons as described above.
The HMI was identified as running Windows CE which is commonly found on embedded systems. Gaining an implant on the HMI was not necessary, as information found on the compromised systems revealed that the HMI was running a VNC server, which required no authentication. A SOCKS proxy was then deployed, which allowed MWR to run a local VNC client on their attacking systems, which communicated with the VNC server via the SCADA workstation over the HTTP beaconing C2 channel.
Diagram of attack path from an external attacker into the ICS network and to various attacks, using HTTP beacons, SMB beacons and SOCKS proxies
A Historian is a database used to record historical ICS sensor data, which is used to monitor trends in specific processes over time. Within SUTD’s SWaT network, the Historian used the commercial PI Server database, which collects, stores and organises data from multiple data sources. In SWaT, this is able to be filtered by “tag”, a textual descriptor of a particular sensor or actuator - for example, as can be seen in the online documentation for SWaT, the tag SWAT_SUTD:RSLinx Enterprise:P1.HMI_LIT101.Pv represents the tank fill level for the raw water process (P1).
Through reconnaissance of this Historian, it was found that it had a custom command-line utility included for viewing and editing historical data. This utility, piconfig.exe, was found at the following location:
C:\Program Files\Rockwell Software\FactoryTalk Historian\Server\adm\piconfig.exe
This tool was found to have a bespoke command language for viewing, editing and deleting arbitrary entries. This required a period of familiarisation, understanding the possible commands, the syntax and valid tags. The following Wiki found here was useful in understanding the options passed into piconfig.exe.
It was possible to enumerate all tags known to the Historian through piconfig.exe, and use this to overwrite arbitrary tag entries. For example, if an attacker was attempting to fill the raw water tank to a dangerous level, they may wish to overwrite the tag corresponding to the raw water tank level sensor in the Historian, in order to hinder an operator’s ability to notice suspicious trends. MWR were able to accomplish this by piping a batch script of the following commands into piconfig.exe:
@istr tag, value, time, mode
This would overwrite the observed tag value for P1’s level sensor at 10:30am with the value 10. This allows attackers to hide potentially dangerous physical process attacks from an operator and multiply the potential impact.
The above batch script could be used to overwrite all values in the last day by replacing the time stamp with *-1d. Historians are often used by organisations to assess performance of a process and make appropriate business decisions, so tampering with the data in this way could also introduce risk to the business as a whole.
After establishing a VNC connection over a SOCKS proxy, it was possible to interact with the HMI as though MWR were a local user on the SCADA workstation. This allows direct control over the SWaT testbed and effectively recreates an insider threat attack scenario, but remotely through a covert C2 channel.
MWR were therefore able to issue HMI commands over HTTP beacons from a remote C2 server, which were then forwarded to the HMI over the SOCKS proxy. This was used to open inlet water valves to the Raw Water process (P1) and close outlet valves, simulating a scenario in which the water tank could be filled to a dangerous level.
Whilst it is possible to physically affect the state of control processes through manual interaction with the HMI, any attempt to overflow a water tank or perform malicious actions is likely to be spotted by an ICS operator legitimately using the HMI on-site. Therefore, this sacrifices the stealth of the external C2 channel and minimises the potential impact.
An alternative way to do this is through overwriting the PLC’s logic code. A PLC is a device that has physical connections to the sensors and actuators, or a logical connection through another device that maintains the physical connections (e.g., as with the RIO in SWaT); its logic code describes the way it should manipulate the physical process (through the actuators) based on the information it receives (through the sensors). MWR launched two successful attacks against the SWaT testbed by modifying the legitimate PLC logic code:
This means an attacker can affect both the sensor data (i.e. the information displayed to an operator on an HMI or SCADA workstation) and the physical state of the process (i.e. the tank fill level, the pressure against valves and pumps, etc.).
PLCs can be programmed in multiple languages, including ladder logic, function block diagrams, and sequential function charts. For the purposes of brevity, pseudo-code is presented below to demonstrate an example of disabling valves so that operators using an HMI can never open them.
Cmd_Open = 0
Cmd_Close = 1
else if HMI.Cmd_Close:
Cmd_Open = 0
Cmd_Close = 1
During the CTF the legitimate PLC logic code was obtained from running devices, modified, and then redeployed using Rockwell Automation’s RSLogix 5000 software for SWaT’s Allen Bradley PLCs.
It would also be possible to create a more sophisticated attack through PLC logic code, although this was not feasible in the time allowed for the CTF. An example of this would be enabling pumps for each of the chemicals in the chemical dosing stage for twice the planned period of time, and only opening the exit valve to the next process when all of these chemicals have been deposited. This avoids obvious signs of compromise such as unresponsive HMI commands or fixed or unrealistic tank fill levels. Yet, it would have potentially fatal effects later on in the testbed cycle, as the water would now have non-safe levels of the chemicals in it.
Furthermore, it is possible to perform all of these logic code tampering actions using the same external C2 channel discussed previously. This may be a slow process due to the use of HTTP beacons, the CPU-intensive RSLogix software and the intrinsic overhead of compiling PLC logic code. However, a sophisticated attacker would likely have no problems taking a slow and stealthy approach to achieve their objectives.
All of the attacks up until this point have been from the perspective of an external attacker, using covert C2 channels. However, there are cases where a physical presence in the environments yields the most effectiveness. For example, during this CTF two bridged Man-in-the-Middle (MITM) attacks were established:
The Level 0 MITM is discussed in detail below. As discussed earlier in this post, Level 0 contains two PLCs (one primary, and one for redundancy) connected in a ring topology to the RIO components (which are in turn connected to the sensors and actuators), by either wired or wireless networking. Only one of these network modes could be operational at one time, and the attack targeted the wired use case.
Topology of MWR/Lancaster’s Level 0 bridged Man-in-the-Middle attack, used against the raw water process of SWaT.
To achieve a MITM at Level 0, a device was physically placed into the ring topology, with two interfaces operating in bridged mode. The attack then consisted of two phases.
First, analysis was performed on network traffic to identify which packets a MITM should edit. The target in this case was the tank fill level, which the MITM sets to a fixed value to hide its increasing value.
Second, a script was designed to perform this modification as packets crossed the bridged interfaces. IP forwarding across a bridge is done within kernel space, but modification of the packets is ideally performed from user space. A Python module called NetfilterQueue solves this issue by taking the packets before they are forwarded, placing them on a queue which can be read by arbitrary user space programs, and then forwarding those packets that are placed back on the queue. To prevent the programs processing those packets from having to process a large number of non-relevant packets, it is possible to selectively choose which packets are placed on the queue through iptables rules. In the example script below, only those packets being forwarded on UDP port 2222 (one of the ports used by the Ethernet/IP protocol) are placed on the queue. The MITM script then used custom Scapy dissectors developed by SUTD to edit both EtherNet/IP and DLR packets. Specifically, the payload of certain packets (i.e., the tank fill level value) was replaced with a set value. For example purposes, the value 331 was used here. The appropriate packets for modification were identified by selecting those with a payload length of 32 bytes. The length was chosen based on the observation during the traffic analysis that different tags were found to have a different payload length.
Security monitoring was being conducted within the Level 0 network; however, the MWR and Lancaster University team identified that if the direction of the flow of traffic within the ring topology could be determined (a viable situation for any insider attacker profile), a strategically placed attacking device could modify traffic before it is first viewed by any security monitoring solutions. This is shown in the diagram above.
A proof-of-concept Python script to perform this MITM is shown below:
#! /usr/bin/env python2.7
from netfilterqueue import NetfilterQueue
from scapy.all import *
from scapyENIP import cip
from scapyENIP import enip_tcp
from scapyENIP import enip_udp
from scapyDLR import DLR
print "Remember - bridge must already be started!"
os.system("echo 1 > /proc/sys/net/ipv4/ip_forward")
os.system("iptables -F -t nat")
os.system("iptables -A FORWARD -p udp --dport 2222 -j NFQUEUE --queue-num 1") # Request (Read||Write)
os.system("iptables -A FORWARD -p udp --sport 2222 -j NFQUEUE --queue-num 1") # Response
nfqueue = NetfilterQueue()
scapyPacket = IP(packet.get_payload())
if len(scapyPacket[Raw].load) == 32: # check packet length is 32
hex_string = '4a10000000000021d2c00061ebe100e1dbe1eba1e8e1eae1eae1e8e1eae1fe0f' # target packet in hex
target_value = 331.0 # target value must be a real number
num_to_put = hex(struct.unpack('<I', struct.pack('<f', target_value)))) # converts real 331.0 (target value) to hex
temp = int(num_to_put,16) >> 1
temp2 = hex(temp).replace('0x', '')
final_string = hex_string.replace('21d2c000',temp2) # place target value into hex string
body = final_string.decode('hex')
scapyPacket[Raw].load = final_string
The final attack attempted during this CTF was targeted at SUTD’s other water testbed, WADI, which simulates the distribution of the water cleansed by SWaT. To introduce diversity for the test environment, WADI uses different industrial equipment to SWaT. Specifically, this attack is concerned with the Schneider RTUs used by WADI.
RTUs are often used to collect information on sensor and actuator state, either through direct connection to these components or through retrieving this information from other devices, such as PLCs. RTUs typically then cache and send this information back to centralised monitoring systems. Schneider RTUs communicate over the Modbus over TCP protocol. Modbus is a plaintext request-response protocol where a Master requests information from a Slave; originally a serial protocol, it also has a TCP version.
Using both the smod framework and the pymodbus Python library, the MWR and Lancaster University team were able to map Modbus “registers” and overwrite arbitrary values. This allows an attacker to disrupt monitoring information in a similar manner to that described against the HMI in SWaT.
To do this, only a simple Python script similar to the following is needed:
from pymodbus.client.sync import ModbusTcpClient
client = ModbusTcpClient(192.168.1.100, 502)
# Write to input register address with specified value
example_register_address = 40100
# Check if value has been overwritten
result = client.read_input_registers(example_register_address)
Overwriting values of industrial devices similar to this has also previously been discussed within the context of Siemens S7 PLCs with sufficiently slow cycle times in the work of MWR’s William Jardine, described here.
Knowing how to spot these types of attack is key to protecting Critical National Infrastructure (CNI), and knowing how to execute these types of attack is key to knowing how to spot them. There are many sources of general guidance on the hardening of IT environments, which are useful for organisations looking to protect themselves against many of the attacks described here, and indeed many of these can be applied to ICS environments too. For example, the use of proper network segregation to separate the critical control devices and the Internet-connected corporate PCs would help limit the risk of the external C2 channel described above.
ICS vendors may also consider the use of Windows Privileged Access Workstations (PAWs), which separate sensitive and administrative tasks (e.g. those interacting with ICS components) from daily use workstations (e.g. read-only interactions like the Historian database). This is similar to the general ICS idea of air-gapping control and corporate networks, but on a more granular level. This could further protect critical devices like PLCs, and also help defend against the lateral movement described above to obtain a cybercriminal-like attacker profile. More info on PAWs can be found here.
Similarly increased security could be gained from more thorough auditing of all the devices in an ICS network and hardening of the user roles, firewall settings, etc. of all PCs on the network. It is important that ICS operators know what devices are on their network; MWR have recently developed a tool to assist in the passive mapping of network devices for critical systems like ICS.
The measures described above will go a long way to defend against many of these attacks, but the following sections will discuss some more tailored, ICS-specific potential security controls.
Detecting physical process attacks executed via the HMI can be difficult, as this can often be indistinguishable from legitimate behaviour. How do you tell the difference between an ICS operator opening a valve on an HMI and a malicious attacker on the network doing the same? One method of doing this would be to monitor key tag values (e.g. the fill level of the raw water tank) and other tags related to it (e.g. the outlet water valve); if the tank is nearly full and the outlet/exit water valve is still closed, malicious behaviour may be underway.
There are two ways defensive teams could detect these sorts of attacks by monitoring tags:
Both of these methods have advantages and disadvantages: analysing based solely on network traffic is beholden to the legitimacy of that traffic (which can be spoofed or interfered with), but actively requesting values can put extra strain on PLCs, which are often legacy devices. The benefits of both of these methods, and potential for combining them, is discussed in the paper “SENAMI: Selective Non-Invasive Active Monitoring for ICS Intrusion Detection” by MWR’s William Jardine.
However, when these attacks are instrumented by changes to a PLC’s logic code, as was done by MWR in S317, this becomes harder to detect. This is affecting the values of the PLC, rather than just the values displayed to an operator, so actively requesting the values from the PLC to monitor would also be fooled.
One method of detecting suspicious physical process manipulation when analysis of PLC values is no longer viable is by considering physical factors and the natural progression of processes. For example, if the value of the tank fill level in process 1 (LIT101) is being overwritten by PLC logic code to show a fixed value of 200, even though the tank is being filled dangerously high (say, to 1000), checking the PLC values would not detect this. However, looking at the PLC value for the tank fill level in process 2 (LIT201) would highlight that LIT201 is far higher than expected if the value of LIT101 was actually 200. This could be used to flag up suspected PLC-level tampering with values.
Alternatively, it is possible to monitor for uploads and downloads of PLC logic code and flag these as events. If these take place at suspicious times outside of normal operation, or are occurring only in specific processes, this may constitute an attack. This is again discussed in “SENAMI: Selective Non-Invasive Active Monitoring for ICS Intrusion Detection.”
When it comes to detecting malicious attempts to overwrite Historian values, a simple fix is to make all entries read-only after they are initially committed. This prevents subsequent doctoring of the data in the manner described above. While full SYSTEM-level compromise of the Historian would allow full control over the data, this mitigation would make it harder to achieve.
Setting values to read-only may not be possible for all Historian implementations though. Where this is the case, an alternate scenario is to log existing Historian values (as stored in the database) at set periods of time (e.g. every two hours). Then, every time the latest backup of that data is updated, it should be compared against the previous backup; if any changes have occurred between these two backups, that data point must have been tampered with.
The issue with this methodology is the ballooning space requirements; suddenly Historians are storing double the amount of data at any given time. At this point, the concern becomes whether space requirements or guaranteed integrity of data is more important to the business’s needs.
Throughout this CTF, MWR and Lancaster University’s team modelled a diverse range of attacks against realistic ICS testbeds, encompassing attacks against physical processes, from both internal and external perspectives. One of the key observations of this event is that, while insider threat is generally considered to be the most expected and realistic scenario, if an ICS network has any components connected to the Internet, a sufficiently motivated attacker can perform the majority of these potentially catastrophic attacks from the Internet.
This event also demonstrates that CTFs offer huge benefits to the ICS industry, and to any security practitioners involved in this space. A red team, offensive perspective on how a real-world attacker might target ICS offers blue teams tangible and solid ideas for how to best detect and defend against these threats.
Relevant scripts and code snippets have been provided within this article which can help security professionals with access to ICS testbeds to explore their security and develop ideas to protect our CNI.