What does the Basecamp project PLC disclosures mean to your operations?

We have been asked by several of our industrial control clients about the Basecamp PLC vulnerability disclosures, the new exploits coming to a Metasploit /SCADA folder near you, and how this might impact their operations. First, if you haven't read or watched the Basecamp PLC vulnerability presentation provided at the S4 conference this year, I would suggest that you watch it now at the Digital Bond web site. It is worth the 90 minutes, and if you were not already aware of how fragile your embedded PLC and RTU devices really are, this will be an eye-opener. (You can go watch it now and come back here...)

Now that you have watched the video, we can speak on the same page. The next knee-jerk reaction that some asset owners have is to look at the list of devices that were involved in the project, then compare those devices to their inventory, and then say... "Man... I'm glad we are not running any of those PLCs". Avoid that reaction. We have worked with a large number of embedded automation controllers and RTUs, and we can safely say that all have major cyber security issues.

The security vulnerabilities that are explained in this disclosure are systemic across almost all embedded devices used in SCADA and Industrial Control Systems. The next negative reaction that most have is to blame Digital Bond or blame the control system vendors. Neither of these two groups can really do anything today or tomorrow that can protect your system. Even if you nag your control system vendor for them to patch the problems discovered in the basecamp project, and they listen to you and repair these issues, more vulnerabilities will just crop up later. Yes, the long term strategy is to get the control system vendors to create hardened and secure devices from the factory, but that is a very long term strategy that will not address all of the millions of vulnerable devices already installed today or being installed tomorrow.

If your field controller is connected to an ethernet network using TCP/IP based protocols, just go ahead and assume that any of the vulnerabilities mentioned in the S4 video are potential security issues with your devices as well. We have seen PLC devices from major control system vendors that freeze or reboot just by sending them a flood of ping requests or by simply scanning them with NMAP or NESSUS. There are several documented examples of PLCs failing because a local printer ran out of ink and flooded the local control network with SNMP packets. So if these devices fail under simple network broadcast storms, it is easy to see how much mischief you can cause to these devices if you actually spend some time to send specifc attacks to these systems.

The point of this discussion is that with or without the recent Basecamp project disclosures, if your field devices are connected over an ethernet network, you are and have already been exposed to cyber risks to your automation or control system performance. Attackers do not need to be physically present on your control network for these vulnerabilities to work, since they can be routed into the control system from other networks, or become activated from a script or batch file on a USB stick. Air-gapped networks are not safe either.

So what do you do about this? Knowing is half the battle, and some will say that getting funding to fix the problems once you know about them is the other half. That is an entirely different topic, but assuming that you have the backing and support to actually make changes to your system and secure it, here are some steps to think about.

6 Steps to Securing Access to Ethernet-Connected PLCs and Field Devices

1. Flush out the logical data flow and architecture - Dive into the weeds and understand your control system in-depth. Build an accurate depiction of how the your PLC and field devices are connected back through the telemetry, LAN/WAN infrastructure, and back up through to your control room HMI and I/O servers. Ensure that everything is documented well, including the IP addresses of all major system components. If you need to, setup a laptop, run Wireshark or TCPDUMP, and capture actual SCADA protocol commands on the network to understand the full route from the control room out to the field devices. Try running tracroute to document all of the hops along the way. Knowing the logical data flows from the plant floor or field systems back to the control room is a great first step.

2. Determine what TCP or UDP ports are required to be open on your PLCs and field devices - This is not only a requirement for utilities that must map out their ports and services for NERC CIP compliance, but is also equally important for all critical infrastructure sectors. You can not adequately secure something when you do not completely understand its attack surface. For instance, if you are running a Rockwell Automation controller, and it uses ports 23, 80, 443, and 2222, you probably do not need the lower ports to be open to the controller, since the protocol operates over 2222. You can block all of the other non-essential ports. If you do not have this information, take a spare PLC off the shelf and scan all TCP and UDP ports to determine what ones answer back. You can also sniff the network while it is in use. Knowing how the devices are connected together (obtained from step 1 above), and then armed with what ports need to be open are the first two steps.

3. Next determine what source and destination IP addresses are used for PLC communications - This step may be a bit easier, since there should be only a finite number of computers that are required to poll the PLC network. In most SCADA systems, this number is less than 10. Do your homework to determine what computers really need to communicate with the controller network, and then conversely, what field devices are required to receive and answer back to these commands. Build a model in Excel using one column to show the list of computers that are communicating, then the next column to show the TCP or UDP ports that are used, then lastly a third column showing the devices that are the destination for the SCADA commands. This is not only an excellent troubleshooting tool for how the ethernet-based SCADA system functions at the PLC level, but can also be used to help secure or lock down the network.

4. Enable Access Control Lists (ACLs) between the computers in the SCADA system and the embedded devices in the SCADA system.  The computers in the SCADA system, such as operator consoles, engineering workstations, data historians, and special-use application computers all typically run on modern fat operating systems like windows or *nix systems. These need a large number of TCP and UDP ports to be open for them to function on the network. The embedded devices do not require the same ports to be open, so you can innoculate your field devices from being vulnerable to viruses, worms, and malware that affect your computers by simply blocking all non-essential ports. This way a virus would have to be written specifically for your field device in order for it to affect your control system. Create additional access controls that limit what source addresses can send packets into the control network. This can also help greatly reduce the attack surface by reducing the potential computers that have digital access to the ethernet-connected field devices. After this step, there should only be a limited number of computers that can access your field devices. If you can not introduce ACLs between your SCADA operations computers and the field devices, then at a minimum, back up to the firewall or DMZ that sits between the SCADA environment and the corporate IT or other networks. At a minimum, access controls should be placed at this level.

5. Enable Application Control, HIPS, or Application Whitelisting in those computers that can communicate with the field devices. If you remember, the Stuxnet attack did not directly infect the Siemens PLCs, but rather first infected a computer workstation that already had ethernet access to the PLC network. The engineering workstations were used as a weapon to deliver the Stuxnet malicious PLC logic to the PLCs. Now, because of the above 4 steps, you are aware of the finite number of computers that have the ability to communicate with the field devices using TCP/IP SCADA protocols. These computers should have end-point security on them, and we STRONGLY suggest application whitelisting. Application whitelisting can essentially freeze the computer systems that have access to the PLCs so that only those approved applications can run and execute on those computers. Any new code that is introduced to the system that is not already whitelisted will be blocked and will not execute.

6. Detection compliments active defense techniques - Now if you have gone through all of the above 5 steps, you should feel fairly good about how the system is locked down. The work is not done just yet. You have to assume that active defenses will eventually fail, and there should be a way to detect when you may be under an attack. Implementing both network and host-based IDS products can help alert you when packets are dropped at the perimeter that was intended for your field devices. Now that you know the TCP or UDP ports that your controllers are using, and their destination IP addresses, you can pull in the switch or firewall logs into a SEM product, and have it alert you when any network traffic was intended for any of your field devices or attempted to utilize their protocol ports, especially if it came from a non-approved source address. Parsing logs from the network devices is a great way to obtain situational awareness. To compliment that further, you can also add an IDS solution to the infrastructure and either write or obtain IDS siguatures for known control system vulnerabilities.

There are additional measures for protecting access to ethernet-connected PLCs and field devices, so this is in no way a complete list. It does provide some helpful tips and actionable things that most control system admins can do with their system to prevent accidental or intentional abuse of ethernet-connected embedded devices.

The main things that I took away from the basecamp PLC disclosures are:

- Almost all ethernet-connected field devices are vulnerable
- You can not secure your environment unless you understand how it works down to the device level
- Place as many access controls in front of the PLC devices as possible to reduce the attack surface
- Block all unnecessary devices and TCP/UDP ports so that only essential traffic is allowed
- Implement strong end-point security on the remaining computers that are allowed to communicate with the PLC devices
- Ensure to extract logs and meaningful security context off of network and host devices to detect abnormal or malicious behavior on the PLC network

Hope my comments in this briefing help illuminate the real issues surrounding the Basecamp PLC disclosures, and provide practical tips for PLC administrators to consider when securing or locking down the PLC network.

Thanks,
Jonathan

Various Musings about the recent Water Plant Hack

• Jonathan Pollet Why would Russian hackers want to burn up a water plant pump? We all knew the capabilities existed.. but.. What is the motivation? 1 day ago
  
  • Stacy Bresler Great question...something I have been recently asking the community in general related to various ICS attack scenarios, water plants included. It seems we often jump to the attack vector analysis and solutions leaving the questions about motivation unanswered or, at least, in the wings of the overall discussions. 1 day ago
    
    
  • Darin Dutcher Agreed that it is a good question, but I think there are more categorically distilled questions that can be asked about threat actors and motivations in relation to this target. 1 day ago
  • Stacy Bresler Absolutely! There is no shortage of questions to be asked :) 1 day ago
    
    
  • Peter H. Hu We may never know the true motivation behind the hack. However, we can attempt to deter their attacks by planning a security model around their potential motivations. C.R.I.M.E. model is a good one that comes to mind. Possible Motivators: Compromise Revenge Ideology Monetary Ego 1 day ago
    
    
  • Alan Rivaldo My response in a haiku --- A motivation | Is not the relevant thing | The end result is. 1 day ago
    
    
  • Alex Domshlak I guess that Russian hackers have not specific interests in Springfield, Ill. From my perspective it looks like kind of Proof of Concept. 23 hours ago
    
    
  • Eric Gallant Probably just a target of opportunity. A system on the public Internet using off the shelf software; easy pickings. Also, as in nearly every cyber attack, the question of attribution is a difficult one to pin down conclusively. Sure they used Russian IPs. But who's to say that means they are actually Russian or even in Russia? 10 hours ago
    
    
  • Jonathan Pollet All excellent responses. I agree about SCADA systems available from the Internet created with COTS software and running on Windows machines are VERY juicy targets. I also think that it was a proof of concept...hopefully all of these incidents are a wakeup call to asset owners. 10 hours ago
    
    
  • Ron Southworth So let's assume that this media news item is real. How do you secure a SCADA system that has been installed to provide a service to 2200 people given that the place is only held together by chewing gum and good luck. Such a place is usually operated by the police chief and the mayor or similar community officials. Believe it or not there are about 1000 water utilities in N America that are exactly the same or in a worse position. Is the local PUC going to approve the expenditure to secure such a system? 7 hours ago
    
    
  • Kelvin Rundle I am not convinced that COTS on Windows makes a target any easier or harder to attack than alternatives. I agree with Ron, small SCADA system operators neither have the resources or budgets to get the help needed to protect these systems, whether they be in the USA or Australia. 6 hours ago
    
    
  • Jonathan Bays Motive is a good question but in trying to assess the motive let's not get hung up on it having to be Russian or Chinese interests just because the more easily traced C&C server appears to be located there. There are so many pirated win machines in both countries that anyone from anywhere could be using them.

Barnaby Jack showcases how medical devices are vulnerable to embedded device hacking

At this year's Hacker Halted conference in Miami, I was in the audience while my friend Barnaby Jack conducted a live hacking demonstration using RF 900 MHz RF and an insulin pump. During the first part of his presentation, he went into great detail describing how he reversed the embedded device firmware and discovered the vulnerabilities that allowed him to send START/STOP/SUSPEND/DUMP ALL INSULIN commands to the pump without any knowledge of the device serial number or PIN.

The presentation and demonstration was a moment of awakening for many in the audiance that were not aware of how prevalent embedded devices are in our society, or how they can be utilized in ways the manufacturer did not intend. 

Update on W32.Duku

Based on research being performed, here is what the community knows about Duku:

• There has only been a few infections to-date
• It does not self-propigate or spread like Stuxnet did, so the victums have to be selectively targeted
• Unlike the initial perception, it does not seem to target control system equipment or vendors
• The attackers are using Duku to look for information, such as blueprints, design documents, and any information that can potentially be used to create a future attack on an industrial facility
• Some of the executables used within the Duku framework shares code with Stuxnet and appears to have been compiled after Stuxnet was discovered
• The malware is designed to self-desctruct after 36 days
• Duku may have other variants that are not currently detectable by Antivirus
• Seems to leverage C&C servers based in India
  
  

Steps asset owners should consider to avoid infection:

• update all AV signatures
• tighten up host prevention strategies and consider application whitelisting technology
• review all ports open on perimeter devices such as firewalls and reduce the attack surface by closing off unnecessary ports on network devices upstream of critical SCADA or DCS systems
• isolate control system elements from the corporate IT networks as much as possible
• operate at a higher treat level and review logs on an increased periodic basis
• monitor sensitive SCADA and DCS hosts for new and unusual services that are not traditionally running on those devices
• enable verbose logging on all host systems, and forward the logs to a centralized SEM or SIEM
• monitor hosts for new files added to system directories such as system32, and system32\drivers
• monitor outgoing traffic leaving the network to unknown destination IP address
• place a sniffer on the external side of the firewall and record traffic for 24 hours, then parse the data and cross check the IP addresses in the PCAP files to see if the traffic contains any IP addresses for known C&C servernet and botnets
• also monitor the dirty side of the firewall for spikes in traffic

Hope this briefing provides helpful insight as to how Duku works, and the migitation efforts that can reduce the chance of getting infected.

thanks,
Jonathan

Patching Issues and Outdated Web Browsers Still Plaguing SCADA and Industrial Control Systems

With each assessment that we perform on live operational SCADA systems, we continue to see themes or trends in the security findings. Some of the security issues that always makes the top 10 list are missing patches and hotfixes and outdated web browsers. While most of the SCADA and ICS community are well aware of the patching problems that continue to be an issue in operating and maintaining secure SCADA systems, the outdated web browser issue is often overlooked and misunderstood.

Older legacy SCADA and Process Control Systems used thick client applications installed on servers and workstations that were purpose-built and engineered for the SCADA application suite. As these systems evolve they become more like traditional IT systems. The IT community has long realized the benefits of moving from thick client to thin client architectures, and now most Enterprise IT applciations are browser based. As with most technology evolution in the SCADA market, the adoption of browser-based applications started to become mainstream several years ago.

About ten years ago, many of the SCADA and Control System vendors began to offer browser-based SCADA HMI, alarm, data historization, and data trending applications. They were marketing this capability as a way for Operations to easily access the data and information that they need to make decisions about the operation of their plant, all from the convienience of a simple web browser. The vendors started by implementing web servers in the SCADA servers, database servers, and view nodes. Eventually this adoption of web technology made its way down into the embedded devices, and even as early as 2001 several PLC and RTU vendors were shipping with embedded web servers running in their hardware. They also began using the term "web" in the name of their products, with examples like the "PlantWeb" system from Emerson - just to name one.

Although browser-based computing provide a greater ease of use for operations, this proliferation of web technologies deep in the SCADA environments creates a challenge for security. Many forms of malware and attack exploitation techniques leverage brower weaknesses, so while web-based SCADA systems are attractive to operations, it opens up the system to a much larger range of cyber attacks than when these systems were based on thick client applications.

We often find SCADA view nodes, HMI consoles, and data trending portals running with Internet Explorer 6, and in some cases we have found they have updated to IE 7, but it is extrememly rare to find a fully patched environment with up-to-date browsers. When we bring this issue up in our briefing sessions with our client's technical teams, the engineering and operations team often downplay the risk and say that they are not using the web browsers to go onto the Internet, so the risk is low. However, they miss the point that even if these older browsers are only used in a closed SCADA-Intranet purpose, just having the older browsers installed on the client and server systems leaves them open to malware and other attacks that can leverage the older browser framework to exploit the host.  

I find this a timely issue, and in fact just today Robert Lemos, Contributing Editor from Dark Reading posted an article about how outdated web browsers are still leaving many organizations vulnerable to attack. His article builds on this issue, and you can find it at the following link:

http://www.darkreading.com/vulnerability-management/167901026/security/attacks-breaches/231602264/outdated-browsers-leave-many-enterprises-vulnerable-to-attack.html

I suggest giving his article a quick read, and while you are reading the article, think about how this relates to SCADA and ICS systems. If you are a systems administrator for a SCADA environment that utilizes browser-based applications, you may want to take a quick inventory of your systems. It would be a great time to clean house, update all of the OS and security patches, and update those old vulnerable browsers.

Cheers!

Jonathan