March 2019 US Grid Cyberattack – An Engineer’s Takeaways

By | Grid, Security, Solutions, VigilantGrid, VulnTracker

First off, let me start by saying that while this event is being regarded as the first successful cyberattack to cause a disruption to the US power grid, there was NO IMPACT TO THE PHYSICAL FLOW OF ELECTRONS. Typically, when words like disruption or disturbance are used in the context of the power grid, somewhere a meter has stopped spinning or some serious transients have been introduced that cause undesirable grid conditions.  I don’t consider a communications failure a “grid disruption” since there is no impact to the physical grid. Also, this specific incident now classified as an “attack” could have been nothing more than an automated bot scanning WAN facing IPs looking for the specific vulnerable firewall to exploit. Nevertheless, as always, events of this nature can serve as an excellent lessons learned opportunity.

So What Happened: From 9:12 AM to 6:57 PM on March 5th 2019 a registered entity (this could be a power company or system operating company) experienced communication failures between a low-impact control center and multiple remote generation sites. The failure, which is described as being “brief (i.e., less than five minutes)”, was the result of multiple firewall reboots spanning over a 10-hour period. The vendor was called in to help diagnose the event and investigate the logs. Later that evening a patch was applied to one of the firewalls and tested in a non-critical environment. Afterwards, the firmware patch was applied to one of the remote generating sites and monitored throughout the night. After seeing no adverse affects of the patch, the rest of the vulnerable assets were patched.

Vulnerability Details: The root cause of this event is believed to be an externally exploitable WAN facing firewall. The Cisco Adaptive Security Appliance (ASA) Web Services Denial of Service vulnerability CVE-2018-0296 was first published on June 6th 2018 and has an impact rating of 8.6. Specifically, the vulnerability allows an unauthenticated, remote attacker to cause an affected device to reload unexpectedly, resulting in a denial of service (DoS) condition. The vulnerability is due to lack of proper input validation of the HTTP URL. An attacker could exploit this vulnerability by sending a crafted HTTP request to a vulnerable device. An exploit could allow the attacker to cause a DoS condition or an unauthenticated disclosure of information. This vulnerability applies to IPv4 and IPv6 HTTP traffic. Cisco has released software updates that address this vulnerability. There are no workarounds that address this vulnerability. Additional details are provided in the Cisco Advisory.

Known Exploits: An exploit intended to test the presence of the CVE-2018-0296 vulnerability was made public on June 21st 2018. This ‘tool’ which is intended to be used for good can instead be used for evil. The exploit code which comes complete with command line instructions is less than 58 lines of Python. The main lines actually exploiting the vulnerability are just 2 lines of code and they leverage the common Python requests package to make the HTTP connection. A user can run the script via the command line by simply typing ‘python <url>’ where url is the node you want to test (or attack). In addition to causing a denial of service attack, the exploit code will return the content of the current directory being traversed on the vulnerable firewall, files in +CSCOE+, active sessions, as well as enumerate valid usernames. Additionally, included in the Readme file of the GitHub repo housing the exploit are instructions on how to find vulnerable nodes using OSINT tools like Shodan and Censys. These tools can automatically return the IP address of the exploitable firewalls.

Here are my preliminary thoughts on the event and key takeaways:

  1. There was no “Grid Disruption” – A grid disruption means a physical impact to the flow of electrons.
  2. Monitoring Firmware and Vulnerabilities – The firewall was externally exploitable because there was a known vulnerability that existed. Exploit code existed before the vendor issued an advisory. This stresses the importance of knowing what assets are installed, the firmware version of those assets, and in using a tool like VigilantGrid to track vulnerabilities and firmware patches. Such tools are capable of notifying key personnel that a vulnerability exists and a patch is available. Additionally, depending on which tool you use, it should offer context as to how critical the patch is in the context of the power system and what the impact would be if the vulnerability is exploited.
  3. Monitoring for Suspicious Activity – Similar to monitoring for vulnerabilities and patches, the assets themselves can be made cyber-aware and can send event logs to a remote SIEM for analysis. In the case of this event, when the directory traversal exploit was initiated, the request likely could have been captured sending a Syslog message to the SIEM for analysis. Once deemed suspicious, the SIEM then sends an alert (text, email, automated phone call, etc) to an individual immediately identifying the cause of the communication failure and source IP of the attacker. SIEMs like VigilantGrid are capable of this type of analysis and can save time when investigating an event.
  4. Network Security Monitoring wouldn’t have Detected the Intrusion – The vulnerable firewall was externally exploited. Most network security monitoring (NSM) tools are on the LAN side of a firewall and therefore don’t inspect communication that terminates at the firewall. One of the main reason for installing a NSM tool on the LAN side of a firewall is to perform deep packet inspection. If placed on the WAN side the payload information would be encrypted making it impossible for the NSM tool to perform any type of deep packet inspection. Additionally, if placed on the WAN side, the NSM tool would likely trigger multiple false positive alarms in response to random drive-by scans. The fact that a NSM would be useless in detecting this type of attack stresses the importance of using and configuring assets to report events to a SIEM.
  5. Was the Grid Really Targeted? – If I were a betting man, my money would be on a bot or a script kiddie that carried out the attack. Given that this vulnerability has been exploited in the wild and it is for a common back-office IT vendor device, it is highly unlikely that the bot or attacker knew that it was targeting a power grid asset owner. Additionally, given the modulatory of the exploit code posted to GitHub and how easy it is to find vulnerable nodes via OSINT, it is more likely a drive-by attack from a bot.
  6. Using Universal Firewalls for the Grid – Cisco firewalls and switches were originally designed for back office IT environments and are commonly used across multiple sectors (finance, healthcare, government, etc). This means Cisco is one of the most targeted vendors on the market. If WAN facing, assets can be identified as being a Cisco product and then vulnerabilities and exploit code can be leveraged to compromise the system. Sadly, this is a side effect of using highly adopted vendors and is something that should be considered when designing power system environments.
  7. Comms Failure Happen all the Time, Grid still works – For the most part, the power grid is tried and true. It is specifically designed to work in isolation and in the presence of a communication failure. Communication failures happen all the time. Additionally, most asset owners at the distribution level have zero remote control over their field assets. In this particular event the comm failure was between a control center and a remote generation site(s). If the generation site(s) is a power plant, there will be staff on site who are capable of locally controlling and monitoring the site. These types of sites are manned 24×7, and if the control center wants to change a set value for the generator they can just pick up the phone and call the on-site operator.
  8. Goes Back to Engineering Design
    1. Whitelist Communication Links – Power system environments are extremely static, and all allowable communication links are known. This means that the network rules and access control lists can be restricted to only allow legitimate communication with the WAN facing asset.
    2. Redundancy – In system protection engineering (relaying and automation), there are multiple requirements to eliminate single points of failure. This has caused a many sleepless nights for engineers as they begin to think about how a system could fail and how to engineer redundancy and resiliency into the system. Some asset owners go to the extreme of having two separate protection relays from two separate vendors to protect the same piece of electrical equipment from damaging system faults. Do we need to start applying that same approach here? If there was a redundant backup communication link that relied on a non-Cisco firewall, comms between the control center and the remote generation site would have never dropped.

Thank you for reading and if you have any questions feel free to reach out at


Special thanks to @BlakeSobczak and @chrissistrunk for linking me back to the original articles.


Securing Your Web Server

By | Cloud, Security | No Comments

Whether you are running a personal website dedicated to pictures of cats or a multi-million-dollar online marketplace, securing your server from hackers remains difficult. Fortunately, there are some common practices that can reduce risk. 3 key practices are listed below:

  1. Restrict access to ports via a firewall. Servers are often misconfigured to allow services to be accessed on the network, but this is bad security practice since it provides bad actors with more points of entry. For SSH access to your server, consider using whitelist rules to prevent clients scanning the server from accessing this service.
  2. Encrypt traffic using SSL/TLS, redirecting all normal HTTP traffic to HTTPS (port 443). This will help prevent data loss via man-in-the-middle attacks.
  3. Keep server software up to date. New software versions often patch against vulnerabilities, so it is important to audit web server software from time to time to ensure the latest patches are installed.


2019 Capital One Breach: A Lesson Offered

By | Cloud, DevOps, Security | No Comments

The Capital One breach wasn’t done through the too-common open or purely misconfigured S3 bucket. It was a hack that serves as a lesson to anyone who deploys production web apps in the cloud. The hacker exploited a Server-Side Request Forgery Request vulnerability in a Web Application Firewall that targeted a tool that AWS itself provides to its EC2 instances. This tool is called the Instance Metadata service, and it allows an EC2 instance to make an HTTP GET request to the link-local IP to retrieve its own metadata. This metadata can include its own AWS security credentials to gain access to other resources in the victim’s AWS infrastructure.

This is a lesson to anyone who deploys applications on cloud platforms such as AWS. The lesson is that it pays to have more than the minimum-to-deployment knowledge of your cloud provider’s architecture and tools. It’s possible for a team to bring an application all the way to production on EC2 machines in AWS without ever knowing its Metadata service exists. Greater knowledge of your particular cloud provider can help you to consider the extra attack vectors that it may introduce.


Why We Created a Mobile App Dedicated to Power System Cybersecurity

By | Security, Solutions, VulnTracker | No Comments

The New Grid

Every aspect of the modernized electric grid relies on a cyber asset. From microprocessor-based protection relays to microwave communication equipment, today’s electric grid is evolving at an astonishing rate. This trend will continue as technologies become even cheaper and the desire for an energy efficient grid exists.

The objective of VulnTracker to help our clients and partners track cybersecurity vulnerabilities, news, and standards specific to the electric utility industry


The following points outline the primary motivation behind the app and why we are making it available for FREE!

Reason 1 – Shifting the Focus: If you work in or around the electric utility industry and hear the words ‘cybersecurity’, what comes to mind? For most, the first and only thing is NERC-CIP. With VulnTracker, users can access the wealth of other power system cybersecurity standards and industry best practices. Though compliance is important, a more sound approach is to first implement the technical standards and industry best practices and let compliance be the by-product. A goal of VulnTracker is to bring awareness to these publications via easy in-app viewing and sharing.

Reason 2 – Moving Past the Fear: Sadly, fear seems to be the primary impetus for implementing cybersecurity. This fear could be fear of a NERC-CIP violation and the associated $1 million/day per violation that follows or the fear of having a company’s name in the press. By focusing on engineering best practices and shedding light on the latest NIST recognized vulnerabilities affecting the grid’s constituent devices, VulnTracker helps asset owners move beyond the fear.

Reason 3 – Risk Managment: Operating and maintaining the power grid is often quoted as being an exercise in risk management. However, this endeavor is often placed on teams who may not understand the operating characteristics or role of say a protection relay or remote terminal unit. One textbook definition of risk is: RISK = (THREAT) x (PROBABILITY) x (IMPACT). Determining if a threat can be carried out requires an understanding of the OT environment. Additionally, to determine the impact of a cyber-event targeting the power grid requires input from power system engineers. By listing the vulnerabilities and providing a description of each, VulnTracker helps engineers and IT professionals promptly gauge the cyber risks for power system environments.

Reason 4 – News Filter: From election hacking to taking out a nation’s power grid, cybersecurity is a hot-button topic for the media. Often these news events are sensationalized and even twisted just to attract mouse clicks. With VulnTracker, we only share news from credible and original sources. This information can be found under Extras and is divided into two categories: Cybirical News and Industry News.

Reason 5 – IT/OT Bridge: With cybersecurity becoming more of an issue, the question is who and what department is responsible for the implementation and maintenance of cybersecurity in power systems. Often this burden is left solely to an IT department forcing them to place restrictions on those engineers who are responsible for designing, implementing, and maintaining a reliable system. The graphic below shows how this has been known to create an adversarial relationship between the IT and OT departments. The first example describes an engineer who actually kept a computer secret from his IT department because he wouldn’t be able to do his job if they knew he had the laptop. The second example describes the current state of several power system devices including relays, remote terminal units, automation controllers, etc. By explaining and organizing applicable cybersecurity and power system related terms and concepts, VulnTracker functions as a pocket reference to help bridge the IT/OT divide.

Last Modified: 11/06/2017

Note: VulnTracker is an evolving project to help asset owners address the cyber challenges of the modernized grid. As more features are added to the app over time, this article will be updated to describe the motivation behind those features.

Download FREE App Today!