Rob, this is a terrific article; it points out a continuing significant problem.
The better Etherenet arcitectures I've seen have an outer office-type network that is connected to the internet, a middle level used for supervisory control of the plant, and an inner for machine level control.
IT departments will have to become more agile. Our continuing trend of doing more with fewer people REQUIRES this.
Last week I was in a plant which had a SCADA server stop communicating with plant-floor HMI terminals. The engineer most knowledgable about the servers was out of the building, but used her smart-phone to remotely reboot it and get the floor functioning again.
Her plant did have very good security yet permitted the flexibility of remote access which permitted rapid response to problems.
Rob, proper security would end her access upon her termination of employment. But that problem isn't limited to remote access. A disgruntled IT employee can cause far more damage from within than without. That is a completely different problem.
Good point, TJ. But I do remember that when I asked what was the greatest threat to plant security systems, time after time, I heard, "A disgruntled former employee. One threat I heard less frequently -- but seems to me a bigger threat -- is the inadvertent attack from a malware bug that enters the system when an employee loads some music onto a workstation.
What a great article. This really points out the serious security threats posed by the plant's connection to the ERP system. Recently, we've heard a lot about theft of corporate intellectual property in big companies. But stuffing documents in a brief case will soon be passe. This is much scarier.
I agree, Chuck, cyber intrusion is scarier than papers in a briefcase. I asked once whether it would have been better if plants had kept their systems issolated as in the past. Apparently the benefits of the connectivity ae just too great to pass up.
TJ, I design, integrate, and maintain a distribution SCADA system and several plant control systems.
What you wrote about is seen by many as wonderful positive case for remote access. However, you have no idea what might happen if the remote access information were ever compromised. Would it be accessed by a vindictive spouse or child? Is the remote access software hardened enough?
Do note that I have seen SCADA systems built around Java, I have seen remote access software with very insecure hashes, I have seen lots of stories of Android and iPhone malware that gets behind the VPN and does all sorts of rude things. The volume of zero-days and patches on these platforms is frightening.
There really is no way to know with any certainty that the remote access software is safe. It wasn't that long ago that VxWorks, the OS for many embedded systems, was discovered to have used an extremely weak hash algorithm for passwords. A brute force hack against a VxWorks password hash file turns out to be trivial. So how many people know this about VxWorks and how many people know to patch this OS? Damned few.
Let's get serious here: It's not just the control engineers who are flummoxed with all this software. It's the IT departments too.
That said, If you expect long windshield times getting access to a site, if you are running extremely thin on staff, if you do not have remote site resiliency features available to you; then remote access is almost a foregone conclusion, regardless of whether you are worried about the security risk.
I believe that instead of expecting superhero engineers, we should be designing systems so that there is no need for these folk to swoop in and save the day from their iPhone while sipping a Daiquiri in Tahiti. In effect, we need to improve the robustness of the design so that remote access is not needed. Your positive story is not something I would highlight as a good thing.
"Can your iPhone bring down a plant?" There's an app for that! :)
Seriously, the three-level approach is the way to go, but many IT groups are not staffed for that. I also feel that many IT groups simply refused to consider dealing with BYOD, and so were caught flat-footed when they got run over by the 21st century.
Well-written and comprehensive article about many aspects to this growing concern as more and more smartphones are making their way into the realm of connected networks. This will become even more pressing as the Internet of things becomes a reality.
Thanks TJ. That remote access, however, is still a vulnerability. So far IT and control managers are not playing well together. Their priorities are in direct conflict. It will be interesting to see where this goes in the future.
Ann, vendors from companies like Rockwell and Siemens say that this arrangement has effectively solved the problem of the conflict between control and IT. The goal apparently is for everyone to take the side of the company and not the side of control or IT.
Rob, that's an amazing change and a long time coming. Hard to believe it's actually happened. I can believe that there's a goal like "taking the side of the company and not the side of control or IT." That reminds me a bit of TQM efforts years ago: it requires a huge change in corporate culture and is not easily done. Any idea how the actual change was implemented, not in the technology, but in people's behavior and minds?
If I remember these stories right, vendors like Rockwell and Siemens were involved in promoting these groups. I remember a Rockwell source shrugging it off, saying, "It's not that hard when you get everyone together."
Wow, that's a very big change from the days of attempts at setting up TQM systems: it was very hard to implement TQM here in the US, and in fact there many failed attempts at many companies. So either that statement is a lot of wishful thinking, or something really is different. If so, I wonder what?
That makes a lot of sense: the narrower goal. TQM required everybody in the whole plant to do everything entirely differently in all areas. It was pretty overwhelming. I can sure see how sacrificing uptime would be a no-op.
I believe TQM ran into problems with populations that were not convinced that changes would be improvements. The attitude seemed to be, "What on earth do you know about what we have to do? If it could be more efficient, we'd make it more efficient."
You're right, Rob. I wrote extensively about implementing TQM for awhile for CMP, including in-depth interviews with several companies that had tried and failed (as well as a few that tried and succeeded). But back then, those resistant "populations" were usually well over 50% of the company's employees.
I think those are good questions, Rob. I don't think younger employees are less territorial, at least not in my experience. It's also true that in the plant in those days they certainly weren't in charge of departments, since that was before the great downsizing that eliminated the middle management layer. But I think one big factor that's changed is that there's more emphasis today on teams than on hierarchy--not that hierarchy doesn't exist but it seems to have moved up the pyramid a ways.
Even more to the point of government intrusion, big corporations who "donate" to the right political party and have the standards slanted in their favor to get a market advantage that buries true innovation!
And with the revelations of IRS political hacks, this is an undeniable truth even if you live in the land of unicorns and believe our government is benevolent!
It is interesting to see these issues cross over from the Telecom world into the plant control world. In the Telecom(Service Provider) world, IT and Control departments use QA Production Labs to validate interoperability and security needs so that they have have confidence in their networks and their ability to handle attacks as well as perform correctly. The issue of a constant barage of software releases and patches dictates the necessity to stand up a lab for testing using automation to keep up with the flow of these releases. New equipment with new features (not to mention new phones and other network devices for the smartgrid) also warrant this sort of testing strategy just to keep the uptime acceptable for your production environment. We are writting up some case studies for companies that have proved out these concepts and will publishing them soon. Please contact me directly if your want to setup a strategy to prevent your plant from being downed from these new threats.
I agree, Notarboca, that standards can be more of a problem than a solution. The WIB certification may be of some help, but it's still an unknown. Meanwhile, plant systems are becoming increasingly open to outside technology
Web browsers on smartphones have gotten a lot better, but the web is a major source of malicious code. With a small screen of smartphones, it's more difficult for users to detect that a site is a phishing site. The malware can then be transferred onto the network from the phone.
Some smart phones OS bypass security mechanisms for user's convinenece. This makes it a lot easier and less frustrating for smart phones to connect to any plant's devices, but it also defeats the purpose of those security measures.
Most of the smartphones users connect to public Wi-Fi. If users connect their phones, containing company information, to an unsecured Wi-Fi network then a real security issue is created. If the same smartphone is connected back to the corporate network over a public Wi-Fi network, it could put the entire company network at risk. Users should be required to connect to the company network via an SSL VPN, so that the data traveling between the phone and the company network will be encrypted in transit and can't be read if it's intercepted.
Many corporations that allow employees to use their own mobile devices at work implement a BYOD security policy. BYOD security can be addressed by having IT provide detailed security requirements for each type of personal device that is used in the workplace and connected to the corporate network.
IT may require devices to be configured with passwords, prohibit specific types of applications from being installed on the device or require all data on the device to be encrypted. Other BYOD security policy initiatives may include limiting activities that employees are allowed to perform on these devices at work like email usage is limited to corporate email accounts only.
It seems to me that if a control system is critical, especially to safety. then it should not send ANYTHING through the aether. It's very hard to interfere, either accidentally or intentionally, with signals going through plain old copper wires.
One of my consulting clients followed a wise policy in this matter. All communications within the plant were wired, and if he needed to update a customer's program, he transmitted the software either by delivering a disk, or using a telephone modem which was plugged in ONLY while the program was being transmitted. Of course that meant his programs had to be elegant and compact enough to be transmitted over a limited bandwidth, but his code writers were good at that.
Disgruntled employees.. Have ALL WAYS been a risk to an organization.
New technology has done little to change this. (Yes, OK, it creates some new "twists" on "how" security measures are defeated).
Companies will always be dependent on the honesty of employees. No system for a manufacturing/processing plant will ever get around this. Accept this and plan security systems accordingly. There will always be an employee with the "keys to the kingdom".
What has changed. Dependence on physical access for security measures.
Any security that depends on physical barriers is no longer valid. Wireless networks, usb drives, CD drives, etc.. are only the beginning of the reasons why limiting physical access is a lousy security method. Very similar to issues relating to car theft. If the thief is determined and clever, the car will be stolen, regardless of the security systems put in place. A better plan.. assume physical security doesn't exist and plan with this limitation in mind.
Any security that depends on "obscurity" of it's system is no longer valid. A better plan assume your custom system (network, control, security) isn't really so "custom". This illusion is similar to Apple users assuming they are immune to computer viruses.
Multiple levels of networks within a facility.. can be good for security , but not great. It can be great for network robustness (keep the traffic where it belongs). Don't confuse the issues.
Educating to these new realities is the first priority. This is NOT easy. People have depended on physical barriers for security for all of history. It is engrained in to the way we view the world - even how we simply talk (whisper).
Technology is just forcing us to review our perspectives on security (and privacy).
Rob--excellent post. A little scary though. I have one client that absolutely prohibits the use of SmartPhones inside the plant facility. Any emergency call must come into a central office and a specific phone number. The individual needed by the caller is then notified to "call home" or whatever. To do this, he or she must use a landline or go outside to make the call. I actually thought the issue was time away from the job or texting on the job but now I'm not that sure. Do you know what protection nuclear power plants have relative to IT security? I hope this would be the ultimate protection.
Bobjengr, I would certainly guess the banning of cell phones is related to security rather than productivity. Cell phones are certainly welcomed in every other job setting. You have a good question about nuclear plants. Yikes.
The way to keep machines on a line secure even while smart phones are all around is to not have any wireless anything connected to those machines. Wired connections to a separate network that does not have internet access at any point, nor wireless access anywhere. Protection schemes that use only software are way less than completely secure, no matter who makes what assertions some may make. Remember "STUXNET" and how well it passed right through all kinds of security walls and fences. Just because one team sees no way to hack into a machine is no reason to believe somebody else can't do it.
Rob, of course it is almost always easier to go wireless, just as it is easier to leave your keys in the door lock, and easier to not lock the car at all. But many times the way to achieve the results that one wants require more than an absolute minimum effort. And those who refuse to deliver more than a minimum effort don't get any sympathy from me.
All of my customers are quite happy with my charges because they know that I never do a job using only the minimum possible effort. Almost always it is possible to deliver a much better value for "a bit" more effort. While delivering only the minimum to achieve the goal, only just barely, represents the best very short term return on investment, it seldom delivers the greatest long term value.
My point being that using a wired connection instead of a wireless connection does in fact deliver more security.
I now realize that STUXNET was not the best example of a serious virus passed through the internet, but that it is defintely an example of how sophisticated hacking can be.
William Your example.. (Stuxnet) illustrates just how pointless the networking security question is.
And how pointless physical security is.
The stuxnet infected machines were not on a network with any connection to the outside world. The machines were infected during the burning of the boot ROMs (installed during the manufacturing of the computer's motherboard) Nothing else was required. No wireless networks. No network connections to the internet. No network connections to other parts of the facility. No getting past physical barriers required. The people building the computers didn't know of stuxnet. The people installing the computers didn't know. The people monitoring the computers didn't know.
The Stuxnet virus looked for a local network connection with specific hardware (xx number of centrifuges of a specific make)... before doing anything (randomly screwing with their speeds).
Do you think any normally implemented hardware or software security system could have detected it? The answer is no.
It took an exceptional individual spending hundreds of hours in a completely different part of the world ..working a completely different problem to discover it.
The solution to security isn't going to come from "the usual suspects".
Just because someone defeated a system once, at manufacture is no reason to think it neccesarily will happen again. many things are done a First Time and then people learn. Remember the passengers brought down flight 88 before it could reach its target once they realized what was happening.
It is just logical that a closed loop wired system will be more secure. I wrote a ton of papers on my old DOS machine that stood alone, created designs delivered by print and/or disk and kept track of all of my billing and bank accounts. My only fear was some clown who would want to shove some disk in to share a "Really Neat Program". Touch it and you die.
All of this automated, wireless, paperless and anything else less, reminds me of lines from an old Creedence Clearwater Rivival song.
"Rushing through the treadmill,
Rushing to get home,
Worry 'bout the time you save".
I would not alllow a smartphone on the shop floor and maybe not even in the company. As for music, I do not think tuning in a local FM station will cause any system crashes.
While I agree that a wired, closed system is likely more secure than a system with internet access or wireless access.. It isn't really that much more secure from a disgruntled employee (my earlier point). Or if you really have a high value target (for industrial espionage or military).. physical barriers can still be breached without notice.. if the incentives are high enough.
And just because Stuxnet was exposed .. this didn't do ANYTHING to eliminate it. It cannot be easily removed from the MILLIONS of computers infected with it (it resides on the motherboard , not the hard drive). And why bother to remove it? It doesn't do anything unless you are running centrifuges with Siemens controllers. It is VERY possible you viewing this on an "Stuxnet infected PC". Do you really know the source of your BIOS? Do you verify your CNC machining centers to be free from hidden access codes? Most motherboard manufacturers use a handful of BIOS suppliers, regardless of PC brand. And the BIOS suppliers? They use programmers from all around the world.
Just because a "system was defeated once" .. doesn't really mean anything in protecting against it in the future in this case. It is beyond many manufacturers of equipment to really review ALL of the code in their machines. They buy device drivers, RTOS, bios, etc.. because the complexity is beyond their ability or the price is right. Should the world require ALL code be re-written by each company making equipment?
Are you prepared to test the security of all the devices on your internal network? Do you have access the the source code for all your equipment?
Can you say you have access to the quality of manpower required to do this job when there is a world of professionals trying to break in - undetected?
The reality: You really don't know how secure your facility is.
The likely situation? Your facilities real security resides in the fact, it is not a worthy target for professionals...
Which leaves you protecting your facility from curious kids or disgruntled employees..
The solution is deny yourself and employes possible methods of be more productive (smart phones/remote access)?
That may be reasonable for a some facilities. Not so reasonable for others.
Those responsible for a facility need to understand the issues, trade offs, risks and actionable items .... regularly.
You obviously know more about this stuff than I do. I tell people, "I know how to do what I know how to do on a computer and everything else is witchcraft."
However, it is apparent that the more we depend on computers to replace things which were once done on paper, the more these sort of problems will exist. I do not want to return to the days of old, but when I see two employees e-mailing each other across a vast chasm of 20-30 feet, methinks this is a ridiculous use of technology. When my wife and kids spend 30 minutes or so texting each other information that could have been handled in a 45 second phone call, it bothers me. When some company has their payroll handled in India because a saves a few bucks, I am not sympathetic when their bank records get hacked.
You are right in that I do practically nothing in which anyother company would be interested and the only hacker I need to worry about is some kid fooling around. I cannot say that I am disappointed by that.
I agree with you. People need to think about their use of technology, before creating life style choices and communications methods based on poor premises.
Sometimes an email .. even across the room.. can help document and clarify an issue within an organization. Other times , it is just a poor substitute for talking.
People need to choose the right tool for the job.
Before anyone down loads and uses a smart phone app... think! (about all the implications/consequences).
Trustworthy employees, educated to think about security and appropriate use of technology.. are likely the most important security features a facility can have.
Trying to provide additional security via standards is a noble idea. Doomed to failure without trustworthy and educated employees in place.
I digress a bit here.....
Much of corporate America has been trained NOT to trust their employees. Reasons are many, but biggest is because much of upper / middle management are not trustworthy themselves.
Good or Bad... NSA has to trust subcontractors like Eric Snowden for their networks. Let's face it .. the best IT administrators generally don't come with military backgrounds (with ideals similar to NSA, CIA, etc...with loyalty to their oaths). In fact, a large portion of network security specialists come with anarchy (hacker) back grounds (independent ideals trump oaths). NSA seeks out new employees at hacker conventions!
The NSA needs and hires these people assuming they can control them (don't they think like us?).
Yea... I don't like generalizations either... but you get the basic idea.
Facility security conflicts will not be resolved with technology alone. Resources should be applied to bigger issues (people) .. before being applied to the lesser (technology).
It is a similar situation with large facilities management. And not any easier to resolve.
I'm not sure of the relevance of the comment I'm about to make, but...I was recently researching a slideshow on engineering criminals and was amazed to discover, not only how young some of the best hackers were, but also how few of them had engineering backgrounds. In fact, the majority had only high school educations.
Rob, what I understood is most of the smartphones are becoming at par with laptop and desktop in terms of computing power. So I think it's possible to control the entire function through smartphones, but how? That answer yet to be find.
The people in IT are tasked with keeping company data safe. A problem arises when users fail to take into consideration the risk that unsecured devices can pose. When IT tells them that they need to time to research securing such devices the end user gets upset and complains that IT is being difficult. In my experience, IT does not start the "war", it is the end user that is trying to introduce an insecure device into the environment. Generally, IT is not given sufficient time to evaluate the device because the end user wants it NOW! The iPhone was a consumer device which was unsecure when it was introduced into the market. At that time Apple was very uncooperative with issues dealing with security. It was only after the IT departments in companies that were concerned about sensitive data starting pushing back that Apple finally started addressing security issues. IT is here to protect the end users from themselves. End users do not understand the implications of bringing untested technology into the environment. We all like to have police officers around except when we get a speeding ticket. Let your IT department do their job and protect your company.
Point well taken, Digerati Ohm. But it works both ways. IT has to understand the plant can't shut down in order for IT to install a patch at 2:00 am. That works for the office PCs, but not for the plant PCs if the plant runs 24/7. I think both conrol and IT have an issue with outside devices. But often the clash between IT and control doesn't have anything to do woth outside devices. it has to do with conflicting mandates.
At least when IT takes down services to install patches, it should be a scheduled maintenance period. Hackers don't care about schedules. I'd rather have a scheduled downtime to install patches than unscheduled downtime because I couldn't. But as I read the article it emphasizes the trend that users are pushing technology into the work environment without understanding the implications. IT must have time to evaluate the risk and do what they can to minimize it. If it cannot be eliminated completely, IT needs to communicate their concerns to management. Then, if management decides to sign off on the risks, IT has done their due dlilgence and responsibility now rests on management. But again, IT must have time to research and test the technology in their environment. This is no different than the standards of good manufacturing and design: R&D and QC.
One of the clashes between control and IT is that IT wants to update the patches more often than control has scheduled downtime. So, to meet IT's desired updating, control would have to shut down the plant more often.
Sometimes it is important to install patches in a timely manner, particularly if a new expoit is a zero day. So it boils down to this: A hacker or an exploit takes down your plant. Who are the executives and shareholders going to lynch, IS or the plant management? I can tell you it will be IS, and the argument will be that IS didn't explain the urgency clearly enough (which they did, but were ignored), and that IS is ultimately responsible for computer security.
Working a truce between the IT and control department seems logical but I think it is more theoretical than practically possible. If the teams couldn't work out an understanding on their own I don't see how they will when sat down together on a round table. Their mantras dangle on the opposite sides of the seesaw thus for the prosperity of one the other will have to take a blow in the neck.
I see your point, AnandY. The mandates of IT and control are absolute. For IT, no security breaches. For control, no downtime. I've heard vendors talk of peace between the two when a committee is set up to solve these issues and both IT and control are represented on the committee.
Cabe, this problem will grow as plants continue to shift to wireless devices. It's one thing to protect a wired network, but a wireless network is even more vulnerable. Wireless is attractive because is costs less and you can put a device in places where it's hard to run wire.
Focus on Fundamentals consists of 45-minute on-line classes that cover a host of technologies. You learn without leaving the comfort of your desk. All classes are taught by subject-matter experts and all are archived. So if you can't attend live, attend at your convenience.