October 18, 1999
The challenge of increased computer power, higher speeds and modern networks.
SCADA systems and the network environment they operate in are changing. More computer power is engineered into the latest SCADA equipment. More data must be moved to meet the information demands of more powerful SCADA systems. And these systems must operate over new types of networks, including frame relay, Ethernet, and IP.
Supervisory Control and Data Acquisition (SCADA) refers to the process of gathering information, often in real time, from remote locations. The data comes from oil wells, pipelines, electric power grids, traffic signals, manufacturing plants, etc. This information is used to analyze and control the performance of these systems. For example, well information might include the volume of liquid pumped over a period of time and the amount of power used. Pipeline information includes the volume of flow of liquids or gases over a period of time. The control aspect may include controls to a well pump to increase or decrease output, or shut down altogether. Pipeline controls may include changing routing, increasing or reducing the flow of the liquids or gases, etc.
From the perspective of the newest data networks based on IP protocols, Ethernet, and frame relay, networking personnel may call SCADA protocols legacy protocols. Legacy means primarily private wire, private microwave and telephone company point to point and multipoint private line networks. The way the term "legacy" comes across, it often sounds like a negative term, and heaping scorn on any protocol that gets labeled "legacy". Do not fall into the trap of thinking that a protocol that you or any one else calls "legacy" is bad, outdated or useless. Some of the most robust protocols and control systems in existence are what we would call legacy protocols. For example, compare the reliability of SCADA systems to that of the desktop PC and its operating systems. Which is likely to stay up and working the longest without requiring a reset or a reboot, the PC or a SCADA system? Just about any SCADA or other type of control systems wins hands down. Or consider the reliability of the Internet over the long term. Today the Internet cannot yet be counted on to deliver its data with the same reliability and timeliness of any SCADA systems. Comparing the Internet, a near universal system, with a SCADA network, a very limited, special purpose network may not be totally fair. So compare the Internet with the underlying reliability of the US telephone network. Clearly, the Internet is less reliable than the US telephone network. In applications throughout the world, SCADA is far more reliable than many of the local telephone networks.
Because of its need for reliability, many SCADA systems use private microwave, private wire, and private optical cable networks. This is necessary because SCADA systems often have very critical control functions. A petroleum pipeline failure can be catastrophic. Leaking oil could contaminate water supplies, cause disastrous fires, etc. An undetected failure of a power sub-station could result in a major power outage. SCADA systems worldwide perform critical functions that require robust networks.
Today, we are seeing data communications networks evolve to a new environment that can cause significant challenges for current SCADA networking practices. In addition, as more computer power is embedded into new SCADA equipment, there is a demand for more information to be transferred, requiring higher data rates. SCADA protocols are the point at which SCADA equipment must meet the challenges presented by modern networks and more computer power. Frame relay, higher speeds, Ethernet, and IP protocols are the new environment for SCADA. Getting SCADA and these networks to work together is the challenge. The average lifetime for most industrial SCADA equipment is well over 10 years. Networking technology changes must faster than that.
SCADA protocols have been in operation for decades and have become widespread and very robust. With decades of development, there is a significant embedded value in the current protocols. Existing SCADA protocols share certain characteristics that make them very robust in what is termed "legacy" networks.
These networks are often lower speed (1200 bps is the most widely used data rate). The data is usually asynchronous, remote terminal units are polled, there are typically many remote terminal units polled over a single multipoint circuit. The polling protocols presume that data blocks sent and received will be contiguous, and that any time gaps in the data or corrupted data is the result of line errors. These protocols will react to the errors by re-polling the remote terminal that failed to correctly respond. On multi-drop lines using modems, the host computer also expects to see the Data Carrier Detect (DCD) signal turn on and off, where DCD is present when data is being received, and DCD is off when no data is being received. A lack of DCD transitions may be an error condition for some SCADA protocols. In networks that have microwave radio links, the DCD signal (or some other RS-232 control signal from the remote terminal unit) is required to key half-duplex radios. And, all this polling and response must happen in a very short time, often measured in fractions of a second.
Characteristics of the new media
The newest networks have their own protocols that must encapsulate and transport the SCADA protocols. These new networks and their protocols interact with existing SCADA protocols and present limiting factors. These network protocols, frame relay, Ethernet, and IP, each have characteristics that will generate delays, cause short gaps in the data, or not transmit DCD transitions. These delays, gaps and lack of DCD transitions may cause SCADA protocols to assume errors in the links. Lack of control signal transitions may also mean the inability to key a microwave radio link.
Frame relay is a packet protocol. The data packets of frame relay networks may have no direct correspondence to the size of SCADA poll/response packets. Therefore, a SCADA packet will often be broken up into several frame relay packets by the network, with delays between the frame relay data packets. These time gaps within a SCADA packet will result in the SCADA polling system assuming a transmission error when in fact there are no errors.
Ethernet also is a packet-oriented protocol. Ethernet packets are generated without regard to the incoming data protocols. Ethernet devices have protocol rules to obey, which are related to the needs of the Ethernet, not the infinite variety of possible devices that may be connected to an Ethernet network.
IP networks have the same packet characteristics of frame relay and Ethernet networks. There is no relationship between the IP packets and the incoming SCADA poll/response data packets. Timing gaps will occur, and the SCADA systems may assume an error condition.
Higher speed SCADA Equipment
SCADA systems are being equipped with more powerful computers. This leads to the need to transfer more information, which requires higher data rates. In the past, most SCADA systems used 1200 bps modems. This allowed the SCADA systems to interrogate remote devices and determine if the remote unit was operating, and also allowed the delivery and collection of small amounts of data. Now, with powerful processors and associated memory being widely available and cheap, the data volume requirements are increasing.
9600 BPS fast poll modems
To meet this higher demand for data, SCADA equipment manufacturers, system designers and users are looking for higher speed devices. There is a very large existing infrastructure of phone lines; private wire and microwave that operate at voice grade frequencies, where there is a bandwidth of about 3000 hertz. This is sufficient to pass 9600 bps using fast poll modems.
Fast poll modems work like 1200 modems to the extent that they pass data over voice grade lines, use 4-wire phone lines and work in multidrop mode. Fast poll modems deliver more data, but at higher data rates. Fast poll modems need additional time to properly acquire the data signal. This is called training time, and is typically known in terms of the Request to Send/Clear to Send (RTS/CTS) delay of the modem. Fast poll modems also modulate the analog signal in a manner different from 1200 bps modems. A 1200 bps modem transfers data by varying a tone's frequency. The 9600 fast poll modems use a more complicated technique known as QAM. The QAM technique results in several milliseconds of delay as several bits of data are sent together and decoded together.
Fast poll modems can deliver more data for SCADA systems, but have an increased RTS/CTS delay and several more milliseconds of propagation delay. Some SCADA polling protocols must be modified, either in their setup criteria, or by the protocol designers, to accommodate this characteristic of the fast poll modem.
Fast poll modems will deliver more data than 1200 bps modems and will poll locations as quickly as 1200 bps modems. The time from a poll to a response may vary slightly between 1200 bps and 9600 bps modems. For example, some traffic control systems expect that after the end of a poll, the beginning of a response (the start of the first character of the response) from a remote unit will occur within 25 milliseconds. That will be the case with 1200 bps modems operating over private wire. A 9600 bps fast poll modem operating over that same wire will result in the beginning of the response from the remote terminal in about 30 milliseconds.
56 Kbps DDS
SCADA systems can also operate over DDS networks provided privately or by telephone companies. DDS networks can provide speeds from 1200 to 57,600 bps over multidrop networks. The good news about DDS is that it has an RTS/CTS delay of less than one millisecond, and has very little propagation delay. In fact, in a bench test using DDS, the time from the end of a poll to the beginning of the response from a remote terminal unit will be less than 2 milliseconds. Therefore, DDS can be an ideal solution, provided it is available and cost effective.
SCADA systems, whether of the latest design or systems that have been in operation for years, can usually be migrated to the high speed networks. System designers must have an awareness of the issues of delay, time gaps in the data, control lead requirements and what equipment infrastructure is available to accommodate these issues.
Data Comm for Business, Inc (DCB) developed several solutions to allow the operation of existing and SCADA systems using today's modern networks. These products allow existing SCADA equipment to be used on existing networks at higher speeds or on new networks like frame relay, Digital Data Systems (DDS), and IP networks.
The LL9.6 fast poll modem has a short RTS/CTS delay. Propagation delay is just a few milliseconds. The LL9.6 fast poll modem will operate over existing 4-wire networks, including private wire, phone company lines and microwave. The LL9.6 fast poll also has several useful diagnostic features, including addressable modems (for diagnostic use), RS-232 control over the diagnostics for remote control and automation of the diagnostics, and line level measurement in dBm.
The D series of dial-up modems include 14.4, 33.6 and 56 Kbps modems. These modems are in a compact metal case that can be mounted in a 1U high (1.75") 19" rack space, with 1, 2 or 3 modems per 1U rack height. The modems can be powered with an external wall mount transformer or can be powered with 12, 24 or -48 volts DC. The ability to rack mount the modem and the DC power capabilities make the D-Series dial-up modems well suited to SCADA requirements.
The DCB DL-56 can be used over phone company DDS lines in point to point or multipoint mode, and point to point over private wire facilities. The DL-56 RTS/CTS delay is less than one millisecond. The propagation delay is less than 2 milliseconds. While it operates at 56 kbps with a synchronous interface, it will also pass 1200 to 9600 bps async data with no modifications. It is easy to setup and comes with both RS232 and V.35 interface.
The DA-56 is similar to the DL-56, but its external interface is asynchronous only. It supports asynchronous data rates from 300 bps to 115,200 bps over 56 kbps DDS facilities. It operates point to point or point to multipoint.
The DCB Frame Relay Broadcast Polling FRAD (BPF) is a FRAD for asynchronous networks that will accommodate almost any byte oriented async polling protocols. The BPF encapsulates async polling protocols into frame relay format for private or public frame relay networks. With the SR-BPF, SCADA equipment with async polling protocols can be used successfully over frame relay networks. The SR-BPF will keep data blocks together, so when the block is delivered to the far end device, there will be no time gaps in the data. The SR-BPF supports up to 40 DLCI's per port. The master unit is available in 1 or 4 ports which support up to 160 or more remote units. Slave units in a single channel version or a 4 port unit can have the same DLCI mapped to all 4 ports, so it can be used as a 4 port sharing unit at remote locations.
The SRVM provides a point to point multiplexer for combining voice and data over point to point links. The SRVM has applications in SCADA over private microwave or other links where there is a need to combine voice and data over a single channel. Channels can be synchronous or asynchronous at speeds up to 128 Kbps. The voice channels can consume as little as 4.8 Kbps of bandwidth.
The SRX/SPL multiplexer is also able to keep the poll and response blocks intact. It can be used to piggyback more than one SCADA system over the same link. One DCB customer, an electric utility, has a 300 bps SCADA system and a 1200 bps SCADA system operating over the same multipoint 9600 bps microwave radio system.
The SR-04W is a 4 port statistical multiplexer with an internal spread spectrum frequency hopping 900 MHz or 2.4 GHz modem. The composite and ports can all operate at asynchronous data rates up to 57.6 Kbps. The spread spectrum modem can operate at distances up to 20 miles or more with good line of sight. The SR-04W multiplexer has an RS232 management port for local or remote setup. The SR-04W is part of a family of multiplexers, the SR Series, that includes 8 channel, 16 channel and 32 channel multiplexers.
The Etherpath provides an asynchronous RS232 entry onto Ethernet networks, using IP protocol to encapsulate SCADA and other asynchronous data. The Etherpath can be set as a server unit that can be connected to from a PC or other devices using telnet. A pair of Etherpath units can be connected to each other by setting one as a server and the other as a client. In the client/server mode, the IP addressing can be fixed, allowing for units that have a permanent "nailed up" connection. The client can also be used with the DCB "Ethermodem" feature which allows the PC attached to the client to connect to the server by sending AT dial commands to the client, where the phone number is in fact in IP address (ATDT 220.127.116.11 for example). The Etherpath can be connected over the same LAN or over routers, since the Etherpath supports a gateway IP address. By early 2000, a version of the Etherpath will be available to deliver multipoint polling protocols to SCADA remote terminal units. One Etherpath serves as the master, communicating to multiple slave units, each with a remote terminal unit attached.
The IP5200 is a low cost, small router, slightly larger than a cigarette package. It has 2 asynchronous com ports and a 10/100 Ethernet port. While Internet access to remote devices is only a near real time access (several seconds delay or more should be expected over the Internet) the IP5200 is compact and low cost, the ideal kind of device required for SCADA type of applications. And over private IP networks, the IP5200 can deliver much better performance than over the Internet, since a private IP network can be much more predictable since the network elements are in the control of the private network owner.
The DCB X2 modem contains both a spread spectrum frequency hopping wireless modem (900 MHz or 2.4 ghz) and a 2 wire dial up modem (14.4, 33.6 or V.90 56 Kbps). The X2 modem is used to extend the reach of dial up modems the last few miles beyond the reach of phone company wires. Applications include retrieving oil field data where it is practical to run one dial up phone line but not additional dial up phone lines to each well. In the oil field application, the X2 wireless modem can be set to operated as a master modem to poll a number of wireless modems that are beyond the reach of the dial up phone line. Other applications include running just a point to point wireless link from the X2 to a wireless modem and terminal device or PC that would otherwise be past the reach of the wired telephone network.
The DCB-115 is an ISM band wireless point-to-point or multi-point modem in a metal case that can be used standalone or rackmount. The DCB-115 operates in point to point or multipoint mode with async speeds up to 57.6 full duplex or 115.2 Kbps half duplex (polling). It is available in 900 MHz and 2.4 GHz versions. Other wireless solutions are provided by DCB, including wireless LAN solutions for connecting desk top computers, lap top computers and hand held Windows CE computers to a wired lan backbone. Unlicensed and licensed point to point links are also available, at 900 MHz, 2.4 GHz, 5.7/5.8 GHz, 23 GHz and other frequencies. Links can be RS232, V.35, T1, E1, LAN, or LAN plus T1, and OC3.
The Access Switch and APS-01 are great devices for collecting remote information or controlling 7remote devices. The Access Switch is not usually used for polling data. It is best suited for someone sitting at a keyboard and controlling or configuring a remote device, or for use with scripts or simple control programs (based on Visual Basic, etc). Devices that send out paging alarms can be connected to the Access Switch. The Access Switch can be attached to an external modem for remote access, and is available with an optional internal dial-in modem. By adding the APS-01, the Access Switch provides a way to control power to remote devices. In short, the Access Switch is great for Supervisory Control and Data Acquisition where a human at a terminal controls the Access Switch or by devices that use something other than polling software.
Will SCADA work well over cellular telephone links, X.25 networks or the Internet? Probably not. Cellular phone networks are designed for short term conversations, have low data rates, and are susceptible to call disconnection. Unless a SCADA system is designed for short term dial up calls, it is unlikely to be a satisfactory alternative.
X.25 networks are error corrected from node to node and often have round trip delays of several seconds. SCADA polling protocols are also error corrected and expect remote terminal response within a few seconds or less. When error corrected SCADA polling protocols operate over X.25 networks, the round trip delay of the X.25 network will often cause timeouts. The SCADA equipment presumes data was lost and will send out a repeat poll. This repeated poll plus the original delayed poll will then be circulated in the X.25 network, causing an error condition that is usually resolved only by resetting the SCADA master polling system.
The Internet, as it operates today, has variable delays similar to that of X.25 networks, and can suffer packet loss that can be as high as 10% or more during severe congestion. The Internet may be acceptable for the retrieval of data on an occasional basis, but not for real time data collection and control requirements.
Private networks, whether private wire, microwave, fiber optic, phone company, or frame relay remain the best choice because of their short propagation delays and reliable delivery of data.
Data Comm for
2949 County Road 1000 E
Dewey, Il 61840