Multiplexer Based Wide Area Networks:
Management Strategies using Inexpensive Tools

Abstract:

As a wide area network grows, it eventually reaches a size where automation becomes the most economical tool in your network maintenance toolkit. This paper describes some common multiplexer network support practices and automation techniques used for larger systems.

The Problem:

We define a large network as one having multiple sites and many pieces of equipment. DCBs customers typically fall into one of several categories ranging from simple to large networks.

1. Small network: Consists of two locations and a dial-up or dedicated analog line or a leased DDS or frame relay circuit and straight multiplexer connections. There usually is no any technical staff on site, it may be a turn-key operation provided by a systems integrator.

2. Medium network: Multiple locations (two to five) normally using frame relay or DDS circuits. May use combined data/voice equipment or fractional T1 transports. Often has a computer person on staff, but that person is not usually proficient in WAN technologies.

3. Large network: More than a few locations and possibly complex equipment such as channel banks, FRADS, and routers as well as multiplexers. Normally has staff with expertise at the host site.

All three size networks may be required to be functional 24 hours per day and 7 days per week, may be able to tolerate an outage of a few hours, or may be able to tolerate a weeks outage. Requirements for spares, redundancy, troubleshooting and diagnostic capability, and staff expertise vary with the business requirements of the network. No one size fits all in this business.

The Solution

In this paper, well discuss the larger networks and what tools can be applied to improve up-time and maintenance economy. These steps fall into several categories.
  1. Pre-installation design.
  2. Documentation and training
  3. Automation
  4. Vendor support

Pre-installation designshould include an analysis of the fault conditions that may be encountered and a plan for each possibility. Where required, spares should be stocked. Vendor agreements should be signed to provide for spares or cross-shipped replacements where needed. A work plan should be written to guide technicians in troubleshooting the system. Should fail-safe operation be required, automated or manual fall-back positions should be designed and installed. Configuration and diagnosis access should be built into the system. DCB Access Switches are quite useful in this regard. All DCB multiplexers have a network management port (executive port) that can be remotely accessed by the technician. At remote sites, these are sometimes connected to dial-in modems for remote diagnosis and configuration. At host sites with multiple multiplexers or a mix of equipment, an Access Switch can be used to provide convenient access to the management port on various pieces of equipment. A network monitoring system can be installed to notify management of outages.

Documentation and training are the most overlooked portion in a network installation. The system design should be on paper (or computer) and available to all technicians working on the system. This documentation should include a one-line diagram or schematic, model and part numbers for all equipment, circuit information including contact names and numbers as well as circuit specifications and numbers for all data circuits. Vendor information should be provided for all equipment including model and serial numbers for all warranty and repair work, maintenance contract information, access information (what room is the remote equipment in, and who do you contact to get into that room). Vendor agreements should identify phone numbers and hours of operation.

This documentation should also include problem diagnosis and repair instructions. It should detail what steps to take and how to escalate the trouble ticket to management when required.

Automation is usually an afterthought on most network systems. There are two main ways to automation the wide area network. The first, trouble determination, is the least used automation technique. Most networks are not monitored by any automated tools. When something goes wrong, network management staff is notified by users that their system is down.

Other networks (usually the largest or most critical ones) are monitored by either SNMP workstations or more economical PCs running home-grown scripts in conjunction with commercial software such as Procomm or Reflection and Windows95.

The SNMP option is the most expensive, requiring a SNMP management platform such as HP Openview or CastleRock SNMPc running on a UNIX or Windows95 workstation. In addition to that platform, another PC is usually required to interface to the multiplexers and datacom equipment because most datacom equipment does not have an ethernet interface as is required by SNMP systems. Since most datacom equipment doesn't directly respond to SNMP throught ethernet, additional programs that run on the SNMP workstation and interact with its API must be written. This method (SNMP) is the most expensive, and requires technicians who have training in the SNMP workstation software as well as scripting for that software. Most datacomm equipment (and all DCB equipment) allows only "traps" and "gets" with SNMP. That is, the equipment can not be managed with SNMP. SNMP only reports equipment status. The operator must then use terminal emulation software or telnet terminal applications to access the management port on the equipment for diagnosis and configuration.

The "home-grown script" option is the more common and more economical method. Using this method, a terminal emulation program is run on a Windows95 workstation. This is connected to one or more Access Switches, which connect to multiple pieces of datacom hardware. A script is written for the terminal emulation program (usually Procomm or Reflection). This script checks each unit in the network for its status on a periodic basis. If a datacom unit reports a problem, the script then changes the color of an icon, sends an email, prints a log entry, or somehow notifies the operator. Then, the operator may use the same workstation to converse with the equipment in trouble. Scripts can be written to interface with management ports on various types of equipment (DSU, multiplexer, router, FRAD, etc) and all can be managed from the same workstation. The Access Switches may be local to the workstation, or accessed via WAN links or TELNET.

Vendor support is another item that is often overlooked at first glance. Too often, users concentrate on installed price instead of total life cycle cost including maintenance support. Your vendor should have a reasonable support policy that includes replacement unit availability (either rentals or spares on site), troubleshooting assistance over the phone (either free as DCB does it or contracted), and maintenance contracts. Inquire about coverage for lightning damage since most maintenance contracts dont cover it (DCB Platinum Protection maintenance contracts cover it, but warranties dont) .

Some Real World Examples:

1. Many DCB customers have very little computer expertise and no datacom expertise. They purchase a computer system from a systems integrator in a turn-key manner. They often purchase the five year "Platinum Protection" maintenance contract. When they sense a network problem, they phone DCB technical support. The technician talks them through problem diagnosis and either refers the problem to the phone company for repair of a bad circuit or sends replacement equipment overnight to the proper site. The old equipment is returned to DCB and the case closed.

2. Some DCB customers with larger networks are in the same situation as example one above, but they installed an Access Switch and modem at the host location. When they call DCB technical support, the technician dials into the Access Switch and performs his own diagnosis operations. This is much quicker since the technician can usually go right to the problem instead of teaching the customer how to diagnose equipment "on the fly".

3. A larger site with technical expertise on staff used WRQ Reflection terminal emulation software on a PC to poll each multiplexer in their system via an Access Switch. The terminal emulation software (Reflection) was run from a script that changed a screen soft button to a different color when it sensed a problem. The user then clicked on that button and was immediately talking to the management port of the device having the problem. Reflection was used because this was an HP mini-computer shop that used Reflection software for its other communications needs.


Rapid indication of network outages is the most commonly requested management function. Since the SR multiplexer has an RS-232 network management port, its relatively simple to write scripts that poll the unit for proper operation. The simplest (and most common) method is to send a "RTY" command to the host multiplexer and wait for a response. If the RTY doesnt respond, or responds with a timeout, then the remote multiplexer isnt on-line with the host. If it responds with the remote multiplexers type, then the system is up and functional. Its also possible to check the "AC" command response to determine network errors or port errors.

This query operation script is run on a PC workstation using the scripting language of a terminal emulation software package. If the program is Procomm running under Windows95 (or equivalent), then the results of the query can even be passed to a SNMP manager program such as CastleRock SNMPc using DDE calls.

When more than one device is to be monitored, the multiple multiplexers should be connected together using an access switch connected to each network management port. Monitoring scripts should be written to access the individual multiplexers through the access switch, test the multiplexer, then close the access switch connection.

Example of a simple test script. (Pseudo code)

 Repeat  every 2 minutes  (or as required)
   begin

SEND "c/r" Wait 15 seconds for "AT YOUR COMMAND" If timeout, then display "NO RESPONSE FROM HOST MULTIPLEXER" If Received "AT YOUR COMMAND" SEND "RTY c/r" Wait 15 seconds for "REMOTE TYPE:" If timeout, then display "NO RESPONSE FROM REMOTE MULTIPLEXER" IF received "REMOTE TYPE" then display "Multiplexer system is operational" end

This example simply asks the remote multiplexer to respond with its type message. If it doesnt we know that the link to it is down. If it responds, we know that the two multiplexers are communicating. In a real system, we would normally format the display and use icons instead of the printed messages. If more information is required, a similar request would be issued for an "AC " command and its response scanned for errors. In an actual system, the hard-coded names would be using variables so the same script is used for multiple multiplexers. To monitor multiple multiplexers, move this script one level deeper in a nest that first connects to the access switch and then connects through it to the desired multiplexer.

Example of Access Switch pass through:

 Repeat  every 2 minutes  (or as required)
   begin
	SEND  "^Z^Z^Z c/r"   (reset the access switch )
Wait 15 seconds for "ACCESS SWITCH"  
	If timeout, then display "NO RESPONSE from Access Switch"
	If Received "ACCESS SWITCH>>" then	
	  begin
  	      SEND "CHICAGOMUX c/r"  (connect to the mux named CHICATOMUX)
	      Wait 15 seconds for "AT YOUR COMMAND:"   (check the status of the host mux)
		If timeout, then display "NO RESPONSE from  Chicago Host Multiplexer" 
		IF received "AT YOUR COMMAND" then 
		begin
      SEND "RTY c/r"   (check the status of the remote mux)
	                     Wait 15 seconds for "REMOTE TYPE:"
		       If timeout, then display "NO RESPONSE from Chicago Remote Multiplexer" 
		       IF received "REMOTE TYPE" then display "Chicago  is operational"
    	  End
	SEND "BYE c/r"  (disconnecting from the access switch)
  end

In summary, there are a number of steps that will help create a more easily managed network. Unfortunately, since multiplexer networks are so much more reliable and simpler than routed ethernet networks, many users choose to ignore the network diagnosis and management functions until after they have a problem.



img
Data Comm for Business Inc.
2949 County Road 1000 E
Dewey, Il 61840
Voice: 217-897-6600
Toll Free: 800-4-DCB-NET
Toll Free: 800-432-2638
Email: Contact Page
Web: www.dcbnet.com
Fax: 217-897-8023
All DCB web pages copyright ©1995--2017 Data Comm for Business, All rights reserved.
EtherPath®, EtherSeries®, EtherPoll®, EtherBridge® and EtherModem® are Registered Trademarks of Data Comm for Business, Inc.