DRAFT V0.4 Network Management System evaluation form NORDUNET Engineering and Technical Forum Network Management working group Editor & Chair: Harri Salminen 1991-11-04 This form can be used in two ways. First it can be used while evaluating different network management systems as an evaluation form or checklist of all questions that should at least be checked on the product. Secondly it can be used to define local requirements for a network management system by setting different priorities for different items in the form and them comparing it to the results of different evaluations that have could already have been done by somebody else. For example following priorities might be used: 0 Don't want it 1 No use for it 2 Don't care if it doesn't cost extra 3 Nice to have but not worth much extra cost 4 Would definitely like to have at least in the future 5 Usefull and worth paying extra to get it now if necessary 6 Necessary to have one way or other 7 Absolutely necessary. No deal otherwise. The requirements could be also verbal specifications or additional questions to the form. If they seem to be of general interest we are willing to intergrate in the future versions of the evaluation form. The completed forms you wish to share with you colleagues should be sent to netf-man@nic.nordu.net which is the mailing list for the netf-netman wg. (should we have separate mailing list for evaluation or collection?). An anonymous ftp archive for the results will be kept in nic.nordu.net directory net/net-man/results and possibly some comparison report could be written if there's enough results and interest. Otherwise the forms could be just processed locally for individual comparison needs. In general most of the questions can be answered with simple Yes/No/NA and more detailed verbal descriptions can be written as well in following linesafter the simple answer. On some questions theres suggested alternatives for the simple answer enclosed in square brackets [Yes|No|Maybe] that should be used.. On some questions a rating on scale from 0 (not available) to 10 (Excellent) should be given and of course a verbal explanation can be added The simple answers should be the first word after the question to facilitate automated statistics and table generation. Additional lines can be used as many as necessary since the question numbers are used as the separators and explanatory comments are enclosed in square brackets. In case there is 'n' or '____' or 'other' questions it means that you can add more alternatives by copying the question. You can also add subquestions or hierachy levels if seen necessary. In case answer is missing from some question it's assumed to mean that the feature in question is missing from that product. Please send your comments directly to the editor at or the netf-netman@nic.nordu.net mailing list to which you can subscribe by sending mail to netf-netman-request@nic.nordu.net. 0. Product indentification 0.1 Name of the product: NMC 3000 Network Manager 0.2 Version of the product: 1991-12-04 V1.2 Information of the coming versions is provided in some points courtesy of Karl Feilder, Network Managers (UK) Ltd. Chapters provided by him at 1991-12-23 are prefixed with kwf:. 0.3 Manufacturer of the product: Network Managers (UK) Ltd 0.3.1 Postall address: Stirling House Stirling Road Surrey Research Park Guildford Surrey GU2 5RF UK 0.3.2 International phone number: +44 483 303223 0.3.3 International fax number: +44 483 303997 0.3.4 Internet address (optional): kwf@netmgrs.co.uk 0.3.5 Contact person(s): Karl W. Feilder, European Sales Manager Mark Dooley, European Support Manager 0.4 International availability: Offices in UK, USA, Germany (Jan 1991) Resellers in every European country Price in the millions.! OEM (re-badged) versions from Proteon, Ascom, Hirschmann, Gambit Pilkington (UK), and others (not allowed to reveal ID) 0.5 Evaluator: FUNET Harri Salminen et al Some clarifications and information on future version from Karl W. Feilder that were received after the first evaluation by FUNET. Most of the additions are prefixed with kwf: to identify the source. 0.5.1 Postal address: VTKK/FUNET P.L. 40 02101 Espoo Finland 0.5.2 International phone number: +358 0 457 2288 0.5.3 International fax number: +358 0 457 2302 0.5.4 Electronic mail address: hks@funet.fi [optional, preferably as seen from the Internet e.g. netf-netman@nic.nordu.net] 0.5.5 Contact person(s): Harri Salminen 0.5.6 Evaluation period (yyyymmdd-yyyymmdd): 19911202-19911205 0.5.7 Description on the evaluation environment: FUNET NOC, X11R5 Color display on a SUN3/60C and software on another SUN SLC. Motif mwm was required. 0.6 Remarks There wasn't enough time for thorough evaluation 1 Overview (free form general description of the product): 2. Management protocol support This section is used to evaluate the support for management procols that are used between the Network Management System and the entities to be managed. Information about customization possibilities for supporting special customer needs should also be included. 2.1 SNMP support [Yes/Optional/Coming/No]?: Yes 2.1.1 SNMP MIB II support [Yes/No]: Yes 2.1.2 List of vendor supplied MIBs: Cisco, Retix, Schneider & Koch FDDI router, WellFleet, Chipcom concentrator, Emulex terminal server, NAT Ethermeter and bridges, Proteon, Novell, Reuters, SynOptics, Generic SNMP router. All these are separately available Product Specific Modules. 2.1.3 Is there support for loading of customer supplied MIBs?: With the programming interface extensions using C++ libraries can be done. Each product is supported via it's own program module including calls to menus and graphics. Kwf: MIB Editor will allow this functionality in a friendly way. MIB Editor will be optional package available 3/92 2.1.4 Are all MIB variables of same type equal?: No, sometimes variables are selectable trough menus (e.g. statistics) but in some other cases predetermined (e.g. fault handler). 2.1.5 General comments: There was no documentation or time to study how easy making modications to the system is. 2.2 CMIP support [Yes|Optional|Coming|No]: Yes 2.2.1 Versions supported: MAP/TOP thin stack protocol V3.0 2.2.2 List of vendor supplied MIBs: Some of the support mentioned in 2.1.2 actually includes MAP/TOP MIBs if the product to be managed has MAP/TOP MIB support. 2.2.3 What are the supported input methods and formats?: See 2.1.3 2.2.4 Are all MIB variables of same type equal?: See 2.1.4 2.2.5 Supported transport and network layers [TP4/CLNS|TP0/CONS]?: MAP/TOP 2.2.6 General comments: No documents or interests to evaluate this further this time 2.3 IEEE 802.1 support [Yes/Optional/Coming/No]?: Yes Version B 2.4 FDDI SMT support [Yes/Optional/Coming/No]?: No 2.5 Type of support for custom programmed protocol modules? Programming Library The support is via C++ libraries. Separate libraries for Database, Graphics, Extended Graphics, MAP/TOP, Network Management, Protocol, PSM Framework, Service and SNMP class libraries. Kwf: The support is offered as a separate Application Programmers Toolkit which wasn't included in the evaluation version but is available upon request. It was released October 1991. 2.6 Ascom X.25 Switch support [Yes/Optional/Coming/No]?: Yes Kwf: Full Ascom Switch PSM available, developed under contract for Ascom 3. Features supported by the network management system 3.1 Configuration management: Y 3.1.1 Graphical representation: Y 3.1.1.1 General description of the user interface: Mousedriven with pulldown menus (Motif style) Multiple menus and relatively easy to use. Delete key didn't work, only backspace. Configuration is integrated with the network management applications but the lines between managed objects (e.g. routers) are mainly static information. Product Modules may contain nicelooking "interactive pictures" of the hardware. Kwf: With some products like Chipcom Hub, Cabeltron Hub, Proteon CNX 500, you can see port status by colour, configure ports from the picture, and collect stats for either the whole module, or per port. 3.1.1.2 Hierarchical logical maps: Yes You double click to a large circle representing lower levels of hieararchy and it opens a new map. 3.1.1.3 Different parallel views: Yes Main setup is hierarchical but one can implement wormholes using dots that can move anywhere in the map hierachy when clicked. Also separate management domains for totally different management hierarchies could be created. 3.1.1.4 Different personal views: Not yet at least (single user database) although one might create multiple parallel views for different users. It's promised as an optional feature that probably comes with the multiuser database when implemented. Also promises for subsystem control by specific users. 3.1.1.5 Support for B/W display: Yes. Configuration is possible without colors although a color display is strongly recommended for fault and performance management. 3.1.1.6 Configurable color schemes: Limited Background map colour, possibly others? 3.1.1.7 Userdefinable icons: Only externally loadable Kwf: Supported in V1.3 (4/92), and now demonstrable. Uses Bitmap Editor within the NMC 3000 environment. Option is selected in the "Customise" pull down menu, under the title "Change Icon". 3.1.1.8 Supported external graphics file input formats: X-windows bitmaps as icons Format for background maps is unknown but probably a vector graphics format 3.1.1.8.1 Can graphics input be done automatically from scripts or only interactively?: At least for icons, otherwise unknown 3.1.1.9 Output file formats supported: Undocumented at least. You can always dump X windows using external programs although resolution and frames might not be the best possible ones. Kwf: Supported in V1.3 (4/92), and now demonstrable. Output is available as either HP Laserjet (4 different dpi densities), or Postscript. Note that Postscript output takes ages !! Typically 1/2 hour for one window. User may printout single or multiple windows, using the "Print" pull-down menu in the "Status Window". 3.1.1.10 Functionality of interactive drawing tools [0-10,remarks]: 3 You can just place the icons and network symbols along with single type of textual names for the objects in the picture. No curved lines or freeform polygons not to mention more elaborate drawing aids available in normal drawing programs are available. Explanatory text unrelated to any objects or multiple text sizes also seem to be unsupported. So it's not suited for drawing fancy presentations, just simple network topologies. 3.1.1.10.1 Supported object types: Icons for the various predefined objects. Lines from those objects. Lines and circles for ring and bus networks. No general purpose object types since it's not really a drawing program. Kwf: V1.3 (4/92) includes a miscellaneous icon type, which has four "flavours". Icons for "ICMP pollable device", "Dumb device", "LLC test frame device", and "ICMP with telnet device". In conjunction with Bitmap Editor, this gives chance to really draw the whole network. 3.1.1.10.2 Support for custom object libraries: Through the C++ programming interface it should be possible to create custom objects but no interactiv graphic object libraries. See above. 3.1.1.10.3 Support for inheritance: At least in a limited fashion since you can set alarm treshold etc. so that it applies to all matching physical subsystems (E.g. cisco AGS+ 8.0+) or draw lines between sites for example in a higher level map and they show up in the lower level as endpoints. Also C++ and Db_Vista are object oriented by design. 3.1.1.10.4 Support for multiple fonts: None apparent. Maybe you can change the default font outside nmc3000 but that seems to be it. 3.1.1.10.5 Support for national character sets or custom defined fonts: Possibly whatever X can support although I did have time to test if 8 bit character support really is there. 7-bit ascii mapping works fine. 3.1.1.10.6 Support for various font sizes: None 3.1.1.10.7 Support for different text orientations: None 3.1.2 Parameter configuration: Y 3.1.2.1 SNMP write access support: Y 3.1.2.2 Interactive console access support: Y 3.1.2.2.1 Telnet to selected object: Y 3.1.2.3.2 Dialup via phonebook to selected object: No 3.1.2.2.3 DEC MOP via node table: Not known since DEC EMA support was not available to test 3.1.2.2.4 X.25 PAD via X.121 table: Not known since no X.25 support was available to test 3.1.2.2.5 VT via X.500 or table: Not known since MAP/TOP support module was not available for test 3.1.2.2.6 Other [possibly user defined?]: kwf: Dialog box design of PSMs is designed to "hide" all variations in the configuration of different devices, and this can be easily seen if one compares the configuration of Retix Map/Top bridge and Retix SNMP bridge or Ascom X.25 Switch. 3.1.3 Shared configuration support with locking and/or syncronisation: Not yet at least. Something is possibly coming in 1992 3.1.4 Emergency configuration possible from a VT100 terminal?: No 3.1.5 Other comments: No autodiscovery support. Support for loading cisco configuration files. kwf: Auto Discovery of IP Addresses, will be available in a new module in 4/92 This area is potentially suicidal, as you could in theory, try to auto discover and ping, every address in the world over Internet. We have thus incorporated a lot of user definable safety features, prohibiting certain routes. Essentially, a reference model is built from the IP routing tables of accessible routers, which is then stored in the database ; the netork is the polled at a user-defined interval to check the consistency with the stored model. Very powerful tool in the right hands, but does require more understanding of IP than we normally expect from our users. 3.2 Fault management: SNMP Poll, SNMP Trap, ICMP Echo, LLC1 Test Frame, possibly product module dependent ones 3.2.1 Indications [text|color|shape|audio|other]: Indications by color and shape changes. Text in fault logs and counters. Audible bell. Seems to be fixed inside the code although it's mentioned that if you're not happy with the application you can replace it with something else you probably programmed yourself... 3.2.1.1 OK state: Green 3.2.1.2 Unknown state: None (state is OK or fault) 3.2.1.3 Alarm threshold exceeded: Five settable alarm levels with colors ranging from light yellow to deep red. 3.2.1.4 Acknowledged but not fixed: Not available. You can view the trouble tickets though. kwf: V1.3 (4/92) and now demonstrable, provides an Event Log, which includes all events which have ever occurred in the same format as the Status Window event report. Thus items which have been cleared from the Status Window, are still present in the event log. This log may be examined using filters eg filter by device name, fault level, or filter by time of occurrence ; you may then print or export the data. 3.2.1.5 Dynamic performance indication by scaling colors/sizes: No 3.2.1.6 Not currently operational or monitored: Grey 3.2.1.7 Other: None 3.2.1.8 Configurable audio indicators: You can turn the beep on and off 3.2.1.9 Other configurable output methods: Not available kwf: V1.3 (4/92) and now demonstrable. The user may define (in a friendly dialog box), an executable which can be triggered from a particular event. This could be an Alarm, Alert, or SNMP trap, which could trigger an E-Mail message or even (with a bit of help) a radio pager. 3.2.2 Handling of faults: 3.2.2.1 Acknowledgement of alarms needed: Yes 3.2.2.2 Trouble ticket support: Ticket is automatically opened for each fault and each new subsystem (even without faults) by the system. Tickets are of predetermined nonconfigurable format that includes associated fault performance graphs which show the periods of downtime or thresholds exceeded and percentage of failure time. 3.2.2.3 Trouble log searching possibilities: Selection from a list. Filtering by subsystem name, type or release and a time interval. 3.2.2.4 Trouble ticket distribution methods: None apparent except perhaps printing or faxing the bitmap displays 3.2.2.4 Support for filtering of specified alarms: User can set five levels of alarms and filter out the lower priority ones. Also faults can be filtered out using subsystem name, error type and date and time range. Finally you can disable temporarily the monitoring of an object. 3.2.2.5 Support for automatic filtering of non-causal alarms?: No yet at least. 3.2.3 Efficiency and responsiveness: Performance on a SUN SLC was acceptable with small test network. There was no time to test performance with hundreds of objects. 3.2.3.1 How many simultaneous processes the system needs? Few dozen at least for different applications. There's only one polling process for each network protocol like SNMP or CMIP and one central event dispatcher along with the database lock manager that are continously running. 3.2.3.2 How much working memory an average monitored object needs? [bytes]: Kwf: No idea, but we recommend a minimum hardware configuration of 16Mb RAM. This is OK for a small network (50 or so nodes) but 24 Mb is noticeably better. 3.2.3.3 Does each operator need to poll separately from his workstation or can a central server do it once for all operators? Not implemented yet. Currently each workstation is totally separate single user system. 3.2.3.4 Can some objects or displays be polled only on demand? Yes, polling could be turned off for some subsystems kwf: V1.3 (4/92) and now demonstrable, the user can select a group turning on or off for polling. Thus one may turn off all stats polling for a device, or site, but leave other polling (eg presence) on. 3.2.3.6 Individually customisable polling intervals: Yes 3.2.4 Distribution support [Yes/No]: Not yet Not Yet Available. Sometime Spring 1992 possibly. 3.2.4.1 Central system with multiple X-windows workstations [Y/N]: Kwf: NMC Voyeur (working name) available early 92. 3.2.4.2 Central system with workstations having local files: Not known yet 3.2.4.3 Distributed co-operating servers: Possibly Promised in brochures but not implemented yet 3.2.4.4 Sharing of information between management domains: Possibly Promised in brochures but not implemented yet 3.2.5 Is there a programming interface to add custom fault detection methods? Yes, C++ Library API Toolkit 3.2.6 Integration of external test tools and analyzers: NAT Ethermeter Kwf: Also R-Mon will be supported before 3/92 3.2.7 Ease of access to contact information (0-10): 1? 3.2.8 Other comments: 3.3 Performance management: 3.3.1 Method of report definition: Variables can be selected from a preconfigured list in the product specific module (not the latest MIB really in the box) and plotted as separate color graphs. Rates are predefined and collected in processed form in the database (e.g. bit/s, packets/hour). More complex things require the use of SQL programming interface or db_Vistas own programming interface. 3.3.2 Automation of regular reports: Not possible apparently without programming in C++. Kwf: Report Writer Module will be available in 4/92, and this will make the collation and presentation of data a lot easier. 3.3.3 Trend graphs on any variable combination over time: Autoscaling and several independent graphs inside the same window. 3.3.4 Relative distributions between variables: Not available 3.3.5 Statistical analysis on time series: maximum, mean values and rate of change in addition to the actual value on the graph. 3.3.6 Annotation of graphs with text and graphics: Not supported. 3.3.7 Output in numerical form to a file for postprocessing: No 3.3.8 User customisable filters or programming interface: Programming interface to the database. 3.3.9 Plotting output in color postscript: Screen dumps with color graphs 3.3.10 Plotting output in b/w postscript: Kwf: Yes 3.3.11 Plotting output in some other form: pixel dump with X tools. See al 3.3.12 How performance data is stored if at all? history data is stored in the predetermined format in the database. Real time statistics means in real time in and out. 3.3.13 Support for IETF recommendations?: Not known. 3.3.14 Other comments: Could have been more flexible reporting tools for producing routine monthly and weekly reports with daytime distributions, error ratios etc. Kwf: see 3.3.2 3.4 Accounting management: Depends on the product modules. Cisco support included IP and TCP accounting. 3.4.1 Billing contact information database: None 3.4.2 Reliable collection and storage of accounting information: Uses the common database 3.4.3 Individually customisable reports and bills: No 3.4.5 Easy interfacing to external accounting systems: C++ and SQL again... 3.4.6 Billing summaries by user groups: No 3.4.7 Other comments: Just raw dumps of the data available for browsing... 3.5 Security management: Not Implemented Yet for ciscos 3.5.1 Individual user authentication: Supported 3.5.2 Authentication method(s) used: user/password 3.5.3 Multiple access levels (Configuration/Operation/Monitor): Administrator and user, subsystem control by specific users is an option 3.5.4 Different groups displays available to different users: Optional feature 3.5.5 Possibility of encrypting files: No 3.5.6 Other comments: Still needs some improvement but is a good start 3.6 General: Not very flexible yet but if the offered features and options suit the particular requirements it's worth looking closer. Also it's very new product that is rapidly developing and worth keeping an eye for. European (UK) design! Kwf: We also derive about 50% of our revenue from Consultancy services, based around customisation of NMC3000. Network Managers is the only vendor independent European Network Management company, and is very proud of its strong financial position. 80% owned by the employees, we are very committed to becoming the Number 1 supplier of Network management in Europe, always try to be friendly and helpful. In 1991 we managed to ship 200 systems, with no advertising. 3.6.1 How many different database(s) are used [none|one|n|many]: One 3.6.1.n Database description: Raima corporation's DB Vista object oriented database with C++ and SQL interface libraries. 3.6.1.1 Database type [single user|centralised|master/slave|multimaster]: Single User, but a "distributed multiuser" one is coming... 3.6.1.2 Database model [text,table,relational,object-oriented,other]: Relational 3.6.1.3 Database management tools [0-10,comments]: 3.6.1.4 External database interface [SQL,C library,other]: SQL interface. C++ Library. DB-Vista proprietary interface? 3.6.1.5 Database product needed if not included: None 3.6.1.6 Support for aging of old information: Yes 3.6.1.7 Support for sharing database information with other NOCs: Not yet at least 3.6.2 Quality of on-line help system [0-10,comments]: 6 Terse context sensitive help in most windows that tells what you see and can do. 3.6.3 Quality of printed manuals [0-10,comments]: 7 Large number of A5 size spiral bound manuals that need to be replaced when updated. Black and white graphics in examples. 3.6.4 Overall ease of use [0-10,comments]: 8 Point and click basic functions, some features do need thorough reading and experimentation. 3.6.5 Distribution restrictions: Single system license 3.6.6 Price range: Starting from around 20KECUS. Additional modules and features are separate. 3.6.7 Support policies: Not clear, they are eager to make improvements and corrections according to a local representative. Supposedly a standard maintenance/update contract could be made after warranty period. Internet email address. 3.6.8 Supported platforms: Color Sparcstation with SunOS 4.1 or higher, Solbourne SPARC OS/MP 4.1, 386 PC with some Unix, others. Only mwm worked although twm was supposed to be supported. X11R4. 3.6.9 Is the implementation derived from some other implementation(s)? Kwf: NMC 3000, resulted from NMC2 project (1988-1990) within Retix UK. Note that NMC 3000 is wholly developed by Network Managers. 3.6.10 Availability of source code Kwf: Price in the millions.! 3.6.11 Other comments: Ask for a test setup from your dealer if you're interested.