Jffnms Manual

JFFNMS Manual Craig Small csmall(AT)small.dropbear.id.au Javier Szyszlican javier(AT)szysz.com 3rd July 2008

2

Contents 1

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

5 5 5 6 6 6

Using JFFNMS 2.1 Overview of JFFNMS Screen . . . . . 2.2 Profiles . . . . . . . . . . . . . . . . . 2.3 Event Viewer . . . . . . . . . . . . . 2.3.1 Events Express Filtering . . . 2.4 Performance Graphs . . . . . . . . . 2.4.1 Interface Selector . . . . . . . 2.4.2 Performance Trend Window

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

9 9 9 9 11 11 12 12

Installing jffnms 3.1 Getting JFFNMS . . . . . . . . . . . 3.2 Hardware Requirements . . . . . . 3.3 Required Packages . . . . . . . . . 3.3.1 Source Code . . . . . . . . . 3.3.2 RedHat-like Packages RPM 3.3.3 Debian Packages . . . . . . 3.4 Initial Configuration . . . . . . . . 3.5 Cron Jobs . . . . . . . . . . . . . . . 3.5.1 What does each task do? . . 3.6 Installing SNMP Extension . . . . . 3.7 JFFNMS Global Setup . . . . . . . 3.7.1 Basic Details . . . . . . . . . 3.7.2 Database Configuration . . 3.7.3 System Configuration . . . 3.7.4 Paths Configuration . . . . 3.7.5 PHP Status . . . . . . . . . . 3.7.6 GUI Options . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

15 15 15 15 15 16 17 17 17 17 18 18 19 19 19 19 20 21

4

Administrating JFFNMS 4.1 Reporting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.1 How JFFNMS calculates Total Availability . . . . . . . . . . . . . . . . . . . . . . .

23 23 23

5

Configuring JFFNMS 5.1 The Consolidation Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

27 27

6

Host Administration 6.1 Zones . . . . . . . . 6.2 Hosts . . . . . . . . 6.3 Interfaces . . . . . . 6.4 Network Discovery

29 29 29 30 31

2

3

Introduction 1.1 What is a NMS . . . . . . . 1.2 Features . . . . . . . . . . 1.3 Definitions . . . . . . . . . 1.4 Polling and Consolidation 1.5 Getting Help . . . . . . . .

. . . .

. . . .

. . . .

. . . .

. . . . .

. . . .

. . . . .

. . . .

. . . . .

. . . .

. . . . .

. . . .

. . . . .

. . . .

. . . . . . . . . . . . . . . . .

. . . .

. . . .

. . . .

. . . .

. . . . 3

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

4

CONTENTS 6.5 6.6 6.7

7

8

9

Hosts Saved Config . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . SubMaps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Satellites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Users and Customers 7.1 Customers . . . . . . . . . . 7.2 Users . . . . . . . . . . . . . 7.2.1 Creating a new User 7.2.2 Editing a User . . . . 7.2.3 User Profiles . . . . . 7.2.4 User Triggers . . . .

31 31 31

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

33 33 33 34 34 34 35

Event Configuration 8.1 Event Severities . . . . . . . . 8.2 Event Types . . . . . . . . . . 8.3 Events from Syslog messages 8.3.1 Syslog Consolidation . 8.4 Alarm States . . . . . . . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

37 37 37 38 39 39

SNMP Traps 9.1 How JFFNMS processes traps 9.2 SNMP Trap Receivers Table . 9.3 Configuring New Traps . . . 9.3.1 Trap Setup Example .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

41 41 42 42 42

10 Polling Configuration 10.1 Interface Types . . . . . . . 10.1.1 Interface Type Fields 10.2 Poller Groups . . . . . . . . 10.3 Poller Items . . . . . . . . . 10.4 Backend Items . . . . . . . . 10.5 Graph Types . . . . . . . . . 10.5.1 Aggregate Graphs . 10.6 Autodiscovery Policies . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

45 45 46 47 47 48 48 49 49

11 Triggers and Actions 11.1 Creating Triggers . . 11.2 Trigger Rules . . . . . 11.3 Actions . . . . . . . . 11.4 Putting it all together 11.5 Deactivating Triggers 11.6 Trigger Logs . . . . . 11.7 Tools . . . . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

51 51 51 52 53 54 55 55

12 Configuring Event Filters 12.1 Filters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.2 Filter Conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.3 Filter Fields . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

57 57 57 58

13 Expanding JFFNMS 13.1 Design Prework . . . . . . . . . . . 13.1.1 Discovery of new interfaces 13.1.2 Manual add of Interfaces . 13.1.3 Find the index . . . . . . . . 13.1.4 Interface Name . . . . . . . 13.1.5 Description Fields . . . . . 13.1.6 Values to collect . . . . . . . 13.1.7 Types of RRD . . . . . . . . 13.1.8 Interface States . . . . . . .

59 59 59 59 59 60 60 60 60 61

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

CONTENTS

5

13.1.9 Events for this Interface . . . . . . . . . . 13.2 Creating a New Interface Type . . . . . . . . . . 13.3 Generic Discovery Plugins . . . . . . . . . . . . . 13.3.1 host information discovery plugin . . . . 13.4 Generic Poller Plugins . . . . . . . . . . . . . . . 13.4.1 snmp counter poller plugin . . . . . . . . 13.4.2 snmp status poller plugin . . . . . . . . . 13.4.3 db poller plugin . . . . . . . . . . . . . . . 13.4.4 buffer poller plugin . . . . . . . . . . . . . 13.4.5 tcp mib pollers . . . . . . . . . . . . . . . 13.5 Generic Backend plugins . . . . . . . . . . . . . . 13.5.1 Alarm backend plugin . . . . . . . . . . . 13.5.2 Buffer backend plugin . . . . . . . . . . . 13.5.3 db backend plugin . . . . . . . . . . . . . 13.5.4 multibuffer backend plugin . . . . . . . . 13.5.5 No backend plugin . . . . . . . . . . . . . 13.5.6 rrd backend plugin . . . . . . . . . . . . . 13.5.7 verify interface number backend plugin . 13.6 Writing a new Poller . . . . . . . . . . . . . . . . 13.7 Writing a Graph Plugin . . . . . . . . . . . . . . . 13.8 Writing a Discovery Plugin . . . . . . . . . . . . 13.9 Creating a simple SNMP-based Interface Type . 13.10Creating more complex Interface Types . . . . . 13.10.1 Determining what OIDs to use . . . . . . 13.10.2 Creating an Auto-discovery Plugin . . . . 13.10.3 SNMP Poller - Counters . . . . . . . . . . 13.10.4 SNMP Poller - Status . . . . . . . . . . . . 13.10.5 Poller Group . . . . . . . . . . . . . . . . . 13.11Creating a New Action . . . . . . . . . . . . . . . 13.12Creating New SLAs . . . . . . . . . . . . . . . . . 13.12.1 Individual SLA conditions . . . . . . . . . 13.12.2 New SLA event type . . . . . . . . . . . . 13.12.3 Creating a new SLA . . . . . . . . . . . . 13.12.4 Relating SLA Conditions to SLA Groups 13.13Host Config Commands . . . . . . . . . . . . . . 13.14Common Parameters . . . . . . . . . . . . . . . . 13.14.1 Interface Parameters . . . . . . . . . . . . 13.14.2 Event Parameters . . . . . . . . . . . . . . 14 ICMP Interface Example 14.1 An ICMP message counter . . . 14.2 Determining what OID to use . 14.3 Configuring a Poller Item . . . 14.4 Configuring the Poller Group . 14.5 Configuring Interface Type . . 14.6 Adding Interface Type Fields . 14.7 Fixing Poller Group . . . . . . . 14.8 Checking the discovery works . 14.9 Checking the poller works . . . 14.10Writing the graph plugin . . . . 14.11Creating new Graph Type . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

61 61 61 62 62 62 62 62 62 63 63 63 63 63 64 64 64 64 64 65 66 66 66 66 67 67 68 68 68 69 69 70 70 70 71 72 72 72

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

75 75 75 75 76 76 77 78 78 79 79 79

15 Working with external programs 15.1 SNMP Traps . . . . . . . . . . . . . . . . 15.2 Syslog . . . . . . . . . . . . . . . . . . . . 15.2.1 Using msyslog to import events 15.2.2 Using syslog-ng to import events 15.3 Smokeping . . . . . . . . . . . . . . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

81 81 81 81 82 82

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

6 16 Interface Types 16.1 Apache Web Server . . . . . 16.1.1 Discovery . . . . . . 16.1.2 Status Polling . . . . 16.1.3 Graphs . . . . . . . . 16.1.4 Caveats . . . . . . . 16.1.5 Configuring Apache 16.2 BGP Peers . . . . . . . . . . 16.2.1 Discovery . . . . . . 16.2.2 Status Polling . . . . 16.2.3 Graphs . . . . . . . . 16.2.4 Caveats . . . . . . . 16.3 Cisco SA Agent . . . . . . . 16.3.1 Discovery . . . . . . 16.3.2 Status Polling . . . . 16.3.3 Graphs . . . . . . . . 16.3.4 Caveats . . . . . . . 16.4 IIS Webserver . . . . . . . . 16.4.1 Discovery . . . . . . 16.4.2 Status Polling . . . . 16.4.3 Graphs . . . . . . . . 16.4.4 Caveats . . . . . . . 16.5 Windows System . . . . . . 16.5.1 Discovery . . . . . . 16.5.2 Status Polling . . . . 16.5.3 Graphs . . . . . . . . 16.5.4 Caveats . . . . . . . 16.6 TCP Ports . . . . . . . . . . 16.6.1 Discovery . . . . . . 16.6.2 Status Polling . . . . 16.6.3 Graphs . . . . . . . . 16.6.4 Caveats . . . . . . . 16.7 Storage . . . . . . . . . . . . 16.7.1 Discovery . . . . . . 16.7.2 Status Polling . . . . 16.7.3 Graphs . . . . . . . . 16.7.4 Caveats . . . . . . . 16.8 Solaris System Info . . . . . 16.8.1 Discovery . . . . . . 16.8.2 Status Polling . . . . 16.8.3 Graphs . . . . . . . . 16.8.4 Caveats . . . . . . . 16.9 Reachability . . . . . . . . . 16.9.1 Discovery . . . . . . 16.9.2 Status Polling . . . . 16.9.3 Graphs . . . . . . . . 16.9.4 Caveats . . . . . . . 16.10Physical Interfaces . . . . . 16.10.1 Discovery . . . . . . 16.10.2 Status Polling . . . . 16.10.3 Graphs . . . . . . . . 16.10.4 Caveats . . . . . . . 16.11blah . . . . . . . . . . . . . . 16.11.1 Discovery . . . . . . 16.11.2 Status Polling . . . . 16.11.3 Graphs . . . . . . . . 16.11.4 Caveats . . . . . . .

CONTENTS

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

83 83 83 83 83 84 84 84 84 84 84 85 85 85 85 85 85 85 85 85 85 86 86 86 86 86 86 86 86 86 86 87 87 87 87 87 87 87 87 87 87 87 87 88 88 88 88 88 88 88 88 88 88 88 88 88 88

CONTENTS

7

17 Satellites 17.1 Relationships between Satellites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

89 89

18 Error Messages 18.1 Running the components on the command line . . . . . . . . . . . 18.1.1 Poller . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18.1.2 Consolidator . . . . . . . . . . . . . . . . . . . . . . . . . . 18.1.3 Interface Discovery . . . . . . . . . . . . . . . . . . . . . . . 18.1.4 RRD Analizer . . . . . . . . . . . . . . . . . . . . . . . . . . 18.2 Database Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . 18.2.1 New installations running selinux . . . . . . . . . . . . . . 18.2.2 Query Failed . . . Table doesn’t exist . . . . . . . . . . . . . . 18.2.3 Query Failed . . . Can’t open file: ’tablename.MYI . . . . . . 18.3 Problems with PHP . . . . . . . . . . . . . . . . . . . . . . . . . . . 18.3.1 Modules for Apache PHP . . . . . . . . . . . . . . . . . . . 18.3.2 Modules for cli PHP . . . . . . . . . . . . . . . . . . . . . . 18.3.3 Webserver displays PHP source . . . . . . . . . . . . . . . 18.3.4 Call to undefined function: mysql connect() . . . . . . . . 18.3.5 Text boxes not updated . . . . . . . . . . . . . . . . . . . . 18.4 Problems with Poller . . . . . . . . . . . . . . . . . . . . . . . . . . 18.4.1 Poller returns with no hosts polled . . . . . . . . . . . . . . 18.4.2 A lot of pollers return blank values . . . . . . . . . . . . . . 18.5 Problems with Hosts Configuration . . . . . . . . . . . . . . . . . 18.5.1 No such variable errors . . . . . . . . . . . . . . . . . . . . 18.6 Problems with Consolidator . . . . . . . . . . . . . . . . . . . . . . 18.6.1 Consolidator wont raise alarms . . . . . . . . . . . . . . . . 18.6.2 Email action returns 0 . . . . . . . . . . . . . . . . . . . . . 18.6.3 No events or Duplicate entry for key . . . . . . . . . . . . . 18.7 Problems with Graphs . . . . . . . . . . . . . . . . . . . . . . . . . 18.7.1 RRD files not created . . . . . . . . . . . . . . . . . . . . . . 18.7.2 Apache Log shows problems with .dat files . . . . . . . . . 18.7.3 Apache Error Log shows .dat files have permission denied 18.7.4 Incompatible libpng version in application and library . . 18.7.5 Gaps in Graphs . . . . . . . . . . . . . . . . . . . . . . . . .

91 91 91 92 92 93 94 95 95 95 95 95 95 96 96 96 96 96 96 96 97 97 97 98 98 98 98 99 99 99 99

19 Random stuff, to go somewhere

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

101

20 Tables of values 103 20.1 User Profiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 21 About this document 105 21.1 Contributing to this document . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 21.2 Copyright and License . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 22 The GNU General Public License

107

8

CONTENTS

Chapter 1

Introduction JFFNMS stands for Just For Fun Network Management System. It uses the PHP scripting language to collect and display the information about various devices that can be found in a computer network. The main building block for JFFNMS is the Interface which are collected together by a Host . The concept of interfaces is expanded from the usual meaning. While it does include physical network ports, for JFFNMS it means something that either has a state, one or more values or both. JFFNMS can report on the Interfaces condition via events, alarms, graphs or actions. It can be setup so that an action, such as an email, is sent on a changed state of an interface, an event happening or even when a value for an interface exceeds a specified level.

1.1

What is a NMS

A Network Management System or NMS is a system of computer hardware and software that is used to manage the state of devices. The devices could be things like routers, switches or servers and they are called network elements generally, though JFFNMS calls them Hosts. There are two ways for a NMS to determine the state of network elements. They can poll for the information or they can wait for triggers. JFFNMS does both types of state checking. Typical polling is SNMP gets and connections to TCP ports, while triggers would be SNMP traps and syslog messages. The lowest level of NMS is the Element Manager, which means it directly looks after the network elements. There can then be additional NMSs that can give a network or system-wide view. JFFNMS is more a element manager, though it does have some features of the higher level NMS.

1.2

Features

JFFNMS is deceptive. Most people use it to satisfy a small set of requirements and initially use it for those reasons alone. Then exploring JFFNMS you can see it does a lot more. Some of the features include: • Alarming of syslog and SNMP trap events • SNMP polling of router, switch and network interface status • Graphing of various statistics of network device interfaces • Graphing of host information such as CPU, memory and disk • Notification via email based upon complex alarm filtering The types of devices, or “Interface Types” that JFFNMS can monitor is large and, thanks to the efforts of the JFFNMS developers and users, growing all the time. For a list of what devices you can monitor, see Interface Types in chapter 16 . 9

10

1.3

CHAPTER 1. INTRODUCTION

Definitions

It is important to understand the various entities and their relationships within JFFNMS. Many hours of wasted debugging and configuration time can be saved if you understand what JFFNMS is trying to do and the terminology used. All network devices or network elements are called hosts. Hosts are basically a container for interfaces. Both hosts and interfaces can have IP addresses as well as other properties. Generally a host is some device out on the network that has an IP address. The definition of an interface is a bit more complex. It’s better to think of an interface as an attribute or property of a host. More specifically, its something you want to monitor on the host. Each interface has an interface type, which describes what information needs to be collected (by a poller group, what information needs to be stored and how it is discovered. The easiest interface type to understand is a physical interface on a router, switch or server; but a process, a TCP port or even a fan is also an interface.

1.4

Polling and Consolidation

The way that information flows within JFFNMS can be a bit daunting at first. This section describes how the information gets into JFFNMS and what happens to it within JFFNMS. There are three main ways that information comes from a Network Element (such as a router, switch or server). They are information from the JFFNMS pollers, syslog messages or SNMP traps. Every 5 minutes cron kicks off the poll sequence. Pollers then do various bits of prodding of the network element. The information that comes from the pollers then ends up either as an event or as a data point in a RRD file. The network element may decide to send a SNMP trap or syslog message to JFFNMS. In this case the piece of information will hit either an external program snmptrapd or syslogd. If they have been setup correctly (see External Programs in chapter 15 for details) then the trap or message will end up in the particular database table for that message type. Once again, cron kicks off another job every minute called the consolidator. The consolidator is actually made up of various types of consolidators that generally work on three sets of tables: An input table, a rule matching table and a output table. The general idea of the consolidators is that they take new rows of the input table, attempt to match them to the rule matching table and if there is a match, modify the data and add them to the output table. The traps and syslog consolidator examine the relevant table, attempt to find known traps or syslog messages and, if matched, will create new events based upon the information for the matching rule in the traps type or syslog type table. Another consolidator is the event consolidator. Unlike most other consolidator, this one has a rather fixed set of criteria. The consolidator takes the event table as its input and the alarm table for its output. It examines each new event and if it can find a matching interface and the event type is one that generates an alarm, it will toggle the alarm state of that interface (if the event would change it). Finally there is the alarm consolidator. This one is used to fire the triggers which then activate the actions. It examines the event and alarm tables for matching enabled triggers. This consolidator is the one responsible for generating emails or pagers when an event or alarm happens. In addition to syslog and snmp events, the pollers can directly or indirectly set events. The direct method means the poller is using the alarm (or similar) backend and is used for things like interface state changes. The indirect method is via the SLA analyzer. The SLA analyzer examines specified interfaces’ RRD files and matches the data against the SLA criteria. If there is a SLA hit, an event is created for that interface. The SLA analyzer is run out of cron every 30 minutes.

1.5

Getting Help

JFFNMS has an email list that is hosted on sourceforge, the subscription information is found at http://lists.sourceforge.n users. If you are stuck then you can try looking in the archives first and if that doesn’t answer your question then feel free to post into the list. You will have to subscribe first.

1.5. GETTING HELP

11

Figure 1.1: Internal Data Flow in JFFNMS

12

CHAPTER 1. INTRODUCTION

Chapter 2

Using JFFNMS If just want to use JFFNMS rather than configuring and expanding it, this is the chapter you are looking for.

2.1

Overview of JFFNMS Screen

JFFNMS has a standard look to all its screens. Right at the top is the control frame. This frame is on every screen and has the same look. Figure 2.1 has an example screen showing interfaces and events. The top line is where the controls go. Every screen has the same set of controls. These allow the operator to move around in the various screens in JFFNMS. Starting on the left is the JFFNMS button. This button takes you to the about page on the sourceforge network. The website is a good place to learn about JFFNMS including updates. There is an online version of this manual linked from the site too. The username appears next. This is the name the operator is logged in as. If you click on the name you will be able to view the user profile. If granted user administration privileges, you will be able to see other users’ profiles. In the middle of the control bar is the view selection controls. Clicking on the Views link takes you to the currently selected view. If you want to see a different view select a different one from the combo box. The icon with the white arrow in a blue box immediately to the right of the combo box works like the Views link. To the right of the views controls is the Performance link. As is implied by its name, clicking on this link will take you to the performance screens which will display graphs. If the person logged in has Administration access, there will be an Administration link. See Chapter 4 Administrating JFFNMS for information on Administration. Clicking on this control will open up the Administration menu, which appears on the right of the window. Finally, the far right hand side has a small button with an X. This button logs the current user out of JFFNMS. The frames below the control bar vary depending on what mode you have JFFNMS is in. There can be generally one or two frames vertically split. In the example shot in Figure 2.1 JFFNMS is displaying the “Interfaces and Events’ view. The top frame is showing the Interfaces, including a pop-up window giving greater details of an interface. The bottom frame is showing events.

2.2

Profiles

The profile menu option allows user to setup preferences and extra information. An important option is the email address of the user. You will need to set this if you want to receive alerts.

2.3

Event Viewer

The event viewer is one of the more often used views in JFFNMS. It displays events as colored horizontal bars. The color is the severity of the event. 13

14

CHAPTER 2. USING JFFNMS

Figure 2.1: Main JFFNMS Window The viewer can be access via the view control either by itself or as a combination of an Map view. In addition the viewer can appear from certain actions such as clicking on an interface icon. By default the events are ordered by date, with the event at the top of the view being the most recent and all events are displayed with the exception of the Unknown events. The top of the event viewer has some icons to assist in filtering and moving around the events. Table 2.1 shows the list of icons and their Functions. On the far top-right of the event viewer there is a drop-down box that allows you to select from the predefined filters. It is by default “All Events” but you can use this when you have consistent filtering requirements. Each event has the following column: Date This is the date the event was correlated Ack Events that have been acknowledged get a green tick, otherwise they get a flashing light logo. Clicking on the event gives an expanded view of that event. See below for more details about acknowledgment

Table 2.1: Event View Icons Icon

Function

two up arrows Go to first (newest) event two left arrows Go to previous event page two right arrowssGot to next event page Arrows in a circleRefresh current event page + Double the maximum number of visible events − Halve the maximum number of visible events all Show all events Speaker Mute/Permit sounds for new events Page with blue X Open another window that will export current events into CSV format Funnel Open express filter - see Event Express Filter( section 2.3.1 )

2.4. PERFORMANCE GRAPHS

15

Figure 2.2: Express Filter Window Type The event type. Should be a short description of the event or class of event. Host & Zone The short names of the host and zone for the event. If there is no relevant host, for example an event about JFFNMS itself, then this column is merged with the Type column. Event The actual message for the event appears here Expanding the event shows the details of the 5 common event fields (Username, Interface, Severity, State and Info). If you then click on one of the values for this field the event viewer is filtered by that item. For example if the event has a Username field of value ‘jsmith’ then clicking on that will show all events which have ‘jsmith’ as their username.

2.3.1

Events Express Filtering

Besides the standard sets of filters, an operator can produce an ad-hoc set of filters for events. The express filter is accessed by clicking on the funnel icon at the top of the event viewer. This brings up a new window, as show in figure 2.2 that has various rows of items that you are able to filter on. Each row starts with 3 radio buttons labelled ‘Yes’, ‘No’ and ‘Dont’ which mean display events that match, display events that don’t match or just don’t use this row for filtering respectively. The contents of the rows is reasonably self explanatory as most are based upon aspects of the event, such as the date it happened or the event fields. The free-text fields are exact matches only, you cannot use regexp there. Once you have selected your filter criteria, Choose if you want the filtered events to appear in a new or existing window and click the Filter Events button. If you decide not to filter events then just click the close link.

2.4

Performance Graphs

JFFNMS is capable of displaying graphs for various values for interfaces. The type of graphs is dependent on the Interface Type, for more information see the section on Interface Types( section 16 ). The graphs can be reached by clicking on the Performance link that appears in the control tool bar. This will first bring up another window that is split horizontally.

16

CHAPTER 2. USING JFFNMS

Figure 2.3: Performance Window

2.4.1

Interface Selector

On the left of the Performance screen is the Interface Selector. This selector is used throughout JFFNMS to choose a list of interfaces based upon various criteria. There are four ways interfaces can be grouped. You can only choose one item from one group. For example you cannot select an interface type and customer at the same time. The four groups you can select interfaces are: Customer This is the customer that has been assigned to the interface. Map Submaps can be used to group interfaces in any way you want. Host This option groups all interfaces found for a specific host. Type Choose all interfaces that are the same interface type. Selecting a group will then display a list below the group boxes that have all interfaces that match that selection. You then have the choice of viewing all interfaces JFFNMS knows, viewing all interfaces for that group or viewing some selected interfaces. The interface list allows you to choose one or many interfaces. The way of choosing multiple interfaces is browser independent, but normally control-click will choose multiple interfaces while shift-click will choose a group of interfaces. Obviously just clicking an interface de-selects others in the list and chooses that one. Click on either the links or the button to choose the interfaces.

2.4.2

Performance Trend Window

The performance trend window will appear to the right of the Interface Selector and is where the graphs will appear. Along the top is the window title and some icons. The icon with the two green arrows will refresh the graph, although the graph will automatically refresh itself at a given amount of time. To stop the automatic refreshing, click the “stop refresh” link. If you have interface administration privileges, you will see a note and pen logo. Clicking on this logo takes you to the interface administration screen with the interface for this graph selected.

2.4. PERFORMANCE GRAPHS

17

The spanner icon will take you to the set of tools for this interface. The last icon is the report screen. The next row have a box selecting the graph type you want to see, and then time selection. Time can either be selected by a span, giving a start and finish time, or it can be selected by a preset time such as “Last 6 Hours”. Finally below the graph and time selection controls is the graphs. If multiple interfaces have been selected then each interface will have its own graph. For graph types that permit aggregation, all interfaces of the same interface type will appear on the aggregated graph as well.

18

CHAPTER 2. USING JFFNMS

Chapter 3

Installing jffnms This chapter describes the method to get and install JFFNMS. You probably also want to setup the external programs too. To do this refer to the External Programs in chapter 15 for more information.

3.1

Getting JFFNMS

JFFNMS is available from SourceForge(see http://www.sf.net/) and its mirrors. For more information on getting JFFNMS look at the JFFNMS website at http://www.jffnms.org/.

3.2

Hardware Requirements

A very commonly asked question on the email list is what sort of hardware is required to monitor X number of hosts or interfaces. The problem is there is no simple answer because there are many variables in play. Not only is there inter-dependencies within the hardware itself (e.g. swapping is less of an issue with faster disks) but each poller consumes different amount of resources. However all is not lost. I have asked members of the mail list to send me their list of hardware and a few other parameters. You have look at the table to get some idea of what people are currently doing. For dual systems that have a separate database server, the database appears on the line below.

3.3

Required Packages

For JFFNMS to work, or to enable some optional features, you will need some packages. For those who don’t have packages, you’ll need to find the various sources and compile it yourself. An important point is one package you should NOT install is the rrdtool module for PHP. This could be called something like php-rrdtool or php4-rrdtool depending on your distribution. In any case don’t install it otherwise you’ll get problems.

3.3.1

Source Code

If you have no packaging system listed below, you will either need to work out what packages corresponds to the ones below or find these source code archives and build the programs yourself. • Apache - Apache web server, essential. • PHP - Needs to have MySQL, GD and SNMP enabled. • net-snmp - Optional if you want SNMP alerts so you use snmptrapd. You can use an alternative one if you prefer, but it has to be able to call an external program. • MySQL - Need this for the libraries (eg to enable PHP MySQL) as well as to supply the main database. • rrdtool - Round Robin Database, for MRTGlike graphs. See http://www.rrdtool.org/ . 19

20

CHAPTER 3. INSTALLING JFFNMS

Table 3.1: Example hardware requirements Hosts

Interfaces

15 125 38 660 41 1531 55 920 67 2017 Database 69 1210 72 1100 82 989 Database 95 624 119 1700 150 2000 Database 362 4570 Database

CPU

Memory

Disk

OS

Notes

PII 266MHz Xeon 3.4GHz P4 2.8Ghz Cel 1.6 Gz Dual 3GHz Sunfire V110 P4 1.7GHz Dual Xeon 2GHz Dual Xeon 3.4 GHz Dual Xeon 3.4 GHz Dual 1.2 GHz Dual P4 3GHz P4 2.8 GHz P4 2.8 GHz Dual Xeon 2.8 GHz Dual Xeon 2.8 GHz

128MB 512MB 1GB 512MB 2GB

SCSI SCSI SCSI IDE SCSI RAID0

768M 4GB 2GB 2GB 1.25GB 1GB 1GB 1GB 2GB 2GB

IDE SCSI RAID5 SCSI SCSI ? SATA SATA SATA SCSI RAID0 SCSI RAID0

RedHat9 Debian sarge Slackware Windows? RedHat ES4 RedHat FreeBSD 4.11 RedHat ES4 Debian Sarge Debian Sarge Windows 2000 Windows 2003 Debian Sarge Debian Sarge Freebsd 4.11 Freebsd 4.11

5avg 3 slow 5avg 1.72 5avg 5.0 35-65% 5avg 45.78 unknown 55% avg 5avg 0.45 5avg 0.13 5avg 0.13 22% bit slow 5avg 2 5avg 2 5avg 4.5 5avg 4.5

• nmap - Provides TCP ports discovery • tac-plus - TACACS+ Server for AAA. Optional but handy to have. (get it from the jffnms site) • msyslog - Modular syslog daemon to insert syslog lines into MySQL or PostgreSQL databases. (get it from the jffnms site)

3.3.2

RedHat-like Packages RPM

• apache - The webserver, essential • cron - Most likely installed but you need it. • mysql-server - Holds all the information for JFFNMS • php - Apache module for running PHP4 scripts • php-snmp - SNMP module for PHP • php-mysql - Mysql library so PHP can talk to MySQL. • php-pgsql - Mysql library so PHP can talk to PostgreSQL. • rrdtool - Examine and manipulate the RRD graphs. (www.rrdtool.org) • fping - Fast Ping program (www.fping.com) • net-snmp* - Provides snmp commands and daemons (was UCD-SNMP) • nmap - Provides TCP ports discovery • tac-plus - TACACS+ Server for AAA. Optional but handy to have. (get it from the jffnms site) • msyslog - Modular syslog daemon to insert syslog lines into MySQL or PostgreSQL databases. (get it from the jffnms site)

3.4. INITIAL CONFIGURATION

3.3.3

21

Debian Packages

• apache2 - The webserver, essential • cron - Most likely installed but you need it. • libapache2-mod-php4 - PHP4 module for apache webserver • msyslog - Modular syslog daemon to insert syslog lines into MySQL or PostgreSQL databases. (get it from the jffnms site) • mysql-server - Holds all the information for JFFNMS • php4-cli - The php cli (command line interface) part • php4-gd - Drawing graphs. • php4-mysql - Mysql library so PHP can talk to database. • rrdtool - Examine and manipulate the RRD graphs. • snmpd - Provides snmptrapd for SNMP events. • tac-plus - TACACS+ Server for AAA. Optional but handy to have. Doesn’t need to be installed on same server

3.4

Initial Configuration

Before you can work with JFFNMS, you will need to tell it where to find the various paths etc. To do this, go to the web location http://sername/admin/setup.php For example if you are installing JFFNMS on your local computer the URL will be http://localhost/admin/setup.php For details about what the various parameters do, see the section JFFNMS Global Setup (see Section 3.7) .

3.5

Cron Jobs

JFFNMS uses cron for a lot of its tasks. Without it no status will be updated or statistics collected. JFFNMS is shipped with an example cron file that you should install.

3.5.1

What does each task do?

• consolidate - Runs the Consolidator Process 3 times every minute. • poller - Executes the Polling Plan every 5 minutes for each host in parallel, one ”thread” for each host. • autodiscovery interfaces - Executes the discovery scripts agains each host every 30 minutes and evaluates if it has to add, modify or delete and interface, based on the applied Autodiscovery Policy. • rrd analizer - Evaluates SLA conformance based on the SLA Definition applied to each interface. • tftp get host config - Gets the Host Configuration file via one plugin and stores it in the database. Currently it works for Cisco Routers and Switches using TFTP and SNMP RW. • cleanup raw tables - Deletes one week older ”raw” events from the trap, syslog and acct tables. • satellite distribution - Distributes Polling information to peer Satellites. (Optional) • tmpwatch.sh - Cleans the Temporary files folders and old logs in GNU/Linux/Unix systems.

22

CHAPTER 3. INSTALLING JFFNMS

Figure 3.1: System Setup Screen

3.6

Installing SNMP Extension

If you want JFFNMS to measure your iptables and tc information on a Linux server, you will need to put a SNMP extension in. This extension allows the SNMP daemon to get values about iptables and tc. These values are in turn read and processed by JFFNMS. The target computer will need PHP4 on it, but only the cli or cgi version. For example, Debian users would install only php4-cli and not php4 (which is the apache version). You will need to note what the php binary is called, on Debian it is called /usr/bin/php. In the directory docs/utils/jffnms-snmp you will find some PHP files. Copy these files to a target host. It doesn’t matter what directory you put them in (but remember which one it is for the next step). Next, change the snmpd configuration, the file will be something like /etc/snmp/snmpd.conf. I put my php files into the directory /usr/local/share/jffnms-snmp so the configuration lines will look like: pass persist .1.3.6.1.4.1.2021.5001 /usr/bin/php -q /usr/local/share/jffnms-snmp/jffnms-snmp.php tc .1.3.6.1.4.1.2021.5001 pass persist .1.3.6.1.4.1.2021.5002 /usr/bin/php -q /usr/local/share/jffnms-snmp.php iptables .1.3.6.1.4.1.2021.5002

Reload the snmpd so it takes the new configuration and you should be complete. To test, use the snmpwalk program. For a remote host with IP address 192.168.1.22 using the community of public, the command is “snmpwalk -v 1 -c public 192.168.1.11 .1.3.6.1.4.1.2021.5001”. Also try it with the last number of 5002. Both should return some tables. If all is well, you should now have two new sets of interfaces if you do a manual discovery.

3.7

JFFNMS Global Setup

The main setup of JFFNMS is done in the Administration menu item System Setup. The page is also displayed when you are initially setup the system with the url http://servername/admin/setup.php. This page will setup things like the paths and directories as well as check that certain modules are loaded. This page is also the one you will first see when setting up JFFNMS as certain paths will need to be specified before it can work properly.

3.7. JFFNMS GLOBAL SETUP

3.7.1

23

Basic Details

The first two items are the version of JFFNMS and the Site Name. You cannot set the version but the Site Name is used for the login screen. Most people put their company’s or department’s name here.

3.7.2

Database Configuration

JFFNMS needs a database to do its work. This section is how you tell it the details of the database. Database Type - JFFNMS can work with either MySQL or PostgreSQL databases. The important thing is to ensure that whatever one you choose that you have the relevant PHP module installed and enabled. Database Host - The host or server where the database is located. If the database is on the same computer as where JFFNMS is installed then this field should be “localhost”. Database Name - Name of the database. This is the same name you created the database with the mysql “mysqladmin create database” or postgresql “createdb” commands. Database Username and Password - The database should be protected with a username and password. Enter them here so JFFNMS knows what values to use. Database Working? - This field will be a green ‘Yes’ if the parameters are correct and the database is available. If there is a problem it will be a red ‘No’. See Database Problems( section 18.2 ) for ways to fix this.

3.7.3

System Configuration

It is best to keep them at the defaults. The Operating System should be set correctly to whatever you are using. Operating System - JFFNMS can usually work out what sort of system it is on. Set this to either UNIX or windows depending what platform you are running it on. GUI Access Methods - Used for satellite mode. Most people should leave this as local. Satellite Server - For a server that is a satellite, this is the hostname or IP address of the main JFFNMS server. For non-satellite mode this is blank. Satellite URI - The URI for the satellite information. This is only set on the satellite server and is in the format http://satellite hostname/admin/satellite.php. If this field is needed, it should be set automatically.

3.7.4

Paths Configuration

Paths have a lot to play with JFFNMS. There are directories for all sorts of information. Each item in this section has a label for the path, a text box to see or change the path and a state to check if the path is ok or not. Generally JFFNMS only checks that the directory exists, it currently does not check if the directory is readable (or writable when appropriate). There can be basically two users that are important to JFFNMS and directory permissions. The first is the “webserver user”, that is the user that the webserver runs as, which would be something like www-data, http or www. On badly setup servers it might be nobody. The second user is the “cron user” which is the user that the cron jobs run as. This is really up to the administrator on how they would like to set it up. Systems using the Debian packages have a JFFNMS user for this purpose. Paths come in two forms. People often get them confused and wonder why some location is not available when they can see it on the server or website. Filesystem Path - This is path that exists on the filesystem or the disk drive. Paths like /opt/jffnms or C: JFFNMS would be filesystem paths. You can check paths using things like command line prompts. JFFNMS sometimes calls these sorts paths absolute paths.

24

CHAPTER 3. INSTALLING JFFNMS

Webserver Path - This path is part of a URL for a webserver. It may map directly to a Filesystem path or, using things like symbolic links or apache aliases for example, be just a virtual directory. Webserver paths are accessed from a browser and JFFNMS sometimes calls them relative paths. Absolute Path - The filesystem path to the top of the JFFNMS directory tree. Subdirectories of this path would be things like htdocs, lib and engine. Both users need to read the files in the subdirectories. Webserver Relative Path - The top of the webserver path or the directory part of the URL. If JFFNMS is installed on a server called mynoc.example.net and you set this field to /jffnms then the URL to get to JFFNMS would be http://mynoc.example.net/jffnms/ TFTP Server Files Path - The filesystem path where TFTP files will be temporarily put. The cron user needs write access to this directory. When setting up the TFTP server its “root directory” needs to be the same as this value. RRD Files Path - The filesystem path where RRD (Round Robin Database) files will be stored. The cron user needs write access and webserver user needs read access. Engine Temp Files Path - A filesystem path for temporary files that the poller and other systems use. Both users need write access. Log Files Path - A filesystem path for the log files, if they are enabled. cron user needs write access. Temp Images Absolute Path - A filesystem path for putting the temporary files that make the images, such as the interface alarm icons. Needs to be writable by webserver user. WebServer Temp Images Relative Path - The webserver path for the URLs for the images. In other words the system should be setup so that requests for items that have this field in their URL will get the webserver serving files from the “Temp Images Absolute Path” real directory. PHP Executable Path - The full filename of the PHP executable. JFFNMS requires both the PHP module and the separate executable. For most installations it would be either /usr/bin/php or /usr/bin/php4. GraphViz Neato Executable Path - Full filename to the neato program which is part of the graphviz package. RRDTool Executable Path - Full filename to the rrdtool program. GNU Diff Executable Path - Full filename to the GNU diff program. The program is used for things like displaying the differences in configurations of hosts. NMAP Portscanner Executable Path - Full filename for the nmap program. Nmap is used for finding TCP and UDP ports on remote hosts. FPING Executable Path - Fping is an improved version of the ping program. This executable is used for the reachability interfaces. SMSClient for SMS via Modem - optional - The full filename for the smsclient program. You only need this if you are going to send pagers via a telephone modem. NTPQ Executable - optional - The full filename for the ntpq program. This is a query client for the NTP protocol and with this you can check that local or remote hosts have their time server synchronized.

3.7.5

PHP Status

The PHP Status section is not for setting parameters, but helps with debugging by checking that various modules and PHP settings have been set. Changing the PHP setting is done with the php.ini file. Adding PHP modules is highly operating system (and distribution) dependent. Remember that for some distributions installing the module is not enough, you will also need to change the php.ini file to load the module too.

3.7. JFFNMS GLOBAL SETUP

25

Register Globals set to On - This means the globals are available everywhere. Most PHP versions now ship with this turned off by default. Allow URL fopen set to On - This option allows PHP programs to open connections to URLs (or remote sites). Required for TCP Port content check to work. SNMP Module Loaded - You need to have SNMP module loaded if you want to do any SNMP gets. JFFNMS could be used without this module loaded but it would be severely limited in what it could do. Sockets Module Loaded - Used to make external connections to remote hosts. GD Module Loaded - The GD module is used to generate some graphics, such as the hosts and interfaces alarm boxes. optional MySQL Module Loaded - This module allows you to store the information in a MySQL database. Either this module or the PgSQL module needs to be enable for JFFNMS to work. optional PgSQL Module Loaded - The module to allow access to a PostgreSQL database. Perl RegExps Module Loaded - If you want to use Perl Compatible Regular Expressions (PCREs) this module needs to be loaded. PCREs are generally better and faster than the old POSIX REs. optional - WDDX Module Loaded - This module can be used for transferring information from the satellite servers. Recommended if you are using satellites but otherwise not needed.

3.7.6

GUI Options

GUI options are only used to set some of the cosmetic aspects of JFFNMS. GUI/Auth Login Method - Most people leave it as login screen, HTTP authentication means you get the standard ugly login box instead. Login Screen Image URL - The URL for the login graphic Login Screen Image Link URL - The link the user will go to if they click the image. Custom CSS Stylesheet URL - If you want to change from the default look, create a CSS Stylesheet and put its URL here.

26

CHAPTER 3. INSTALLING JFFNMS

Chapter 4

Administrating JFFNMS Apart from configuration, there are some tasks that the administrator may need or want to do with JFFNMS. This chapter describes these tasks, when they are needed and what to do.

4.1

Reporting

JFFNMS can report on the availability of a system of a set of interface or on individual interfaces themselves. The reports are found at the Administration menu item Reports ⇒State & Availability. The interface selector is like the one you see for the interface administration screen and the performance screen. For a report you select the interfaces you want to report on, using the interface selector and the time period you want the report for. You can also choose what event types you will consider for an interface being up and down. In the top right corner, there is a view details option box. Clicking this box will show the details of the alarms, including their start and stop times plus their duration. For each interface selected, you get a report on: round trip time and packet loss (where relevant); Unavailable time ; Availability. Unavailable time is the total time that the interface has been considered in the down state for the selected time period. The availability is: (totalselectedtime − unavailabletime) ∗ 100/totalselectedtime Right at the bottom of the list is the Total Unavailable Time and Total Availability. These figures are for the “system” of the selected interfaces. It’s important to know they are a system availability and not just an average.

4.1.1

How JFFNMS calculates Total Availability

JFFNMS does not just do a simple (and often incorrect) average of system availability for the Total availability. It doesn’t do a multiplication of the individual availabilities either. The next few paragraphs describe how Total Availability is calculated. JFFNMS scans the database for all Ups and Downs of the selected interfaces for the requested time interval. It then orders these Ups and Downs in chronological (time event occurred) order. The ordered list is then scanned and an “interfaces down” counter is initially set to 0 and is incremented when the scanner finds a Down alarm and decremented when it finds and Up. If the counter leaves 0, the system notes the time of the Down alarm. If the counter returns the 0 (meaning no interfaces are down at that time) the time difference between this Up alarm time and the time of the Down alarm that incremented the counter from 0 is added to the Total Unavailable time. For example, if there are a series of events for the selected time period 00:00 to 24:00 that look like table 4.1 then the consolidator will process the events into a series of alarms that will look like table 4.2. When the report needs to work out the Total Availability it ignores which particular interface was affected by the Up or Down event but instead uses a counter of down interfaces. A count of 0 means the system is available. Table 4.3 shows the how the times for the interface availability are calculated. There are some important boundary conditions and assumptions that are made here that are useful to know. The first is only alarms that start and stop within the time period selected are considered. A 27

28

CHAPTER 4. ADMINISTRATING JFFNMS

Table 4.1: Example list of interface events Time

InterfaceState

10:00hs 11:00hs 12:00hs 13:00hs 14:00hs 15:00hs

A B B C A C

Down Down Up Down Up Up

Table 4.2: Processed alarms from example events Start

Stop

Duration (hours)

10:00 11:00 13:00

14:00 12:00 15:00

4 1 2

Interface A B C

Table 4.3: Interface State Calculation Time

State

00:00hs 10:00hs 11:00hs 12:00hs 13:00hs 14:00hs 15:00hs

— Down Down Up Down Up Up

CounterNotes 0 1 2 1 2 1 0

Initially assume all interfaces up Counter was 0 so record time as start

it is now 0 so record time as stop of this outage

4.1. REPORTING

29

side effect of this is that all interfaces are assumed up at the start of the report and have to be up at the finish of it (otherwise their stop time is past the end of the time period). The table above shows a system outage of 5 hours long, starting at 10:00 when interface A went down and finishing at 15:00 when interface C comes up. Using a method such as this one means you get a Total Unavailable Time of 5 hours instead of 7, which is the sum of the 3 outage times. 5 hours is correct, because this is the time that one or more of the interfaces being considered was down. The Total Availability is now easily worked out using the Total Unavailable Time and Total Selected Time (totalselectedtime − totalunavailabletime) ∗ 100/totalselectedtime In our example, it is (24h − 5h) ∗ 100/24h = 19 ∗ 100/24 = 79.17% So the Total Unavailable Time for our report is 5 hours and the Total Availability is 79.19%.

30

CHAPTER 4. ADMINISTRATING JFFNMS

Chapter 5

Configuring JFFNMS There are a wide variety of objects (internally called structures) within JFFNMS. To really understand JFFNMS, you have to get a good understanding of its objects and the relationships between them. The objects are arranged in order as you would see them in the administration screens. This is done to help you find the object you need and also the administration menus are arranged in a logical order.

5.1

The Consolidation Process

Before getting into how to configure JFFNMS it is important to understand how it handles various pieces of information and what definitions it uses for the pieces of data. Raw events are brought in via their relevant processes. That is msyslog or syslog-ng import raw events into the syslog table, or snmptrapd imports raw events into the traps table (with the variables going into the varbinds). Every minute the consolidator is run from cron. There are two main parts of the consolidator. The first is the raw event specific part. This takes the raw events and compares them to a match list. For example the syslog consolidator will look at all non-analyzed raw events in the syslog table and will compare each one with the list that is found in the table you create on the Syslog Events screen and place any matching into the events table. An event is a one-off occurrence that is essentially a cleaning up and categorizing of a raw event. A syslog raw event has a time, host and a message. The consolidator, if there is a match, will give it an event type, a host and then optionally an interface, state, username and information. Events are visible in the event viewer (the display with the horizontal colored bars). The second part of the consolidator is the event consolidator. It is run after all the raw event consolidators have run and it looks at the new events that have been created. It’s job is to convert an event into an alarm. An alarm has a type (the same as the event being processed), a start and stop time, an interface id (which also specifies its host) and a state (from Alarm States) called active. An alarm is started and stopped by the consolidation of events. For this to happen, the consolidator needs to find an event that has a known host, interface and state. There is no concept of an alarm with no interface. Alarms are visible by the colors of the interfaces (or their hosts) in the map views. When the consolidator finds an event with Alarm State DOWN (ie the interface is now down or broken) it first goes to find any existing alarms for the same interface and type if so makes them inactive and sets the stop time based on the event time. It then creates a new alarm with the event time as the start time. A new event with state UP will make the consolidator set any alarms with matching interfaces and type to inactive and set their stop time to the event time. This means the interface on the map should go back to OK state. The Start Time, Stop Time and Duration are used in the State Report to calculate Availability. To find out what interface an potential alarm is for, the consolidator matches the event’s host with each host. If that matches it then tries to match the event’s interface field with the interfaces name (or interfaces.interfaces variable internally)

31

32

CHAPTER 5. CONFIGURING JFFNMS

Chapter 6

Host Administration Hosts are the reason for wanting JFFNMS in the first place. They are the devices that have interfaces that need to be monitored. This could be an ethernet port, a program running or even some environmental variable like temperature. Hosts are grouped together in Zones but they also can be arbitarily grouped into one or more submaps. The actual thing that is monitored on a host is called, within JFFNMS, an interface.

6.1

Zones

In earlier versions of JFFNMS, Zones were merely a grouping of Hosts( section 6.2 ) which could be done in any way that made sense to you. Newer versions now use the Zones for Network Autodiscovery policies. Zone configuration is found at Administration menu item Hosts and Interfaces ⇒Zones. Table 6.1 describes the contents of the Zone table. The section on The Seed CIDR parameter tells the Network Discovery where to initally go looking for hosts. There can be multiple networks specified as long as they are separated by a comma. The CIDR notation is network/bits. For example to scan all of the Class A 10.0.0.0 network and the Class C 172.16.1.0 network, the parameter would be 10.0.0.0/8,172.16.1.0/24 The Max Hops parameter is used to limit how far out JFFNMS goes looking for hosts. A level of 1 means only hosts contained in the CIDR block column will be scanned. 2 means all of level 1 but also any immediately adjacent subnets would be scanned. 3 means everything 2 does, but also adjacent subnets to the ones found in 2, or subnets that are 2 hops away from your seed subnets.

6.2

Hosts

Hosts are the actual servers or computers that you are monitoring. It’s essentially something that has an IP address. Hosts are also containers for interfaces. A host has the following fields: Name name of the host Zone Which Zone (see Section 6.1) the Host is contained in IP Address The IP address of the host that is used for the poller Polling Check this box to enable polling for the host Read Only Community The SNMP community string to get read only access Read/Write Community The SNMP community string to get read/write access Visibility FIXME Autodiscovery Policy The policy used for autodiscovery of interfaces for this host, see AD Policies in section 10.6 for information. 33

34

CHAPTER 6. HOST ADMINISTRATION

Table 6.1: Zone Table Definition Column

Description

Zone Short Name Image

Description or name of the Zone Short name of the zone, often the same as Zone column A location of a small icon for the zone that appears in the event view. Image files are located at JF F N M S/htdocs/images. Visibility Determines if the hosts and interfaces are Shown, Hidden or Marked disabled. ND Enabled If this box is checked, Network Discovery is enable for this zone. Seeds CIDR Seed networks to start autodiscovery in CIDR (network/bits) notation. Allow Private IP If checked, Network Discovery will include Hosts that use Private IP addresses. SNMP CommunitiesA comma-separated list of SNMP communities to try on newly found hosts. Max Hops How far wide to scan for networks? Re-Scan How often to go and and re-scan the networks for new hosts.

AD Default Customer the customer that auto-discovery will assign to a newly discovered interface, if you leave Unknown, you will see a message every 30 minutes saying that the interface setup is incomplete. Customers have to be configured as described in the User Administration( section 7.2 ). Main Interface if the interface(s) selected here are down, the poller will only check them (and not all other host interfaces) until they are up. TFTP Server IP The server to transfer the configuration file to, this should be configured to the JFFNMS server IP address from the Router point of view. Only used for hosts that you will be transferring configuration files from. Config Transfer Mode For Cisco routers and switches, which method to get the configuration. Tacacs+ Source IP Useful when you run TACACS over tunnels or something (not the same IP for polling) this is the Source IP of the TACACS request (could be the same as the IP if you have configured your router that way). Satellite Set to Local for most circumstances Latitude Longitude You can either add the hosts here or get JFFNMS to do it automatically using Network Discovery( section 6.4 ).

6.3

Interfaces

Interfaces in jffnms are not only the physical devices that connect a Host to the network. They also include service/daemon ports or some SNMP parameter. You can think of an Interface as some attribute of a host that you need to watch (can I ping this IP address, is the mail port up?) Interfaces can be viewed by Administration menu Hosts and Interfaces ⇒Interfaces. This brings up the Interface selector. You can then choose interfaces using various criteria. After selecting a group of interfaces, the interface table appears in the middle frame. The table has the following parameters Host The host name and Zone the host is in Interface Name Display name of interface, (ie. if it is a Cisco Router Serial1/0). This name should be whatever the device calls it.

6.4. NETWORK DISCOVERY

35

Customer This is the owner of the interface. The customer can also login and see their interfaces. Poller Group The group of pollers/backend that is defined in the interface type. for instance the Cisco Interface Poller Groups defines all the OID’s and the way to use them in the Physical Interface Type. Check Status SLA The SLA that is used against the interface. See section 13.12 about SLAs. Enable Sound Enables audible alerts on the Interfaces Screen if the interface changes states. Show in Maps This option will determine if the interface is visible in the root map or not. You also have the option of showing the interface disabled,which means it will be gray on the map. The interface table will also display other columns that are specific for that interface type. These columns will be different for different types of interfaces.

6.4

Network Discovery

There are two ways that hosts can be added to JFFNMS. The first way is to manually at them using the Hosts table, see the Hosts Section( section 6.2 ) for details on manually adding Hosts. The second way is for JFFNMS to find the hosts itself. This is called Network Discovery, Network Autodiscovery or NAD. This document will use Network Discovery throughout to reduce confusion. The idea behind Network Discovery is for a NMS to be told some initial parameters and some boundaries about what sort of devices to find and then it goes looking for them. If they meet a certain criteria then they are added to the list of Hosts that the NMS knows about. While they sound similiar, Network Discovery is different to Autodiscovery Policies( section 10.6 ) which are used to find interfaces on known hosts. Possibly a less confusing term would be to call them Interface Discovery. The Network Discovery is kicked off by adding or editing entries in the Zones Table( section 6.1 ). The Zones drive and specify the discovery.

6.5

Hosts Saved Config

This is where the Router configuration files are stored, JFFNMS uses TFTP and SNMP to tell the router to send the configuration. and then it Stores it here. A TFTP Server and SNMP RW community are needed. You configure the community and the TFTP setup in the Hosts(see Section 6.2 screen.

6.6

SubMaps

Submaps are a way of grouping the display of hosts or interfaces. They are in a hierachial tree setup with the root of the tree being the map called “Root Map”. Each submap can be a direct child of the Root Map or a child of another submap. A submap has a name and a background colour. A User can optionally have a profile setting that gives them a base map. This means they will only be able to see their base map and any of its decendants. Clicking on the View item for a submap shows all the interfaces that are visible for that submap. You can add more interfaces to a submap by clicking the add button above the SubMaps/Interfaces Administration list or use the add to submap feature in the interfaces configuration.

6.7

Satellites

Satellites is the JFFNMS RPC (Remote Procedure Call) System, its used for Distributed (Load Balanced and Fault Resistant) Polling, and also allows External programs to call JFFNMS public API’s, this is useful to integrate JFFNMS with your other applications. There are various protocols available, like HTTP/Serialize, SOAP, TCP/WDDX, etc.

36

CHAPTER 6. HOST ADMINISTRATION

Chapter 7

Users and Customers JFFNMS has two types of people that can have access to it: Customers and Users. Customers only get the output or a view of what JFFNMS monitors while Users interact and change JFFNMS and the monitoring itself. Customers are the customers of the servers or services that you are providing. They will be interested in know how well a particular server or interface is performing and how often an interface has gone offline. They get read-only access of the interfaces that they are the owners for. Users are the administrators and operators of JFFNMS. With the exception of access levels or enabled features, there is no distinct difference between users who are network operators and users who are administers of JFFNMS itself.

7.1

Customers

Customers are people or organizations that own or use a service, which in JFFNMS is called an Interface . You will need to make at least one customer (eg “Common” or “myISP”) so that other objects are owned. Customers may also have a Username and Password. This is not the same as a standard user but gives a limited access to the system so the Customer can look at some performance graphs for their equipment. The Customer list is found in the Administration menu item Users and Customers ⇒Customers. Each customer has the following properties: Customer Name - The long or full name of the customer Customer ShortName - Shortened version of the customer name Username - A Username to login to JFFNMS Password - Password to authenticate to JFFNMS. JFFNMS allows Customers to login and view the performance graphs of interfaces that have been assigned to them. To do this they need to login with the username and password that you have entered into this table.

7.2

Users

Users are the operators and administrators of JFFNMS. These would be the people who are responsible for the ongoing maintenance and monitoring of the network elements, as well as the people who configure and customize JFFNMS itself. Unlike customers( section 7.1 ), Users cannot own interfaces. However if you want to limit the interfaces they can see you can do this with Maps. Selecting the Administration menu item Users and Customers ⇒Users shows the table of all configured users. Each row of the table shows the following features of the user: 37

38

CHAPTER 7. USERS AND CUSTOMERS

Figure 7.1: Users Table Table 7.1: Users Table Fields Field

Description

Action A combo box allowing you to perform various actions on this user Username The username this user uses to login to JFFNMS Password Password used to login to JFFNMS Full NameFull name of the user Router Used to link to modified TACACS. Users with this clicked will be able to authenticate via the linked TACACS.

7.2.1

Creating a new User

JFFNMS ships with a single user called Admin. Follow the steps below to add more users to the table. 1. Bring up the Users Table (see Figure 7.1) by going to the Administration menu and selecting Users and Customers ⇒Users. 2. Click the Add button in the blue bar at the top of the table. 3. Fill in the relevant fields. 4. Click Save button.

7.2.2

Editing a User

By editing a user you can change the username, password, fullname or the fact the user has router access. 1. Bring up the Users Table (see Figure 7.1) by going to the Administration menu and selecting Users and Customers ⇒Users. 2. Find the user in the table, on that row make sure the drop-down box is showing “Edit User”. 3. Click the blue arrow immediately to the right of the drop-down box to be able to edit the fields. 4. Change the fields to the new values. 5. Click the Save button

7.2.3

User Profiles

User profiles are flags or values that are applied to a user. The profile of a user is used to enable certain functions in JFFNMS or change the default behavior. Table 20.1 shows the list of Profile Values and what they mean. The set of profiles for a user can be seen by going to the Users table, choosing the View User Profiles action and clicking the blue arrow. Most people leave the Profiles alone, but one that almost always needs to be changed is the email address. Without setting an email address this user will not receive any triggers.

7.2. USERS

7.2.4

39

User Triggers

This screen selects which Users( section 7.2 ) have what Triggers( section 11.1 ) enabled for them. For example, a certain user can receive an email when an interface drops. For a trigger to work, it needs to be applied to a user and then made Active. Applying a Trigger to a User To add a trigger to a user, you need to do the following: 1. Bring up the Users Table (see Figure 7.1) by going to the Administration menu and selecting Users and Customers ⇒Users. 2. Find the user in the table, on that row make sure the drop-down box is showing “View User Triggers”. 3. Click the blue arrow immediately to the right of the drop-down box to view the triggers for this user. 4. Click on the Add button at the top of the trigger table. 5. Choose the trigger you want to use 6. Optionally, click the active check-box 7. Click save button 8. Click the Save button

40

CHAPTER 7. USERS AND CUSTOMERS

Chapter 8

Event Configuration Events are generated by the different consolidators that are run out of cron every minute. They are some piece of information that has come into JFFNMS that it recognises. Events come from pollers, syslog and SNMP traps. They have a type and a severity. Events then may cause an interface to have an alarm or for an alarm on an interface to be removed. Events have 4 fields; Username, Interface, State and Info. The Username and Info fields are just generic containers to provide extra information about the event. Interface and State are used by the consolidators to map an event to a specific interface and set the alarm severity of a subsequent alarm respectively. The event consolidator runs over the new list of events that have come in recently. An event that has its type configured in the Event Types Table( section 8.2 ) to generate an alarm is then examined. The Event needs to have an interface associated with it, which means there is a host defined and the interface field in the event matches the name of one interface in the host. In addition the state field of the event needs to match one of the Descriptions in the Alarm States Table( section 8.4 ).

8.1

Event Severities

Each event has a severity. The severity changes the colour of the event and can be used for filtering. The list of event severities is found at Administration ⇒Event Analyzer ⇒Severities. The table lists each severity, the level of the severity (the higher being events that are more critical or important) and then the foreground and background colours used in the event viewer for events of that severity. Do not get event severity confused with alarm levels which has its own table at Alarm States( section 8.4 ).

8.2

Event Types

This page determines how a particular event is displayed. This does none of the selection but just shows what severity and text will be displayed. The following fields are used to define an event type. Description A short description of the event. This will appear in the event viewer. Severity The severity of the event chosen from the list defined in the Event Severities Table( section 8.1 ) Event Text A string template that will display in the message column of the event viewer. Show in Event Viewer? An event can always be displayed, never be displayed or only displayed if the filter specifies it. Event Generates an Alarm? Check this is the event is supposed to generate an alarm for this interface on the event. Different Alarm for Up Event If you want the event to be different when the state is up change this to the Up event, otherwise leave it as Unknown. 41

42

CHAPTER 8. EVENT CONFIGURATION

Table 8.1: Syslog Field Matches Field Description DigitMatches the Nth captured subpattern, or the thing in brackets in PCRE matches or the Nth word after the literal text. ∗ The field will have the whole syslog message D Everything past the match otherAnything else the field gets the value configured

Alarm Duration If the event generates the alarm, how long is the alarm for. Show Host Field Some events have no host, such as internal JFFNMS events. Most events have this item checked. Event Text is a mixture of plain text and parameters derived from the event and the interface the event has occurred on. Parameters are described in <> signs, eg < user > for the user field, the following parameters are defined: The standard set of fields that can come from all events are user, state, info which come from the syslog or SNMP trap consolidation. If the event also is associated with an interface then the Interface Parameters( section 13.14.1 ) can also be used in the template. The “Show in Event Viewer?” means by default you will see the alarm. If you want to record the event but don’t want to see it by default don’t click this option. You can use Filters(see Section 12.1) to display the other events.

8.3

Events from Syslog messages

One of the sources for events is from the syslog daemon. The first thing to do is to get the relevant syslog messages into the JFFNMS Table, which is described in External Programs -Syslog( section 15.2 ) Now that the messages are coming into the raw syslog events table, JFFNMS needs to be told how to convert these items into events. The Syslog Events Types table is found in Administration menu item Internal Configuration ⇒Event Analyzer ⇒Syslog Message Rules. Each row starts with the Text Match. This can either be a Perl-compatible regular expression (PCRE) or a literal string. The literal string cannot contain any brackets ( or ). Due to the flexibility and unambiguity, PCRE matches are preferred. Next are the Interface, Username, State and Extra Info Fields. These can have various values and are described in table 8.1. If the syslog messages matches the Text Match, an event defined by the Event Type column, which is a selection of all event types from the Event Types( section 8.2 ) table, is raised. The fields that are defined are put into the relevant event fields. There may be many matches for the same syslog message. The position column does the order of evaluation. The syslog consolidator will examine each syslog match rule in order of position (lowest number to highest). This way if you have more specific rules give them a lower Position than more generic rules. If you want the syslog messages to raise alarms on interfaces then the interface and state fields must match the name of an interface for the host and a Alarm State( section 8.4 ) respectively. For example, if you have the syslog message show below and the fields you want to have filled in are show in table 8.2. 1473: 4w2d: %SYS-5-CONFIG_I: Configured from console by craig on vty0 (192.168.1.25) You would use text match field like this: %SYS-5-CONFIG_I: Configured from (\S+) by (\S+) on (\S+) We have three sets of brackets, so we have three substrings: substring[1] = ”console” substring[2] = ”craig”

8.4. ALARM STATES

43

Table 8.2: Required Syslog field matches Field

Value

Interface vty0 Username craig Extra Info console Event TypeConfiguration

Table 8.3: Example Syslog Configuration Action Edit Del

ID 10001

Text Match %SYS-5-CONFIG I: Configured from (\ S+) by (\ S+) on (\ S+)

Interface Field 3

Username Field 2

State Field

Extra Info Field 1

Event Type Configuration

Position 1

substring[3] = ”vty0” We want the Interface field to be filled in with ”vty0”, so in the Syslog Message Rules, we put 3 the Interface Field column. That tells jffnms to fill in the Interface field with the first substring match. Similarly, Username has 2 and Extra Info is 1. There is no need for the numbers to be sequential or even for all of them to be there, however the number of bracket pairs must equal the highest index; eg in our example putting 4 in any of the Fields columns is invalid. We’re not using the State field here, so its column is blank. The full syslog event line in JFFNMS will look like table 8.3. You would then setup an event in the Event Types to display the information.

8.3.1

Syslog Consolidation

Every time the consolidator runs, it goes through the syslog table looking for any rows it has not yet analyzed. If it finds a new row, it then goes through the syslog types table looking for a PCRE match to the syslog string. If there is a match, the syslog consolidator will then fill in the fields using the given substring indexes and using the Event Id given for that particular PCRE. If there is no match with any of the PCREs, the consolidator will use the default syslog event. PCREs are checked by their ID number. Like all consolidator modules, the syslog consolidator will output any log information to jffnms/logs/consolida date>.log The date is in the format YYYY-MM-DD

8.4

Alarm States

It is the job of this table to convert from the various classes of alarm states and sort them into a smaller list of internal alarm states. The wide variety of Interface Types( section 10.1 ) means there is a need for this mapping. In addition this table sorts the alarm levels and enables the sounds associated with the alarms. The Alarm State table is found at Administration menu Event Analyzer ⇒Alarm States & Sounds. Table 8.4 describes what each column does. Of the alarm states foundin Table 8.5, the first three states get their inspiration off the ifOperStatus SNMP OID as described in section 5.2.1 of RFC 1156 and you could use that as a guide for what to map your own Alarm Levels. The Alert level means that something bad has happened to the Interface but not too bad to take the interface offline.

44

CHAPTER 8. EVENT CONFIGURATION

Table 8.4: Alarm States and Sounds Table Column

Description

Description A short description of the alarm level. This field is used by the event consolidator to match the state field to an alarm state. Alarm Level The importance of this level of state. The lower the number the more important. For example, a down state is more important than an up, so down has a default level of 10 and up has a level of 100. This means the state of an interface, if it has both active events with a state of up and down will be down. Sound In If an interface goes into this alarm level, send this sound to the GUI. Internal StateThe internal alarm state that this state should map to. There are only 4 alarm states, see table 8.5.

Table 8.5: Internal Alarm State Levels State

Description

Up Interface is up and working and online Down Interface is down or not working or offline TestingInterface is testing or training, it cannot be currently used but is enabled Alert Interface is up but has had some warning or degradation

Chapter 9

SNMP Traps SNMP stands for Simple Network Management Protocol. The funnny thing about the name is that SNMP is most definitely not simple! At pretty much any stage of understanding SNMP; from the concepts, through to the data layout, right down to the coding; it is not simple. Fortunately you can often ignore all the gritty complex detail and make vast simplifications. One part of SNMP is a trap. A trap is a one-way message from a network element; such as a router, switch or server; to the NMS. The messages are sent via UDP, which means they are not guaranteed to arrive. The most common messages are the 6 generic messages. The main ones being link up, link down and agent reset (which usually means the device was rebooted). There are also whats called “enterprise specific” traps, these are the ones the vendors create and can give a rich set of traps, if they will tell you what they mean.

9.1

How JFFNMS processes traps

By itself, JFFNMS cannot receive traps. It requires an external program called snmptrapd to receive and decode the trap from the network. Figure 9.1 gives an overview of how SNMP traps are handled in JFFNMS. snmptrapd is configured to call a script within JFFNMS which inserts the trap and its variables (call varbinds) into the JFFNMS database. All that JFFNMS knows at this point is the source IP of the trap, the trap id (OID) and a set of varbinds identifed by order. The snmp consolidator then processes the new traps my comparing the trap OID to the ones in the Trap Receivers table. If there is a match, the varbinds are passed to the Receiver Command and the results of the command are then passed to the backend. The backend may do whatever it wants with the trap information, but a very common thing is to

Figure 9.1: SNMP trap consolidation process 45

46

CHAPTER 9. SNMP TRAPS

Table 9.1: SNMP Trap Receviers Field

Description

ID Position

The internal ID of this rule, only needed for debugging Releative position of this rule to the others in the table. JFFNMS processes the rules from lowest position to highest. Description Description of the trap of what this rule does. Interface Type The Interface Type that this trap should apply to. OID Match SNMP traps are identified by OID, traps that have this OID will be processed by this rule. Receiver CommandWhich script in the directory engine/trap recievers will be used to process the varbinds into event fields. Parameters Parameters passed to the Receiver Command script. Backend What backend to process the output of the Receiver Command. They are identical as the poller backends, see Backend Items( section 10.4 ) for details about them. Stop If Matches Do no more checks for this trap if the OID matches.

use the event backend plugin which will put a new row into the event database table. This new event will then show on the operator screens.

9.2

SNMP Trap Receivers Table

This table is used to tell JFFNMS which traps you are interested in and if it finds traps matching the OID, what it should do with them. You can access it via the Administration menu item Event Analyzer ⇒SNMP Trap Receivers. It has the following fields:

9.3

Configuring New Traps

To configure a new SNMP trap, you first need to do some homework about the trap and gather the following information: • What is the OID of the trap? This identifies the trap from all other ones. • What, if any, Interface Type would this trap apply to? A trap about a phyiscal interface down would apply to the “Physical Interfaces” interface type. • Are the varbinds going to be processed and if so what event fields will they become? For example the first varbind might be the interface state • What backend to use, or if the trap is received, what do you want to happen? Most people want an event to be raised, in which case you use the event backend.

9.3.1

Trap Setup Example

You’re worried the fan on your router might be broken, so you want to know if the fan is broken trap is sent, 1.3.6.1.4.1.9.9.13.3.0.4 This trap has 5 variable bindings: 1. the enterprise of the trap, for this trap it is enterprises.9.9.13.3 2. Name of the fan that has the problem, eg ps2 3. The IP address of the source of the trap 4. The SNMP community of the trap 5. The state of the trap (its a number 1-5)

9.3. CONFIGURING NEW TRAPS

47

Most of this is pretty useless, we only really need to know something has happened and then to what fan is it. So we setup the event to fill in varbind 2 into the info field and then setup a new event which prints Problem with fan at ¡info¿. So the result on the status screen is something like Problem with fan at ps2. The idea with the SNMP traps is to plan forward but implement it backwards. A lot of the other objects should be done this way too. So start with the event first. Create a new event type, you can see this in the Event Types( section 8.2 ) section. Note down the event ID. Create a new Backend( section 10.4 ). The Backend Command will be “event” and the Paramter will be the id of the new Event Type you created in the previous step. You now need to create a Receiver Command. As it is infinitely easier for me to code one than explain it ill just assume I wrote it for now. As we are mapping varbinds to variables, well assume it is called mapping. Finally create the new SNMP Trap Receiver( section 9.2 ). The OID is the OID of the trap. The Receiver Command is mapping, while the parameters are info=2 (which means the event field called info will get the contents of the second varbind). Backend is the backend you created previously.

48

CHAPTER 9. SNMP TRAPS

Chapter 10

Polling Configuration This chapter describes all the backend collection and collation processes. It is used to describe new host or interface types and is how you add new sorts of checks. Unfortunately it is pretty complicated to wander your way around. The good news is you may not need to edit any of these tables for a default JFFNMS setup. At first there are pollers, that collect data, and backends, that do something with the data that are collected by the poller. An interface type determines what values are stored and how autodiscovery is used. Finally a Poller Group creates a type which is a single interface types and a collection of pollers. FIXME - Poller Group and Interface types have a cross reference, why? FIXME - it was for filtering ease, and Interface Types needs a Default PollerGroup and PollerGroups need to be filtered by InterfaceType on the Interfaces Screen.

10.1

Interface Types

The Interface Types are what drive the polling part of JFFNMS. These are the key to all other attributes such as what values to get, the graphs that are displayed and how interfaces are discovered and polled. The list of Interface Types is shown with the Administration menu item Internal Configuration ⇒Polling & Discovery ⇒Interface Types. The table has a set of actions that can be performed on each Interface Type. Edit and Del edit and delete the Interface Type respectively. The View link displays the Interface Type Fields( section 10.1.1 ) while the rest of the columns contain attributes about the Interface Type. Description A Description of the Interface Type. Such as Noise Level. Autodiscovery Enabled Check this if you are going to create an autodiscovery plugin and you want JFFNMS to attempt to find these sorts of interfaces on its regular interface discovery polls. Validate in Autodiscovery Each time the poller is run the interface IDs are checked again. Autodiscovery Function The name of a Discovery Script( section 13.8 ), in the engine/discovery directory without the .inc.php only the base name. Autodiscovery Parameters This string is passed to the Discovery Script( section 13.8 ) directly. (useful to write a discovery script only once and use it different things. Autodiscovery Default Poller If an interface of this type is discovered by the autodiscovery script then the specified Poller Group( section 10.2 ) will be used for the new interface. Most of the time there is only one Poller Group for each Interface Type. Internal Update Handler Only used for some special interface types but it is a handler that is called each time the record of that type is updated for manipulating data. Generally leave blank. Manual Interface Add If this field is checked, the Administrator can manually add interfaces of this type. Used for things like TCP ports that are not automatically found. Break By Card Display the interfaces grouped by Card. Useful for things such as router or switch ports that have cards or blades. 49

50

CHAPTER 10. POLLING CONFIGURATION

Table 10.1: Interface Type Field types Field

Description

Index

A field that is unique for this Interface Type for a particular host. This is used by JFFNMS to determine which interface is what. For example, for SNMP interfaces JFFNMS uses the ifIndex SNMP OID. Boolean A value that can only be Yes or No. Description Interface icons that appear in the maps have the value for the description fields in them. Other A text field that is not part of the Index or used for Description. RRDTool DSA Data Source (DS) definition for a RRD file.

Have Tools If checked interfaces of this type have the tool icon displayed which takes the user to the tools. Have Graph Cosmetic Option, show a link to the performance (graph) system from the Interface Map. Default Graph Cosmetic Option, which graph to show by default on the Performance Viewer, from the Graph Types( section 10.5 ) Table. Default SLA New interfaces of this type are given this SLA if discovered. RRDTool Structure RRA A standard RRA definition as specified in RRDTool manual page. Try to use RRA:AVERAGE:0.5:1: ¡resolution¿ is replaced by the interface type resolution defined in the RRDTool Resolution Field. RRDTool Resolution - this is RRDTool resolution (how many datapoints do we need to store) 103680 is one year of 5 minutes intervals. RRDTool Step The step in a RRD file in seconds. This should be 300 (5 minutes) for almost all cases.

10.1.1

Interface Type Fields

The fields are used by the discovery, poller and graph plugins so whether or not you need any fields depends on if there is a value you want either an operator to put in or for the poller to track. You may need to put some RRDToolDS type fields if you want to monitor some value. Without these type fields no RRD file is created. Each interface type requires a set of interface fields. These fields are the containers which hold values for the interfaces. The fields are the same for each Interface Type, but the values that are held in the fields are different for each interface. Table 10.1 gives the list types of Interface Type fields. There should be only one field that has the Index type. Multiple Index fields will cause parts of JFFNMS to fail, such as the verify interface number( section 13.5.7 ) backend. Interface Type Fields are displayed by clicking on the View link for the desired Interface Type row. A new table appears below the Interface Type table that shows all the selected fields. Each row has an Edit and Del link as well as attributes of the fields Description A simple description of the field. For fields of Description type this string is displayed in the Interface icon on the maps. Internal Name This string is used by things such as the poller name and interface fields. Position The position orders fields of the same type. Interface Type The Interface Type this field is applied to. Field Type See table 10.1

10.2. POLLER GROUPS

51

Show This option will decide if the field is not shown anyway, shown in discovery and interface editing tables or just the table. Overwritable With this field checked, Administrators are able to overwrite the value Tracked Fields with this value checked will cause alarms if the autodiscovery detects a change. Default Value For most field types this is the value of the field if the autodiscovery does not fill it in. For RRDTool DS types it is the definition of the data source. As previously mentioned, the Position attribute sets the orders of fields of the same type. For Description field types, it is the order of the fields that are displayed in the Interface maps. The top-most field having the lowest position number. RRDTool DS fields also use the Position attribute for ordering. Generally a Poller Group( section 10.2 ) will collect information. The definition of RRD files is out of the scope of this document, the general idea is the same for all RRD files not just ones defined in JFFNMS. Each RRDTool DS field has 4 attributes that define the Data Source. The first is the DS type which uses the standard definitions of an RRD. The type can be Counter, Gauge or Absolute. The Min field is for the minimal value. Any value under this minimum is not added to the RRD. Maximum value can be fixed or be derived from another field. If it is fixed then the value goes in the box immediately to the right of the Max: label. For maximums that come from another field, put the field name (Internal Name) in the Use box and check the “for Max:” checkbox.

10.2

Poller Groups

A Poller Group can be consider a device type where you want to collect the same information and status. It uses a single interface type which you then apply multiple pairs of pollers and backends. Generally for most poller groups, they follow a standard order of things that need doing. 1. Verify that the id of the interfaces is still correct using an interface specific poller and the verify interface number backend. 2. Check the status of the interface and use the alarm backend to signal if there is a problem (or there was and now the interface is OK). 3. Collect some statistics about the interface, such as byte counts and store in temporal buffer. 4. Move information out of temporal buffer into a RRD file.

10.3

Poller Items

For JFFNMS to determine the state of some host’s interface, it needs to measure something using some method. The poller table determines what command is used and what parameters as passed to that command. The commands are found in ¡JFFNMS¿/engine/pollers directory. The pollers are called from the main program engine/poller.php The table has the following fields: • Description - Text description of the poller. • Name - This must match an RRD DS column that has been defined in the RRDTool Structure DEF column in the Interface Types( section 10.1 ) table if you want this data to be stored on a RRDTool file. • Poller Command - What program in engine/pollers will be used, “.inc.php” is appended to the name. See below to see what commands are already defined. • Parameters - What parameters are passed to the command. This column is dependent on what command you use. Predefined variables are explained further on.

52

CHAPTER 10. POLLING CONFIGURATION The following commands are defined: • snmp counter - Measure the value of some SNMP variable. Parameter is the OID to measure. • snmp interface status - Returns the value of the IfOperStatus SNMP table of the interface. • verify bgp peer number - Checks that the BGP peer for internal Indexes change, this is to keep track of the BGP Peer when you add a new BGP Peer to the router • bgp peer status - Checks the status of your BGP peers, a status of down or active returns down. Requires BGP peer IP address. • tcp - Makes a TCP connection to the Host IP Address, using the port specified as the INum of the interface.

The following predefined values are available on this table. They are written in the Parameter column with angles like ¡fieldname¿ • host ip - IP address of the Host. Defined in hosts( section 6.2 ) Table. • interface - Interface Name, comes from the Interface Name column in the Interfaces Table. • interfacenumber - Interface Number, comes from INum column in Interfaces Table. • address - IP Address of the interface, comes from the Address column in the Interfaces Table. • peer - Peer IP address, comes from the Peer column in the Interfaces Table. • description - Interface Description, comes from the Description column in the Interfaces Table. The job of the poller is to get some value or string which is then passed onto the specified backend. It is there the backend that determines what value the poller function should return. See the next section about backends for a description of the types and their expected values.

10.4

Backend Items

Backends are programs that are found in engine/backends. Like the pollers, the backends are called by engine/poller.php. The main difference between a poller and a backend is the poller collects more data while a backend works on existing or received from the poller data. There are 4 supplied backend types that come with JFFNMS: alarm, verify interface number, rrd and buffer. The alarm backend is used for sending alarms to interfaces. Unless you want the backend to do absolutely nothing, it must have a parameter that is the ID event type to raise or clear. The type is found in the Event Types( section 8.2 ) table and is the ID column. The relevant poller will supply the backend with the string “up” or “down” which generates the relevant event for that interface, using the event type supplied. The verify interface number has no parameters. The poller gives it the current interface id for the specific interface. The backend then checks this id with the existing one in the database and updates the database with the current one if they are different. The RRD backend is used to update and create the RRD files. Unlike any of the other backends, this one does not use a corresponding poller (use No Poller). It retrieves the information from the temporal buffer and then saves the information in the RRD files. The buffer backend takes the information from its poller and stores it in a buffer. This buffer is used by subsequent RRD backends. It takes no parameters.

10.5

Graph Types

JFFNMS can produce graphs for anything the pollers collect and put into an RRD file. The Graph Types table is used to tell JFFNMS what graphs are available for what Interface Types. Assuming they use the same variables, different Interface Types can use the same graph backend. For each graph type, you can have two graphs on the same page. Not every graph type needs two graphs.

10.6. AUTODISCOVERY POLICIES

53

Table 10.2: Autodiscovery Policies Options Option

Desctiption

Default Poller

New interfaces use the default poller as defined by the interface’s Interface Type( section 10.1 ). If this option is not used, then No polling will be performed on new interfaces. Permit Add Allow the Autodiscovery process to add new interfaces if an interface is found but not in the database. Permit Delete Allow the Autodiscovery process to delete existing interfaces when they are not found in host. Alert on Delete Create an event when deleting an interface. Permit Modify Allow the Autodiscovery process to change interface attributes. Permit Disable Allow the Autodiscovery process to disable an interface. Skip Loopback Do not discover loopback interfaces (Interface name begins with “loopback”, “lo” or “null” or IP address is 127.0.0.1) . Check State Only interfaces that are operationally up will be discovered. Check Valid AddressInterfaces need a valid IP address before they are discovered.

Description The name of the graph. Appears in the drop-down box on the Performance pages. Interface Type The Interface Type the graph is for. Allow Aggregation Does this graph have permit an aggregation graph, that is a single graph that can show multiple lines of data of the same type. For more information see Aggregate Graphs. Graph 1 This is used to find the name of the file the graph definition is located. The filename will be JF F N M S/engine/graphs/ < name > .inc.php Width 1 Width of the first graph Height 1 Height of the first graph Graph 2 Filename of second graph Width 2 Width of second graph Height 2 Height of second graph

10.5.1

Aggregate Graphs

An aggregate graph is one that is able to show simultaneously data from multiple graphs of the same Interface Type. The filename for the aggregate is like the standard graph, except it has aggregate in it. For example if the graph filename is traffic.inc.php, the aggregate filename will be traffic aggregate.inc.php The function arguments are similar to that of the standard graph function except that the interfaces are in an array with an index of the interface ID. The array contains the standard Interface Parameters (see table 13.7 page 73 ).

10.6

Autodiscovery Policies

The Autodiscovery Policies are used by JFFNMS to change what it does for the autodiscovery polling. Each policy can have enabled any of the options from list below. The Policies are found in the Administration menu item Polling & Discovery ⇒Autodiscovery Policy. It is unlikely that you will need to change anything on this table but JFFNMS does allow you to edit or add new policies. The options are hard coded deep inside the JFFNMS discovery code and would be difficult to change. The standard set of Autodiscovery Policies should cover most situations and have the following features.

54

CHAPTER 10. POLLING CONFIGURATION

Table 10.3: Autodiscovery Policies Policy Name

Description

Standard (for switches)As can be assumed from the name, use this for switches. They can be added or deleted but must be operationally up. This is the only policy that doesn’t need a valid address for the interface. Standard Most network devices should use this policy. Interfaces are added and alarmed if deleted. Just Inform This policy will only alert on changes to the list of interfaces and does not make and changes to the database. Automagic Allows the process to do everything it needs to. Administrative This policy is like Standard except it doesn’t use the default poller and it can delete interfaces.

Chapter 11

Triggers and Actions While they are two separate things, triggers and actions work together. There is no point in only having one without the other. A trigger and action are like a sentence such as: IF trigger THEN action For example, you may light to send an email if an interface for a particular host goes down, the sentence would read: IF an Interface on Host ’Claymore’ changes then Send an Email Triggers are part of the system for JFFNMS to send out notifications when certain specific things happen. The most common example of a trigger is JFFNMS sending an email when either an event or alarm occurs. To setup a trigger, there are a few steps involved, this chapter will explain what those steps are and how to get triggers working. In brief, they steps are: 1. Create the trigger, that is a container to hold the rules 2. Define the trigger by adding trigger rules 3. Optionally, create a new action or decide which existing action you will use 4. Add the trigger to a user

11.1

Creating Triggers

You can see the current list of triggers with the Administration menu item Internal Configuration ⇒Triggers & Filters ⇒Triggers Configuration. This will bring up a table of triggers. Each trigger has a small description and a Type field which determines if the trigger works off an event or an alarm. Generally it is better to work off an alarm because they have a more definite state but it is really dependent on what the trigger is for.

11.2

Trigger Rules

In each row of the trigger table is a View Rules link. Clicking on this link brings up the rules that are used to evaluate the trigger. Table 11.1 explains what each column in the trigger rules table does. You will be unable to edit all the fields in one go as some are dependent on others. You cannot edit the Value column until the Field column is selected. You cannot edit the Action Parameters until the action is selected. Currently the only action is Send Email. Actions are defined in the Actions( section 11.3 ) table but also need to be created in the code. The Boolean at the end can be used so that the trigger will fire only if all rules match or if one of them does. The AND and OR can be used in combination with care. For example to fire the trigger if condition A and either B or C are true you would set the rules up like: 55

56

CHAPTER 11. TRIGGERS AND ACTIONS

Table 11.1: Trigger Rules definition Column

Description

Position

Rules can be ordered within a trigger. They are evaluated from the lowest position to the highest. Field What value of the alarm or event is used for the matching? For example selecting Host here will mean this row uses a set of hosts specified in the Value table to determine if it matches. Operator A comparison operator for comparing the value of the field with the value given in the Value column. Value The value that the Field is compared to. It may be a select box or a text field depending if JFFNMS can determine the valid values. Action Which Action to execute if this line matches. Set to No Action if there are subsequent rules. Parameters- Fill in the parameters Asked by this Action. These are defined in the action table by the User Parameters field. if Match This field determines if any more rules are evaluated if this rule matches. Boolean The last column has no title, but is used to determine how the next row compares to the last.

Table 11.2: Action Definition Table Column

Description

Description Command

A small description of what this action does. Command name, used to determine what action plugin is included, see notes above for details. Internal ParametersA comma-separated list of key:value pairs. These values will be available to the action plugin. See Creating a New Action( section 13.11 ) for details on how these values are used. User Parameters Another commas-separated list of key:label pairs. They are used in the Trigger Rules( section 11.2 ) table for the Parameters column.

Position 10 20 30

Condition A B C

AND OR

However to fire the trigger if A happens or if B and C happen, then the table would be reconstructed like this Position 10 20 30

11.3

Condition B C A

AND OR

Actions

Actions need to be defined in this table as well as created in the code. The action Description is referenced by the Trigger Rules( section 11.2 ) and gives a name of what the action is trying to do. The command is used by the code to load the engine/actions/commandname.inc.php file and call the action commandname() function. Actions are define in the table found in the Administration menu Internal Configuration ⇒Triggers & Filters ⇒Actions Definition. The table described in Table 11.2 appears.

11.4. PUTTING IT ALL TOGETHER

57

The Parameters can have tokens in them, so JFFNMS can fill in some specific details about the alarm or event that trigged the action. The tokens are written as “¡object-parameter¿. For example, to print the hostname of an interface, you would put “¡interface-host name¿”. The objects are interface, action and event. The parameters are found Common Parameters( section 13.14 ) section. To see how the various Parameter fields work together, consider the email action. It requires input from the user for each trigger that uses that action. The input is the subject and a comment. The User Parameters field is “subject:Subject,comment:Comment” meaning there will be two fields with the labels Subject and Comment and the email action will receive the value of those fields in the array value parameters[‘subject’] and parameters[’comment’].

11.4

Putting it all together

OK, so now we have talked about Triggers and Actions, it is time to use an example to get a trigger working. The important points to consider are the start and end of the whole process, in other words what do you want to trigger off and what action. In this example, assume you want to send an email any time a host called Claymore has an interface changing state. The following shows what is required. 1. Bring up the Triggers table by selecting Administration menu item Internal Configuration ⇒Triggers & Filters ⇒Triggers Configuration. 2. Click Add at the top of the Triggers table, a new Trigger row opens in edit mode. 3. Enter “Claymore Interfaces” as the trigger Description 4. Make sure “Match Alarms“ is selected for the Type 5. Click the Save button. 6. The Triggers table show be redisplayed with the new Trigger you have just created, now it is time to enter in the rules. 7. Click ”View Rules“ on the line showing the Trigger, a Triggers Rules table appears below the Triggers table. 8. Click Add from the Triggers Rules table, a new Trigger Rule row opens in edit mode. 9. Change the Field column (the second one) to Host and then click Save. You have to edit the row twice to get the list of Hosts. Leave everything else the same. 10. Click edit for the Trigger row, the row now changes to edit mode. 11. Choose the Host called Claymore from the list in the Value column. 12. Check that Action column is ”No Action“ and the last column is ”AND“. If so click the Save button. 13. Click the Add from the Triggers Rules table, yet another new Trigger Rule opens in edit mode. 14. Change the Position column to 20. 15. Change the Field column to Type, this means we will be matching on Alarm type. 16. Change Action to ”Send Email“, the parameters are not shown at the moment. 17. Change ”if Match“ column to ”Stop this Trigger“. 18. Click on the Save button, the Value and Parameters columns change because the Field and Action were changed respectively. 19. Click on the Edit button for the row just created (Position 20). 20. Select ”Interface Protocol“ from the Value column. 21. Put in some values in the Parameters column, perhaps giving some information about this host.

58

CHAPTER 11. TRIGGERS AND ACTIONS

Figure 11.1: Trigger tables with example trigger created 22. Click Save You now have a new Trigger which will send email if an interface on the host Claymore changes state. Your screen should look something like Figure 11.1. Next you have to tell JFFNMS who will be getting these emails and what their email address is. First of all, make sure that the user has an email address set. This is done in the User Profiles Table( section 7.2.3 ) by entering in the user’s email address in the eMail option. Next you need to apply the trigger to the user, this is explained in the User Triggers( section 7.2.4 ) section, more specifically the part about adding a new trigger for a user. Your newly created trigger should be in the list of available triggers when you click on the Add button.

11.5

Deactivating Triggers

There are times when you will need to undertake maintenance on some of your networking equipment. At those times you probably do not want to send alarms for an interface dropping because you have unplugged it, for example. This is especially so if the work is at night and the support engineer is asleep! Deactivating Triggers is done by removing the active flag for that trigger, applied to a specific user. There is currently no simple way of disabling a Trigger for all users, or even all Triggers for a specific user. The User Trigger table is accessed via the Administration menu item User and Customers ⇒Triggers Users. The far-right column is labelled Active. If there is a tick-box in that column for that particular User/Trigger row, then that User will get the notification if that Trigger fires. Deactivating a Trigger for an User is a reasonably simple excercise: 1. Bring up the User Trigger table by selecting Administration menu item User and Customers ⇒Triggers Users. 2. Find the relevant row for the User and Trigger Pair. 3. Click on the Edit button, the row re-appears in edit mode. 4. Un-check the Active column so the tick disappears. 5. Click the Save button, the row reappears in view mode.

11.6. TRIGGER LOGS

59

To re-activate the Trigger, follow the above process only ensure that the Active column has a tick in it.

11.6

Trigger Logs

If you have logging/debugging enabled, then the trigger log will display JFFNMS evaluating the trigger for each alarm or event. For each alarm raised, each trigger is evaluated. Triggers that match the alarm then cause actions to happen. A sample log is shown below. 19:10:22 19:10:22 Stop 19:10:22 19:10:22

================================================== alarm 16: T 3 - P 10 - R 3 If ’interface_host(2) = 3’ (0)

alarm 16: --------------------------------------------------------------------alarm 16: T 4 - P 10 - R 5 If ’any’ (1) Then email (1)

Each line shows, from left to right: localtime, alarm id, Trigger id, Position of rule in trigger group, id of trigger rule, then the evaluation. Alarm 16 has been raised (this is just the alarms internal ID). There are two triggers defined, one with Trigger ID 3 and one with 4. Trigger 3 has its first rule at position 10, which is rule number 3. This rule tries to match the host. It wants to match to host with id 2, but the alarm is for host 3, therefore it is not a match (the “(0)”). In addition the rule says to stop evaulating for this trigger. Trigger 4 is rather simple, if there is something in the alarm, it is a match. Naturally this is only a test alarm. As there is a match, the action is Then performed, which was the email action. The email action returned 1, which means it was successfull.

11.7

Tools

The tools are links that can be found on the expanded view of the events table. Clicking on the flashing light or tick icon will expand the relevant event. A tool consists of a description, an URL, an optional image for an icon and a filter. The URL can have the usual macros, e.g. ¡interface id¿. The filter is the same set as the express filters used on the events viewer itself.

60

CHAPTER 11. TRIGGERS AND ACTIONS

Chapter 12

Configuring Event Filters 12.1

Filters

Filters are used to filter in or out different events based upon some fields in the event types. By default all events that have had their “Show in Event Viewer?” option ticked are shown which is set in the Event Types( section 8.2 ) table. They are also used for Tools to determine which tools are visible in the expanded view. Unlike SLAs, Filters do not use RPN logic but are written in a standard fashion. To access the filters, select the Administration menu item Triggers & Filters ⇒Event Filters. You will get the table of currently defined filters. Filters are only a container that has a Description and is used to hold Filter Conditions( section 12.2 ). The Description column is what appears in the Event Viewer. To see the the Filter Conditions, click on the View button for the required Filter.

12.2

Filter Conditions

As previously mentioned, a Filter is made up of one or more Filter Conditions. The filter conditions can be joined together with AND or OR operators and the conditions can be based upon different attributes of an event. Table 12.1 describes what each column in the Filter Conditions table does. To join multiple rows, use the AND and OR rows. These rows have the relevant Field set to AND and OR. So to make a filter that will show all events of type Configuration or Logins you would make a table like Position 1 2 3

Field Event Type OR Event Type

Operator = =

Value Configuration Logins

Table 12.1: Filter Conditions Field

Description

Position Each condition within a filter is ordered. The conditions are evaluated from the lowest position value to the highest. Field The field that will be evaluated in the event against the value column. OperatorHow the Field and Value columns are compared. The operators are: equal; not equal; less than; more than. Value The value that is compared to the Field, using the Operator.

61

62

12.3

CHAPTER 12. CONFIGURING EVENT FILTERS

Filter Fields

The Filter Fields table is used to create new Fields you can compare things to in the Filter Conditions Table( section 12.2 ). For most uses of JFFNMS, this table is left alone as it depends on the internal structure of the databases.

Chapter 13

Expanding JFFNMS While JFFNMS comes with many interface types, the sad fact is that the world is filled with a large variety of things you can monitor, graph and alarm. This chapter describes how you can expand JFFNMS so it is able to monitor something new. If you are happy with your creation and think others could benefit, it would be a nice idea to contribute to JFFNMS by sending in your patches and SQL statements.

13.1

Design Prework

Before starting to create your new Interface Type, you should do some planning. Quite often even experienced developers have had to re-work the descriptions so some care and time now is a good idea. The next sections describe points you should consider before starting to write any code.

13.1.1

Discovery of new interfaces

What is the least intrusive way of finding these sorts of interfaces? A large majority of interfaces are found by doing an SNMP query to a specific OID. If the OID returns a value, then there could be an interface. Be careful with assumptions. You may discover half-configured or useless interfaces. The Cisco SAA interface type scans a SNMP table for its interfaces, however only some types of interfaces return useful information. Possibly polling another SNMP table to check for type and suppressing known “bad” types would give a cleaner result. Design the discovery plug-in to determine with the least amount of prodding and time that an interface of its type doesn’t exist on the Host. Is there one SNMP query, for instance, that if it returns nothing means there are no interfaces of this type? If it returns nothing then stop right there; there is no need to continue.

13.1.2

Manual add of Interfaces

Does it make sense for this type of interface to be manually added by the Administrator. The TCP port interface type does some probing for know TCP ports but does not do an exhaustive check of all 65,535 TCP ports so it does make sense for this interface to have a manual add feature. Try to avoid using this feature as a work-around for a badly written discovery plugin. If it is not too intrusive the plugin should find all valid interfaces of this type.

13.1.3

Find the index

Examine the data that you can get about the interface. Look for something that could be used as a short-cut to say “this interface”. If your interfaces are part of a well-formed SNMP table, there is most likely to be an index column (probably the first OID in the table). The index is used as a key to access information about the interface. It should rarely change and if it does, there should be a poller that detects and optionally corrects for this. The ifDescr to ifIndex 63

64

CHAPTER 13. EXPANDING JFFNMS

poller which changes the index field to match a fixed description field is an example of this corrective poller. The TCP port number for the TCP port interfaces is the best sort of index field because the index completely describes the interface; you cannot have TCP port 53 moving to TCP port 80. Usually there can be some fallback index which is the order that the discovery plugin detects them. This is definitely not recommended, especially for interfaces that can be dynamically added and removed, but sometimes it is the only index you have. All interface types require an index.

13.1.4

Interface Name

The interface name goes by the confusing variable of interfaces.interface. This field is used to match events to interfaces by the event consolidator to create alarms. If you are receiving syslog messages and what to create alarms for interfaces from those messages then part of the syslog message needs to be extracted into the event “interface” field and that field should match the interfaces “Interface Name”.

13.1.5

Description Fields

Description fields are interface values that better help describe the interface. They are used to give the operators a prompting about what this interface is used for. A good description will be short and to the point, a brief look at it will give most of the information you require. While a PID (process ID) makes a good index, it doesn’t seem to convey any real meaning. Knowing that PID 1234 is still alive is good, but knowing that the Apache process is running is even better. Most interface types use one or more descriptions in this way. They can either be directly obtained from the device, such as Interface descriptions for physical interfaces or found elsewhere, such as the protocol name for TCP ports which is found locally and changes TCP port 25 into SMTP. An important point about description fields is they must not confuse or misinterpret their information. A serious problem could occur where for some reason important interface ‘A’ gets unimportant interface ‘B’ description. To make it clearer, an Interface Name( section 13.1.4 ) is usually the name given to the interface or port by the device itself, while the name is something the Administrator does. For a physical interface on a Cisco router, the Name “FastEthernet0/1” is defined by the router’s software, while the description “webserver 1” comes from the configuration set by the administrator.

13.1.6

Values to collect

Are there any values or metrics that you would like to collect about this interface type? Traffic or load counters, response times, temperature and errors are things that are typically good to be collecting and displaying. JFFNMS doesn’t really have any limit on what you collect. If it produces a number that makes some sort of sense then it can be graphed. Of course only you can decide if it is worth-while collecting that information. Values should only be collected if they are to be graphed and/or examined with the RRD analyzer for SLA alarms. If neither of these things will happen then there is no point collecting the information. Most of the time you will need to know what units the values are being collected in and the sort of value. The value might be a counter, which means it is counting things and increments each time. Alternatively it may be a gauge which is like a level that can go up or down, such as a CPU load.

13.1.7

Types of RRD

If you are collecting some data (see previous subsection) you need to know what sort of data it is. There are three types of RRD files: the counter, gauge and absolute. A counter is the most common type of RRDfile. This is a continuously increasing value and it means you want to measure a rate of something. Other than wrap-around, where the number is so big it goes back to zero again, every time you measure this value it is either the same or bigger than last time. The byte and packet counters of an interface are the most common example of this sort of file. Next most likely is the gauge. This is a level and is not expressed as a rate. The load on the system, free memory or percent battery charge are examples of a gauge. Finally there is the absolute type. This is also a counter like the first one but the counter is reset when read. It means an absolute type is something of a mixture of the other two.

13.2. CREATING A NEW INTERFACE TYPE

65

Quite often it is easy to get these wrong. The absolute one is, in the authors experience, quite rare so unless it is something very strange you are measuring then it won’t be this one. If you are using a counter type and the values are going negative and they shouldn’t be (eg packets per second) then this item is a gauge. Alternatively if your gauge item is steadily increasing and you think it should be approximately level then you probably should be using a counter.

13.1.8

Interface States

There is generally two types of states for all interfaces: administrative or operational. However there is no real limit on this, it just means if you have other types of states then you need to work out how to interpret or display this information. Administrative state is due to actions by an Administrator. These states usually only change when someone makes them change. An interface that is shutdown, deconfigured, enabled or taken out of service would have an administrative state change. Administrative state really has only two states: up or down. Half-configured or enabled interfaces are either ready for some service (up) or their not (down). Administratively down interfaces are marked on the map as gray with a colored square denoting the operational state. Generally administrative state trumps operational; you cannot have an administratively down but operationally up interface. Operational state is used to determine if an interface is working and can be used. Ethernet interfaces that are enabled, but not plugged in are administratively up but operationally down. If the interface is not usable because something is wrong and it not due to its configuration, then its operational state is down. For a new device, you will need to determine how, if possible, to detect the administrative and operational state of an interface. There may also be syslog messages or SNMP traps that will change the state of the interface. Administrative state pollers generally use a backend based on the DB (database) backend plugin to directly change the show field. There is generally no event or alarm created for this change in state. The pollers return a number and the backend equates the number to changing the field or not. Operational state pollers in contrast return a string, such as “up” or “down”. To work correctly the string must be one from the Description field of the Alarm States( section 8.4 ). The backend is based on the alarm plugin.

13.1.9

Events for this Interface

Events are something that happens to the Interface. If you have an operational state poller( section 13.1.8 ) then its corresponding backend should be based on an alarm plugin and reference an Event. The event consolidator will then pick up this event and use it to change the alarm (color in map) state of the interface. Besides the state poller events, you may also like to consider what syslog messages or SNMP traps are worthy of their own event for this interface. Information that you think would be useful to someone administering the device is a good candidate for an Event. Events come in two types, those that create alarms and those that don’t. The alarming events need to have something in them, for instance the syslog message, that can be put into the ‘interface’ field by the consolidator and then can be used to marry up to the interface name by the event consolidator.

13.2

Creating a New Interface Type

13.3

Generic Discovery Plugins

Of the various plugins, the most difficult to make generic is the Discovery ones. This is because there usually has to be specific careful checks to determine if an interface of a particular type. However there are a few Discovery Plugins that could be reused. For interface types of a certain family there is benefits in using the same discovery plugin.

66

13.3.1

CHAPTER 13. EXPANDING JFFNMS

host information discovery plugin

This discovery plugin is really a SNMP system OID checking plugin. It is given a comma-separated list of matches and compares each match with the system OID. If there is a match it returns a single interface called a CPU. This discovery plugin is useful for discovering a type or family of the same systems. For example, you may have a new brand of Ethernet switch. You could discover and match against this model’s system OID and if it matches create a CPU interface. By JFFNMS convention, a “CPU interface” is one describing the host itself. It usually has graphs showing CPU load, memory usage and the TCP MIB pollers( section 13.4.5 ). Generally the CPU load and memory statistics are host type-specific while the TCP MIB is a generic SNMP MIB.

13.4

Generic Poller Plugins

There are a quite a few Poller Items that could be reused. It is a good idea to check through this list first to see if you can avoid re-writing a poller plugin and use the existing ones instead. There is a difference between a Poller Plugin and a Poller Item. A plugin is a code fragment that is placed in the engine/pollers directory. Poller Items are instances of pollers that use the plugins and give them parameters. The name given put in the “Poller Command (file)” column of the Poller Item table tells JFFNMS that this Poller Item uses the (file) as its poller backend.

13.4.1

snmp counter poller plugin

The snmp counter poller plugin is one of the most used plugins in JFFNMS. It takes only one parameter, which is the SNMP OID the poller should get. The item returns a single value and puts it into the field that matches the Name column. If you need to get a number out of a device via SNMP, this is probably the plugin to use. Generally you would couple this with the Temporal Buffer backend.

13.4.2

snmp status poller plugin

This plugin is like the snmp counter one, except that it is used for obtaining status via SNMP. It takes two parameters that are separated by a comma. The first one is the SNMP OID to poll. The second is a pipe-separated list of match and value pairs. The pairs are written in match=value format. For example, to poll OID 123 and return ‘yes’ if the SNMP reply is 1 and ‘no’ if the SNMP reply is 2 the parameter would be “123,1=yes—2=no”. If SNMP returns any other number, the plugin returns ‘down’.

13.4.3

db poller plugin

The db name is a mis-nomer because this plugin extracts values out of fields that appear in the poller array (see table 13.1 , some of which are derived from database values. The plugin can take one or two parameters. The first parameter is the key so the poller knows which field you want. The second optional parameter, separated by a comma, is “to bytes”. If this parameter is present the value obtained from the table is divided by 8.

13.4.4

buffer poller plugin

The buffer plugin can yank values out of the buffer that have been previously put there by Poller Items that have used the Buffer or Multibuffer backend. This is especially useful when you have a single poller that obtains multiple values in one hit and now you want to extract a subset of those values. It is important that the Poller Item that puts the values in the buffer happens earlier (Has a lower sequence number) in the poller group. The plugin takes a comma separated list of buffer keys. These keys would refer to previous buffer names. If there are multiple keys then the values are returned separated by the pipe symbol |. Do not confuse this poller plugin with the Buffer backend plugin. The poller gets values out of the buffer while the backend puts values into the buffer.

13.5. GENERIC BACKEND PLUGINS

13.4.5

67

tcp mib pollers

The TCP MIB is a generic MIB, or one that is not usually dependent on the particular SNMP agent. Most devices that have SNMP will have this MIB. It’s purpose is to measure the number of current established TCP connections, pending passive (incoming) TCP connections and pending active (outgoing) TCP connections. The set of 3 TCP MIB pollers use the SNMP counter( section 13.4.1 ) to obtain their information. They are usually used in poller groups for interfaces found using the Host Information( section 13.3.1 ) discovery plugin.

13.5

Generic Backend plugins

The comment that you should look at Poller plugins before writing your own is even more important when looking at the backends. Almost every Interface Type uses the same set of backend pollers. The difference is quite clear with JFFNMS shipping approximately 40 poller plugins but only 7 backend plugins. In fact the backend plugins are so generic you can often use the existing backend definitions. This is mainly due to the fact that a lot of backend obtain their information from the Poller Item name and therefore the same backend can examine or update different fields.

13.5.1

Alarm backend plugin

The Alarm backend plugin doesn’t directly create an alarm. What it does do is create an event and then that event may create an alarm when the event consolidator runs next. The plugin requires the Event ID of the event to generate and, optionally, two other parameters. The first optional parameter is what level event to generate if the input from the poller is empty. If this parameter is set to ‘nothing’ then no event is generated. The second optional parameter is number of seconds to wait before adding a event in case the interface is flapping. The default wait time is 60 seconds. The parameters are separated by commas and are in the order of event id, empty event level, wait time. If you want to default empty event level (which is down) then put two commas between the event id and wait time. The backend expects from the poller an alarm level or the alarm level and optional description. A pipe character separates the level from the description. The level must match one of the Descriptions in the Alarm States( section 8.4 ) table. If the level is not found in that table, the alarm backend will not work and returns “Invalid Result”. The optional description from the poller is fed into the alarm under the Other Info field. It is then up to the event definition to decide how to present that information.

13.5.2

Buffer backend plugin

This backend takes the value from the poller and stores it into a buffer for later use by other pollers or backends in the same Poller Group. It is important to remember that this buffer is destroyed at the end of the Poller Group run. There are two ways that the buffer is named. If the Backend table has a parameter, then this parameter is the buffer name. A far more common use is that the buffer name comes from the Name column of the Poller Item definition.

13.5.3

db backend plugin

There is only one table that can be changed with the db backend and that is the interface table. The backend takes three parameters. The first being the database table field to be changed. Next is a list of key value pair that match the poller result. Finally there is a skip value. The list of matches are separated by the pipe character and are in the format of match=value. This means if the poller output is match, then the database field will be changed to value. If there are no matches defined, then the database field is changed to the poller output value. The skip value is used to suppress changes to the database. If the old value of the database field is equal to the skip value, then no changes are made.

68

CHAPTER 13. EXPANDING JFFNMS

To illustrate how it all works we will use the Admin Status Change View backend which is used by Physical interfaces. It has the Parameters of “show rootmap ,down=2—up=1,0”. This will mean that if the field show rootmap is not already 0 and the poller output is “down” then show rootmap is changed to 2. Similarly if the value is not already 0 but the poller output is “up” then show rootmap is set to 1. Finally if the show rootmap value is 0 or the poller output is not “down” or “up” then the field is unchanged. The backend returns 1 if there is a database change and -1 if there was none.

13.5.4

multibuffer backend plugin

The multibuffer is very similar to the standard buffer plugin except that it can take multiple values. The names of the values comes from the Name column of the Poller and are separated by commas. The backend expects its values from the poller separated by the pipe character. The order of the parameters and poller output is important as the first output variable is put into the first named buffer and so on. The multibuffer will only set buffers that previously have no value. On other words if a previous poller/backend pair set the buffer of the same name then this backend will not update or modify that buffer. The backend returns the size of the buffer.

13.5.5

No backend plugin

This is by far the simplest backend plugin. It does absolutely nothing except return 0. This backend is generally used when the poller is doing direct operations on certain tables. However the use of this backend generally means you have combined your poller and backend into one.

13.5.6

rrd backend plugin

The RRD backend is used to insert values into the RRD files. It can either have a parameter of “∗” or no parameter. With the parameter of ∗ the backend takes a look at all interface buffers and tries to match the buffers to RRD files by using the buffer name and the RRD interface field Internal Name. This is why it is important your poller items name match the RRD Interface Type value name. No parameter means that the backend will attempt to update a RRD file by finding an interface field that has the Internal Name that matches the pollers name. The backend returns a string showing all DS that are changed in the format ds name: value.

13.5.7

verify interface number backend plugin

The name for this plugin comes from its original purpose which was to check that the interface number was still the same. This backend now has far more flexibility and is used to check that the interface index is still correct. The poller that feeds this backend needs to independently determine the “correct” value for this interface. For example with physical interfaces the poller could examine the description (ifDescr) table to find the correct index. The point is that the poller has to be sure that the index value is correct and do this without using the index value (because the value will always agree with itself. The backend takes the correct index value from the poller, compares this value to the current value and then changes it if they are different. For this backend to work, there can be one, and only one, interface type field that is of Index type, see Interface Type Fields( section 10.1.1 ).

13.6

Writing a new Poller

If you need to get information from some other source that is not available using the standard pollers, then you will need to write your own. The program has to be written in PHP and is put into engine/poller/¡command name¿.php Inside the file, a single function called poller ¡command name¿ is created. This function is called by the main poller and has a single argument which is an array. The array contains many fields, the first

13.7. WRITING A GRAPH PLUGIN

69

Table 13.1: Options given to Poller Item functions Field

Description

host id The Host ID from the Hosts Table host ip IP Address of the host. SNMP Read/Write Community from Host. rw community ro community SNMP Read-Only Community from host. Internal sequence number in database for this interface. interface id interface number IfNum column. interface descriptionDescription column. address IP Address of the interface. peer Peer IP address (for BGP interfaces). Incoming (downloading) bandwidth. bandwidth in bandwidth out Outgoing (uploading) bandwidth. Name of poller from Pollers Table. poller name poller parameters Parameters column from Pollers table. random Random number between 10 and 99. Start time of poller, from time(). time start noerrors Has the No Poll Errors been selected. (deprecated)

lot are taken directly from the database and describe the interface being checked. They get their values mainly from the Interfaces table. The parameter can be described in any way, because is supplied to the poller in the poller parameters variable of the options array and the poller has to parse it. The poller function returns a value which is then directly sent to the backend which also gets the same parameters described above. Useful values to return depend on the backend. For the alarm backend ”up” and ”down” are generally used. For most others a number of the measurement is used.

13.7

Writing a Graph Plugin

JFFNMS can display information collected in its RRD files. The purpose of the graph plugin is to describe to the RRD graphing engine the graph. Items such as what sort of graph, which RRD files to use and graph attributes such as titles, units and colors as defined by the plugin. The plugin has a file in engine/graphs/graphname .inc.php and has a single function graph graphname . This function has one parameter which is an array. The array contains standard interface parameters for the specific interface that is going to be graphed, see Table Interface Parameters( section 13.7 ). rrdgraph parameters can be broken up into three types. Data definitions, graph descriptions and other parameters. The data definitions and graph descriptions are required for all graphs, the other parameters are useful, but not essential. To make sense of the graph, plugins you really need to read the rrdgraph manual pages. The function returns an array of two items. The first item is the other parameters and the second are the combined data definitions and graph descriptions. Data definitions are usually found by using the rrdtool get def() function. The first parameter of the function is the same parameter of the plugin function. The second parameter is a an array. The array is either a standard array that lists which interface variables (that are RRDTool type) you will use. In this format the rrdgraph binary will use the same name as the RRDTool interface value. The second way that the array is used is with an associative array, in the form of “alias” => ”name”. The “alias” is the name that will be used in the other RRD parameters. The “name” is used to link to the interface values. The graph description is used to create the virtual data definitions (lines that start with CDEF) and the graphs themselves, using the usual rrdgraph parameters such as LINE1, LINE2, AREA and STACK. Axillary items such as GPRINT, HRULE and COMMENT are also defined at this point. The other parameters are for any of the command line arguments that you can send to rrdgraph. Most common parameter used is for the vertical label, which is defined by “–vertical-label=’Bits per

70

CHAPTER 13. EXPANDING JFFNMS

Second’ ”.

13.8

Writing a Discovery Plugin

The discovery plugins are found in engine/discovery directory. To make a new discovery plugin, create a file in there called pluginname.inc.php The file contains one public function, called discovery pluginname which has 4 parameters: The host IP address, the SNMP read-only community, the hostid and the param field from the discovery table. The function returns a single array of array. The outer array uses a index number that starts at 1. The inner array contains information for each interface found. The inner array has the following keys: interface Interface Name (ie. Serial0/1) admin Administrative status: up, down oper Operating status: up, down Starting in 0.7.1 you can return any field from the Interfaces table in the inner array, this allows better control (you can make your discovery script select an specific SLA or Poller) There should be at least one field that is an index and you should fill in something in the other 3 common fields. If you do not, then JFFNMS might act strangely. To debug the plugin, use the following steps: 1. Write the plugin, you can send debug to stdout for now 2. Create a new interface type with the discovery field using this plugin, also note the ID that this interface type gets 3. Find a host that has this type of interface and note the host ID 4. Run the Interface Discovery( section 18.1.3 ) component off the command-line, see the link for details.

13.9

Creating a simple SNMP-based Interface Type

Easily the most common request on the JFFNMS email list is someone has some SNMP-capable device along with the MIB and they want to monitor some value out of the MIB. In JFFNMS this would mean the device will become a host with the value being an interface. This section deals with how to create a simple interface type that will poll for one counter only. It is probably the most common type of interface you will need for starters at least.

13.10

Creating more complex Interface Types

You first need to do your prework, so review the section on the Prework required( section 13.1 ) first and make sure you have considered each point. Good interface design takes some time, but gives you a much better result in the end. It is easier to use an example to illustrate how to do this work then make it all theoretical. The example I’ll use is a “Real Server” from an Alteon Load Balancer. The basic idea is web browsers connect to a Virtual Server on the Load Balancer which then forwards the request to a Real Server (eg something running Apache). The benefit is that you can have many real servers for the one website.

13.10.1

Determining what OIDs to use

For the Alteons I grabbed the MIBS and ran the following commands: $ snmpwalk -v 1 -c public -m all -M.:/usr/share/snmp/mibs 10.1.2.3 private $¿$ snmpwalk1.txt $ snmpwalk -v 1 -c public -On 10.1.2.3 private $¿$ snmpwalk2.txt

13.10. CREATING MORE COMPLEX INTERFACE TYPES

71

Table 13.2: Return values of Discovery Plugin Field

Value

real serverIndex number from index row interface IP address of real server admin Administrative state of server Oper Operational state of server hostname Real Server name

Both commands run a snmpwalk on the device. The first loads all MIBs in the current directory (where the Alteon MIB files are) and the common MIB directory. The second command shows the same information but the OID is now numeric with the -On flag. The hard slog is to work out what OIDS are useful and what are not. Quite often SNMP items are in tables, and we find that the Real Servers are in two tables, so lets start documenting the information we will require. I have only used the shortened SNMP information here. Index: slbCurCfgRealServerIndex Description: slbCurCfgRealServerName IP address: slbCurCfgRealServerIpAddr Maximum Connections: slbCurCfgRealServerMaxConns Admin State: slbCurCfgRealServerState Oper State: slbRealServerInfoState Total Sessions: slbStatRServerTotalSessions Octets: slbStatRServerHCOctetsLow32 We have a nice set of OIDs to work with, coming from the configuration, information and stat tables of the real server. There is an explicit index field, but what should be the “Interface Name”? The name should mean “this interface”, a unique property that sets it apart from all the other interfaces of this type. A real server is defined by its number and IP address, however it would be nice to correlate the syslog messages into events and alarms. The messages look like WebOS <slb>: cannot contact real server 10.20.100.3 WebOS <slb>: real server 10.20.100.3 operational The syslog messages reference the server by its IP address, so this should be its name.

13.10.2

Creating an Auto-discovery Plugin

Refer to the previous chapter about how to create an auto-discovery plugin. The plugin has to check to see if the host returns values for the OID and if so show that interface in the discovery page. The easiest way to check if the device understands your OID is to query for it and see if it returns something. If it does, you add that interface into an array. For our real servers, we first query the Index row. If we receive nothing then return FALSE right away. A common mistake is to query all columns first then check to see if valid information is returned. The plugin then does some more SNMP walks to get the IP address hostname, Admin and Oper state. For each row in the Real Server index table, a row is created in an array which is a row itself. The fields and values are described in Table 13.2. Next you need to create a new interface type. This interface type will use the poller just created. See the section Creating New Interface Type for details. The interface, admin and Oper fields do not need to be defined as they are there for all interface types. The fields will also need two RRD types for collecting the Sessions and Octets. These fields will be RRD counters and will look like table 13.3

13.10.3

SNMP Poller - Counters

There is already a poller plugin for getting SNMP values. All you need to do is create enough Poller Items for each value you are tracking for the interface and then create a Poller Group that holds the Items.

72

CHAPTER 13. EXPANDING JFFNMS

Table 13.3: SNMP Example RRD fields Description

Internal NameType and Value

Sessions Octets

sessions octets

RRDTool DS Type: Counter RRDTool DS Type: Counter

Table 13.4: SNMP Example: Poller Items Description

Name (Match RRD Struct DS)

Poller Command (file)

Parameters

octets sessions

snmp counter snmp counter

.1.3.6.1.4 . . . 7.¡index¿ .1.3.6.1.4 . . . 8.¡index¿

Alteon RealServer Octets Alteon RealServer Sessions bottomrule

The poller command is snmp counter and the Parameters are the SNMP OID you want to collect. The Name comes from the Internal Name that you put in the Interface Type fields for your new interface for the RRD fields. The OID also needs the instance or the row of the table’s column, put “.¡index¿” at the end of the OID to get the right instance. Table 13.4 has the poller items, notice the Name column is the same as the Internal Name column in Table 13.3.

13.10.4

SNMP Poller - Status

The Admin poller uses the snmp counter to return a number that corresponds to the Administrative status. This is fed into a specific DB backend that changes the show interface field if the number matches the value for Administratively down. The Oper poller uses the snmp status poller. This poller takes an OID and a list of value = returnstring pairs. As it returns ’down’ on no match, only put the values for up in the poller parameters. This poller feeds the string ’up’ or ’down’ to a newly created alarm backend. The backend creates an Alteon Real Server specific alarm if the string fed to it is ’down’.

13.10.5

Poller Group

For the interface I always create the group in this order: • Admin Status poller to Admin Verify backend • Oper Status poller to Alarm Oper backend • SNMP Counter poller to Temporal Buffer backend • No Poller to RRD all DS backend The following table ?? displays the poller items

13.11

Creating a New Action

Actions are things that JFFNMS can do to respond to an Alarm or an Event. Typical actions are sending messages via email or pagers, but an action could be anything you want it to be, it is just a matter of writing the script and setting up the action in JFFNMS. Action scripts are found in the directory jffnms/engine/actions/ and have the filename action name.inc.php In this example, we will call our action “testaction” which means the file will be called jffnms/engine/actions/testaction.inc.php

13.12. CREATING NEW SLAS

73

Table 13.5: Position Poller 10 15 20 25 90

Backend Alteon RealServer Admin Alteon Admin Status View Alteon RealServer Oper Alarm Verify Operational Alteon RealServer SessionsTemporal Buffer Alteon RealServer Octets Temporal Buffer No Poller RRDTool AllDSs

The file must have one function called “action tt action name”, in our case it would be action testaction . The function takes one parameter which is an array. The array consists on the following items: • parameters - Another array of Internal and User Parameters

13.12

Creating New SLAs

SLA stands for a Service Level Agreement. JFFNMS can continuously monitor network interfaces and check you are within a SLA. The setting of a NMS SLA does not necessarily have to be linked with a contractual SLA with your customers; you may want to use slightly tougher settings within a NMS or just use some design goals. A SLA is for a certain interface type and consists of several SLA conditions, each of which is checked individually. The information comes from the RRDTool files, these files are updated by the poller.php every 5 minutes. The steps on creating a SLA are as follows: 1. Determine what individual conditions you want to check. 2. Create new event type, if required. 3. Create the new SLA. 4. Group the individual SLA conditions to the SLA. 5. Assign the SLA to the relevant interface

13.12.1

Individual SLA conditions

The individual SLA conditions are the low-level checks that can be performed on the available data. Variables are written between greater and less than signs like < variablename > and normal mathematical symbols are used. JFFNMS ships with a set of individual conditions that may do everything you need. It is a good idea to check the list of existing conditions first. To create a new individual condition, go to the Administration menu Internal Configuration ⇒SLA Definitions ⇒Individual Definitions. You will get a list of the existing conditions. The table for individual conditions has the following columns. Actions Click the Edit or Del link to edit or delete an existing condition. Description Description of the condition, appears in drop box when setting UP a SLA. It is a good idea to put the set-point value in the description. For example, “Packet Loss” is no good because there is no set point, “Packet Loss ¿ 10%” is better. Show Info A small message about what SLA condition has been violated. Condition This is the condition that the SLA event triggers on, it can use mathematical symbols as well as numbers and values.

74

CHAPTER 13. EXPANDING JFFNMS

Show Expression The value that will be presented. Show Unit The unit of the show expression. The three Show parameters are used to create the event. For example, if Show Info is “IN ¿ 90%”, Show Expression is “(< in > /1024)” and Show Unit is “kbps” and assuming the value for in is 1024000 and the SLA triggers, the resulting message will look like “IN ¿ 90%: 1000 kbps”.

13.12.2

New SLA event type

There is already a SLA event type. For most uses this event will do what you need. However if you require a different SLA event type then this is the time to do it. The method for doing this is exactly the same as making any other event type, see Event Types( section 8.2 ) for details on how to create a new event type.

13.12.3

Creating a new SLA

Once the SLA individual conditions and SLA event type are ready, it is time to create a new SLA. This is found in the Administration menu item Internal Configuration ⇒SLA Definitions ⇒Condition Groups. That will bring up the SLA group table, each SLA has the following parameters: Action By clicking on the Edit or Del buttons you can edit or delete the SLA group. Clicking on the View button will show the conditions within a group, see section 13.12.4 SLAs -SLA Condition Relation for details of that table. Description Description of SLA, used for drop down boxes in interface tables. Usually describes what sort of customer or use the SLA is for. Interface Type The type of interface the SLA applies for. In the interface table only the SLAs that match the interface type will be shown. Event Type The event type, taken from the Event Types( section 8.2 ) Event Types area for a list. Either use the default SLA or the one you created in previous section. Alarm State What level alarm the event should be, comes from the Alarm States(see Section 8.4) for a list of states. Event Text A string that is prepended to the event message. The SLA condition message follows this. Threshold This value is used in Reports to discard measures above that percent of utilization. This is useful to not count high packet loss measures on a saturated link for reports.

13.12.4

Relating SLA Conditions to SLA Groups

One a new SLA Group is created, it needs to have conditions bound to it so that JFFNMS can check if an interface is within its SLA or not. Clicking on the View link from the SLA group table creates another table underneath it which is used to apply the conditions. Action Click Edit or Del to edit or delete a condition in this group SLA The group name of this SLA. It is unlikely you will ever need to change this here. Position The position is used to work out the order of evaluation for the RRD Analyzer. Condition This is the condition used. It comes from the description column of the SLA Condition Definition table. Show Check this button if you want this SLA Condition to appear in the event if it fires.

13.13. HOST CONFIG COMMANDS

75

Table 13.6: Example JFFNMS SLAs - SLA Conditions Relation Table Position Condition 10 20 30 40 50 60 70 72 73 74 75

Input Error Rate > 10% Output Traffic < 95% Input Traffic < 95% RoundTrip Time > 700ms Packet Loss > 10% Drops > 1% OR OR AND AND OR

The logic used for the SLA Grouping is RPN. The group is evaluated in order of the Position column and each item is placed onto the stack. If the item is an operator then some items may be removed from the stack and replaced with the result. The only operators allowed are AND and OR. For example, the “Customer Satellite Link” has the following logic Alarm if (((Drops > 1% ) OR (Packet Loss > 10% ) OR (RTT > 700ms )) AND (Input Traffic < 95% ) AND (Output Traffic < 95% )) OR (Input Error Rate > 10% ). In other words alarm if both input and output traffic are below 95% at the same time there are large drops, packet loss or RTT. Alternatively alarm if the input error rate is more than 10% . The result would be like table 13.6. The lines are evaluated from the lowest ID to the highest. When the SLA parser finds a Boolean Operator, it takes the last 2 values from the stack and apply the function, the result is put back on the top of the stack. All Individual Conditions return only TRUE or FALSE. (if their conditions are met or not)

13.13

Host Config Commands

JFFNMS is capable of transferring the configuration off a host and storing it locally for later analysis. To do this there are Host Config plugins. JFFNMS is able to transfer configuration from Cisco routers and switches, however if you have another device type you may be able to create a new Host Config Command. The first thing to do is to determine how the configuration transfer will work. The general method of the transfer is to create a globally writable temporary file in the TFTP directory, issue some SNMP set commands to the host, wait (possibly checking for some SNMP value) and then examine the resulting configuration file in the TFTP directory. You should try to do the transfer using a shell script first to determine that it is going to work. That way any problems will be due to something within JFFNMS and not because, for example, you are using the wrong SNMP OID. Each configuration transfer type is stored in JF F N M S/engine/conf igs/ directory and the filename is “< transf er command >.inc.php”. The file has two functions: “config < transf er command > get” and “config < transf er command > wait”. The get function takes 4 parameters: IP address of the host, Read/Write Community, IP address of the TFTP server, file-name that the configuration will be stored on TFTP server. This function is used by JFFNMS to tell the host to send the configuration to the specified file on the TFTP server. If the transfer initiation was successful return true, otherwise false. The wait function is used by JFFNMS to determine if the transfer is complete. The function should only return if the transfer completed or if a certain amount of time has expired. Like the get function, it should return true on success and false on failure or timeout. Once the file is created, you can add a new Host Config Commands type. You will then find the new type in the Host Table (see section 6.2).

76

13.14

CHAPTER 13. EXPANDING JFFNMS

Common Parameters

A lot of the plugins are able get a series of parameters about various items such as users, hosts and interfaces. This section describes what sort of parameters are available and what they can be used for.

13.14.1

Interface Parameters

13.14.2

Event Parameters

The event object may have one or two fields depending if it is an Up event or a Down one. A down event only has a “start” field while an Up event has “start” and “stop” fields. Within each field there is a number of values. The interface parameters refer to the interface that has had the event on it. The host will refer to the host of that interface. Some of the names of the fields differ between the alarm table and the interface table, but see Interface Parameters( section 13.14.1 ) for details about those parameters. . . . more . . .

13.14. COMMON PARAMETERS

77

Table 13.7: Interface Parameters Parameter interface host name host ip client name client shortname sla description sla threshold poller group description type description zone name zone image description address mask peer bandwidthin bandwidtout interfacenumber flipinout show rootmap check status make sound host show creation date modification date last poll date id host type sla client poll zone rrd mode poll interval aux table index

Type number text text text text text number text text text text

Description

The name of the interface Name of the host the interface is on IP address of the host the interface is on Long name of the customer of the interface short name of the customer of the interface Name of the SLA on the interface FIXME Name of the poller group used to poll interface Name of the Interface Type of the interface Name of the zone the interface is in file-name of the small icon for the zone the interface is in text Description of the interface text IP address of the interface text IP network mask of the interface text IP address of the peer for the interface number Incoming/Input Bandwidth of the interface number Outgoing/Output Bandwidth of the interface number FIXME boolean Flip In/Out on graphs? boolean Is the Interface visible in the root map? boolean Is status checking of the interface enabled? boolean Will JFFNMS make a sound if the interface state changes? boolean Is the host visible? timestampThe time the interface was created in JFFNMS timestampThe time the interface parameters were changed in JFFNMS timestampThe time the poller was last run on the interface ID This interface ID ID The host the interface is on ID The interface type ID The SLA applied to the interface ID The Customer that owns the interface ID The poller group used on the interface ID The zone the interface is in number How RRD data is stored, new interfaces use 2 number FIXME number FIXME

78

CHAPTER 13. EXPANDING JFFNMS

Table 13.8: Event Parameters Parameter id date severity severity level fgcolor bgcolor user state info text ack interface id

Type Description ID text text number text text text text text text number ID

interface customer

text

type id

text

client

ID

host id

text

host ip

text

host name

text

show host

number

zone zone id

text ID

zone image

text

zone name

ID

Event ID Date of the event change in the format YYYY-MMDD hh:mm:ss String representation of the event severity Level of severity of the event Hex triple for foreground color of event as displayed in viewer Hex triple for background color of event as displayed in viewer Username field for event State field for event Extra Info field for event Event message as displayed in viewer Has event been acknowledged? 1=No 1=Yes See interface field in Interface Parameters Table (Table 13.7) See client name field in Interface Parameters Table (Table 13.7) See type field in Interface Parameters Table (Table 13.7) See client field in Interface Parameters Table (Table 13.7) See host field in Interface Parameters Table (Table 13.7) See host ip field in Interface Parameters Table (Table 13.7) See host name field in Interface Parameters Table (Table 13.7) See host show field in Interface Parameters Table (Table 13.7) Short Name of the Zone the host is located in See zone field in Interface Parameters Table (Table 13.7) See zone image field in Interface Parameters Table (Table 13.7) See zone name field in Interface Parameters Table (Table 13.7)

Chapter 14

ICMP Interface Example The previous chapter described how to expand JFFNMS in some detail, but often it helps to have a concrete example of what you are trying to do for illustration. This chapter will guide you, step by step, through all the things needed to get a simple poller going. Use this as a guide or a template for your own pollers.

14.1

An ICMP message counter

We will use the example of creating a graph that shows the number of incoming ICMP messages coming into a server. ICMP stands for Internet Control Message Protocol and is used by IP networks to send control messages. The most well-known ICMP messages are echo reply and request, which are used by the ping program. When you “ping” something you are sending an ICMP echo request packet and the thing you just pinged replies with an ICMP echo reply. This counter is a good one to use for an example because it is found on on most systems and is easy to increment by pinging the device.

14.2

Determining what OID to use

Most people assume you plug the MIB somehow into JFFNMS and it will poll everything fine. This is an incorrect assumption as you need to work out which value you want to measure. MIBs are also in a tree structure, with the root being . and the leaf being the value you want to measure, while JFFNMS uses an OID, which is a chain of nodes from the root to your leaf. It is best to work backwards in the MIB. Find the values you want to monitor and find their parents and then grandparents until you get back to the top of the tree. The particular OID we want to use is the icmpInMsgs counter, found in the IP MIB. It’s OID is .1.3.6.1.2.1.5.1.0 I can check this counter with a simple command line: Listing 1 $ snmpget -v 1 -c public localhost .1.3.6.1.2.1.5.1.0 IP-MIB::icmpInMsgs.0 = Counter32: 1397 We can see that the host responds with a value, and there have been 1397 incoming ICMP messages for this host since the SNMP daemon was last started.

14.3

Configuring a Poller Item

A Poller Item is referenced in a Poller Group (see next). For our Interface Type, we want to use the snmp counter plugin to poll the host our OID. To create the Poller Item: 1. Bring up the Pollers table by selecting Administration then the menu item Polling & Discovery ⇒Poller Items. 2. Click the Add button on the top of the table, a new poller item row is show 3. In the new item row, put the following values: 79

80

CHAPTER 14. ICMP INTERFACE EXAMPLE

Figure 14.1: Example Poller Item • Description : ICMP Incoming Msgs • Name : icmp inmsgs • Poller Command : snmp counter • Parameters : 1.3.6.1.2.1.5.1.0 4. Once you have typed in the right values, click the Save button on the left side of the row. Remember what you have put into the Name column as that is used elsewhere in the setup. The Description is just information for you so you know what the poller does.

14.4

Configuring the Poller Group

Next is to get the Poller Group to join the Poller Item created in the previous step to the Backend that takes the value from the poller and puts it into the RRD file for later graphing. As we are only polling 1 item, the group is rather simple: 1. Bring up the Poller Groups table by selecting Administration then the menu item Polling & Discovery ⇒Poller Groups. 2. Click the Add button on the top of the table, a new poller item row is shown 3. Type a name, such as “ICMP InMsgs Counter” for the Group. The Interface Type will have to stay as “No Interface Type” for now. 4. Click the Save button, which will take you back to the Poller Groups table. 5. Find your newly created group and click the “View” button. There will now be a split screen with the groups up the top and an empty list of items down below. 6. At the top of the Pollers/Backend relation table (at the bottom pane) click the Add button. 7. In the new row, type in the following parameters • Position : 10 • Poller : The Name of the Poller Item you just created, we called ours “ICMP Incoming Msgs” • Backend : RRD Individual Vale 8. Click the Save button and your poller group should have 1 row

14.5

Configuring Interface Type

Next step on our list is to create the Interface Type. This is the object that holds all the other pieces together. It also allows us to setup the values the Interface Type has, see next section. 1. Bring up the Interface Types table by selecting Administration menu item Internal Configuration ⇒Polling & Discovery ⇒Interface Types. 2. Click Add to create a new Interface Type 3. in the new dialog, type the following values. Anything not listed below just leave as the default.

14.6. ADDING INTERFACE TYPE FIELDS

81

Figure 14.2: Example Poller Group • Description : ICMP In Messages • Discovery Function : simple • Discovery Parameters : .1.3.6.1.2.1.5.1.0,IcmpInMsgs • Discovery Default Poller : The Poller Group you just created, for our example it is “ICMP InMsgs Counter”. 4. Click Save and you have your new Interface Type

14.6

Adding Interface Type Fields

Interface Type Fields are the containers that are used to hold parameters for a particular interface type. It tells JFFNMS, for example, what RRD files to create. For our example, we just need the index and a single RRD file. We’re more interested in the rate of incoming ICMP packets than the absolute value of packets received since last reboot, so the RRD counter type is the one to use. 1. Find the new Interface Type you just created in the previous step, click the Fields link to bring up a new table below your Interface Types table. As it is a new Interface Type, there are no fields. 2. On the Interface Type Fields (the lower table) click Add 3. A new field row appears, enter the following parameters: • Description : Index • Internal Name : index • Field Type : Index 4. Click Save to save the index row. 5. Next, to add the RRD file definition, again on the Interface Type Fields (the lower table) click Add 6. Another new row appears, enter the following parameters: • Description : ICMP In Messages • Internal Name : This must match the “Name” column for your Poller Item you created, it is how JFFNMS knows to link the value found by the poller to the RRD file. In our case it is icmp inmsgs • Position : 20 • Field Type : RRDToolDS 7. You need to save this row before continuing. JFFNMS needs to know this row is a RRDToolDS type before those RRD parameters appear, click Save. 8. Click Edit on the row for the RRDToolDS field, the Default Value column now has the RRD parameters.

82

CHAPTER 14. ICMP INTERFACE EXAMPLE

Figure 14.3: Example Interface Type 9. As what we are measuring is a counter, the only parameter to set is Max, so set it to some large number, like 100000. You must have a value for Max. 10. Click Save.

14.7

Fixing Poller Group

The Poller Group references an Interface Type. The Interface Type references a Poller Group. There is a chicken and egg problem here as you need to create one before the other. As we created the Interface Type after the Poller Group, it is OK and has the Default Poller column set, but now we need to fix the Poller Group so it will appear in the list of pollers available for interfaces of this type. 1. Bring up the Poller Group table again. 2. Find the Poller Group you previously created and click the Edit link. 3. JFFNMS now changes that Group so you can edit it. 4. Change the Interface Type column so it refers to your newly created Interface Type. 5. Click the Save button to save your changes.

14.8

Checking the discovery works

Now we are ready for some testing. Make sure whatever host you run it on has SNMP and it does give you a value for this counter. Run the snmpwalk command described at the start of this chapter. Next go into the Administration menu item Hosts and Interfaces ⇒Hosts, this should then show you the host table. Find the Host you are testing, then set the combo box on the left of that row to “Manual Discovery w/o Portscan” and click the white/blue arrow next to it. You should see another pane open below the Host table, showing a whole lot of Interfaces. If your changes are working, there will also be a new interface having a type of “ICMP In Message”. This means the discovery part of your changes is working.

14.9. CHECKING THE POLLER WORKS

83

Take note of the ID of the host. It is the number in bold immediately to the right of the white/blue arrow. We will need this host ID in the next stage. Click the checkbox on the left side of your interface, then click the button marked “Add Marked Interfaces” right down the bottom of the interface table. This will change the windows to bring up an Interface edit window. The interface has an ID, make note of that too. This interface ID is different to the host ID.

14.9

Checking the poller works

To check the poller, once you have an interface of this type find the host ID and interface ID and run it on the command line. $ php -q poller.php 42 123 23:37:52 : H 42 : Poller Start : 6 Items. 23:37:55 : H 42 : I 123 : P 10 : snmp counter:icmp inmsgs(.1.3..1.0):1402 -> rrd(): icmp inmsgs:402 (time P:3002.8 | 0.53) 23:37:55 : H 42 : I 123 : P LPD : last poll date(): 1206707875 -> db(last poll date): 1 (time P:0.38 | 3.38) 23:37:55 : H 42 : Poller End, Total Time: 3055.86 msec. The poller is working nicely, the rrd backend is sending the value 402 into the RRD file labelled icmp inmsgs.

14.10

Writing the graph plugin

OK, so we’re collecting data, what now? We need to graph it. To do this we either need to use an existing graph plugin or write one. The plug-in is there to tell JFFNMS what to do with the graphs and is basically a bunch of lines that get sent to rrdtool. We have only 1 RRD file so our plugin is pretty simple. Make a file called graphs/icmp inmsgs.inc.php function graph_icmp_inmsgs ($data) { $opts_DEF = rrdtool_get_def($data,array("icmp_inmsgs")); $opts_GRAPH = array( "AREA:icmp_inmsgs#00CC00:’ICMP Messages ’", "GPRINT:icmp_inmsgs:AVERAGE:’Average\:%4.0lf %sEps’", "GPRINT:icmp_inmsgs:LAST:’Last\:%4.0lf %sEps\\n’"); $opts_header[] = "--vertical-label=’Messages per Second’"; return array ($opts_header, @array_merge($opts_DEF,$opts_GRAPH)); } The function name has to be whatever you call the filename with graph at the front of it. This filename (without the .inc.php) is used in the GUI to create a new graph type.

14.11

Creating new Graph Type

Now to tell JFFNMS about the plug-in. Go to Internal Configuation ⇒Polling & Discovery ⇒Graph Types. Click on the Add button at the top of the table to make a new Graph Type. Enter in the following values: • Description : Incoming Messages • Interface Type : ICMP In Messages • Allow Aggregation : Keep this unchecked

84

CHAPTER 14. ICMP INTERFACE EXAMPLE • graph1 : icmp inmsgs (This is from the filename of the graph plugin) • Width 1 : 500 • Width 2 : 150

As long as you have already checked the “Have Graph” box in the Interface Types table, you should be able to see this graph in the Performance windows.

Chapter 15

Working with external programs JFFNMS is able to work with other programs to bring in different status information. This chapter describes the various types of external programs and how to get them working with JFFNMS.

15.1

SNMP Traps

To get jffnms receiving traps, you will need to install a SNMP trapd from net-snmp (formerly CMU snmp) package or one that works similarly. In the configuration file for snmptrapd, eg /etc/snmp/snmptrapd.conf put a line like: traphandle default /opt/jffnms/engine/trap_receiver.sh This tells snmptrapd to run that trap receiver program every time it gets a trap. See snmptrapd.conf(5) manual page for details about what this does.

15.2

Syslog

There are two ways of working with syslog. The original way was to use a syslog program called msyslog which is shipped from the same site as JFFNMS. The second is to use syslog-ng and make a simple script to input the data.

15.2.1

Using msyslog to import events

To get JFFNMS to receive syslog messages from your routers you have to install a syslog daemon that sends messages to a MySQL database. Some distributions have msyslog included, which in that case just install that package. If your distribution doesn’t, you can install the provided msyslog package found on the JFFNMS Website. To install it, follow the instructions below. tar xvzf msyslog-v1.08a cd msyslog-v1.08a ./configure make make install Then you have to modify you syslog start script (/etc/init.d/syslog) to use the new server, and new configuration, examples are provided in the docs/unix folder. You need to start msyslog as something like: /usr/local/sbin/syslogd -i linux -i unix -i udp You also need to configure your routers to send syslog messages to an specific facility (again examples are provided). 85

86

CHAPTER 15. WORKING WITH EXTERNAL PROGRAMS

15.2.2

Using syslog-ng to import events

Some distributions do not ship msyslogd or it has been withdrawn (due to stability and maintenance problems). In that case you can use syslog-ng for importing events from syslog. You need to use a pipe for this method, the messages are sent out the pipe and then collected by a script that is running a MySQL client. This document cannot be a full manual on syslog-ng, but essentially the configuration file creates sources (where the information came from), filters (based on syslog facility and level) and destinations, which are like the things in the right-hand-side of a traditional syslog file. To start you off, the example given in the JFFNMS documents for msyslogd sends all messages for facility local6 to JFFNMS, for syslog-ng it would look like: source s_jffnms { unix-dgram("/dev/log"); internal(); udp(); }; filter f_jffnms { facility(local6); };

destination d_jffnms { pipe("/var/run/syslogmysql.pipe" template("INSERT INTO syslog (date, date_logged, host, message) VALUES (’$YEAR-$MONTH}; log { source{s_jffnms}; filter{f_jffnms}; destination{d_jffnms}; }; Next you need to make the script, it essentially keeps the mysql client running. An example script is: #!/bin/sh MYPIPE="/var/run/syslogmysql.pipe" if [ ! -e $MYPIPE ] ; then mkfifo $MYPIPE fi while [ -e $MYPIPE ] ; do mysql -u jffnms --password=jffnms jffnms < $MYPIPE done Then you just need to run it. Once you checked it actually works, you can get it running from init, by putting something like the following into /etc/inittab N1:23:/usr/local/sbin/mysqlpipe.sh

15.3

Smokeping

Smokeping is a great program for measuring round trip times (RTT) and packet loss to remote destinations. It not only prints these two figures but the distribution of the RTT so you can see if there is variance in the ping times. Integration has been available since JFFNMS version 0.7.0. The best way to use it is to create a new host specially for Smokeping with no IP address. For Smokeping to work, JFFNMS needs to know the top of the data directory (where the RRDs are found). You tell JFFNMS by setting the SNMP read-only community to the value of the top directory. If you don’t know what your data directory is, look in the Smokeping configuration file and find the line beginning with datadir. Once you have set the directory, you should see the Smokeping interfaces when you use the “Get Interfaces” command from the host table. To alarm a Smokeping interface, such as letting you know if the RTT or packet loss is too high, you would use a SLA just like checking for any other interface parameter. Just like a normal interface, the performance and reporting features are available.

Chapter 16

Interface Types The various interface types in JFFNMS determine what sort of things can be monitored, graphed and alarmed. A particular device, such as a Cisco router, consists of many interface types. Some of these interface types will be specific for that class of equipment, such as Cisco CPU Load, while there could also be generic interface types, such as a physical interface. This chapter describes what interface types JFFNMS has, what JFFNMS can do with them and what you will be able to see. Armed with that knowledge you will know how useful JFFNMS is for monitoring a particular sort of equipment.

16.1

Apache Web Server

The Apache webserver is able to provide a set of statistics about how busy the processes serving webpages are. JFFNMS uses this information to present some graphs. As long as Apache is configured correctly, any webserver that is reachable from the JFFNMS host can be monitored this way. By default, most Apache webservers do not provide this information, you need to enable and configure the mod status module first.

16.1.1

Discovery

Apache status is found by connecting to the HTTP port (TCP port 80) and successfully geting the URL “/server-status?auto”.

16.1.2

Status Polling

The apache poller regularly visits the URL (shown above) and then parses the results. The following 7 values are displayed in the associated graphs: • Number of Accesses (HTTP GETS, POSTS etc) to the server • Total amount of traffic sent in kiloBytes • CPU Load • Uptime of server • Average Bytes per Request • The number of busy workers • The number of idle workers

16.1.3

Graphs

The graphs get their data directly from the Apache status poller. Some values are combined into a single graph to aid in comparison. 87

88

CHAPTER 16. INTERFACE TYPES

Table 16.1: Graphs for Apache Graph

Description

Hits Througput of the webserver in accesses per second KBytes Throughput of the webserver in Bytes per second Apache CPU LoadThe load that the Apache processes place on CPU Bytes Per Request Bytes per Request. The average size of the web accesses Workers A graph showing the red busy workers and green idle workers

16.1.4

Caveats

The target Apache webserver needs to be correctly setup to first enable the status module and then to allow access from the JFFNMS server.

16.1.5

Configuring Apache

For the Apache server to display status, first enable the status module, you will then need to permit access to the status URL from the JFFNMS server. For example, if your JFFNMS server’s IP address is 192.168.1.10 then the listing would look like: SetHandler server-status Order deny,allow Deny from all Allow from 192.168.1.10 ExtendedStatus On The Location clause can either go in the main section of the configuration file or into a virtual host. The ExtendedStatus line needs to go into the main section.

16.2

BGP Peers

BGP stands for Border Gateway Protocol. It is the routing protcol that is used by ISPs to transfer routes across The Internet. An ISP’s routers speak BGP to other ISP’s routers. These remote routers are called BGP peers.

16.2.1

Discovery

JFFNMS discovers BGP peers by determining if the BGP peer OIDs are valid. If they are each entry in the BGP Peer table is a new interface.

16.2.2

Status Polling

The poller checks the BGP operational status and alarms if it changes from up. In addition it collects inbound and outbound updates plus BGP peer uptime.

16.2.3

Graphs

There is only one graph for BGP Interfaces. This graph shows the number of Inbound (peer to your router) or outbound updates per 5 minutes. A large and sustained level of updates may mean route flapping is occuring. Be careful if you have a large sustained level of outbound route updates because you may be dampened.

16.3. CISCO SA AGENT

16.2.4

89

Caveats

None.

16.3

Cisco SA Agent

Cisco Service Assurance Agent is a feature of Cisco devices where the device can inject packets into the network and measure various attributes of the network between devices. SAA is part of Cisco’s IPSLA system.

16.3.1

Discovery

Once a device that is capable of SAA is configured to start collecting information a new SNMP table is created. The JFFNMS discovery engine, if it finds this table populated, will display new SAA interfaces.

16.3.2

Status Polling

The Cisco SA Agent poller collects the following information for each SA group: forward and backward jitter, round trip latency, forward and backward packet loss. There is no status polling.

16.3.3

Graphs

Interfaces of this type have 3 graphs. Round Trip Latency, Forward and Backward Jitter, and Forward and Backward Packet loss (as a percentage).

16.3.4

Caveats

While there are many types of SAA probes, the only ones that are correctly detected and polled are the jitter probes. Any other type of probe will be detected but will probably display graphs with 0 all the time for all attributes. There are many parameters that can go into the router configuration and you should check with Cisco documentation about how to configure the router correctly so the probes do not cause any undue performance hits. However a basic configuration, assuming the remote end has IP address 192.168.100.1, would look like: rtr 1 type jitter dest-ipaddr 192.168.100.1 dest-port 16384 rtr schedule 1 start-time now

16.4

IIS Webserver

IIS is the default Webserver that comes with most Microsoft operating systems.

16.4.1

Discovery

Discovery of an IIS server happens when the plugin is able to SNMP poll a certain OID that sits under the Microsoft MIB. If the SNMP get was successful an IIS Webserver interface is created.

16.4.2

Status Polling

The IIS pollers use SNMP to find the Total numbers of: bytes received, CGI requests, files sent, GETs and POSTs.

16.4.3

Graphs

There are 4 graphs for an IIS Webserver interface. One each showing the values for bytes received, CGI requests and files sent. The number of GETs and POSTs are combined on one graph with a green area for POSTs and then a blue area on top for GETs.

90

16.4.4

CHAPTER 16. INTERFACE TYPES

Caveats

Unline the Apache module, this one is dependent on SNMP setup correctly on both JFFNMS and the target servers, as well as a path between them. Devices such as firewalls may allow HTTP traffic but drop SNMP.

16.5

Windows System

This uses a Windows-specific MIB to find a CPU. You can then see information about Load.

16.5.1

Discovery

SNMP is used for the discovery. Presence of the specific private OID is enough for the interface to be found. The parameters tell which OID to look for to find the CPU.

16.5.2

Status Polling

Nil.

16.5.3

Graphs

The poller collects information on the following: Average CPU Utilization, Number of Processes, Number of Users, Number of Active Opens, Number of Passive Opens, Number of Established connections. The TCP information (active and passive opens, established connections) are graphed together on the TCP Connection Status Graph. Likewise the number of users and processes are on the Processes/Users graph. Only CPU utilization gets its own graph.

16.5.4

Caveats

Currently the poller assumes there is a single CPU.

16.6

TCP Ports

This is the externally found TCP ports. The host running JFFNMS actively connects to the target using this interface. From there JFFNMS can determine how long it takes to connect and do some simple content checking. This interface type requires the nmap program to be installed and JFFNMS to be told of the binary’s location. The Administrator has the option of sending something to the port and doing a simple check of the return information. This option is called Content Checking.

16.6.1

Discovery

The Auto-discovery parameters determine the range of TCP ports to probe. This range is then connected by the nmap program. Any open ports are then created into interfaces.

16.6.2

Status Polling

The poller checks that the port is open (contactable from JFFNMS) and, optionally that the returned content matches what has been given. Failure of either check raises an alarm.

16.6.3

Graphs

Response time is collected for all interfaces. If there is SNMP, the number of established connections is also collected and graphed. That feature requires access to the tcpConnEntry table.

16.7. STORAGE

16.6.4

91

Caveats

The text sent needs to be a URL starting with http, https, ftp or ftps and the check is a simple needle in haystack non-regular expression check.

16.7

Storage

The Storage interface is used to collect information about disks. It is for hosts that are running the UCD/NET snmp daemon.

16.7.1

Discovery

Discovery is done by polling the specific UCD-SNMP OID. The number of blocks on the device, total and used, are collected. In addition, the block size (to convert blocks to bytes) is also found.

16.7.2

Status Polling

None.

16.7.3

Graphs

Values for Total and Used Blocks plus block size are regularly collected. The graph shows Total and Used values in Bytes.

16.7.4

Caveats

16.8

Solaris System Info

Another platform-specific system information interface type, this time for hosts running Solaris.

16.8.1

Discovery

Interfaces of this type are discovered by polling the system OID and matching it to a list of known host OIDs.

16.8.2

Status Polling

None

16.8.3

Graphs

The pollers collect information about the CPU, Swap, Memory and Load. The graphs are the same as in the UCD Host interface type.

16.8.4

Caveats

It is assumed there is only one CPU.

16.9

Reachability

Reachability Interface types are ones that are used to ping remote hosts. The pings originate from the server running the JFFNMS pollers so there needs to be an IP path from the polling server to the remote location.

92

16.9.1

CHAPTER 16. INTERFACE TYPES

Discovery

If JFFNMS has been setup with the fping binary and it can execute it then any host with an IP address has a Reachability interface.

16.9.2

Status Polling

The poller will obtain the round trip time (RTT) and packet loss to the remote host. If the packet loss exceeds the threshold for the interface (the default is 70

16.9.3

Graphs

For each reachability interface JFFNMS can show graphs of the RTT and packet loss.

16.9.4

16.10

Caveats

Physical Interfaces

Almost any SNMP-capable device will have Physical Interfaces. These are the entries found in the ifTable part of the MIB. However each class of device has a different way of presenting its interfaces. Depending on the device, physical interfaces could include actual device interfaces, Etherchannel groups, vlans, layer-3 interfaces in switch/router combinations and loopback/localhost interfaces. Problems occur with several popular SNMP-capable devices that do not set their description correctly or non-uniquely. In addition, on some devices, such as Cisco routers and switches, the poller can set

16.10.1

Discovery

JFFNMS scans the ifTable SNMP interface table to discover Physical Interfaces. Some interfaces, such as ATM aal5, ISL and 802.1q trunks have their descriptions slightly modified to remove the identifier.

16.10.2

Status Polling

The Physical Interface poller collects several statistics about the interface. The first two are the administrative and operational status of the interface. The states are then used to determine the state of the interfaces alarm icon. Input and Output rate of packets, octets and errors are collected. As well as the

16.10.3

Graphs

16.10.4

Caveats

Problems are likely to occur if the description (ifDescr OID) is the same for different interfaces.

16.11

blah

16.11.1

Discovery

16.11.2

Status Polling

16.11.3

Graphs

16.11.4

Caveats

Chapter 17

Satellites You can increase the capacity or the reliability of JFFNMS by using satellites. Satellites are other JFFNMS installations that can either have a sibling or master/slave relationship. By default, JFFNMS runs in Master mode with no satellite enabled. This means that it will not query remote systems for interface status. At the same time it will not allow itself to be queried for status FIXME - more stuff about what is passed to whom.

17.1

Relationships between Satellites

A JFFNMS system can be one of the following: • Master • Master Backup • Satellite

93

94

CHAPTER 17. SATELLITES

Chapter 18

Error Messages Quite often on the email lists users are getting the same error messages. The solutions are usually quite simple but can the messages can initially be baffling. This chapter is here to list the more common problems and their solutions.

18.1

Running the components on the command line

Various components can be run at the command line instead as a cron job for debugging purposes. It is important to ensure that you run the job as the correct user (by using su or sudo) and that you are in the right directory, which should be jffnms/engine directory.

18.1.1

Poller

The most common component to run on the command line is the poller. This is because most problems stem from something not updating, such as a RRD file filled with NaN or the status of an interface not changing. The poller command is run with the options: Listing 2 PHP poller.php host id interface id sat id poller pos interface type The IDs are not things like IP addresses, but the internal number that JFFNMS assigns. You can see the ID form each interface and host by viewing the Host Table( section 6.2 ) or Interface Table( section 6.3 ) and looking in the first column after the actions. Any parameter that is 0 or is not specified means all of that particular type of things. For example, if you want to poll all interfaces for a host, you would just specify that host’s id only and leave the other fields blank. To poll host that has ID 42 but only its TCP ports (interface type 2), the command line would be: php poller.php 42 0 0 2 A typical poller output will look like the following: 18:46:02 : H 2 : Poller Start : 43 Items. 18:46:02 : H 2 : I 2 : P 10 : snmp_counter:storage_block_size( .1.3..4.2): 1024 -> buffer(: 1 (time P: 19.17 | B: 0.16)) [.. multiple poller item lines ..] 18:46:04 : H 2 : I 2 : P 60 : no_poller(: 0 -> rrd(*): storage_block_size:1024 - storage_block_count:386772 - storage_used_blocks :255424 (time P: 0.1 | B: 24.15)) Each line starts with the local time. The first line for each host tells you what is the host ID and how many interfaces there are. In the example we are polling host id 2 “H 2” and there are 43 interfaces. A poller group is a series of poller item → backend pairs that are arranged in priority. Each pair is displayed. The second line shows the following information. 95

96

CHAPTER 18. ERROR MESSAGES • The polling is for host id 2 “H 2”, interface id 4 “I 4” and the poller priority is 10 “P 10”. • the poller plugin is called snmp counter • the poller name is storage block size, this could be the RRD name too • the poller parameters are cut short in the middle, but they have “.1.3..4.2” • the poller returned the value 1024. • the backend plugin is called buffer. • the backend plugin returned 1, which usually means it is happy. • it took 19.17 and 0.16 microseconds to run the poller and backend, respectively.

The third line in the example, after the break, shows a typical response for a poller/backend pair for putting the data into the RRD files. The first fields are like for any poller line. The important thing to notice is that each field has a value and that the value makes sense for whatever you are counting. Also notice that the storage block size value is the same value as was given in the snmp counter poller. JFFNMS poller groups work by the poller item that fetches the value putting it into the buffer, and then the RRD backend yanking the value out to put into the RRD file. Notice the rrd has the asterix (or star) in the brackets. This means we have used the RRDTool All DS backend. A common mistake is the use the RRD Individual Value poller here. That is only good if you are directly taking a poller value and putting into a RRD file.

18.1.2

Consolidator

The consolidator takes one set of information and transforms it into another. Examples include events to alarms and syslog messages to events. The consolidator can run with the following parameters: php -q consolidate.php < times > The times parameter specifies the number of times the consolidators are run, with the default being 3 times to cover any things that need to be consolidated a few times.

18.1.3

Interface Discovery

The Interface Discovery component polls each host that has autodiscovery turned on for each interface type. Depending on the autodiscovery setting, it will then: do nothing; add the interface; generate an event saying it found the interface. This command is often used when you are trying to debug a new Discovery Plugin( section 13.8 ). In that case you would just enter the host’s ID and new interface type ID. There are 4 paramters that you can use to launch the interface discovery program: Host ID : The numeric ID of the host you want to run discovery on. Interface Type : ID for the Interface type, use 0 if you want to discover all types of interfaces. Output : Changes the output of the plugin, normally set this to 0. Log Events : By default if the poller finds a new interface it will generate an event. Putting a 0 for the loge vents parameter will suppress this event generation. Now, let’s say you have created a new interface type and it has been given the ID of 10001. You have already put your target host in JFFNMS which said it was host number 42. To check your discovery plugin works, with no alarms raised for finding interfaces, run the command: $ php -q autodiscovery interfaces.php 42 1001 0 0 A common problem with this sort of testing is to test a host where you have autodiscovery turned off. You don’t get very much useful information.

18.1. RUNNING THE COMPONENTS ON THE COMMAND LINE

97

$ php -q autodiscovery interfaces.php 42 22:57:42 H 42 : Autodiscovery took 6 msec. There is nothing at all going on here, go back to the Hosts table in the GUI and enable autodiscovery. You should then get some more information, like the following: $ php -q autodiscovery interfaces.php 42 10001 0 0 23:02:43 H 42: IT 10001 : Autodiscovering My New Type 23:02:45 H 42: IT 10001 : I 1 : DB : Not Found 23:02:45 H 42: IT 10001 : I 1 : HOST: inte: eth0 | desc: 23:02:45 H 4 : IT 4 : I 1 : RES : Nothing Done. 22:57:42 H 42 : Autodiscovery took 2600 msec.

primary

It’s found a new interface of type 10001, but nothing was done. Most likely the autodiscovery type is notify or at least not automagic. The important thing is it has found your new interface type if you get these messages.

18.1.4

RRD Analizer

Every 30 minutes the RRD Analizer is run, checking certain values are within pre-defined limits. If you are attemping to create an email if some value goes too high, running the RRD analizer is your first step. Just like the poller, the RRD Analizer will print out a lot of useful information. 20:14:38 I90 : =========================================================== =============================== 20:14:38 I90 : Start: 2006-01-13 19:40:00 Stop: 2006-01-13 20:05:00 Measures: 5 20:14:38 I90 : cpu user ticks(27) cpu idle ticks(65) cpu nice ticks(0) cpu system ticks(2) load average 1(1) load average 5(1) load average 15(0) nu m users(0) num procs(67) tcp active(0) tcp passive(1) tcp established(0) 20:14:38 I90 : ----------------------------------------------------------------------------------------20:14:38 I90 : sla : Cond0: ( 1 > 5 ) = 0 -FALSE- Load Average > 5: 1 20:14:38 I90 : sla : Cond1: (((( 27 + 0 + 2 ) * 100 ) / ( 27 + 65 + 0 + 2 )) > 80) = 0 -FALSE- Usage > 80%: 30.85 % 20:14:38 I90 : sla : Cond0= Eval 1 vs 0: 0 OR 0 = 0 20:14:38 I90 : sla : Final Eval: False.

Each line begins with the server time and the interface number. In this example, we have started the analizer just after 8pm and it is showing the SLA applied to interface 90 (I90). The first line is just a separator, so you know this group of evaluations is for this interface only. Line two is displaying the time range (from 19:40:00 to 20:05:00) that this analisis will be for and that there are 5 matching measurements. As each poll is done 5 minutes apart and the analysis is for 30 minutes, it should always be 5. Line 3 shows all the RRD values we have for this interface with the actual average value in brackets. These are the measurements that we will be using. Line 4 is another separator with hyphens, and so it is onto the RRD evaluation. Before we get to line 5, it will make a lot more sense if we look at the actual SLA group we have applied to Interface 90. Interface 90 is a “Linux System Info” interface type. The SLA is the “Linux/Unix CPU” SLA. It’s definition is in table 18.1 A simple enough definition. It triggers if the Load Average is above 5 or the CPU Utilization is about 80% (or both). Now, re-list each sla line:

98

CHAPTER 18. ERROR MESSAGES

Table 18.1: Linux CPU SLA Definition PositionCondition 10 20 30

Load Average > 5 CPU Utilization > 80% OR

20:14:38 I90 : 1

sla :

Cond0:

( 1 > 5 ) = 0 -FALSE- Load Average > 5:

Condition 0 is the first line of the SLA: Is Load Average higher than 5? The Individual Definition for this SLA is (< load average 5 >> 5) The analizer has plugged in the value for the 5 minute load average (see line 3 above and the load average 5 value ) into the equation, and correctly worked out that 1 not greater than 5 so the line is FALSE. 20:14:38 I90 : sla : Cond1: (((( 27 + 0 + 2 ) * 100 ) / ( 27 + 65 + 0 + 2 )) > 80) = 0 -FALSE- Usage > 80%: 30.85 % Next is CPU Utilization > 80%. The problem here is that there is no direct value for this, so we need to calculate it. Again looking at the individual conditions we get (cpu user ticks + cpu nice ticks + cpu system ticks)100 > 80 cpu user ticks + cpu idle ticks + cpu nice ticks + cpu system ticks Plugging the numbers given in line 5, JFFNMS has calculated that the load is 30.85%. This is not over 80% so again the result is false. 20:14:38 I90 :

sla :

Cond0= Eval 1 vs 0:

0 OR 0 = 0

Position 30 in the ruleset was the OR. The above line ORs 0 (false) with 0, giving 0. 20:14:38 I90 :

sla :

Final Eval:

False.

Finally we do our final evaluation. As we know from the previous line the result is False, so we do not raise an alarm. This logic has worked out correctly because the servers 5 minute load average was not above 5, nor was the CPU Utlization over 80%.

18.2

Database Problems

The first thing to check with the database is go to the Global Setup( section 3.7 ) page and make sure it says the database is OK. Database problems may not be caused by JFFNMS but more likely to be a problem with PHP or how you have setup the database. This section is not a tutorial about how to fix a database, there are better locations for that. Both types of databases have their own set of problems and fixes that you probably want to check first. For MySQL check Problems and Common Errors (http://dev.mysql.com/doc/mysql/en/Problems.html) appendix on the MySQL website. If you are running PostgreSQL then the FAQ for PostgreSQL (http://www.postgresql.o is a good source of information for that database type. By default JFFNMS suppresses the database error messages. If you are having problems with the database, you need to re-enable these messages. The file lib/api.db.inc.php has a function called db open that then calls either mysql connect or pg connect. Both these functions have an at @ symbol in front of them which means suppress errors. Temporarily remove this symbol and reload the page to obtain the error. Most errors with the database start with the lines “db ping(mysql) Connection to DB Restored. . . ”. This is just JFFNMS trying to get to the database, the real error will be after these lines. If you want to test it on the command line, make sure you put in the host -t parameter as JFFNMS currently cannot use the Unix socket. The command would be $ mysql -h localhost -u username -p databasename Enter password: yourpassword

18.3. PROBLEMS WITH PHP

18.2.1

99

New installations running selinux

A very popular email on the JFFNMS list starts of with something like I am running Fedora Core 4 and nothing works . . . Almost always this is due to FC4’s botched implementation of selinux and you either need to fix it (if you understand how) or disable it. For more information about Fedora and selinux, visit Fedora Core 3 SELinux FAQ (http://fedora.redhat.com/docs/selinux-faq-fc3/) .

18.2.2

Query Failed . . . Table doesn’t exist

A common error for first-time users, or once you upgrade to a newer version of JFFNMS. The problem is that you have either no loaded the SQL tables into the database or you have not run the new SQL commands to update the database structure for the new version.

18.2.3

Query Failed . . . Can’t open file: ’tablename.MYI

MySQL uses files that end with the suffix .MYI to store the actual information from a table of the same name. For example the table ‘events’ is stored in a file called ‘events.MYI’. Any error message that mentions these MYI files means the database itself is in trouble, usually some sort of corruption. The MySQL website has a section on Corrupted MyISAM Tables (http://dev.mysql.com/doc/mysql/en/corruptedmyisam-tables.html) . The short answer is if you see this problem run the command “REPAIR TABLE tablename” on the MySQL client. The website has links to the REPAIR TABLE and CHECK TABLE syntax.

18.3

Problems with PHP

Some problems are not really JFFNMS’ fault but are due to something not setup correctly in PHP. However how these problems appear is dependent on how JFFNMS uses PHP. The PHP website has some good information about PHP Installation and Configuration ( http://www.php.net/manual/en/install.php) also some Frequently Asked Questions in their PHP Instalation FAQ (http://www.php.net/manual/en/faq.installation. PHP is used by JFFNMS in both “Apache module” and “cgi” modes. Generally speaking if you are trying to do something via a web page, it is running as a Apache module and everything else is via cgi. The way that PHP is used changes the configuration file location. The following table shows the locations of the main PHP configuration file, php.ini for the various types of systems: System CGI directory Apache directory Debian /etc/php/cgi/ /etc/php/apache/

18.3.1

Modules for Apache PHP

The Apache PHP modules needs to have certain modules loaded. You can see which ones are active with the System Setup administration menu item. If you believe that the module is loaded but is not shown on this page, you have to make sure the PHP ini file tells it to load the module with the extension= line and that apache is restarted. If the System Setup screen doesn’t show the modules are loaded then there is no point continuing until that is fixed as a lot of JFFNMS will not work correctly. The exception to this is only one of the database (MySQL or PostgreSQL) modules needs to be loaded and WDDX is not essential.

18.3.2

Modules for cli PHP

Just like the Apache PHP module needs the modules, the command line needs the same ones. You can find what modules will be loaded with the “php -m” command.

100

CHAPTER 18. ERROR MESSAGES

18.3.3

Webserver displays PHP source

If you load up the JFFNMS screen and only get PHP source code and not a properly rendered web page then that means that the webserver is not interpreting PHP but just displaying the raw pages. There can be many reasons for this happening, including: 1. PHP module is not loaded in Apache 2. Apache does not know it is supposed to use the PHP handler for those files 3. PHP ini setting shorto pent ag not set, see PHP core php.ini Directives .

18.3.4

Call to undefined function: mysql connect()

This error message PHP doesn’t know anything about the function mysql connect(). The problem here is this function is part of the PHP mysql module and that module is not loaded. To load the module, you need to make sure the following line is in your php.ini configuration file: extension=mysql.so

18.3.5

Text boxes not updated

This problem is best described by you trying to type in some new value in a text box, you the click submit and the old value appears. No matter what box you try, you always get the old value. JFFNMS needs to have register globals turned on. The problem is that newer versions of PHP have this off by default, which is different to the old default. The most common time you will see this is when you have just installed JFFNMS and you are trying to update the setup JFFNMS for the first time.

18.4

Problems with Poller

The first thing to do when you suspect there is a problem with the poller is to run it on the command line, see Running poller on command line( section 18.1.1 ) for information on how to do this.

18.4.1

Poller returns with no hosts polled

On new installations, if you have hosts configured and interfaces setup for those interfaces, running the poller on the command line should give you some debugging output. However if you get no output at all, the most likely cause is the php module for the database you are using is not being loaded for the cgi/cli PHP version. See Modules for cli PHP( section 18.3.2 ) for details.

18.4.2

A lot of pollers return blank values

The section on Running poller on command line( section 18.1.1 ) describes the output of the poller. If you are getting nothing between the : and the -¿ of the poller lines then it means the poller itself is returning nothing. For SNMP-based pollers, this could mean either the SNMP module is not loaded or the snmpget command is not found if you are using some higher versions of SNMP.

18.5

Problems with Hosts Configuration

There are only so many things that can go wrong with the TFTP transfer of the hosts configuration. The first thing to know is that this only works with certain Cisco devices. Assuming you have the correct device, check the following: • Correct SNMP Write Community - JFFNMS needs to send some SNMP sets to tell the device to write the file. If the community is missing or incorrect then the device will ignore the request.

18.6. PROBLEMS WITH CONSOLIDATOR

101

• Correct Config Transfer Mode - Different Cisco devices need different transfer mode (which means the SNMP OID set that is used). See next section “No such variable errors” for more information. • Correct TFTP Server IP - The host needs to have the IP address to send the configuration file. This is almost always the IP address that the JFFNMS program sits on. • TFTPd running - For TFTP to work, there has to be a server running on the JFFNMS host. • TFTP server reachable - The device needs to be able to get back to the TFTP server. Firewalls need to permit TFTP (UDP port 69) from the device to the server. • TFTP directory writable - The directory that the configuration files need to be written to needs to be writable by whatever user id you run the cron job as. Most likely it is either www-data, http or jffnms. • Cron job enabled - Make sure the cron configuration runs the tftp get host config script at some regular intervals. You can manually run the TFTP configuration collector, but make sure you do it as the right user id, by running php4 -q tftp get host config.php . You need to be in the engine directory and the PHP binary may be called php and not php4.

18.5.1

No such variable errors

When you run the script on the command line you may get an error like Warning: Error in packet. Reason: (noSuchName) There is no such variable name in this MIB. in /usr/share/jffnms/engine/configs/cisco catos.inc.php on line 16

Warning: This name does not exist: enterprises.9.5.1.5.1.0 in /usr/share/jffnms/engine/configs/cisco catos.inc.php on line 16
This means you have the wrong Config transfer mode.

18.6

Problems with Consolidator

The consolidator is responsible for getting the events happening from the raw tables. If your NMS is going strangely silent or you are getting strange alarms then it might be the consolidator.

18.6.1

Consolidator wont raise alarms

You may have an event that should be raising an alarm for an interface, but for some reason this is not happening. This problem usually only happens when you are creating new interface types. The first thing is to check that the event is actually supposed to raise an alarm. Go to the Event Types which is found in the Administration menu item Internal Configuration ⇒Event Analyzer ⇒Event Types. Find the event you are working on and make sure that the column “Event Generates an Alarm?” is checked. Another common reason for no alarm is because the consolidator cannot link the event to the interface. For an event to be consolidated into an alarm, it must have a known host, a valid interface field that matches the “interface.interface” field of the interface, and a valid state field. Also read the section on Triggers in chapter 11 , especially the part about the debug log format. If you see a “Then email (1)” then look at your server and how it handles email. If you don’t see a line ending with that, then the problem is (at least) within JFFNMS.

102

CHAPTER 18. ERROR MESSAGES

18.6.2

Email action returns 0

The email action is pretty simple, it takes a bunch of fields and uses the PHP mail function to send the email. If the trigger debug logs show a return value of 0, it can only be a few things. Those things are, in order of most to least likelyhood: • The user that the trigger is applied to has no email address, check the user profiles. • The PHP mail function is not available in PHP, due to how PHP was compilied • The PHP mail function returned an error.

18.6.3

No events or Duplicate entry for key

This problem is not mysql’s fault but a problem with the mysql databases. The problem is the consolidator attempts to insert new events and it cannot due to a duplicate key. Running the consolidator on the command line shows the problem:

jffnmshost:/usr/share/jffnms/engine# sudo -u jffnms php4 -q consolidate.php SYSLOG Events to Process: 212857 string(67) ‘‘rsyncd[20922]: rsync to blah from fred.mynet (192.168.10.1)’’ SYSLOG Message ID: 2433507 // Host: 18 // interface(: // state(: // username(: // inf Query Failed - table_insert(events) - insert into events (date,type,host,interface,state The important thing is the error message “‘Query Failed . . . Duplicate entry (some number) for key 1. This means you are trying to add a new line to the database and the value for the primary key is already in there. The problem is that you are not specifying the primary key column (”id“), so how can you be adding a duplicate? The underlying problem is that the mysql table is corrupt and has forgotten what the next id number should be. It’s gotten terribly confused and has one part of thinking the maximum value for id is 2581292 but another part thinking it is some other, higher number. The solution is simple, login to mysql and repair the table: mysql> repair table events; +---------------+--------+----------+----------------------------------------------+ | Table | Op | Msg_type | Msg_text | +---------------+--------+----------+----------------------------------------------+ | jffnms.events | repair | warning | Number of rows changed from 337934 to 337941 | | jffnms.events | repair | status | OK | +---------------+--------+----------+----------------------------------------------+ 2 rows in set (43.63 sec) Then re-run consolidator again and wait for the rush of new events. I have never heard of this problem with PostgreSQL, if you do see it and are using PostgreSQL please let us know.

18.7

Problems with Graphs

18.7.1

RRD files not created

First check that the files really are not being created. The JFFNMS performance screens can sometimes say there are no files for an interface when the files are there but there is a problem with the code or the file. The files are created in the RRD Files Path configuration item, which you can see in the Setup screen (System Setup in the Administration Menu ). You can then check in that directory to see if files interfaceNN-Y.rrd are there. NN will be the interface ID and the Y will be a number like 0 or 1. Next place to check is the log files of the Apache server. Any child processes, such as PHP or RRDTool, that print to stderr will get their information logged here, so it is a good place to see if one of those tools is mis-behaving.

18.7. PROBLEMS WITH GRAPHS

18.7.2

103

Apache Log shows problems with .dat files

An example error message is ERROR: Opening ’var/lib/jffnms/tempengine/3f6e91e17a5d1.dat’ for write: No such file or directory In this case there is a missing / (var instead of /var) that is causing the problem. To fix this edit the config.ini either directly or via the System Setup menu. No such file or directory means either the tempengine directory doesn’t exist or the file could not be created.

18.7.3

Apache Error Log shows .dat files have permission denied

A related error message in the apache error logs is ERROR: Opening ’/var/lib/jffnms/tempengine/3f6e92a0bf810.dat’ for write: Permission denied This means the user the web server is running under is unable to write files to that directory. Make sure that whatever user or group the webserver is running as is able to write files into that directory.

18.7.4

Incompatible libpng version in application and library

You will see this error message in the Apache error log. This problem is due to the incompatible linking between RRDTool the libpng and libgd libraries. It is also a reasonably difficult problem to fix as it involves recompiling. You can see the double-linking with the ldd command, such as $ ldd /usr/bin/rrdtool librrd.so.0 => /usr/lib/librrd.so.0 (0x000002000002a000) libpng.so.2 => /usr/lib/libpng.so.2 (0x000002000005c000) libgd-gif.so.1 => /usr/lib/libgd-gif.so.1 (0x00000200000ae000) libm.so.6.1 => /lib/libm.so.6.1 (0x00000200000f6000) libc.so.6.1 => /lib/libc.so.6.1 (0x0000020000184000) libpng12.so.0 => /usr/lib/libpng12.so.0 (0x000002000031a000) libz.so.1 => /usr/lib/libz.so.1 (0x000002000035a000) /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x0000020000000000) libpng is linked twice in this case, and there are two different versions, namely libpng.so.2 and libpng12.so.0. A setup such as this is almost guaranteed to give some sort of problems.

18.7.5

Gaps in Graphs

There is essentially three known reasons why there are gaps in the graphs. The first is that the polled data has exceeded maximum value for the RRD file. This is more likely to happen if the gaps appear around where the graph has values 15 or 150 Mbps. You will get 0 for the times the graph exceeds the limit. The RRD file is created using the parameters in the RRDTool Structure DEF column of the Interface Types table. If the column uses the tag ¡bandwidth¿ then the values of Bandwidth IN and Bandwidth OUT are used in the following formula: Bandwidth = 1.5 ∗ M ax(Bandwidth IN, Bandwidth OU T ) If you change the maximum value then the RRD file is updated next poll using the rrdtune feature of RRDTool. Another reason for gaps in graphs is the poller fails to run, or fails to complete. A heavily loaded server or a problem with JFFNMS and/or the database will show up as random missing samples in the graph, with values of 0. Finally, if you have very high speed interfaces you may get some strange results. The graphs look OK until you reach a certain rate, then they have very low values before finally going back up to where

104

CHAPTER 18. ERROR MESSAGES

they were. This will occur around the 120-150Mbps mark where it will suddenly show traffic rates in the small 10s of Mbps. This problem is caused by the SNMP counters wrapping around in the 5 minute polling interval ( 65535 ∗ 5 ∗ 60 = 19M Bps = 152M bps ). There is no simple solution to this problem. Changing the polling rate, and associated RRD definitions to reflect the new rate, will fix it. Later versions of JFFNMS will use the 64 bit counters to get around this problem, until 10,000 GigE comes along.

Chapter 19

Random stuff, to go somewhere - there is some correlation, same events with all fields equal right after each other are joined.

105

106

CHAPTER 19. RANDOM STUFF, TO GO SOMEWHERE Appendix

Chapter 20

Tables of values 20.1

User Profiles

107

108

CHAPTER 20. TABLES OF VALUES

Table 20.1: Table of User Profile Values Option

Type Description

Administration Access View All Interfaces User Administration Map Sound eMail Base Map Events Sound System Administration Host Administration

selectAccess to Administration menu selectView all interfaces selectView and edit all users selectMaps can play sounds on events text Email address of user, for triggers text Top of the tree of visible maps selectEvent viewer can play sounds selectAccess allowed to Internal Configuration selectAccess allowed to “Hosts and Interfaces” menu item selectAccess to report pages selectFIXME selectAccess to statistics page text Default filter used for event viewer text Refresh time in seconds for event viewer text Refresh time in seconds for maps text Aliased used for sms client selectView type of map selectDefault view for maps text If used, show only these customers

Reports Access Disable Popups View Start Page Stats Events Default Filter Events Refresh Interval Map Refresh Interval SMS Pager Alias Default View Type Default View Customer Filter

Chapter 21

About this document 21.1

Contributing to this document

Contributions or comments are welcome about this document. It is only with feedback from other JFFNMS users that it can improve. Something that may appear to be obvious to the authors could prove an insurmountable block for you, and possibly many other JFFNMS users. Your feedback will make future users’ paths hopefully a lot smoother. You can find information about the email list in the Getting Help( section 1.5 ) section. Altenatively, if you don’t want to send suggestions or comments to the list email one of the authors, their email address is at the front of this document.

21.2

Copyright and License

This document is copyright 2004,2005 by the following authors: • Craig Small • Javier Szyszlican This is free documentation; you may redistribute it and/or modify it under the terms of the GNU General Public License in chapter 22 , version 2, as published by the Free Software Foundation; eiher version 2 of the License, or (at your option) any later version. This work is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. The copyright holders regard the LATEX form of this work as the Source Code of this Work for the purposes of the GNU General Public License.

109

110

CHAPTER 21. ABOUT THIS DOCUMENT

Chapter 22

The GNU General Public License Version 2, June 1991 c 1989, 1991 Free Software Foundation, Inc. Copyright 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed.

Preamble The licenses for most software are designed to take away your freedom to share and change it. By contrast, the GNU General Public License is intended to guarantee your freedom to share and change free software—to make sure the software is free for all its users. This General Public License applies to most of the Free Software Foundation’s software and to any other program whose authors commit to using it. (Some other Free Software Foundation software is covered by the GNU Library General Public License instead.) You can apply it to your programs, too. When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for this service if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs; and that you know you can do these things. To protect your rights, we need to make restrictions that forbid anyone to deny you these rights or to ask you to surrender the rights. These restrictions translate to certain responsibilities for you if you distribute copies of the software, or if you modify it. For example, if you distribute copies of such a program, whether gratis or for a fee, you must give the recipients all the rights that you have. You must make sure that they, too, receive or can get the source code. And you must show them these terms so they know their rights. We protect your rights with two steps: (1) copyright the software, and (2) offer you this license which gives you legal permission to copy, distribute and/or modify the software. Also, for each author’s protection and ours, we want to make certain that everyone understands that there is no warranty for this free software. If the software is modified by someone else and passed on, we want its recipients to know that what they have is not the original, so that any problems introduced by others will not reflect on the original authors’ reputations. Finally, any free program is threatened constantly by software patents. We wish to avoid the danger that redistributors of a free program will individually obtain patent licenses, in effect making the program proprietary. To prevent this, we have made it clear that any patent must be licensed for everyone’s free use or not licensed at all. The precise terms and conditions for copying, distribution and modification follow.

T ERMS AND C ONDITIONS F OR C OPYING , D ISTRIBUTION AND M ODIFICATION 0. This License applies to any program or other work which contains a notice placed by the copyright holder saying it may be distributed under the terms of this General Public License. The “Program”, below, refers to any such program or work, and a “work based on the Program” 111

112

CHAPTER 22. THE GNU GENERAL PUBLIC LICENSE means either the Program or any derivative work under copyright law: that is to say, a work containing the Program or a portion of it, either verbatim or with modifications and/or translated into another language. (Hereinafter, translation is included without limitation in the term “modification”.) Each licensee is addressed as “you”. Activities other than copying, distribution and modification are not covered by this License; they are outside its scope. The act of running the Program is not restricted, and the output from the Program is covered only if its contents constitute a work based on the Program (independent of having been made by running the Program). Whether that is true depends on what the Program does.

1. You may copy and distribute verbatim copies of the Program’s source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice and disclaimer of warranty; keep intact all the notices that refer to this License and to the absence of any warranty; and give any other recipients of the Program a copy of this License along with the Program. You may charge a fee for the physical act of transferring a copy, and you may at your option offer warranty protection in exchange for a fee. 2. You may modify your copy or copies of the Program or any portion of it, thus forming a work based on the Program, and copy and distribute such modifications or work under the terms of Section 1 above, provided that you also meet all of these conditions: (a) You must cause the modified files to carry prominent notices stating that you changed the files and the date of any change. (b) You must cause any work that you distribute or publish, that in whole or in part contains or is derived from the Program or any part thereof, to be licensed as a whole at no charge to all third parties under the terms of this License. (c) If the modified program normally reads commands interactively when run, you must cause it, when started running for such interactive use in the most ordinary way, to print or display an announcement including an appropriate copyright notice and a notice that there is no warranty (or else, saying that you provide a warranty) and that users may redistribute the program under these conditions, and telling the user how to view a copy of this License. (Exception: if the Program itself is interactive but does not normally print such an announcement, your work based on the Program is not required to print an announcement.) These requirements apply to the modified work as a whole. If identifiable sections of that work are not derived from the Program, and can be reasonably considered independent and separate works in themselves, then this License, and its terms, do not apply to those sections when you distribute them as separate works. But when you distribute the same sections as part of a whole which is a work based on the Program, the distribution of the whole must be on the terms of this License, whose permissions for other licensees extend to the entire whole, and thus to each and every part regardless of who wrote it. Thus, it is not the intent of this section to claim rights or contest your rights to work written entirely by you; rather, the intent is to exercise the right to control the distribution of derivative or collective works based on the Program. In addition, mere aggregation of another work not based on the Program with the Program (or with a work based on the Program) on a volume of a storage or distribution medium does not bring the other work under the scope of this License. 3. You may copy and distribute the Program (or a work based on it, under Section 2) in object code or executable form under the terms of Sections 1 and 2 above provided that you also do one of the following: (a) Accompany it with the complete corresponding machine-readable source code, which must be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange; or,

113 (b) Accompany it with a written offer, valid for at least three years, to give any third party, for a charge no more than your cost of physically performing source distribution, a complete machine-readable copy of the corresponding source code, to be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange; or, (c) Accompany it with the information you received as to the offer to distribute corresponding source code. (This alternative is allowed only for noncommercial distribution and only if you received the program in object code or executable form with such an offer, in accord with Subsection b above.) The source code for a work means the preferred form of the work for making modifications to it. For an executable work, complete source code means all the source code for all modules it contains, plus any associated interface definition files, plus the scripts used to control compilation and installation of the executable. However, as a special exception, the source code distributed need not include anything that is normally distributed (in either source or binary form) with the major components (compiler, kernel, and so on) of the operating system on which the executable runs, unless that component itself accompanies the executable. If distribution of executable or object code is made by offering access to copy from a designated place, then offering equivalent access to copy the source code from the same place counts as distribution of the source code, even though third parties are not compelled to copy the source along with the object code. 4. You may not copy, modify, sublicense, or distribute the Program except as expressly provided under this License. Any attempt otherwise to copy, modify, sublicense or distribute the Program is void, and will automatically terminate your rights under this License. However, parties who have received copies, or rights, from you under this License will not have their licenses terminated so long as such parties remain in full compliance. 5. You are not required to accept this License, since you have not signed it. However, nothing else grants you permission to modify or distribute the Program or its derivative works. These actions are prohibited by law if you do not accept this License. Therefore, by modifying or distributing the Program (or any work based on the Program), you indicate your acceptance of this License to do so, and all its terms and conditions for copying, distributing or modifying the Program or works based on it. 6. Each time you redistribute the Program (or any work based on the Program), the recipient automatically receives a license from the original licensor to copy, distribute or modify the Program subject to these terms and conditions. You may not impose any further restrictions on the recipients’ exercise of the rights granted herein. You are not responsible for enforcing compliance by third parties to this License. 7. If, as a consequence of a court judgment or allegation of patent infringement or for any other reason (not limited to patent issues), conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot distribute so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not distribute the Program at all. For example, if a patent license would not permit royaltyfree redistribution of the Program by all those who receive copies directly or indirectly through you, then the only way you could satisfy both it and this License would be to refrain entirely from distribution of the Program. If any portion of this section is held invalid or unenforceable under any particular circumstance, the balance of the section is intended to apply and the section as a whole is intended to apply in other circumstances. It is not the purpose of this section to induce you to infringe any patents or other property right claims or to contest validity of any such claims; this section has the sole purpose of protecting the integrity of the free software distribution system, which is implemented by public license practices. Many people have made generous contributions to the wide range of software distributed through that system in reliance on consistent application of that system; it is up to the author/donor to decide if he or she is willing to distribute software through any other system and a licensee cannot impose that choice.

114

CHAPTER 22. THE GNU GENERAL PUBLIC LICENSE This section is intended to make thoroughly clear what is believed to be a consequence of the rest of this License.

8. If the distribution and/or use of the Program is restricted in certain countries either by patents or by copyrighted interfaces, the original copyright holder who places the Program under this License may add an explicit geographical distribution limitation excluding those countries, so that distribution is permitted only in or among countries not thus excluded. In such case, this License incorporates the limitation as if written in the body of this License. 9. The Free Software Foundation may publish revised and/or new versions of the General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. Each version is given a distinguishing version number. If the Program specifies a version number of this License which applies to it and “any later version”, you have the option of following the terms and conditions either of that version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of this License, you may choose any version ever published by the Free Software Foundation. 10. If you wish to incorporate parts of the Program into other free programs whose distribution conditions are different, write to the author to ask for permission. For software which is copyrighted by the Free Software Foundation, write to the Free Software Foundation; we sometimes make exceptions for this. Our decision will be guided by the two goals of preserving the free status of all derivatives of our free software and of promoting the sharing and reuse of software generally.

N O WARRANTY 11. B ECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE , THERE IS NO WARRANTY FOR THE PRO GRAM , TO THE EXTENT PERMITTED BY APPLICABLE LAW. E XCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND / OR OTHER PARTIES PROVIDE THE PROGRAM “ AS IS ” WITHOUT WARRANTY OF ANY KIND , EITHER EXPRESSED OR IMPLIED , INCLUDING , BUT NOT LIMITED TO , THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE . T HE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU . S HOULD THE PROGRAM PROVE DEFECTIVE , YOU ASSUME THE COST OF ALL NECESSARY SERVICING , REPAIR OR CORRECTION . 12. I N NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER , OR ANY OTHER PARTY WHO MAY MODIFY AND / OR REDISTRIBUTE THE PRO GRAM AS PERMITTED ABOVE , BE LIABLE TO YOU FOR DAMAGES , INCLUDING ANY GENERAL , SPE CIAL , INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM ( INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS ), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES .

E ND OF T ERMS AND C ONDITIONS Appendix: How to Apply These Terms to Your New Programs If you develop a new program, and you want it to be of the greatest possible use to the public, the best way to achieve this is to make it free software which everyone can redistribute and change under these terms. To do so, attach the following notices to the program. It is safest to attach them to the start of each source file to most effectively convey the exclusion of warranty; and each file should have at least the “copyright” line and a pointer to where the full notice is found. one line to give the program’s name and a brief idea of what it does. Copyright (C) yyyy name of author

115 This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. Also add information on how to contact you by electronic and paper mail. If the program is interactive, make it output a short notice like this when it starts in an interactive mode: Gnomovision version 69, Copyright (C) yyyy name of author Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type ‘show w’. This is free software, and you are welcome to redistribute it under certain conditions; type ‘show c’ for details. The hypothetical commands show w and show c should show the appropriate parts of the General Public License. Of course, the commands you use may be called something other than show w and show c; they could even be mouse-clicks or menu items—whatever suits your program. You should also get your employer (if you work as a programmer) or your school, if any, to sign a “copyright disclaimer” for the program, if necessary. Here is a sample; alter the names: Yoyodyne, Inc., hereby disclaims all copyright interest in the program ‘Gnomovision’ (which makes passes at compilers) written by James Hacker. signature of Ty Coon, 1 April 1989 Ty Coon, President of Vice This General Public License does not permit incorporating your program into proprietary programs. If your program is a subroutine library, you may consider it more useful to permit linking proprietary applications with the library. If this is what you want to do, use the GNU Library General Public License instead of this License.