NICE Perform® Release 3.1
Troubleshooting Guide February 2009 385A0755-01 Rev. A2
Insight from Interactions
® TM
Information in this document is subject to change without notice and does not represent a commitment on the part of NICE Systems Ltd. The systems described in this document are furnished under a license agreement or nondisclosure agreement. All information included in this document, such as text, graphics, photos, logos and images, is the exclusive property of NICE Systems Ltd. and protected by United States and international copyright laws. Permission is granted to view and photocopy (or print) materials from this document for personal, non-commercial use only. Any other copying, distribution, retransmission or modification of the information in this document, whether in electronic or hard copy form, without the express prior written permission of NICE Systems Ltd., is strictly prohibited. In the event of any permitted copying, redistribution or publication of copyrighted material, no changes in, or deletion of, author attribution, trademark legend or copyright notice shall be made. All contents of this document are: Copyright © 2009 NICE Systems Ltd. All rights reserved. This product is covered by one or more of the following US patents: 4,893,197
5,185,780
5,216,744
5,274,738
5,289,368
5,325,292
5,339,203
5,396,371
5,446,603
5,457,782
5,819,005
5,911,134
5,937,029
6,044,355
6,115,746
6,122,665
6,192,346
6,246,752
6,249,570
6,252,946
6,252,947
6,330,025
6,542,602
6,564,368
6,694,374
6,728,345
6,775,372
6,785,369
6,785,370
6,856,343
6,865,604
6,870,920
6,871,229
6,880,004
6,937,706
6,959,079
6,965,886
6,970,829
7,010,106
7,010,109
7,058,589
7,085,728
7,203,655
7,240,328
7,305,082
7,333,445
7,346,186
7,383,199
7,386,105
7,436,887
360o View, ACTIMIZE, Actimize logo, Alpha, Customer Feedback, Dispatcher Assessment, Encorder, eNiceLink, Executive Connect, Executive Insight, FAST, FAST alpha Blue, FAST alpha Silver, FAST Video Security, Freedom, Freedom Connect, IEX, Interaction Capture Unit, Insight from Interactions, Investigator, Last Message Replay, Mirra, My Universe, NICE, NICE logo, NICE Analyzer, NiceCall, NiceCall Focus, NiceCLS, NICE Inform, NICE Learning, NiceLog, NICE Perform, NiceScreen, NICE SmartCenter, NICE Storage Center, NiceTrack, NiceUniverse, NiceUniverse Compact, NiceVision, NiceVision Alto, NiceVision Analytics, NiceVision ControlCenter, NiceVision Digital, NiceVision Harmony, NiceVision Mobile, NiceVision Net, NiceVision NVSAT, NiceVision Pro, Performix, Playback Organizer, Renaissance, Scenario Replay, ScreenSense, Tienna, TotalNet, TotalView, Universe, Wordnet are trademarks and registered trademarks of NICE Systems Ltd. All other registered and unregistered trademarks are the property of their respective owners. Applications to register certain of these marks have been filed in certain countries, including Australia, Brazil, the European Union, Israel, Japan, Mexico, Argentina and the United States. Some of such registrations have matured to registrations.
385A0755-01 Rev. A2
For assistance please contact your local supplier or the nearest NICE Systems Customer Service Center: EMEA Region: (Europe, Middle East, Africa) Tel: +972-9-775-3800 Fax: +972-9-775-3000 email:
[email protected] APAC Region: (Asia/Pacific) Tel: +852-8338-9818 Fax: +852-2802-1800 email:
[email protected] The Americas Region: (North, Central, South America) Tel: 1-800-NICE-611 Fax: +720-264-4012 email:
[email protected] Israel: Tel: 09-775-3333 Fax: 09-775-3000 email:
[email protected]
NICE invites you to join the NICE User Group (NUG). Visit the NUG Website at www.niceusergroup.org, and follow the online instructions. International Headquarters-Israel
North America
Tel: +972-9-775-3100 Fax: +972-9-775-3070 email:
[email protected]
Tel: 1-800-663-5601 Fax: +201-356-2197 email:
[email protected]
United Kingdom
Germany
Tel: +44-8707-22-4000 Fax: +44-8707-22-4500
Tel: +49-(0)-69-97177-0 Fax: +49-(0)-69-97177-200
France
Hong-Kong
Tel: +33-(0)1-41-38-5000 Fax: +33-(0)1-41-38-5001
Tel: +852-2598-3838 Fax: +852-2802-1800
Feedback
All queries, comments, and suggestions are welcome!
•
If you have access to niceweb.nice.com, click here to open a feedback form -or-
•
Email:
[email protected]
For more information about NICE, visit www.nice.com
Revision History Troubleshooting Guide
Revision A1
Modification Date
Description
December 2008
•
Added the following cross-server troubleshooting flows:
• • •
A2
February
Playback Troubleshooting Flow on page 77 Archiving Troubleshooting Flow on page 117 Multi Data Hub Troubleshooting Flow on page 141
•
Added a section for the playback error “There are no recordings to retrieve”. See Understanding Playback Retrieval on page 116.
•
Added a section for the Log Analyzer tool. See Using Log Analyzer to Check Logs on page 312.
•
Added an explanation for the “Chain not found” message. See Troubleshooting the “Chain Not Found” Error Message on page 352.
•
Added appendix for displaying NICE system information in the System Administrator. See Enabling Display of NICE System Information in System Administrator on page 573.
•
Added cross references and small changes to the Playback Troubleshooting Flow on page 77.
•
Added cross references and small changes to the Archiving Troubleshooting Flow on page 117.
Blank page for double-sided printing.
Contents 1 Introducing NICE Perform Troubleshooting
23
About this Troubleshooting Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 Sources of Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 Finding the Technical and User Guides . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 Setting an Alert for any Changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 Synchronizing the Guides to Your Local Machine . . . . . . . . . . . . . . . . . . . . 33 Indexing and Searching the Guides . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
2 Recording Troubleshooting Flow
37
Gathering Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 Checking Exceptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 General Recording Troubleshooting Flow . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 Troubleshooting Flow Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 Troubleshooting Call Failures using Exceptions . . . . . . . . . . . . . . . . . . . . . 54 Troubleshooting No New Calls in Business Analyzer . . . . . . . . . . . . . . . . . . 59 Troubleshooting Calls with Poor Audio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 Troubleshooting Calls with Audio from Several Calls . . . . . . . . . . . . . . . . . 67 Troubleshooting TRS Recordings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 Troubleshooting Integration Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
3 Playback Troubleshooting Flow
77
Gathering Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 Understanding the Playback Flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 General Troubleshooting Flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 Contents NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
7
Troubleshooting Playback Problem for All Calls . . . . . . . . . . . . . . . . . . . . . 83 Troubleshooting Playback Problem for Specific Calls . . . . . . . . . . . . . . . . . 85 Setting the Log Level in System Administrator . . . . . . . . . . . . . . . . . . . . . . . 87 Setting the Log Level in the .Config File . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 Setting the Player Log Levels. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 Checking the Workstation Ports. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 Checking the File Versions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 Checking the Locate Order . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 Checking the Storage Center . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 Troubleshooting Player Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 Playback Logic Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 Logger Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 Storage Center Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 Checking the Playback Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 Sample Player Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 Sample Log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 Storage Center Disconnection Log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 Storage Center with Playback Logic Log . . . . . . . . . . . . . . . . . . . . . . . . . . 102 Logger Disconnection Log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 Checking the NiceScreen Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 Example of Troubleshooting using the Logs. . . . . . . . . . . . . . . . . . . . . . . . 108 PlaybackAdministration.txt . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 LocateLog.txt . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 Collecting the Server Logs using the NICE Perform Logs Collector Tool . 113 Collecting the Client Logs using NICE Perform Logs Collector Tool. . . . . 113 Copying the Playlist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 Dumping the Client Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 Understanding Playback Retrieval . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 Other Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
4 Archiving Troubleshooting Flow
117
General Archiving Troubleshooting Flow . . . . . . . . . . . . . . . . . . . . . . . . . . 118 Tape/DVD Archiving Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 Contents NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
8
Media Library Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 Backup Server Device Detection Troubleshooting . . . . . . . . . . . . . . . . . . . 123 DVD-RAM Devices Not Detected . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 USB Tapes Not Detected . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124 SCSI Tapes Not Detected . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124 Other Backup Server Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 Storage Center Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 Setting the Log Level to DEBUG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 Checking for Archived Calls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 Gathering Storage Center Information . . . . . . . . . . . . . . . . . . . . . . . . . . . 132 Checking the Storage Retention and Deletion Values . . . . . . . . . . . . . . . . 133 Verifying that the Storage Center Database is Selected . . . . . . . . . . . . . . 133 Verifying the Used and Free Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 Verifying the Storage Unit Threshold . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134 Checking that the Archiving Rule Inserts New Calls . . . . . . . . . . . . . . . . . 135 Troubleshooting the Archiving Rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 Checking the Storage Center Archiving Tables . . . . . . . . . . . . . . . . . . . . . 137 Checking the Storage Center Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138 Checking the Nice Storage Streaming Service Logs . . . . . . . . . . . . . . . . . 139 Checking the Player Logs for Storage Center Errors . . . . . . . . . . . . . . . . . 139 Other Troubleshooting Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
5 Multi Data Hub Troubleshooting Flow
141
Gathering Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142 Multi Data Hub Implications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 System Administrator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 Rule Engine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 Business Analyzer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 Playback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 Monitor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 Quality Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 My Universe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 Contents NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
9
Interaction Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 Storage Center . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 Checking Connectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 Checking Replication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 Troubleshooting Replication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 Troubleshooting Communication Problems between Primary and Secondary Data Hubs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 Troubleshooting Database Structure Differences . . . . . . . . . . . . . . . . . . . 147 Stopping Replication When Upgrading All Data Hubs . . . . . . . . . . . . . . . . 147 Other Troubleshooting Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148 Troubleshooting Saving in a Multi Data Hub Environment . . . . . . . . . . . . . 148 Changing Passwords . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148 Enabling MSDTC to Function in Multi Site Environments. . . . . . . . . . . . . . 148 Other Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
6 Media Encryption Troubleshooting Flow
151
Troubleshooting Workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 Gathering Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 Changing the Logging Level . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 Checking Server Connections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 Checking the Logger/ScreenAgent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 Checking the SNMP Traps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 Checking the KSM Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 Verifying the KSM is Functioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 Checking the Client-side Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156 Checking the Server-side Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156 Checking for Certificate Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157 Trap Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159 Trap Messages for CIM Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159 Trap Messages for KSM Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160 SAMS Log Messages. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 ScreenAgent Module Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 Verifying the IIS User Permissions for the KSM Configuration . . . . . . . . . 164 Contents NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
10
Using a Fully Qualified Domain Name instead of a Host Name . . . . . . . . . 166 Data Not Entered in the nice_crypto Database . . . . . . . . . . . . . . . . . . . . . . 169 Renewing Certificates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171 Screens Not Recorded . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171 ScreenAgent Not Working Properly . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172 Ensuring Connection between SQL Server and Applications Server . . . . 175 Installing the CA certificate on Vista Workstations. . . . . . . . . . . . . . . . . . . 175
7 Channel Mapping Troubleshooting Flow
177
Channel Mapping Troubleshooting Workflow . . . . . . . . . . . . . . . . . . . . . . 179 Gathering Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 Identifying a Channel Mapping Problem . . . . . . . . . . . . . . . . . . . . . . . . . . 179 Verifying the Channel Mapping Configuration . . . . . . . . . . . . . . . . . . . . . . 181 Troubleshooting an Out of Memory Error . . . . . . . . . . . . . . . . . . . . . . . . . 183 Troubleshooting Two Users Updating Channel Mapping Concurrently . . . 184 Troubleshooting Saving in a Multi Data Hub Environment . . . . . . . . . . . . . 184 Adding a New Agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184 Other Troubleshooting Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185 Open Case Procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186 Open Case Procedures for System Administrator Parameters (CLS) . . . . 186 Open Case Procedures for the RCM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187 Troubleshooting Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190 Error Message appears when Updating Configuration . . . . . . . . . . . . . . . 190 A New or Modified Mapping Definition Does Not Work . . . . . . . . . . . . . . . 191 New Channel Mapping Definitions are not Saved . . . . . . . . . . . . . . . . . . . 191 Channel Mapping does not Work! . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191 Channel Mapping in System Administrator Differs from what is in Effect . . 192 Channels Definitions: Logger does not Appear . . . . . . . . . . . . . . . . . . . . . 192 Channels Definitions: Input Channels do not Appear . . . . . . . . . . . . . . . . 193 Source Definitions: Cannot define Virtual Extensions . . . . . . . . . . . . . . . . 193 Source Definitions: Physical Switch does not Appear . . . . . . . . . . . . . . . . 193 Dynamic Mapping: Screen Logger does not Appear . . . . . . . . . . . . . . . . . 194 Dynamic Mapping: Virtual Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194 Contents NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
11
Deleting or Changing Logger, Switch, or CLS Definitions . . . . . . . . . . . . . 194 Procedure for Reassigning a Logger to a Different CLS . . . . . . . . . . . . . . 195
8 Applications Troubleshooting Flow
197
Workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198 Gathering Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199 Checking the Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200 Troubleshooting a Communications Failure . . . . . . . . . . . . . . . . . . . . . . . . 202 Troubleshooting Communications Failure with Importer or Reporter . . . . 203 Checking the Applications Server Services . . . . . . . . . . . . . . . . . . . . . . . . 205 Using NICE Services Configuration Manager . . . . . . . . . . . . . . . . . . . . . . 206 Solving Application Services User Rights Problem . . . . . . . . . . . . . . . . . . 207 Checking the Installation Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208 Checking the Log Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209 Setting the Log Level in System Administrator . . . . . . . . . . . . . . . . . . . . . 209 Setting the Log Level in the Config File . . . . . . . . . . . . . . . . . . . . . . . . . . . 210 Checking the Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214 Solving Desktop Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217 Verifying the .NET Framework Versions . . . . . . . . . . . . . . . . . . . . . . . . . . 217 Set Security Wizard Continues to Appear After Installation . . . . . . . . . . . . 218 Problem Occurs on One Workstation . . . . . . . . . . . . . . . . . . . . . . . . . . 218 Problem Occurs on All Workstations . . . . . . . . . . . . . . . . . . . . . . . . . . 220 Solving Set Security Wizard Hangs or Displays an Error . . . . . . . . . . . . . . 221 Set Security Manually . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222 Set Security Manually Using Caspol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224 Solving Post Installation HTTP Error . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224 Verifying the Applications are Not Loaded under the Internet Zone . . . . . . 228 Configuring NICE Perform as a Trusted Web Site . . . . . . . . . . . . . . . . . . . 228 Checking NICE Perform Initialization Errors . . . . . . . . . . . . . . . . . . . . . . . 229 Solving Applications Server Configuration Problem . . . . . . . . . . . . . . . 229 Troubleshooting Remote Name Could Not be Resolved Error . . . . . . . 230 Troubleshooting Unable to Connect to the Remote Server Error . . . . . 230 Troubleshooting Serialization Problem . . . . . . . . . . . . . . . . . . . . . . . . 230 Contents NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
12
Solving Service Unavailable Error . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230 IIS AppPool Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231 Non-Paged Pool Memory Leak Problem . . . . . . . . . . . . . . . . . . . . . . . 233 Disabling Custom Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233 Solving ActiveX Control Error . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233 Troubleshooting Application Access (Power Management Issue) . . . . . . . 234 Troubleshooting Browser Problems Using Logs . . . . . . . . . . . . . . . . . . . . 237 Explorer Log (on the Client Machine) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237 Desktop Client Logs (on the Client Machine) . . . . . . . . . . . . . . . . . . . . . . 237 IIS - WWW Log File (on the Applications Server) . . . . . . . . . . . . . . . . . . . 238 IIS - HTTPERR Log File (on the Applications Server) . . . . . . . . . . . . . . . . 239 Application Event Viewer (on the Applications Server) . . . . . . . . . . . . . . . 239 Solving ActiveX Control Error on the Applications Server . . . . . . . . . . . . . 241 Optimizing Client Server Performance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242 Optimizing Client Server Performance on Computers without Internet . . . 244 Redirecting the Client Request to the Local Server . . . . . . . . . . . . . . . . . . 244 Modifying the Security Settings in the Internet Explorer . . . . . . . . . . . . . . 245 Solving Email Failure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246 Verifying the SMTP Service Installation and Configuration . . . . . . . . . . . . 246 Troubleshooting SMTP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248 Troubleshooting Crystal Reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250 Using the Process Explorer Utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251 Using the Desktop Tester . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252
9 Databases Troubleshooting Flow
253
General Databases Troubleshooting Flow . . . . . . . . . . . . . . . . . . . . . . . . . . 255 General Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256 Checking the Windows Event Viewer. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256 Checking the SQL Server Error Logs and Events . . . . . . . . . . . . . . . . . . . . 256 Checking SQL Jobs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257 Checking the Connection to the NICE Perform Database Server . . . . . . . . 257 Checking the Database Connections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257 Checking for Missing Databases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258 Contents NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
13
Checking Free Space in the Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259 Verifying the Log on as a Service User Rights Assignment . . . . . . . . . . . . 260 Verifying the Service Account . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260 Verifying Domain Users Assigned to the User Rights Assignments . . . . . . 261 Checking the SQL Server Installation Folder . . . . . . . . . . . . . . . . . . . . . . . 263 Monitoring Performance Objects and Counters . . . . . . . . . . . . . . . . . . . . . 265 Checking for Performance Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266 Scanning the Disk for Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267 Checking Replication in Multi Data-Hub Environments . . . . . . . . . . . . . . . 268 Changing the Database Passwords in SQL Authentication . . . . . . . . . . . . 269 Changing the nice sa Login Password . . . . . . . . . . . . . . . . . . . . . . . . . . . 270 Changing the nice Login Password . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270 Checking the Database Backup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271 Increasing the Size of a Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273 Checking Performance Using the NICE Performance Collector . . . . . . . . . 274 Defining Autogrowth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276 Using the NICE DBSpace Utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278 Analyzing the Database Using NICE DBAgnose . . . . . . . . . . . . . . . . . . . . . 284 General Data Mart Troubleshooting Flow . . . . . . . . . . . . . . . . . . . . . . . . . . 287 Data Mart Troubleshooting Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288 Checking the Database Size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288 Recreating Links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288 Checking the Data Mart Database Version . . . . . . . . . . . . . . . . . . . . . . . . 289 Checking Connectivity Between the Database and Data Mart Servers . . . 289 Checking a Data Mart Job Failure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 290
10 Customer Feedback Installation Troubleshooting Flow
291
General Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292 Troubleshooting Customer Feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294 Troubleshooting Flow Expanded . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295 Troubleshooting - General Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295 Troubleshooting - DTMF Routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296 Troubleshooting - CTI Info or Business Data Routing . . . . . . . . . . . . . 298 Contents NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
14
Surveys and Interactions are not Properly Correlated . . . . . . . . . . . . . . . . 301
11 High Density Logger Troubleshooting Flow
303
General High Density Logger Troubleshooting Flow . . . . . . . . . . . . . . . . . 304 Gathering Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305 Checking the Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305 Verifying the Logger Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306 Generating a Build Content Report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307 Checking Logger Initiation Failure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308 Setting the Debug Level . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309 Checking the Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309 Using Bug Find to Check Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309 Using Log Analyzer to Check Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . 312 Checking for Logger Configuration Errors . . . . . . . . . . . . . . . . . . . . . . . . . 314 Checking Logger Activity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316 Checking Map Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316 Checking RCM Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317 Synchronizing the Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317 Testing the Logger COTS using the NICE Test Kit . . . . . . . . . . . . . . . . . . . 318 Tests and Log File Locations in a Windows Environment . . . . . . . . . . . . . 318 Tests and Log File Locations in a DOS Environment . . . . . . . . . . . . . . . . 320 Testing the NICE Boards using the NICE Test Kit . . . . . . . . . . . . . . . . . . . . 320 Testing the Logger using the NICE PST. . . . . . . . . . . . . . . . . . . . . . . . . . . . 321 Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321 Shutting Down and Starting Up the Logger . . . . . . . . . . . . . . . . . . . . . . . . . 322 Shutting Down and Starting CLS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323 Using the Nicelog Setup Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323 Other Troubleshooting Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325 Other Useful Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 326
Contents NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
15
12 VoIP Troubleshooting Flow
329
Gathering Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330 Isolating the VoIP Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330 Troubleshooting VoIP Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331 Troubleshooting the VoIP Logger LAN Connection . . . . . . . . . . . . . . . . . . 332 Using Logs to Check that the VoIP Logger is Configured to Receive RTP Packets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 334 Checking the RTP Packets using the Sniffer . . . . . . . . . . . . . . . . . . . . . . . 337 Troubleshooting Network and Environment Issues . . . . . . . . . . . . . . . . . . 338 Checking the CLS Exceptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 339 Checking Channel Mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 340 Using the Sniffer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 340 Troubleshooting N+1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 347 Other Troubleshooting Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 349 mascm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 349 Configuring after Replacing a NIC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 349 Replacing a CISCO Phone . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 349 Troubleshooting a Packet Size Problem . . . . . . . . . . . . . . . . . . . . . . . . . . 349 Troubleshooting RTP Timestamp . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 350 Troubleshooting RDP Delay . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 351 Troubleshooting the “Chain Not Found” Error Message . . . . . . . . . . . . . . 352 Troubleshooting Active VoIP Integrations . . . . . . . . . . . . . . . . . . . . . . . . . 353
13 Interactions Center Troubleshooting Flow
355
Interactions Center Troubleshooting Workflow . . . . . . . . . . . . . . . . . . . . . 356 Interactions-based Call Recording Data Flow . . . . . . . . . . . . . . . . . . . . . . . 357 Interactions-based Call Recording Data Flow Details . . . . . . . . . . . . . . . . 358 Understanding the Total Call Recording Data Flow . . . . . . . . . . . . . . . . . . 360 Understanding the Total Call Recording Data Flow Details . . . . . . . . . . . . 361 Understanding the Screen Call Recording Data Flow . . . . . . . . . . . . . . . . . 363 Screen Call Recording Flow Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363 Gathering Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 366 Contents NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
16
Open Case Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 366 RecordingDecisions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 366 Call Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 367 DB Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 368 TRS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 369 RCM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 369 System Administrator Parameters (CLS Plug-ins) . . . . . . . . . . . . . . . . . . . 371 Checking the Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373 Checking the SNMP Traps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 374 Setting the Reporting Level . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 375 Running the RCM Information Dumper . . . . . . . . . . . . . . . . . . . . . . . . . . . . 376 Checking the CLS Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 377 Starting/Stopping the Interactions Center Services . . . . . . . . . . . . . . . . . . 378 Checking the Status of Interactions Center Components. . . . . . . . . . . . . . 379 Checking Components with the NICE Services Configuration Manager . . 379 Checking Processes Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 380 Troubleshooting using the Interactions Center Logs . . . . . . . . . . . . . . . . . 381 Examples of Troubleshooting using the Logs. . . . . . . . . . . . . . . . . . . . . . . 383 Example 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383 Example 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 384 Checking Ports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 408 Checking the CLS Exceptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 410 Query the Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 410 In the Business Analyzer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 410 Fine-tuning the CLS Server Configuration Parameters. . . . . . . . . . . . . . . . 411 Isolating a Screen Problem Further . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 412 Shutting Down and Starting CLS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 413 Other Troubleshooting Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 414 Other Useful Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 415
14 NMS Troubleshooting Flow
417
Workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 418 Gathering Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 419 Contents NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
17
Troubleshooting NMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 420 Client-Side Troubleshooting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 421 Pinging the Monitored Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 421 Checking the SNMP Service Configuration . . . . . . . . . . . . . . . . . . . . . . . . 421 Verifying that SNMP Agent is Sending Traps . . . . . . . . . . . . . . . . . . . . . . 425 Repairing the SNMP Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 425 Reinstalling the NICE Agents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 426 Server-Side Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 427 Collecting the System Log Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 427 Checking the NMS Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 428 Checking the CA-Unicenter Service and Sub-Services . . . . . . . . . . . . 428 Checking the Unicenter TND Agent Technology Service . . . . . . . . . . . 432 Checking the SNMP Trap Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . 434 Checking the SNMP Nice Trap Formatter Service . . . . . . . . . . . . . . . . 434 Checking the Trap Mechanism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 435 Checking the MIB File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 439 Checking Polling Timeout Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 440 Troubleshooting Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 442 Scenario 1: Monitored Server Does Not Appear in the 2D Map . . . . . . 442 Scenario 2: Drilldown Fails in the 2D Map . . . . . . . . . . . . . . . . . . . . . . 442 Scenario 3: Only Some Agents are Reporting . . . . . . . . . . . . . . . . . . . 442 Scenario 4: There are No Traps in the Console . . . . . . . . . . . . . . . . . . 442 Scenario 5: No Email Notifications Are Received . . . . . . . . . . . . . . . . 443 Changing the System Administrator Server Name . . . . . . . . . . . . . . . . . . . 445 Checking for a CAClient Serialization Issue . . . . . . . . . . . . . . . . . . . . . . . . 445 Other Troubleshooting Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 445 Using the NMSUtil Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 446
15 Telephony Services Server Troubleshooting Flow
449
Troubleshooting Workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 450 Gathering Information. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 451 Checking the Telephony Services Server Configuration . . . . . . . . . . . . . . 452 Checking the Dialogic Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 453 Contents NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
18
Troubleshooting the Dialogic Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . 454 Troubleshooting the Dialogic Analog Interface . . . . . . . . . . . . . . . . . . . . . 454 Troubleshooting the Dialogic Digital Interface . . . . . . . . . . . . . . . . . . . . . . 457 Troubleshooting the Dialogic IP Interface . . . . . . . . . . . . . . . . . . . . . . . . . 460 Using the ISDIAG Utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 461 Testing the Extension Using the Dialogic Voice Demo . . . . . . . . . . . . . . . 464 Testing an Analog Extension Using the Dialogic Voice Demo . . . . . . . 464 Testing a Digital Extension Using the Dialogic Voice Demo . . . . . . . . . 467 Using the Universal Dialogic Diagnostics Utility . . . . . . . . . . . . . . . . . . . . 470 Troubleshooting using the NICE Playback Telephony Layer Tester . . . . . 474 Checking the Telephony Services Server Service . . . . . . . . . . . . . . . . . . . 478 Checking the Applications Server Service . . . . . . . . . . . . . . . . . . . . . . . . . 478 Checking the Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 479 Setting the Logs Reporting Level . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 479 Viewing the Telephony Services Server Logs . . . . . . . . . . . . . . . . . . . . . . 480 Viewing the NICE Playback Streaming Services Logs . . . . . . . . . . . . . . . . 481 Viewing the NICE Playback Administration Service Logs . . . . . . . . . . . . . 481 Viewing the NICE Player Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 481 Checking Ports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 482 Troubleshooting Playback to Extension Problems . . . . . . . . . . . . . . . . . . . 482 Checking Ports Using Port Tester
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 483
Shutting Down and Starting the Telephony Services Server . . . . . . . . . . . 485 Troubleshooting Dial Plans . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 486 Other Troubleshooting Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 486
16 NiceScreen Troubleshooting Flow
489
Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 491 Isolating the Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 492 Gathering Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 492 Checking the NiceScreen Logger Service Status . . . . . . . . . . . . . . . . . . . 492 Checking the ScreenAgent Interactions Center Registration . . . . . . . . . . . 493 Checking the Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 494 Checking SAMS for Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 494 Contents NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
19
Isolating the Problem Further . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 496 Checking the NiceScreen Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 496 Checking the ScreenAgent Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 496 Checking the RCM Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 497 Checking the Call Server Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 497 Checking the Call Status in the Database . . . . . . . . . . . . . . . . . . . . . . . . . 497 Checking the Rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 497 Checking the Naming Resolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 498 Checking Connectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 498 Verifying the ScreenAgent Configuration . . . . . . . . . . . . . . . . . . . . . . . . . 500 Testing the ScreenAgent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 500 Other Troubleshooting Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 501 Other Useful Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 501 NICE ScreenAgent Installation/Operational Problems . . . . . . . . . . . . . . . . 503 NICE ScreenAgent Optimization and Performance Issues . . . . . . . . . . . . . 505 NiceScreen Logger Configuration Parameters . . . . . . . . . . . . . . . . . . . . . . 507
17 Storage Center Troubleshooting Flow
513
Common Troubleshooting Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 514 Installation Freezes: SNMP Dependencies . . . . . . . . . . . . . . . . . . . . . . . . 514 Archiving Failure: Multiple Network Interface Cards . . . . . . . . . . . . . . . . . 515 Storage Center Is Not Communicating with the NMS . . . . . . . . . . . . . . . . 515 Verify SCLoader Service User . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 516 Verify the SnmpAgent Object . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 517 Verify the SNMP Service Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . 520 Allowing Duplicate Archiving . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 521 Storage Center Does Not Archive Calls . . . . . . . . . . . . . . . . . . . . . . . . . . 521 Other Useful Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 524 Verifying the Storage Center Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . 526 Implementing the Troubleshooting Workflow . . . . . . . . . . . . . . . . . . . . . . . 527 Verifying that the Storage Center Database is Selected . . . . . . . . . . . . . . 527 Verifying the Used and Free Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 528 Gathering Storage Center Activity Statistics . . . . . . . . . . . . . . . . . . . . . . . 528 Contents NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
20
Checking that the Archiving Rule Inserts New Calls . . . . . . . . . . . . . . . . . 528 Troubleshooting the Archiving Rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 529 Checking the Storage Center Archiving . . . . . . . . . . . . . . . . . . . . . . . . . . 531 Checking the Storage Center Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 531 Checking the Nice Storage Streaming Service Logs . . . . . . . . . . . . . . . . . 532 Verifying Playback from Storage Center Only . . . . . . . . . . . . . . . . . . . . . . . 533
A Collecting Information Using the Log Collector
537
What is the Log Collector? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 538 What is a Component? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 538 What is a Report? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 539 FAQs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 540 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 541 Getting Started . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 542 Collecting Information from NICE Perform Servers . . . . . . . . . . . . . . . . . . 543 Collecting Information from Your Local Machine . . . . . . . . . . . . . . . . . . . . 550 Saving and Exporting Reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 555 Importing Saved Reports for Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 556 Scheduling Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 558 Viewing Collected Reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 560 Viewing and Saving Reports in other Applications . . . . . . . . . . . . . . . . . . 562 Reports Created with the Log Collector. . . . . . . . . . . . . . . . . . . . . . . . . . . . 563 Server-Side Reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 563 General Server-Side Reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 563 Applications Suite Reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 564 Playback Telephony Services Server Reports . . . . . . . . . . . . . . . . . . . 566 Logger Reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 566 SQL Server Reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 567 Interactions Center Reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 567 Storage Center Reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 567 Reporter Reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 568 NiceScreen Logger Reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 568 ScreenSense Reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 568 Contents NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
21
Stream Server Reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 568 Business Services Framework (BSF) Server Reports . . . . . . . . . . . . . 569 CTI Integrations Reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 569 Media Library Reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 569 Audio Analysis Reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 569 VRG Reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 570 MPCM (Media Provider Controller Manager) Reports . . . . . . . . . . . . . 570 RTS Reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 570 Client-Side Reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 571 General Client-Side Reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 571 Player Reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 571 ScreenAgent Reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 571 ScreenSense Agent Reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 571
B Enabling Display of NICE System Information in System Administrator
573
Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 574 Workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 574 Common Terms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 575 Verifying the System Administrator User . . . . . . . . . . . . . . . . . . . . . . . . . . 576 Adding the System Administrator User to Computers not in an Active Directory Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 578 Editing DCOM Permissions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 582 Setting Security Permissions through the WMI Control . . . . . . . . . . . . . . . 588
Contents NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
22
1 Introducing NICE Perform Troubleshooting This troubleshooting guide includes troubleshooting flows and procedures that are intended for use by support staff and NICE support engineers in Tier 1, 2 and 3. This guide does not include procedures for Tier 4.
Contents About this Troubleshooting Guide................................................................................ 24 Sources of Information .................................................................................................. 26 Finding the Technical and User Guides ....................................................................... 27 Setting an Alert for any Changes .................................................................................. 30 Synchronizing the Guides to Your Local Machine ...................................................... 33 Indexing and Searching the Guides.............................................................................. 34
Chapter 1: Introducing NICE Perform Troubleshooting NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
23
About this Troubleshooting Guide
About this Troubleshooting Guide This troubleshooting guide includes troubleshooting flows and procedures that are intended for use by support staff and NICE support engineers in Tier 1, 2 and 3. This guide does not include procedures for Tier 4. This guide includes the following troubleshooting:
•
Recording Troubleshooting Flow: Includes a general recording troubleshooting flow, as
well as several recording troubleshooting scenarios. This flow refers to the relevant components, as required.
•
Media Encryption Troubleshooting Flow: Includes a troubleshooting workflow, as well as
tables of trap messages and ScreenAgent Management System (SAMS) log messages.
•
Channel Mapping Troubleshooting Flow: Includes a channel mapping troubleshooting
flow, as well as provides quick solutions for common problems you face when defining channel mapping. The procedures are designed for the seasoned user who is familiar with the NICE Perform system and the System Administrator application. For complete instructions on using the System Administrator, see the System Administrator’s Guide.
•
Applications Server Troubleshooting Flow: Contains a troubleshooting workflow for the
NICE Perform Applications Server, including solving Desktop problems and troubleshooting Browser problems using Logs. A list of the services, config and log files is also provided.
•
Databases Troubleshooting Flow: Includes procedures for troubleshooting NICE Database
Server and Data Mart. These procedures must be perform only by authorized personnel. For full assistance, contact NICE Systems Customer Services.
•
Customer Feedback Installation Troubleshooting Flow: Provides a review of some information that will aid the installer in understanding the Customer Feedback workflow. Also provided is a workflow recommending how to proceed when troubleshooting a Customer Feedback system.
•
High Density Logger Troubleshooting Flow: Contains a troubleshooting flow with
troubleshooting tips and procedures for High Density Loggers.
•
Interactions Center Troubleshooting Flow: Includes a Interactions Center troubleshooting
workflow, Interactions-based call recording data flow, screen call recording data flow, total call recording data flow, and other troubleshooting issues.
•
NMS Troubleshooting Flow: Includes an NMS troubleshooting flow and other
troubleshooting issues.
•
Playback Telephony Services Server Troubleshooting Flow: Includes a troubleshooting
flow and other troubleshooting issues.
•
NiceScreen Troubleshooting Flow: Lists some of the most common problems you may
encounter and solutions.
•
Storage Center Troubleshooting Flow: Describes common troubleshooting issues and
how to verify that the Storage Center was properly installed. It also includes a troubleshooting workflow to solve various issues.
Chapter 1: Introducing NICE Perform Troubleshooting NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
24
About this Troubleshooting Guide
•
VoIP Troubleshooting Flow: Contains troubleshooting tips and procedures for when you
experience a problem with VoIP.
•
Collection Information Using the Log Collector: The Log Collector is an easy-to-use utility that gathers the information from all your NICE Perform servers and local workstation, in the form of reports, for easy transference for analysis.
Chapter 1: Introducing NICE Perform Troubleshooting NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
25
Sources of Information
Sources of Information There are several additional sources of information. 1. There are also over 150 Technical and User Guides on ExtraNICE (http://www.extranice.com)
•
Hardware Guides
•
History Release Notes
•
Installation Guides
•
Reference Guides
•
Release Notes
•
Switch/Driver Integration Guides
•
Upgrade/Migration Information
•
User Guides
2. The NICE Perform Release 3.1 Maintenance Guide is especially useful and should be read
together with this guide. 3. Internal (for NICE-use only) sources include:
•
Technical Documentation Team Site for technical documents in progress: http://niceweb.nice.com/sites/Technical_Writers_Documents/default.aspx
•
Tier 4 Team Site for tools and utilities: http://niceweb.nice.com/teams/Tier4_RD/default.aspx
•
Tier 3 Team Site: http://niceweb/sites/Tier3_CS/Tier3 working documents and manuals/Forms/AllItems.aspx
•
Product Training Team Site: http://niceweb.nice.com/sites/ProductTraining_RD/default.aspx
Chapter 1: Introducing NICE Perform Troubleshooting NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
26
Finding the Technical and User Guides
Finding the Technical and User Guides There are over 150 guides on ExtraNICE. 1. Browse to http://www.extranice.com/. You must have access to the relevant content on
ExtraNICE. 2. If you are accessing ExtraNICE from outside of the NICE network, the following window
appears. Figure 1-1
ExtraNICE Portal
3. Enter your User name and Password and click on Log On to ExtraNICE. The following
window appears.
Chapter 1: Introducing NICE Perform Troubleshooting NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
27
Finding the Technical and User Guides
Figure 1-2
ExtraNICE Portal Home Page
4. Select Products > NICE Perform > NICE Perform Release 3.1. The following window
appears. Figure 1-3
NICE Perform Release 3.1
5. Expand the Content Type: Technical and User Guides. 6. Expand the required Files Types to view a list of manuals.
Chapter 1: Introducing NICE Perform Troubleshooting NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
28
Finding the Technical and User Guides
Figure 1-4
List of Manuals
Chapter 1: Introducing NICE Perform Troubleshooting NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
29
Setting an Alert for any Changes
Setting an Alert for any Changes You can set a daily alert of any changes to the content in SharePoint. 1. In ExtraNICE, select Products > NICE Perform > NICE Perform Release 3.1. The
following window appears. Figure 1-5
ExtraNICE Portal - NICE Perform Release 3.1
2. Click on NICE Perform Release 3.1. The following window appears.
Chapter 1: Introducing NICE Perform Troubleshooting NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
30
Setting an Alert for any Changes
Figure 1-6
NICE Perform Release 3.1 - Actions
3. Select the Actions drop-down menu. Figure 1-7
Actions Drop-down Menu
4. Select Alert Me. The following window appears.
Chapter 1: Introducing NICE Perform Troubleshooting NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
31
Setting an Alert for any Changes
Figure 1-8
Alerts Page
5. Select Send a daily summary. 6. Click OK. A daily alert is emailed to you detailing any changes or updates made to the
selected page. 7. Copy the changed files to your local folder.
Chapter 1: Introducing NICE Perform Troubleshooting NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
32
Synchronizing the Guides to Your Local Machine
Synchronizing the Guides to Your Local Machine Use a tool of your choice to synchronize the Guides from SharePoint to your local machine. For demonstration purposes only we describe how to use Colligo Reader. 1. Download and install http://colligo.com/products/sharepoint/reader.asp. 2. Select Start > Program > Colligo Reader > Colligo Reader. 3. From the Site menu, select Download New Sites. The following window appears. Figure 1-9
Download New Site Window
4. In the URL field, add http://www.extranice.com/EIS/Products/NICEPerform. 5. Add your User name and Password. 6. Click Download. The files are synchronized to your local machine. 7. The files are located in a folder, for example, C:\Documents and Settings\<username>\Local Settings\Application Data\ColligoOfflineClient\Storage5\Files\Sites\1\NICE Perform Release 3.1 .
Chapter 1: Introducing NICE Perform Troubleshooting NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
33
Indexing and Searching the Guides
Indexing and Searching the Guides Use a search tool of your choice to search for information. For demonstration purposes only we describe how to use Windows Desktop Search. 1. Download and install Windows Desktop Search
http://www.microsoft.com/windows/products/winfamily/desktopsearch/getitnow.mspx. 2. Click on the Indexing icon in the Taskbar. Figure 1-10
Index Now
3. Select Windows Desktop Search Options. The following window appears. Figure 1-11 Indexing Options
4. Click Modify. The following window appears.
Chapter 1: Introducing NICE Perform Troubleshooting NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
34
Indexing and Searching the Guides
Figure 1-12
Indexed Locations
5. Select the location with the manuals and click OK. 6. Use your selected search tool to locate the relevant information. Figure 1-13
Search Results
Chapter 1: Introducing NICE Perform Troubleshooting NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
35
Blank page for double-sided printing.
2 Recording Troubleshooting Flow This section contains troubleshooting tips and procedures for when you experience a problem with recording.
Contents Gathering Information .................................................................................................... 38 Checking Exceptions ..................................................................................................... 39 General Recording Troubleshooting Flow ................................................................... 50 Troubleshooting Call Failures using Exceptions ........................................................ 54 Troubleshooting No New Calls in Business Analyzer................................................. 59 Troubleshooting Calls with Poor Audio ....................................................................... 66 Troubleshooting Calls with Audio from Several Calls ................................................ 67 Troubleshooting TRS Recordings................................................................................. 69 Troubleshooting Integration Issues.............................................................................. 75
Chapter 2: Recording Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
37
Gathering Information
Gathering Information In order to narrow down the problem answer the following questions: 1. Is the recording TDM or VoIP? 2. For VoIP, is the recording environment passive or active? 3. For TDM, is the recording environment trunk-side, extension-side or observation? 4. Is the recording total or QM (interaction based)? 5. In a total recording environment is TRS configured? 6. Is the site N+1? 7. Is the recording summed or with speaker separation? 8. Is the channel mapping static or dynamic? If necessary export the channel mapping
definitions. See Exporting Channel Mapping Definitions in the Channel Mapping Guide. 9. Did the problem occur for a specific agent, group of agents or for all agents? 10. Which CTI integration and what version? 11. Check for exceptions by checking the tblExceptionXX table. See Checking Exceptions
on page 39.
Chapter 2: Recording Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
38
Checking Exceptions
Checking Exceptions Query the database to see the exceptions. Most of the time exceptions are connected to recordings and not directly to the interaction IDs. In order to see the connection between an exception and interaction ID, query both the exception details and the related recording (which includes the interaction ID). To check the CLS Exceptions in the Database: 1. Query the tblInteractionCatalog table to check for which period the calls occur. 2. Get all the exceptions related to interactions: SELECT * FROM tblExceptionXX, tblInteractionXX WHERE tblExceptionXX.iInteractionID = tblInteractionXX.iInteractionID 3. Get all exceptions related to recordings: SELECT * FROM tblExceptionXX, tblRecordingXX WHERE tblExceptionXX.iRecordingID = tblRecordingXX.iRecordingID 4. Check the vcExceptionDetail field in tblException. To check the CLS Exceptions in Business Analyzer:
•
In Business Analyzer, select the interaction and click the Exceptions tab in the Details pane. Figure 2-1
Checking the CLS Exceptions in Business Analyzer
Chapter 2: Recording Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
39
Checking Exceptions
ID
Exception Short Description
1
Duplicate call start.
Meaning
Possible Cause
Recommended Action
Duplicate start of call – another call with the same call key has started while this call was open.
Problem with the driver or PABX reports.
Check the driver or PABX. Collect the driver and CLS log files.
2
Maximum duration exceeded.
Call too long – the call was open for more seconds than the value of the registry parameter MaxOpenCallDuration and was therefore forcibly closed.
Either the call was longer than the parameters value, or there is a problem with the driver or PABX reports.
Check if the value of the Call Server parameter op_MaxOpenCallDuration matches the length of calls in the site.
3
Call flushed while open.
Call flushed – a flush command was executed while the call was open.
The driver may have gone down and come up.
Check that the driver is up.
4
N/A
N/A
N/A
N/A
5
Call start not reported.
End call without start (Default Start Time).
Problem with the driver or PABX reports.
Check the driver or PABX.
Could be followed by reports with exp02 because the start and end call reports have different information. 6
N/A
N/A
N/A
Review the driver log files, look for errors. If there is no error no action is needed.
Review the driver log files, look for errors. If there are errors, collect the driver and CLS log files.
N/A
Chapter 2: Recording Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
40
Checking Exceptions
Exception Short Description
Meaning
Possible Cause
Recommended Action
7
No available recording resource.
No available recording resource.
There are not enough voice recording channels to meet the number of recording requests in the site.
Could be caused by unclosed recorded calls – check for exp02 calls. If there is a substantial amount of calls with these exceptions, check RCM logs for Logger disconnections (Logger down) and if channels were allocated after connection reestablished with Logger.
8
Logger not responding.
Logger not responding.
Problem with Logger or connection to Logger.
Check the Logger.
ID
Check the network connectivity between the Logger and the CLS. Check the RCM logs.
9
Unspecified recording failure.
Unspecified recording failure.
Unknown failure cause. Received from the RCM.
Collect the Integration and CLS log files.
10
Agent logout during call.
Agent logout during the call.
Problem with the driver or PABX call or logout reports.
Check if it is physically possible to logout during a call. Collect the Integration and CLS log files.
11
Too many calls for agent/extension.
MaxExtentionOpenCall - too many open calls for the same agent/extension when compared to the op_MaxCallPerExt registry parameter.
The agent had more simultaneous open segments than the parameter value. Could be a problem with the driver or PABX call reports.
Check if the parameter value meets the requirements of the site. Look for calls with exp02 to see if there is a problem with the call reports.
Chapter 2: Recording Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
41
Checking Exceptions
ID 12
13
Exception Short Description Voice recording failed.
Screen Logger not responding.
Meaning
Possible Cause
Recommended Action
Voice recording failed.
Error code received from the telephony server. If this appears as e12 check the sub-exception ID for more information.
•
Collect the Integration and CLS log files.
•
Sub-exception 12 310 indicates the Logger is down.
•
Sub-exception 12 1702 indicates no VoIP audio due to a configuration problem. This is dependant on the VoIP integration. Check the configuration, the forwarding data passed to the Capture (RCM logs), the Capture logs, as well as the forwarding device configuration. For example, no audio is received if no or the wrong forwarding information is passed to Capture, or if the audio is not forwarded by the telephony switch/forwarding device.
Screen logger not responding.
Logger is down or network issue.
Check the NiceScreen Logger and ScreenAgent. Check the RCM logs for errors when calling screen capture to start record.
Chapter 2: Recording Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
42
Checking Exceptions
ID 14
15
Exception Short Description Screen recording failed.
Unmapped voice recording.
Meaning
Possible Cause
Recommended Action
Screen recording failed.
If this appears as e14, check the sub-exception ID for more information.
•
Sub-exception 14 1002 indicates an error was received during screen recording. Either ScreenAgent was disconnected from the Interactions Center or ScreenAgent recording failed. Partial recording (up until the time of the error) may be available.
•
Sub-exception 14 4 indicates there was an unspecified failure.
Unmapped voice recording.
Problem in the voice channel configuration.
Switch the logs to DEBUG. Check channels configuration. See if there is a mapping configured for this call. Check the RCM logs. Check what the RCM received in the Start request.
16
Unmapped screen recording.
Unmapped screen recording.
The recording request was received with empty Station or IP address (depends on the screen allocation mode).
If the allocation mode is by IP address then check if the agent logged in. Check ports. Verify in RCM logs that the start request for recording the agent's screen contained the screen agent's IP.
17
Voice recording retry.
Recording voice succeeded only after retry (partial retry).
May be a temporary failure on the Logger.
Check Logger for possible reasons for temporary failures.
Chapter 2: Recording Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
43
Checking Exceptions
Exception Short Description
Meaning
Possible Cause
Recommended Action
Call Server service shutdown.
Call Server was down during the call.
Call Server was down.
Check reason for Call Server failure.
19
N/A
N/A
N/A
N/A
20
Logger not responding.
The Logger did not respond to the start record command.
Stop record command arrived before a response for the start record request arrived. This may occur for one of the following reasons:
No action is needed if there was indeed a short call; otherwise collect CLS and driver log files.
ID 18
Collect CLS logs, event viewer and CPU Performance Monitor.
•
The call was very short (1 or 2 seconds).
•
The request was for 2 medias.
•
Success in both was required (usually QA).
•
One media failed immediately.
•
Stop record is sent for both medias.
If the request was for 2 medias, try to understand the recording problem with the first media. Check the RCM logs. Check the Call Server logs.
21
N/A
N/A
N/A
N/A
22
N/A
N/A
N/A
N/A
Chapter 2: Recording Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
44
Checking Exceptions
ID 23
Exception Short Description
Meaning
Possible Cause
Recommended Action
Call start not reported.
Stop record without start (screen or voice).
Usually happens with very short calls.
No action is needed if there was indeed a short call; otherwise collect CLS and driver log files.
The request was for 2 medias. Success in both was required (usually QA). One media failed immediately. Stop record was sent for both medias. Or may be an internal problem. 24
Error in stop record request.
Stop record with wrong ID. No start call request was found with this CLS Call ID.
Usually happens with very short calls. The request was for 2 medias. Success in both was required (usually QA). One media failed immediately. Stop record was sent for both medias. Or may be an internal problem.
25
26
If the request was for 2 medias, try to understand the recording problem with the first media. Check the RCM logs. Check the Call Server logs.
No action is needed if there was indeed a short call; otherwise collect CLS and driver log files. If the request was for 2 medias, try to understand the recording problem with the first media. Check the RCM logs. Check the Call Server logs.
Too many requests for channel.
Too many recording requests for the same Logger and channel (more than 30).
May occur due to a problem with the driver or PABX call reports – calls are not closed.
Collect CLS and driver log files.
RCM service down during call.
RCM was down during the call.
RCM was down.
Collect CLS logs, event viewer and CPU Performance Monitor.
Check the Call Server logs.
Check the RCM logs. Check the reason that the RCM went down. Chapter 2: Recording Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
45
Checking Exceptions
ID 27
Exception Short Description
Meaning
Possible Cause
Recommended Action
Error on complete interaction start.
The contact started after its segment.
Problem in the driver or PABX call or logout reports.
Check the driver or PABX. Collect CLS and driver log files. Check the RCM logs for a long period in DEBUG mode.
28
29 30
31
No available recording resource.
No available recording resource.
Not enough Logger recording channels.
See if the configuration meets the site needs. If not change it.
The Resource Call Manager cannot allocate more resources for this initiator.
Bad resource management configuration.
Collect CLS and driver log files.
There maybe a problem with calls that are not closed and therefore resources are not being freed up.
Check the RCM logs.
Restart RCM. Could be a problem with the driver or PABX reports.
Error on complete interaction close.
Segment was open when the contact closed.
Could be a problem with the drivers or PABX call reports.
Check the driver or PABX.
Time Interval recording aborted.
Block dummy call was closed due to a Call Server restart.
Call Server restarted.
Collect CLS log files.
Client disconnect.
Client was disconnected during the open block.
Client was disconnected.
Collect CLS and driver log files. Check why Call Server was closed. Collect CLS log files. Check why the client was disconnected. Check the Call Server logs.
32
N/A
N/A
N/A
N/A
Chapter 2: Recording Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
46
Checking Exceptions
ID
Exception Short Description
Meaning
Possible Cause
Recommended Action
33
Stop on demand not by initiator.
Stop on demand was performed on the interaction recording by a client who was not the recording initiator.
Stop on demand was performed on the interaction recording by a client who was not the recording initiator.
No action required.
34
Invalid call time report.
Time field was changed by the DB Server. Interaction was inserted with time value lower then 1970. Stop time was lower than start time.
Call Server error. Illegal time parameters were reported.
Collect CLS log files.
RCM service not responding.
The Call Server received a timeout error on the request to the RCM.
RCM is too busy.
Collect CLS logs, event viewer and CPU Performance Monitor.
Event caused the RCM not to respond, for example, detected many Loggers initializing.
Check the RCM logs.
String field was truncated by the DB Server. Call Server sent a string longer than allowed by the DB schema.
Business data length is incorrectly specified.
Collect CLS and driver log files.
35
36
Business data value too long.
Internal RCM error.
Check the Call Server logs.
Check the maximum business data length, change it in the DB and restart the CLS.
Chapter 2: Recording Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
47
Checking Exceptions
ID
Exception Short Description
37
Failed to record VoIP.
Meaning
Possible Cause
Recommended Action
Failed to record. Audio not found in VoIP recording.
This usually results from a failure to sniff VoIP packets by the Logger.
•
Run a sniffer to check if packets were coming in.
• •
Collect CLS and Logger log files.
•
Check the RCM logs for the allocated channel.
•
Check the Capture logs.
After the Start command to the Logger, the RCM waits 2.5 seconds and checks if the audio packets were received.
Check for channel mapping failures.
38
Failed to update VoIP data.
Update VoIP (which updates the IP for an open request and its participant) sent by the Driver failed.
Occurs, for example, when the update is for an unmapped device or a closed request, or because the update was not received by the RCM.
Check that the device in the Update VoIP sent from driver is mapped.
39
Complete interaction handled on multiple servers.
Split Contact - Segments for contact handled on more than one Interactions Center.
Interaction Router could not find mapped segment on the Interactions Center that was selected for the other open segment under the same contact.
No action required.
40
Recording time mismatch.
Recording Time Mismatch – recording time was out of call time range.
Problem with integration. Calls may be reported late to the CLS when the system is under stress.
Check the Call Server logs.
Chapter 2: Recording Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
48
Checking Exceptions
ID
Exception Short Description
41
Unknown Initiator.
Meaning
Possible Cause
Recommended Action
At the start of the call, the initiator participant has no User ID, that is, the User ID was not found in the UserAdmin Database.
User is not defined under UserAdmin.
Ensure the user exists and is defined in the DB.
Chapter 2: Recording Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
49
General Recording Troubleshooting Flow
General Recording Troubleshooting Flow See Troubleshooting Flow Details on page 52 for a description of every step mentioned in the following flow. Figure 2-2
General Recording Troubleshooting Flow
Chapter 2: Recording Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
50
General Recording Troubleshooting Flow
Chapter 2: Recording Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
51
General Recording Troubleshooting Flow
Troubleshooting Flow Details 1. Is the problem a system-level problem, that is, no recording at all? 2. If the problem is a system-level problem:
•
Check that all services are up and running (Applications Server, Interactions Center and Loggers).
•
Check that the Loggers are up.
•
Check the communication between servers.
•
Check that there is time synchronization between the various NICE servers.
3. Check if the problem is a data or audio problem, by running a query in Business Analyzer. If
there are no interactions, continue with Step 7. 4. Is the problem a Logger-level problem? 5. If the problem is a Logger-level problem: a.
Verify the Logger configuration in System Administrator, check that the Logger was added to the CLS and that it is up (by selecting it and verifying that you see all the Logger's details).
b. Check the Logger communication with the Applications Server and Interactions Center. c.
Check that the Loggers are time synchronized.
d. For TDM environments, see the High Density Logger Troubleshooting Flow. e.
For VoIP environments, see the VoIP Troubleshooting Flow.
6. Make a test call: a.
Monitor the test call.
b. If the test call does not appear in the Monitor, proceed with Step 7. c.
If the test call appears in Monitor, try to ROD the test call.
d. If the test call is not recorded, check the mapping.
•
For a trunk-side recording environment, add the participant trunk group and participant trunk number columns in Business Analyzer, and verify which trunk groups and trunk numbers were assigned to the interactions. Check the mapping for these trunks.
e.
For TDM trunk-side recording, check the trunk connections.
f.
If the test call is recorded, try to play back the test call in Business Analyzer.
g. If the test call can be played back in Business Analyzer, check if there are Scheduler rules.
Verify that you can see the rule in the Rules Manager table in the DB. Verify that you do not see errors in the Scheduler and schedule sync. h. If the test call cannot be played back in Business Analyzer, see the Playback
Troubleshooting Flow. i.
For VoIP recording problems, see the VoIP Troubleshooting Guide. Chapter 2: Recording Troubleshooting Flow
NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
52
General Recording Troubleshooting Flow
7. Is the problem a CTI Server to Driver problem? 8. If the problem is a CTI Server to Driver problem: a.
Open the driver window.
b. Make a call. c.
Check for event errors in the window (errors appear in red).
d. Trace the call in the Connection Manager and Driver logs (D:\Program Files\Nice Systems\CTI\Log) and try to locate errors. e.
See the relevant Integration Guide to troubleshoot the problem.
9. Is the problem a Driver to CLS problem? 10. If the problem is a Driver to CLS problem: a.
Trace the call in the Call Server and RCM logs (D:\Program Files\Nice Systems\Interactions Center\Log) and try to locate errors. Search by time or extension number to identify the problem.
b. See the Interactions Center Troubleshooting Flow. 11. Is the problem an Interactions Center process problem? 12. If the problem is an Interactions Center process problem: a.
If the event triggers a recording, identify the call initiator: ScreenSense, Total, QA/Rule-based, or ROD.
b. Check the call events and mappings in the Call Server logs and RCM logs (D:\Program Files\Nice Systems\Interactions Center\Log). For ScreenSense tagging requests there
are no RCM logs. c.
Check the DBServer logs to check the end of the call.
d. Check the CLS exceptions. See Checking Exceptions on page 39. e.
See the NICE Interactions Center Troubleshooting Flow.
13. Is the problem a CLS to Database problem? 14. If the problem is a CLS to Database problem: a.
Query SQL Server database (tblInteractions). See the Interaction Data Reference Guide.
b. If the interaction is missing or not complete, check for a problem between the CLS and the
Database Server. c.
See the NICE Interactions Center Troubleshooting Flow.
d. See the Databases Troubleshooting Flow.
Chapter 2: Recording Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
53
Troubleshooting Call Failures using Exceptions
Troubleshooting Call Failures using Exceptions This flow is appropriate for silent calls and other call failures. 1. In Business Analyzer, note the Interaction ID of the calls. 2. Check for interaction exceptions: a.
In the database expand nice_interactions > Tables.
b. Query the tblInteractionCatalog table to check for which period the calls occur. Figure 2-3
c.
tbllnteractionCatalog Table
Run a query, where XX is the period in which the exceptions occur, and YY is the interaction ID: SELECT * FROM tblExceptionXX where iInteractionID = YY
Chapter 2: Recording Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
54
Troubleshooting Call Failures using Exceptions
Figure 2-4
Running a Query
d. Check the vcExceptionDetail field to identify the exception. e.
Troubleshoot the exception using Checking the CLS Exceptions in the Interactions Center Troubleshooting Flow.
3. Check the recordings. a.
Run a query, where XX is the period in which the exceptions occur, and YY is the interaction ID: SELECT * FROM tblRecordingXX where iInteractionID = YY.
Chapter 2: Recording Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
55
Troubleshooting Call Failures using Exceptions
Figure 2-5
Check the Recordings
b. Note the iLogger, iChannel and time of the recording. This will be used later to
troubleshoot the problem. 4. Depending on the type of recording make a note of the following information. Table 2-1: Recording Type Information Recording Type
Information
Recording Type
Information
Trunk/Digital Matrix
Trunk number, time slot, Logger, as well as agent and agent group (for interaction based recording)
Extension-side
Extension number, channel mapped to the extension, Logger, as well as agent and agent group (for interaction based recording)
Passive VoIP
Extension number, IP address, MAC address, Logger, as well as agent and agent group (for interaction based recording)
Active VoIP (Service Observation)
Virtual extension, extension associated with the virtual extension, Logger, as well as agent and agent group (for interaction based recording)
5. For interaction based recording, check the rule: a.
On the Interactions Center open the D:\program files\Nice systems\Interactions Center\BIN\Log\Call Server.log.xxx.txt log files.
Chapter 2: Recording Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
56
Troubleshooting Call Failures using Exceptions
b. The following line indicates that the rule engine is checking if the call should be recorded: 18/07/08 16:04:38.317 DEBUG [Command Thread, 5680]: RecordingDecisions::Selector: Recieved a call for decision: Agent_ID=179389, Extension=25938, Switch_ID=25, Direction=0, Dnis=17209, Dialed_Num=, CLI=7172472759 ,BusinessData= c.
One of the following lines appears after the rule check to see if the call matches any of the active recording rules:
•
If the call does not match any of the active recording rules: 18/07/08 16:04:38.317 DEBUG [Command Thread, 5680]: CALLSRVR: No Recording Command Exist for AgentID 179389, Ext 25938
a.
If the call does match one of the active recording rules: 18/07/08 15:58:34.662 INFO [Command Thread, 5680]: Start record Request to RCM.
b. If the call is not recorded because it does not match one of the active rules, check in Rules
Manager that the Scheduler rule that is recording this agent or group of agents is valid and active. 6. Connect to the Logger using the Remote Desktop and check if there are any errors (use mstsc /console and not mstsc). See the High Density Logger Troubleshooting Flow. For the VoIP Logger, also check the IPCaptureLog.sav file. This contains the first error in the current
session, or if there are no errors in the current session, the first error in the last session when an error occurred. See the VoIP Troubleshooting Flow. 7. For trunk-side recording: a.
Use the Functional Tester to check the trunk status.
b. If there is a problem with frame loss, trunk not synchronized, etc, use the NiceLog Setup
Tool to solve the problem. See the Using the Nicelog Setup Tool in the High Density Logger Troubleshooting Flow. 8. For extension-side recording: a.
Make a call from the extension, speak and leave the call open.
b. Use the Functional Tester to check which channel has activity. c.
If the activity is on a channel different to that noted in Step 4, check the channel mapping.
d. If there is no activity, check the Logger. See to the High Density Logger Troubleshooting
Flow. 9. For VoIP recording: a.
Use the Functional Tester to check which channel has activity. If there is no channel activity, the problem is most likely a VoIP Logger problem.
b. Use the sniffer to check the RTP packets. See Checking the RTP Packets Using the Sniffer
in the VoIP Troubleshooting Flow. If there are packets, check the sniffing NIC configuration in the Logger IP tool.
Chapter 2: Recording Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
57
Troubleshooting Call Failures using Exceptions
c.
If there are no packets, troubleshoot the network and environment. See the VoIP Logger Troubleshooting Flow. In particular check:
•
Passive VoIP: Check the mirroring and network topology.
•
Active VoIP: Check that the NIC IP is routable from the source IP address, and verify that the port intervals configured in the Logger IP Tool are open on the firewall for UDP packets.
10. For Avaya DMCC active VoIP recording: a.
Check in the Observer logs that the virtual extension is registered to the correct extension.
b. If the correct extension was used, run the sniffer to check the packets. c.
If the correct extension was not used, check the configuration.
Chapter 2: Recording Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
58
Troubleshooting No New Calls in Business Analyzer
Troubleshooting No New Calls in Business Analyzer 1. Checking free space in the NICE_Interactions database. If required, increase the database
size, see the Databases Troubleshooting Flow. a.
In the Start menu, select Programs > Microsoft SQL Server 2005 > SQL Server Management Studio, and click Connect.
b. In the Object Explorer, expand Databases, right-click on the relevant database, select Properties, and check the Space available. This provides the space of the entire DB with
no distinction between the data size and log size. c.
Right-click the database, select Reports > Standard reports > Disk usage. The usage is displayed graphically with the separation for data and log.
d. In System Administrator, expand the Master Site > Database Servers > Databases
and select each database to see the used and free space percentages. Figure 2-6
Database Size
2. Check the NMS for trap 1023. To find the reason for the failure to insert the record, the DBSrvr log files (on the Interactions Center in D:\program files\Nice systems\Interactions Center\BIN\Log\DBServer.log.xxx.txt files) should be backed up
and sent to NICE Customer Support. The metadata for the Call might be inserted into the database by the TRS (if it exists), but in order to ensure that this is the case, NICE Customer Support should be consulted. 3. Check in the Driver window that calls are being sent to the CLS by looking for Establish
events. 13699296: 13699296: 13699296: 13699296: 13699296: 13699296: 13699296: 13699296: 13699296: 13699296: 13699296: 13699296:
[07/21/06 18:37:49] DELIVERED TO APPLICATION: MonitorCrossRefID 63 EstablishedEvent ::= { establishedConnection { callID 15008, deviceID "5204", devIDType 0 }, answeringDevice Chapter 2: Recording Troubleshooting Flow
NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
59
Troubleshooting No New Calls in Business Analyzer
13699296: 13699296: 13699296: 13699296: 13699296: 13699296: 13699296: 13699296: 13699296: 13699296: 13699296: 13699296: 13699296: 13699296: 13699296: 13699296:
{ deviceID "5204", deviceIDType 55, deviceIDStatus 0 }, callingDevice { deviceID "97228776", deviceIDType 55, deviceIDStatus 0 }, calledDevice { deviceID "1691", deviceIDType 55, deviceIDStatus 0
4. Check the Call Server log files (on the Interactions Center in D:\program files\Nice systems\Interactions Center\BIN\Log\Call Server.log.xxx.txt files) for the call reported event. For more information on troubleshooting using log files, see the NICE Interactions Center Troubleshooting Flow.
•
Insert Interaction to DB InteractionData *********** properties: ======== LocalStartTime = 17/02/2008 13:16:34 . LocalStopTime = 17/02/2008 13:16:39 . GMTStartTime = 17/02/2008 11:16:34 . GMTStopTime = 17/02/2008 11:16:39 . OriginatorInteractionID 11269314 . members: ======== InteractionId : 11269314 interactionOriginalID : 11269314 Duration : 00:00:05 OpenReasonId : 128 CloseReasonId : 128 SwitchId : 1 InitiatorUserId : 10 QATypeId : False InteractionTypeId : 2 RecordedTypeId : 0 TypeData : CallInteractionData *********** properties: ======== members: ======== PbxCallIndex :
=
11269314 Chapter 2: Recording Troubleshooting Flow
NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
60
Troubleshooting No New Calls in Business Analyzer
CallDirectionTypeId : 1 CompoundId : 11269311 IsPlaybackCall : False ********************* IsEvaluated : False iMediaTypesId : 2 additionalDataCollection[0] :
1
participantsArray[0] : ParticipantData *********** properties: ======== members: ======== ParticipantId : 11269312 Station : 1 SwitchId : 1 IsFirstUser : True IsInteractionInitiator : False DeviceTypeId : 0 DeviceId : 0 ParticipantTypeId : 1 UserId : 10 ********************* participantsArray[1] : ParticipantData *********** properties: ======== members: ======== ParticipantId : 11269313 Station : 144 SwitchId : 0 IsFirstUser : False IsInteractionInitiator : True DeviceTypeId : 0 DeviceId : 0 ClientId : 0 ParticipantTypeId : 2
Chapter 2: Recording Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
61
Troubleshooting No New Calls in Business Analyzer
********************* recordingsArray[0] : RecordingData *********** properties: ======== RecordingGMTStartTime = 17/02/2008 13:51:26 . RecordingGMTStopTime = 17/02/2008 13:51:29 . members: ======== Logger : 10225 Channel : 5 iMediaTypeId : 2 InitiatorTypeId : 64 InitiatorUserId : 0 RecordingSideTypeId : 3 RecordingRecordedTypeId : 3 RecordedParticipantId : 11269312 TimeDiff : -02:34:50.2600000 WrapupTime : 00:00:00 m_iTransientIndex : 0 ********************* recordingsArray[1] : RecordingData *********** properties: ======== RecordingGMTStartTime = 17/02/2008 13:51:26 . RecordingGMTStopTime = 17/02/2008 13:51:29 . members: ======== Logger : 10225 Channel : 5 iMediaTypeId : 2 InitiatorTypeId : 64 InitiatorUserId : 0 RecordingSideTypeId : 3 RecordingRecordedTypeId : 3 RecordedParticipantId : 11269313 TimeDiff : -02:34:50.2600000 WrapupTime : 00:00:00 m_iTransientIndex : 0 *********************
Chapter 2: Recording Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
62
Troubleshooting No New Calls in Business Analyzer
*********************
•
Insert Contact (If Compound Ended) ContactData *********** properties: ======== GMTStartTime = 17/02/2008 11:16:34 . GMTStopTime 11:16:39 . members: ======== ContactId : 11269311 Duration : 00:00:05 OpenReasonId : 256 CloseReasonId : 256 TransferSiteId : 0 TransferContactId : 0 RecordedTypeId : 0 QATypeId : False DirectionTypeId : 1 IsEvaluated : False iMediaTypesId : 2 InitiatorSwitchId : 1 InitiatorStation : 1 InitiatorUserId : 10 m_attachedInteractionInfo : InteractionUpdatableData *********** properties: ========
=
17/02/2008
members: ======== m_interactionsInfo[0] : InteractionUpdatableInfo *********** properties: ======== InteractionId = 11269314 . QATypeId False . members: ========
=
False . IsOverwrite
=
********************* ********************* ********************* Chapter 2: Recording Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
63
Troubleshooting No New Calls in Business Analyzer
5. Check the Call Server log files (on the Interactions Center in D:\program files\Nice systems\Interactions Center\BIN\Log\DBServer.log.xxx.txt files) for the call inserted
event. For more information on troubleshooting using log files, see the NICE Interactions Center Troubleshooting Flow.
Chapter 2: Recording Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
64
Troubleshooting No New Calls in Business Analyzer
6. Check if the call was inserted into the database table: a.
Query the tblInteractionCatalog table to check for which period the calls occur.
b. Check the tblInteractionXX and tblRecordingXX tables. 7. If the calls are in the database the problem may be in the Business Analyzer query: a.
Try to use the NICE user when querying.
b. Ensure that you do not have any expression or filter applied to the query. c.
Collect the logs from the Applications Server.
Chapter 2: Recording Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
65
Troubleshooting Calls with Poor Audio
Troubleshooting Calls with Poor Audio 1. If VoIP calls are recorded with poor audio, see the VoIP Troubleshooting Flow. 2. For extension-side recording, check the connections between the handset, PABX, ALI board
and Punch Down Block (Krone). 3. For trunk-side recording, check for an error on the channel using the Functional Tester.
Chapter 2: Recording Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
66
Troubleshooting Calls with Audio from Several Calls
Troubleshooting Calls with Audio from Several Calls The following symptoms may occur:
•
Call duration equals the "op_MaxOpenCallDuration" parameter (by default 5 hours).
•
tblException shows INF002, EXP002, Exp005, INF029.
•
Calls have the wrong voice, calls are cut off or start from the middle and this is not a mapping problem.
To troubleshoot the problem: 1. Check the integration troubleshooting in the relevant Integration Guide. 2. Ensure that all devices are monitored. Some integrations require that all ACDs, VDNs, hunt
group IVRs be monitored. 3. Ensure that the latest Update is installed. 4. Check what information is required to open a case as described in the Integration Guide. 5. Check which call to analyze: a.
Business Analyzer, note the Interaction ID (XXXX) of the call.
b. In the database expand nice_interactions > Tables. c.
Query the tblInteractionCatalog table to check for which period the calls occur. Figure 2-7
tblInteractionCatalog Table
d. Check the Compound ID for this call, by running the following query: select iCompoundID from tblCallInteractionXX where iInteractionID = 'XXXX' e.
Check how many calls belong to the Compound ID, by running the following query: select * from tblCallInteractionXX where iCompoundID = 'XXXX'
6. If there are multiple results it means that the compound was wrongly associated with other
calls. Focus on the first call and note the: Chapter 2: Recording Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
67
Troubleshooting Calls with Audio from Several Calls
•
PBX Call ID: select IPBXCallID from tblCallInteraction where iInteractionID = 'XXXX'
•
Compound ID select IcompoundID from tblCallInteraction where iInteractionID = 'XXXX'
•
Station: select nvcStation from tblParticipantXX where iInteractionID = 'XXXX'
•
Switch ID: select iSwitchID from tblParticipantXX where iInteractionID = 'XXXX'
•
Time of the call: select dtInteractionLocalStarttime from tblInteractionXX where iInteractionID = 'XXXX'
7. Investigate the call events: init: The caller picks up the phone. Delivered: The call between the two parties is ringing. Establish: The recipient picks up the phone. Connection clear: The call is over. Transfer: The call is transferred from one agent to another. Held / Retrieve: One of the parties pressed hold or retrieved the call. a.
Look for the Call ID and time in the driver and CTI logs (D:\NICECTI\Integration\Log).
b. Look for the device number and try to follow the flow of events up to the connection clear
or transfer. If there is no connection clear or transfer, then it is a third party problem or not all devices are being monitored (see Step 2). c.
If there is a transfer event then there will be a new call ID, so continue the check using the device number.
Chapter 2: Recording Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
68
Troubleshooting TRS Recordings
Troubleshooting TRS Recordings 1. Switch the log reporting level to debug: a.
In System Administrator, verify that you are working in Technician mode.
b. In the Organization tree, select CLS Definitions > CLS Server. Click the Report Level
tab. c.
Expand TRS.
d. Set the File to Debug. Do not set the Screen to Debug as this will affect performance. e.
Repeat for Call Server and RCM.
2. Switch the driver logs to Debug: a.
In System Administrator, verify that you are working in Technician mode.
b. In the Organization tree, select CTI Integrations > Drivers, and select the relevant
driver. c.
Expand the Driver Reporting Level.
d. Select the desired row and click Edit. e.
Select the checkboxes for the reporting levels you want to include, and click OK.
3. Check the volume of TRS calls. In Reporter, run the TRS Interaction Volume Report and TRS Interaction Volume by Logger Report . In order to identify the TRS status, this report
presents the interaction volume up until to the latest TRS run. Figure 2-8
TRS Interaction Volume Report
4. Are there too many TRS calls? Continue with step 15.
Chapter 2: Recording Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
69
Troubleshooting TRS Recordings
5. If there are fewer calls than expected or no calls, check if the CLS/Driver was down for an
extended period while the Logger was recording. 6. If there are fewer calls than expected or no calls, check if the problem is related to specific
Loggers by checking the TRS Interaction Volume by Logger report. If it is a general problem, continue with Step 9. 7. Check the connectivity between the Loggers and the Interactions Center: a.
Ping the Loggers and Interactions Center.
b. Check if Trap ID 1095 clsTRSFailed QueryLogger or 1096 clsTRSSucceededQueryLogger appears. For further details, see the NMS User's
Guide. c.
Check the TRS logs (on the Interactions Center in D:\program files\Nice systems\Interactions Center\BIN\Log\TRS.log.xxx.txt files).
d. Check the Logger logs (in D:\NTLogger\logger\log\LogFilex.DAT). See the High
Density Logger Troubleshooting Flow. 8. Check if the problematic Loggers are associated with the same Interactions Center. If the
problem is related to a specific Interactions Center: a.
Check if the NICE CLS TRS service is started.
b. Check the TRS traps. For further details, see the NMS User's Guide. Table 2-2: TRS Traps Trap ID
Description
Recommended Action
1089 clsTRSisDown
The TRS service is not running. Interactions Center Module: TRS
Check the SNMP messages and the TRS log files and try to determine the reason that the service is down. Check if the service was closed down manually.
1090 clsTRSIsUp
TRS Service has started and initialized properly. Interactions Center Module: TRS
1093 clsTRSFailed toQueryCLS DB
The TRS has failed to execute an SQL command. Interactions Center Module: TRS
Check if the SQL Server is up and running. Verify that the server machine where the Interactions Center runs has connection to the server machine where the nice_interactions database is installed. Analyze the TRS log files.
Chapter 2: Recording Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
70
Troubleshooting TRS Recordings
Table 2-2: TRS Traps (Continued) Trap ID
Description
1094 clsTRS SucceededTo QueryCLSDB
TRS successfully connected and queried the nice_interactions database.
1095 clsTRSFailed QueryLogger
Indicates a communication problem with the Logger.
Verify that the Logger is up and running.
Interactions Center Module: TRS
Verify that the Interactions Center is properly connected to the Logger. Check if RCM also has a communication problem with the same Logger.
1096 clsTRS Succeeded QueryLogger
TRS successfully connected and queried the Logger.
c.
Recommended Action
Interactions Center Module: TRS
Interactions Center Module: TRS
In System Administrator, check the TRS configuration.
d. Check the TRS logs (on the Interactions Center in D:\program files\Nice systems\Interactions Center\BIN\Log\TRS.log.xxx.txt files). 9. If the problem is not related to a specific Interactions Center: a.
Change the Logger log levels to debug:
•
Open d:\NTlogger\logger\config\SysRC.dat in Notepad.
•
In [system -default] set the LogeLevel2File and LogLevel2Screen to 5 instead of 3.
•
Reboot the Logger. See Shutting Down and Starting Up the Logger in the High Density Logger Troubleshooting Flow.
b. Check that the Logger is up. c.
Check the Logger logs. See the High Density Logger Troubleshooting Flow.
10. Check that the Data Mart Population Schedule job ran: a.
In System Administrator, expand the Master Site > Data Marts and select the required Data Mart server.
b. Click the Monitor tab. c.
In the SQL Jobs area, select the Data Mart Population Schedule job. The job steps are detailed in the History area.
Chapter 2: Recording Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
71
Troubleshooting TRS Recordings
Figure 2-9
Data Mart Population Schedule Job
11. Collect the relevant log files using Perform Mate including the Driver, Call Server, RCM,
TRS, DBServer and any problem Logger logs. See the High Density Logger Troubleshooting Flow and NICE Interactions Center Troubleshooting Flow. 12. Collect the following data tables:
•
tblInserterLoggers
•
tblInserterCalls (relevant part only):
13. In System Administrator, check the TRS parameters: a.
In the Organization tree, select CLS Definitions > CLS Server.
b. Select the required CLS Server, and click the Configuration tab. c.
Click
next to TRS. Note the parameter values.
IMPORTANT Any change of parameters should be done ONLY in consultation with NICE Support. d. For a list of the parameters see the NICE Interactions Center Troubleshooting Flow. 14. Check if the TRS parameters were modified: a.
In Audit Trail create and run the following advanced query.
Chapter 2: Recording Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
72
Troubleshooting TRS Recordings
Figure 2-10
Advanced Query
b. Check if the parameter value has changed, and what the previous and new values are. Figure 2-11 Audit Trail
15. If there are too many TRS calls: a.
In Business Analyzer, check if the same calls appear as inserted by both the Interactions Center (an agent name appears) AND the TRS (User- Unmapped appears). If this is the case, provide a list with these calls with their recording times, Logger ID and channel ID.
b. In Reporter, run the TRS Interaction Volume Report and TRS Interaction Volume by Logger Report in order to identify whether the problem is related to a specific TRS or
specific Loggers.
Chapter 2: Recording Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
73
Troubleshooting TRS Recordings
c.
If the problem is related to a specific Logger:
•
Run a query in Business Analyzer to see whether the increase is for a specific channel.
•
Try to playback several of these calls and see whether the calls are invalid, noisy, ring tones, DTMF or IVR calls.
d. If the increase is not related to a specific Logger, check the following:
•
Check for driver problems, for example, internal errors, disconnections between the drivers and the PABX or internal PABX, and so on. See the troubleshooting in the relevant Integrations Guide.
•
Check for Interactions Center or Interactions Center subcomponents problems. For further information, see the NICE Interactions Center Troubleshooting Flow.
e.
Collect the relevant log files using Perform Mate including the Driver, Call Server, RCM, TRS, DBServer and any problem Logger logs. See the High Density Logger Troubleshooting Flow and NICE Interactions Center Troubleshooting Flow.
f.
Collect the following data tables:
•
tblInserterLoggers
•
tblInserterCalls (relevant part only):
16. Switch all log levels (CLS Server, Drivers and Loggers) back to what they were before you
started this troubleshooting procedure.
Chapter 2: Recording Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
74
Troubleshooting Integration Issues
Troubleshooting Integration Issues Various tools and utilities for various additional troubleshooting and performance issues are detailed in the relevant Integration Guide:
•
NICE Events Spy enables you to trace events after they were transferred from the PABX to the Connection Manager, enabling you to detect bugs or malfunctions.
•
NICE Debug Service enables you to gather data critical for solving problems stemming from
the transfer of events between the switch and the Connection Manager.
•
NICE Connection Manager Monitor tool enables you to view the contents of the
Connection Manager's tables. It also enables you to verify if:
•
•
Devices are monitored
•
Monitored devices are filtered
•
Displays the loaded CTI links
•
Displays connected clients.
Log Manager system logs all significant system activity and maintains a log of all data,
enabling you to view the history of all relevant system activity. The Log Manager system has three main components:
•
•
Log Manager
•
Log Manager Services
•
Log Viewer
CAPI Spy enables you to monitor all messages sent by the CTI driver to the CLS CAPI (Call
Server). Examination of these messages enables you to pinpoint whether the problem is in the CTI driver or in the CLS CAPI server. CAPI Spy has two main components:
•
CAPI Spy Plug-in
•
CAPI Spy Utility
Chapter 2: Recording Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
75
Blank page for double-sided printing.
3 Playback Troubleshooting Flow This chapter describes the playback flow and common troubleshooting issues when playback is not functioning properly.
Contents Gathering Information .................................................................................................... 78 Understanding the Playback Flow ................................................................................ 79 General Troubleshooting Flow ...................................................................................... 81 Troubleshooting Playback Problem for All Calls......................................................... 83 Troubleshooting Playback Problem for Specific Calls ............................................... 85 Setting the Log Level in System Administrator........................................................... 87 Setting the Log Level in the .Config File ...................................................................... 88 Setting the Player Log Levels ....................................................................................... 90 Checking the Workstation Ports ................................................................................... 92 Checking the File Versions ............................................................................................ 92 Checking the Locate Order............................................................................................ 94 Checking the Storage Center ........................................................................................ 95 Troubleshooting Player Errors ...................................................................................... 96 Example of Troubleshooting using the Logs.............................................................108 Example of Troubleshooting using the Logs.............................................................108 Collecting the Server Logs using the NICE Perform Logs Collector Tool .............. 113 Collecting the Client Logs using NICE Perform Logs Collector Tool...................... 113 Copying the Playlist ..................................................................................................... 114 Dumping the Client Memory ........................................................................................ 115 Understanding Playback Retrieval.............................................................................. 116 Other Troubleshooting ................................................................................................. 116 Chapter 3: Playback Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
77
Gathering Information
Gathering Information Important issues to consider when troubleshooting for playback misfunction are:
•
Is this a new installation? If yes, when was the installation executed?
•
Were there any changes to the environment that might have affected the system? If yes, check these changes first.
•
How often does the problem occur?
•
Can the problem be retraced? If yes, specify the steps.
•
Were the Servers restarted? If yes, did this solve the problem?
Chapter 3: Playback Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
78
Understanding the Playback Flow
Understanding the Playback Flow The Playback Flow is described in the chart and procedure below. Figure 3-1
Playback Flow
1. The user selects an interaction in Business Analyzer (Client) and clicks Play. 2. The Business Analyzer (Client) sends the playback request to the Player. 3. The Player sends the playback request, including the call parameters (site ID, call ID, segment
or complete, user ID of users requesting playback, user ID of the agent to play back) to the Playback Administration on the Applications Server. 4. The Playback Administration checks the privileges with the User Admin to ensure that the
user is allowed to playback the interaction. 5. The Playback Administration then gathers the interaction data from the Database Server:
•
tblRecordingXX
• •
tblInteractionXX
•
tblContacXX for complete calls
tblParticipantXX
6. The Playback Administration uses tblRecordingXX to locate where the interaction is stored
based on the locate order. NOTE: Recordings with Recording Type = Not Recorded are ignored.
Chapter 3: Playback Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
79
Understanding the Playback Flow
7. The Playback Administration gathers the locate results. 8. The Playback Administration builds the playlist (xml) for each located media, including the
start and end times and where the media is stored. 9. The playlist is sent to the Player. If the playlist contains no medias, the user receives a No recordings were found message. See Understanding Playback Retrieval on page 116 for
an explanation of this message. 10. The NICE Player eliminates any redundant medias according to the participants. 11. The NICE Player/Stream Server fetches the media from the Loggers and/or Storage Center,
and plays the call.
Chapter 3: Playback Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
80
General Troubleshooting Flow
General Troubleshooting Flow General troubleshooting is described in the flowchart and procedures below. Figure 3-2
Troubleshooting Flow
1. What method of playback are you using?
•
Playing back to an extension.
•
Playing back directly from the Logger or Storage Center, that is, with a direction connection between the Player and the Logger or Storage Center.
•
Playing back via a Stream Server. See Choosing a Stream Server Configuration in the System Administrator's Guide.
2. If your problem is playing back to an extension, see Playback Telephony Server Services
Troubleshooting. 3. Is the problem playing back a specific call or all calls? Chapter 3: Playback Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
81
General Troubleshooting Flow
4. If the problem is for all calls, see Troubleshooting Playback Problem for All Calls
on page 83. 5. If the problem is for specific calls only, see Troubleshooting Playback Problem for Specific
Calls on page 85 6. If the site is a Multi Data Hub environment:
•
Check if the agent and the Logger are located in the same site and physical location.
•
Try to playback the call locally. If this works, the problem is most likely a network issue, for example, timeout, latency, and so on.
7. If you have N+1 or 1+1 the call may be recorded on a couple of Loggers:
•
Check where the call is physically located in locatelog.txt. See Example of Troubleshooting using the Logs on page 108.
Chapter 3: Playback Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
82
Troubleshooting Playback Problem for All Calls
Troubleshooting Playback Problem for All Calls Troubleshooting playback problems for all calls is described in the flowchart and procedures below. Figure 3-3
Troubleshooting Flow for Playing Back All Calls
Chapter 3: Playback Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
83
Troubleshooting Playback Problem for All Calls
If the problem is for all calls: 1. Set the appropriate log level:
•
Set the Log Level in the System Administrator. Setting the Log Level in System Administrator on page 87.
•
Set the Log Level in the Config File. See Setting the Log Level in the .Config File on page 88.
•
Set the Player Log Level. See Setting the Player Log Levels on page 90.
2. Check that the Applications Server and Database Servers are up. See Applications
Troubleshooting Flow on page 197 and Databases Troubleshooting Flow on page 253. 3. Check the time synchronization between the Applications Server and Database Server. 4. If playing back over a WAN, check if there is a latency problem or communication errors. 5. If other users can play back the calls on the same workstation when logging in, check the user
permissions in the User’s Admininstrator application. 6. If you can play back the calls when logged-in to a different workstation, check the workstation
connectivity and also check workstation problems. See Solving Desktop Problems on page 217. 7. In System Administrator, check the range of the workstation ports, and ensure that these ports
are open. See Checking the Workstation Ports on page 92. 8. On the Applications Server, check for a conflict between the NICE updates in the ServerBin
and ClientBin folders. See Checking the File Versions on page 92. 9. If a No Recording Found message appears for the calls in the Player, check the Locate Order. See Checking the Locate Order on page 94. 10. Check that the Storage Center is archiving. See Checking the Locate Order on page 94 and
Checking the Storage Center on page 95. 11. If an error message other than No Recording Found message appears in the Player, see
Troubleshooting Player Errors on page 96. 12. Troubleshoot using the Playback Administration Service Logs. See Example of
Troubleshooting using the Logs on page 108. 13. If this is an encrypted environment, see Media Encryption Troubleshooting Flow
on page 151. 14. Submit information to NICE Customer Support:
•
Dump the Client Memory. See Dumping the Client Memory on page 115.
•
Copy the Playlist. See Copying the Playlist on page 114.
•
Collect the Server Logs using the NICE Perform Logs Collector Tool. See Collecting Information Using the Log Collector on page 537.
•
Collect the Client Logs using the NICE Perform Logs Collector Tool. See Collecting Information Using the Log Collector on page 537. Chapter 3: Playback Troubleshooting Flow
NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
84
Troubleshooting Playback Problem for Specific Calls
Troubleshooting Playback Problem for Specific Calls Figure 3-4
Troubleshooting Flow for Playing Back Specific Calls
If the problem is for specific calls: 1. Set the appropriate log level:
•
Set the Log Level in the System Administrator. See Setting the Log Level in System Administrator on page 87.
•
Set the Log Level in the Config File. See Setting the Log Level in the .Config File on page 88. Chapter 3: Playback Troubleshooting Flow
NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
85
Troubleshooting Playback Problem for Specific Calls
•
Set the Player Log Level. See Setting the Player Log Levels on page 90.
2. If other users can play back the calls on the same workstation when logging in, check the user permissions in User Admininstration. 3. Is there anything unique about the calls? (for example, were they all recorded on the same
Logger, or at the same time, or are they recordings of the same agent or phone number). 4. If the problems relate to a specific remote Logger, check for Firewall or network issues, for
example DNS configuration. See High Density Logger Troubleshooting Flow on page 303. 5. If a No Recording Found Message appears for the calls in Player, check the Locate Order.
See Checking the Locate Order on page 94. 6. Check that the Storage Center is archiving. See Checking the Storage Center on page 95. 7. If an error message other than a No Recording Found message appears in the Player, see
Troubleshooting Player Errors on page 96. 8. Troubleshoot using the Playback Administration Service Logs. See Example of
Troubleshooting using the Logs on page 108. 9. If this is an encrypted environment, see Media Encryption Troubleshooting Flow
on page 151. 10. Submit information to NICE Customer Support:
•
Dump the Client Memory. See Dumping the Client Memory on page 115.
•
Copy the Playlist. See Copying the Playlist on page 114.
•
Collect the Server Logs using the NICE Perform Logs Collector Tool. See Collecting Information Using the Log Collector on page 537.
•
Collect the Client Logs using the NICE Perform Logs Collector Tool. See Collecting Information Using the Log Collector on page 537.
Chapter 3: Playback Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
86
Setting the Log Level in System Administrator
Setting the Log Level in System Administrator To set the log level in System Administrator: 1. In System Administrator, verify that you are working in Technician mode: In the Organization tree, select Organization. Then mark the Technician Mode checkbox and click Save. 2. In the Organization tree, select Applications > Configurations > General. 3. Select the Logs Level tab. Figure 3-5
Logs Level Tab
4. In the Master Level and File Level columns, note the current log level for Playback Administration and Playback Locate. 5. In the Master Level and File Level columns, change the log level for Playback Administration and Playback Locate to Debug. 6. After troubleshooting ensure that you reset these values according to the values you noted in
Step 4 on page 87. WARNING Failure to switch the log levels back to the default values might cause system problems due to a performance overload.
Chapter 3: Playback Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
87
Setting the Log Level in the .Config File
Setting the Log Level in the .Config File Each of the components on the Applications Server has .config files stored in the following folder: <Application Server Installation Path>\ServerBin\.
Playback Administration
NiceApplications.Playback.Administration.Startup.exe. config
Loggers
LocateService PBResourceManager
IMPORTANT Backup your .config files before editing them. If you edit the configuration files, ensure that you use a plain text editor such as Notepad without any options. DO NOT use text wrap as this will corrupt your configuration file. In any event, any change of configuration files should be done ONLY in consultation with NICE Support!!!!.
To set the log level in the .config file: 1. Navigate to <Application Server Installation Path>\ServerBin\. 2. In Notepad, open the relevant .config file. 3. Expand the
section. 4. Expand the required logger name sections (, and ). 5. Note the values in both Level and Threshold. 6. Set the debug level, by changing the values on both Level and Threshold to be defined as ALL.
Chapter 3: Playback Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
88
Setting the Log Level in the .Config File
Figure 3-6
Log Levels in the .config File
7. Save and close the .config file. There is no need to restart the Applications Server. 8. After troubleshooting ensure that you reset these values according to the values you noted in
Step 5 on page 88. WARNING Failure to switch the log levels back to the default values might cause system problems due to a performance overload.
Chapter 3: Playback Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
89
Setting the Player Log Levels
Setting the Player Log Levels You need to set the Player log level in System Administrator and in the NiceApplications.Playback.GUI.exe.config file to enables playback logic logging. For further information, see Troubleshooting Player Errors on page 96. To set the Player log levels: 1. To set the Player log levels in System Administrator: a.
Verify that you are working in Technician mode.
b. In the Organization tree, select Playback > Player Settings. c.
Click the Log Level tab. Figure 3-7
Log Level Tab
d. Note the current Log Level setting. e.
Set the Log Level to Debug.
f.
Logout and then login for the change to take effect.
g. After troubleshooting ensure that you reset these values according to the values you noted
in Step d on page 90. IMPORTANT Failure to switch the log levels back to the default values might cause system problems due to a performance overload.
To create a log file of the playback logic errors: 1. To create a log file of the playback logic errors in the NiceApplications.Playback.GUI.exe.config config file: a.
Set the EnablePBLogicLog value to true:
b. In the Registry Editor, navigate to: HKEY_LOCAL_MACHINE\SOFTWARE\Nice Systems\PBLogic\Log, and note the values of the MMLClient_LogLevel and the PBLogic_LogLevel. c.
Set the MMLClient_LogLevel and PBLogic_LogLevel to 0 for maximum details. Chapter 3: Playback Troubleshooting Flow
NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
90
Setting the Player Log Levels
Figure 3-8
Registry Key
d. After troubleshooting ensure that you reset these values.
•
Set the EnablePBLogicLog value to false.
•
In the Registry Editor, reset the MMLClient_LogLevel and PBLogic_LogLevel values according to the values you noted in Step b on page 90.
IMPORTANT Failure to switch the log levels back to the default values might cause system problems due to a performance overload.
Chapter 3: Playback Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
91
Checking the Workstation Ports
Checking the Workstation Ports The settings apply to all Workstations! To check the Workstation ports: 1. In System Administrator, navigate to Applications > Configurations > General. 2. Expand the Workstation Settings area. Then select as follows:
•
Random Ports - all ports on each workstation must be open.
•
Specific Ports Range - a range between 1024 and 5000 (the minimum range is 11 ports.
For example, 2001 - 2011 inclusive).
Checking the File Versions On the Applications Server, check for a mismatch between the NICE updates in the ServerBin folder (contains binaries and DLL files for server-based applications) and the ClientBin folder (contains the binaries and DLL files needed by clients). To check the file versions: 1. Browse to D:\Program Files\Nice Systems\Applications\ServerBin. 2. Check the NiceApplications.Playback.xxx.dll version numbers: a.
In Explorer, right-click on Name, and select More. Figure 3-9
Choose Details Window
b. Select File Version and check OK. c.
Check the file version numbers against the D:\Program Files\Nice Systems\Applications\ClientBin. Chapter 3: Playback Troubleshooting Flow
NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
92
Checking the File Versions
3. If the NICE Playback Telephony Services service or the NICE Playback Streaming
service is installed, ensure that all the required NICE updates are installed. Check the following version numbers:
•
NiceApplications.Playback.Streaming.xxx.dll
•
NiceApplications.Playback.Telephony Services.xxx.dll
Chapter 3: Playback Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
93
Checking the Locate Order
Checking the Locate Order The Locate Order is the order in which all storage media are searched when a query is run. If your site is installed with a Storage Center, you can select to search the Storage Center before the Loggers when running a query. If most of the interactions in your query results are stored on a Storage Center, this will produce faster results. Locate Order can also be defined per Location. If a user is defined on a location which has a Locate Order defined, the locate order of the Location takes precedence over the locate order defined for the site and is used. For more information on the logs, Example of Troubleshooting using the Logs on page 108. Locate Order
Description
Troubleshooting
No Storage Center
Do not search the Storage Center (there is no Storage Center at the site).
If there is a Storage Center and this is incorrectly configured the calls will not be played back.
Before Source Logger
First, search the Storage Center and then search the Loggers. If most of the interactions in your query results are stored on a Storage Center, this will produce faster results.
You will not be able to playback interactions that are not yet archived to the Storage Center if the Logger that they are stored on is down.
After Source Logger
First, search the Logger where the interaction was recorded and then search the Storage Center.
You will not be able to playback interactions that are not yet archived to the Storage Center if the Logger that they are stored on is down.
After All Loggers
Search all the Voice Loggers and then search the Storage Center.
Storage Center Only
Search only the Storage Center and not the Loggers.
You will not be able to playback interactions that are not yet archived to the Storage Center.
Customized Order
You define which Loggers to search, and in what order to search them.
If a Logger is missing from the customized order you will be unable to playback those interactions.
The source and spare Logger will always be checked first, prior to this customized order.
Chapter 3: Playback Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
94
Checking the Storage Center
Checking the Storage Center Check if the Storage Center is archiving the calls. To check the Storage Center: 1. Check the directories on the Storage Center to check if the calls are archiving there. 2. Check the archiving to the Storage Center. a.
In Reporter, import the Health Checks Report template.
b. Run the Archiving Coverage Report. c.
Check the Archiving Coverage per Storage Center View.
Figure 3-10
Archiving Coverage Report
3. To further troubleshoot Storage Center problems, see Storage Center Troubleshooting Flow
on page 513.
Chapter 3: Playback Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
95
Troubleshooting Player Errors
Troubleshooting Player Errors The player errors are recorded in the Player log files, *.txt, in %APP_DATA%\NicePlayer\Release3. There are various types of errors that may occur in the Player:
•
Playback errors
•
Logger errors
•
Storage Center errors
Playback Logic Errors The following playback logic errors may appear in the Player Logs. These errors have the MMLAPIBase.PbLogicLog prefix. Error
No.
Description
Action Required
OK
1
OK.
None
Fail
-1
General failure.
Collect the Player logs and submit them to NICE Customer Support. Refer to Collecting the Client Logs using NICE Perform Logs Collector Tool on page 113.
Illegal parameter
-3
One or more of the input parameters has an invalid value.
Collect the Player logs and submit them to NICE Customer Support. Refer to Collecting the Client Logs using NICE Perform Logs Collector Tool on page 113.
Improper mode
-4
The playback logic is not in a proper mode to perform the requested operation.
Collect the Player logs and submit them to NICE Customer Support. Refer to Collecting the Client Logs using NICE Perform Logs Collector Tool on page 113.
Connection failed
-5
Failed to connect to (or disconnect from) the NiceScreen Logger or the Storage Center. NiceScreen Loggers are sometimes referred to as Multi Media Loggers (MML).
Check the connectivity to the Storage Center or the NiceScreen Logger. Troubleshoot the Storage Center or NiceScreen Logger. See the Storage Center Troubleshooting Flow on page 513 or the NiceScreen Troubleshooting Flow on page 489.
Chapter 3: Playback Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
96
Logger Errors
Error
No.
Description
Action Required
End of requested segment
-7
No more buffers are available for the requested segments.
Collect the Player logs and submit them to NICE Customer Support. See the Collecting the Client Logs using NICE Perform Logs Collector Tool on page 113.
No packets yet (for monitoring)
-8
The segment exists but its next requested buffer was not found.
Collect the Player logs and submit them to NICE Customer Support. See Collecting the Client Logs using NICE Perform Logs Collector Tool on page 113.
Segment not found
-9
The requested segment was not found in the NiceScreen Logger or the Storage Center.
Collect the Player logs and submit them to NICE Customer Support. See Collecting the Client Logs using NICE Perform Logs Collector Tool on page 113.
Logger Errors The following Logger errors may appear in the Player Logs. Error No.
Description
-9
Handler is invalid.
-1
Communication failure.
0
Operation successful.
1
Command failed.
11
Invalid parameter, for example, invalid tag.
12
Token does not exist.
57
Not enough resources for playback.
200
Audio Over LAN & Import: Invalid token
201
Audio Over LAN & Import: DB error or record aborted.
Action Required
Failed to connect to the Logger. To check this, ping the Logger from the client, and check any Firewall settings between the Logger and client.
Chapter 3: Playback Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
97
Logger Errors
Error No. 203
Description
Action Required
Audio Over LAN & Import: No free virtual output channel.
The Logger has no free connections. Each media uses a connection to the Logger. To check this, wait a while and then retry playing back the interaction.
Chapter 3: Playback Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
98
Storage Center Errors
Storage Center Errors The following Storage Center errors may appear in the Player Logs. Error No.
Description
Action Required
300007
The Storage Center failed to connect to a backup device on a Logger. This should not be considered an error if there are no backup devices in the site.
Check the Logger where the backup device is located.
300008
The Storage Center did not archive or it had an error in the locate flow.
See:
•
Checking the Locate Order on page 94
•
Checking the Storage Center on page 95
•
Setting the Player Log Levels on page 90
•
Storage Center Troubleshooting Flow on page 513.
Checking the Playback Logs The server log files are found in the <Application Server Installation Path>\ServerBin\ folder unless specified otherwise. PlaybackAdministration.txt.x Playback Administration
NiceApplications.PlaybackAdministration.SystemFramework.log PlaybackRM.txt.x LocateLog.txt.x
Business Analyzer
NBA.log. NBADefault.log.
Chapter 3: Playback Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
99
Sample Player Logs
Sample Player Logs Sample Log 2008-07-31 13:39:15,781 DEBUG; [MMLAPIBASE.PBLOGICLOG] - DebugInfo - CNMLApi : CNMLApi : NML_ReleaseBuffer() - Released buffer context 20000004. [Issued by component] [ThreadId 12144, ErrorCode 0] 2008-07-31 13:39:15,781 DEBUG; [MMLAPIBase.PbLogicLog] - DebugInfo - CNMLApi : CNMLApi : NML_ReleaseBuffer() - Released buffer context 20000004. [Issued by component] [ThreadId 12144, ErrorCode 0] 2008-07-31 13:39:15,781 DEBUG; [MMLAPIBase.PbLogicLog] - DebugInfo - LogicImp : LogicImp : [I] GetNextPacket() - Got Packet (pContext=536870916, Type=0, Size=4096, TimeTag=29941032.3332150528, Flags=0, HeaderSize=24, DataSize=4096 [ThreadId 12144, ErrorCode 0] 2008-07-31 13:39:15,781 DEBUG; [MMLAPIBase.PbLogicLog] - DebugInfo - LogicImp : LogicImp : [I] GetNextPacket() - Got Packet (pContext=536870916, Type=0, Size=4096, TimeTag=29941032.3332150528, Flags=0, HeaderSize=24, DataSize=4096 [ThreadId 12144, ErrorCode 0] 2008-07-31 13:39:15,781 DEBUG; [MMLAPIBase.PbLogicLog] - DebugInfo - LogicImp : LogicImp : [I] GetNextPacket() - Got Packet (PacketGUID=0-0-0-0-0-0-0-0-0-0-0-0-0-0-0-0, MACSize=0, PacketMAC=) [ThreadId 12144, ErrorCode 0] 2008-07-31 13:39:15,781 DEBUG; [MMLAPIBase.PbLogicLog] - DebugInfo - LogicImp : LogicImp : [I] GetNextPacket() - Got Packet (PacketGUID=0-0-0-0-0-0-0-0-0-0-0-0-0-0-0-0, MACSize=0, PacketMAC=) [ThreadId 12144, ErrorCode 0] 2008-07-31 13:39:15,781 DEBUG; [MMLAPIBase.PbLogicLog] - DebugInfo - LogicImp : LogicImp : <
Storage Center Disconnection Log 2008-09-25 12:33:05,250 INFO ; [ExpandView.OnTmrScaleTick] - Position: 32515 2008-09-25 12:33:05,500 INFO ; [ExpandView.OnTmrScaleTick] - Position: 32765 2008-09-25 12:33:05,750 INFO ; [ExpandView.OnTmrScaleTick] - Position: 33015 2008-09-25 12:33:05,906 ERROR; [MMLAPIBase.GetPacket] m_pbLogic.GetNextPacket failed res = CONNECTIONFAILED 2008-09-25 12:33:05,921 INFO ; [VoiceMMLServer.GetPacket] - screen GetNextPacket failed at NiceApplications.Playback.MediaSourceAPI.MMLAPIBase.GetPacket(Boolean& lastPacket, Byte[]& data, tSecuredPacketParams& packetParams) at NiceApplications.Playback.MediaSourceAPI.VoiceAPI.GetActivePacket(Boolean& lastPacket, Byte[]& audioBuffer, DateTime& startTime, DateTime& stopTime, CompressionType& compression, Guid& guid, UInt32& actualDataPosition, UInt32& actualDataSize) Chapter 3: Playback Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
100
Sample Player Logs
at NiceApplications.Playback.MediaSourceAPI.VoiceAPI.GetPacket(Boolean& lastPacket, Boolean& active, Byte[]& audioBuffer, DateTime& startTime, DateTime& stopTime, CompressionType& compression, Guid& guid, UInt32& actualDataPosition, UInt32& actualDataSize) at NiceApplications.Playback.Streaming.Nice.VoiceMMLServer.GetPacket() 2008-09-25 12:33:05,921 ERROR; [AudioBufferBase.FetchingThreadFunction] screen GetNextPacket failed at NiceApplications.Playback.Streaming.Nice.VoiceMMLServer.GetPacket() at NiceApplications.Playback.MediaServices.Logic.AudioFetcher.GetPacket(Boolean& fetchDone) at NiceApplications.Playback.MediaServices.Logic.CompressedInBuffer.GetPacket() at NiceApplications.Playback.MediaServices.Logic.CompressedBuffer.Pop() at NiceApplications.Playback.MediaServices.Logic.AudioBuffer.GetPacket() at NiceApplications.Playback.MediaServices.Logic.AudioBufferBase.FetchingThreadFunction() 2008-09-25 12:33:05,921 INFO ; [AudioBufferBase.FetchingThreadFunction] - m_state changed to Seek 2008-09-25 12:33:05,921 ERROR; [MMLAPIBase.GetPacket] m_pbLogic.GetNextPacket failed res = CONNECTIONFAILED 2008-09-25 12:33:05,921 INFO ; [VoiceMMLServer.GetPacket] - screen GetNextPacket failed at NiceApplications.Playback.MediaSourceAPI.MMLAPIBase.GetPacket(Boolean& lastPacket, Byte[]& data, tSecuredPacketParams& packetParams) at NiceApplications.Playback.MediaSourceAPI.VoiceAPI.GetActivePacket(Boolean& lastPacket, Byte[]& audioBuffer, DateTime& startTime, DateTime& stopTime, CompressionType& compression, Guid& guid, UInt32& actualDataPosition, UInt32& actualDataSize) at NiceApplications.Playback.MediaSourceAPI.VoiceAPI.GetPacket(Boolean& lastPacket, Boolean& active, Byte[]& audioBuffer, DateTime& startTime, DateTime& stopTime, CompressionType& compression, Guid& guid, UInt32& actualDataPosition, UInt32& actualDataSize) at NiceApplications.Playback.Streaming.Nice.VoiceMMLServer.GetPacket() 2008-09-25 12:33:05,921 ERROR; [AudioBufferBase.FetchingThreadFunction] screen GetNextPacket failed at NiceApplications.Playback.Streaming.Nice.VoiceMMLServer.GetPacket() at NiceApplications.Playback.MediaServices.Logic.AudioFetcher.GetPacket(Boolean& fetchDone) at NiceApplications.Playback.MediaServices.Logic.CompressedInBuffer.GetPacket() at NiceApplications.Playback.MediaServices.Logic.CompressedBuffer.Pop() at NiceApplications.Playback.MediaServices.Logic.AudioBuffer.GetPacket() at NiceApplications.Playback.MediaServices.Logic.AudioBufferBase.FetchingThreadFunction()
Chapter 3: Playback Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
101
Sample Player Logs
2008-09-25 12:33:05,921 INFO ; [AudioBufferBase.FetchingThreadFunction] - m_state changed to Seek 2008-09-25 12:33:05,921 ERROR; [CommonViewManager.ErrorOccurred] ErrorOccurred, description: SCREEN_SERVER_SCREEN_FAILED_TO_GET_NEXT_PACKET 2008-09-25 12:33:05,921 ERROR; [CommonViewManager.ErrorOccurred] ErrorOccurred, description: SCREEN_SERVER_SCREEN_FAILED_TO_GET_NEXT_PACKET 2008-09-25 12:33:05,921 DEBUG; [CommonViewManager.DisableCloseForms] >>DisableCloseForms
Storage Center with Playback Logic Log 2008-09-25 12:58:41,218 WARN ; [MMLAPIBASE.PBLOGICLOG] - Warning MMLPoint::SendBlockingMsgToChannel : timeout expired. ChannelId(1), Opcode(203) [ThreadId 5464, ErrorCode 0] 2008-09-25 12:58:41,218 WARN ; [MMLAPIBase.PbLogicLog] - Warning MMLPoint::SendBlockingMsgToChannel : timeout expired. ChannelId(1), Opcode(203) [ThreadId 5464, ErrorCode 0] 2008-09-25 12:58:41,218 WARN ; [MMLAPIBase.PbLogicLog] - Warning MMLPoint::SendBlockingMsgToChannel : error detected while waiting. ChannelId(1), Opcode(203), lastError(553648143) [ThreadId 5464, ErrorCode 0] 2008-09-25 12:58:41,218 WARN ; [MMLAPIBase.PbLogicLog] - Warning MMLPoint::SendBlockingMsgToChannel : error detected while waiting. ChannelId(1), Opcode(203), lastError(553648143) [ThreadId 5464, ErrorCode 0] 2008-09-25 12:58:41,218 FATAL; [MMLAPIBase.PbLogicLog] FatalError PlaybackPoint::GetBuffer : SendSyncCommandToChannel failed. Channel id(1) [ThreadId 5464, ErrorCode 0] 2008-09-25 12:58:41,218 FATAL; [MMLAPIBase.PbLogicLog] FatalError PlaybackPoint::GetBuffer : SendSyncCommandToChannel failed. Channel id(1) [ThreadId 5464, ErrorCode 0] 2008-09-25 12:58:41,218 FATAL; [MMLAPIBase.PbLogicLog] FatalError - CNMLApi : CNMLApi : NML_GetBuffer() - NMLAPI_PB_GetBuffer() returned 0,0.GetLastError returned 0 [ThreadId 5464, ErrorCode 0] 2008-09-25 12:58:41,218 FATAL; [MMLAPIBase.PbLogicLog] FatalError - CNMLApi : CNMLApi : NML_GetBuffer() - NMLAPI_PB_GetBuffer() returned 0,0.GetLastError returned 0 [ThreadId 5464, ErrorCode 0] 2008-09-25 12:58:41,218 ERROR; [MMLAPIBase.PbLogicLog] Error - CNMLApi : CNMLApi : ConnectionFailureCallback() - Disconnection occured for ClientHandle a86860. [ThreadId 5964, ErrorCode 0] 2008-09-25 12:58:41,218 ERROR; [MMLAPIBase.GetPacket] m_pbLogic.GetNextPacket failed res = ConnectionFailed
Chapter 3: Playback Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
102
Sample Player Logs
2008-09-25 12:58:41,218 ERROR; [MMLAPIBase.PbLogicLog] Error - CNMLApi : CNMLApi : ConnectionFailureCallback() - Disconnection occured for ClientHandle a86860. [ThreadId 5964, ErrorCode 0] 2008-09-25 12:58:41,218 INFO ; [VoiceMMLServer.GetPacket] - screen GetNextPacket failed at NiceApplications.Playback.MediaSourceAPI.MMLAPIBase.GetPacket(Boolean& lastPacket, Byte[]& data, tSecuredPacketParams& packetParams) at NiceApplications.Playback.MediaSourceAPI.VoiceAPI.GetActivePacket(Boolean& lastPacket, Byte[]& audioBuffer, DateTime& startTime, DateTime& stopTime, CompressionType& compression, Guid& guid, UInt32& actualDataPosition, UInt32& actualDataSize) at NiceApplications.Playback.MediaSourceAPI.VoiceAPI.GetPacket(Boolean& lastPacket, Boolean& active, Byte[]& audioBuffer, DateTime& startTime, DateTime& stopTime, CompressionType& compression, Guid& guid, UInt32& actualDataPosition, UInt32& actualDataSize) at NiceApplications.Playback.Streaming.Nice.VoiceMMLServer.GetPacket() 2008-09-25 12:58:41,218 ERROR; [AudioBufferBase.FetchingThreadFunction] screen GetNextPacket failed at NiceApplications.Playback.Streaming.Nice.VoiceMMLServer.GetPacket() at NiceApplications.Playback.MediaServices.Logic.AudioFetcher.GetPacket(Boolean& fetchDone) at NiceApplications.Playback.MediaServices.Logic.CompressedInBuffer.GetPacket() at NiceApplications.Playback.MediaServices.Logic.CompressedBuffer.Pop() at NiceApplications.Playback.MediaServices.Logic.AudioBuffer.GetPacket() at NiceApplications.Playback.MediaServices.Logic.AudioBufferBase.FetchingThreadFunction() 2008-09-25 12:58:41,218 INFO ; [AudioBufferBase.FetchingThreadFunction] - m_state changed to Seek 2008-09-25 12:58:41,218 ERROR; [CommonViewManager.ErrorOccurred] ErrorOccurred, description: SCREEN_SERVER_SCREEN_FAILED_TO_GET_NEXT_PACKET 2008-09-25 12:58:41,234 WARN ; [MMLAPIBase.PbLogicLog] - Warning MMLPoint::SendBlockingMsgToChannel : timeout expired. ChannelId(2), Opcode(203) [ThreadId 4100, ErrorCode 0] 2008-09-25 12:58:41,234 WARN ; [MMLAPIBase.PbLogicLog] - Warning MMLPoint::SendBlockingMsgToChannel : timeout expired. ChannelId(2), Opcode(203) [ThreadId 4100, ErrorCode 0] 2008-09-25 12:58:41,234 WARN ; [MMLAPIBase.PbLogicLog] - Warning MMLPoint::SendBlockingMsgToChannel : error detected while waiting. ChannelId(2), Opcode(203), lastError(553648143) [ThreadId 4100, ErrorCode 0] 2008-09-25 12:58:41,234 WARN ; [MMLAPIBase.PbLogicLog] - Warning MMLPoint::SendBlockingMsgToChannel : error detected while waiting. ChannelId(2), Opcode(203), lastError(553648143) [ThreadId 4100, ErrorCode 0]
Chapter 3: Playback Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
103
Sample Player Logs
2008-09-25 12:58:41,234 FATAL; [MMLAPIBase.PbLogicLog] FatalError PlaybackPoint::GetBuffer : SendSyncCommandToChannel failed. Channel id(2) [ThreadId 4100, ErrorCode 0] 2008-09-25 12:58:41,234 FATAL; [MMLAPIBase.PbLogicLog] FatalError PlaybackPoint::GetBuffer : SendSyncCommandToChannel failed. Channel id(2) [ThreadId 4100, ErrorCode 0] 2008-09-25 12:58:41,234 FATAL; [MMLAPIBase.PbLogicLog] FatalError - CNMLApi : CNMLApi : NML_GetBuffer() - NMLAPI_PB_GetBuffer() returned 0,0.GetLastError returned 0 [ThreadId 4100, ErrorCode 0] 2008-09-25 12:58:41,234 FATAL; [MMLAPIBase.PbLogicLog] FatalError - CNMLApi : CNMLApi : NML_GetBuffer() - NMLAPI_PB_GetBuffer() returned 0,0.GetLastError returned 0 [ThreadId 4100, ErrorCode 0] 2008-09-25 12:58:41,234 ERROR; [MMLAPIBase.GetPacket] m_pbLogic.GetNextPacket failed res = ConnectionFailed 2008-09-25 12:58:41,234 INFO ; [VoiceMMLServer.GetPacket] - screen GetNextPacket failed at NiceApplications.Playback.MediaSourceAPI.MMLAPIBase.GetPacket(Boolean& lastPacket, Byte[]& data, tSecuredPacketParams& packetParams) at NiceApplications.Playback.MediaSourceAPI.VoiceAPI.GetActivePacket(Boolean& lastPacket, Byte[]& audioBuffer, DateTime& startTime, DateTime& stopTime, CompressionType& compression, Guid& guid, UInt32& actualDataPosition, UInt32& actualDataSize) at NiceApplications.Playback.MediaSourceAPI.VoiceAPI.GetPacket(Boolean& lastPacket, Boolean& active, Byte[]& audioBuffer, DateTime& startTime, DateTime& stopTime, CompressionType& compression, Guid& guid, UInt32& actualDataPosition, UInt32& actualDataSize) at NiceApplications.Playback.Streaming.Nice.VoiceMMLServer.GetPacket() 2008-09-25 12:58:41,234 ERROR; [AudioBufferBase.FetchingThreadFunction] screen GetNextPacket failed at NiceApplications.Playback.Streaming.Nice.VoiceMMLServer.GetPacket() at NiceApplications.Playback.MediaServices.Logic.AudioFetcher.GetPacket(Boolean& fetchDone) at NiceApplications.Playback.MediaServices.Logic.CompressedInBuffer.GetPacket() at NiceApplications.Playback.MediaServices.Logic.CompressedBuffer.Pop() at NiceApplications.Playback.MediaServices.Logic.AudioBuffer.GetPacket() at NiceApplications.Playback.MediaServices.Logic.AudioBufferBase.FetchingThreadFunction() 2008-09-25 12:58:41,234 INFO ; [AudioBufferBase.FetchingThreadFunction] - m_state changed to Seek 2008-09-25 12:58:41,234 ERROR; [CommonViewManager.ErrorOccurred] ErrorOccurred, description: SCREEN_SERVER_SCREEN_FAILED_TO_GET_NEXT_PACKET
Chapter 3: Playback Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
104
Sample Player Logs
Logger Disconnection Log 2008-09-25 12:43:38,828 INFO ; [RenderVoiceMediaBase.WriteAudioToDSBuffer] - MediaId 3 writeSize 16000 clentBufferLength 16000 2008-09-25 12:43:38,843 ERROR; [LoggerPBAPI880.GetNextPacketFromLogger] NC_StreamFetchData failed res = -16 2008-09-25 12:43:38,843 ERROR; [VoiceServer.GetPacket] GetPacket error VOICE_SERVER_CONNECTION_FAILURE 2008-09-25 12:43:38,843 ERROR; [AudioBufferBase.FetchingThreadFunction] FETCH DATA FAILED at NiceApplications.Playback.Streaming.Nice.VoiceServer.GetPacket() at NiceApplications.Playback.MediaServices.Logic.AudioFetcher.GetPacket(Boolean& fetchDone) at NiceApplications.Playback.MediaServices.Logic.CompressedInBuffer.GetPacket() at NiceApplications.Playback.MediaServices.Logic.CompressedBuffer.Pop() at NiceApplications.Playback.MediaServices.Logic.AudioBuffer.GetPacket() at NiceApplications.Playback.MediaServices.Logic.AudioBufferBase.FetchingThreadFunction() 2008-09-25 12:43:38,843 INFO ; [AudioBufferBase.FetchingThreadFunction] - m_state changed to Seek 2008-09-25 12:43:38,843 INFO ; [RenderVoiceMediaBase.OnAudioBufferError] - Got buffering event while playing propogating event 2008-09-25 12:43:38,859 INFO ; [PlaybackController.OnBufferingEvent] - Controller got Buffering event while playing or seeking 44 2008-09-25 12:43:38,859 INFO ; [PlaybackController.PlayFrom] - PlayFrom 9250 2008-09-25 12:43:38,859 INFO ; [PlaybackController.Seek] - Seek 9250 2008-09-25 12:43:38,890 INFO ; [RenderVoiceMediaBase.GetFreeSpaceSize] - playCursor 151948, LastPlayCursorPosition 135118 2008-09-25 12:43:38,890 INFO ; [RenderVoiceMediaBase.WriteAudioToDSBuffer] - MediaId 1 writeSize 16000 clentBufferLength 16000 2008-09-25 12:43:38,921 INFO ; [PBViewManager.OnBuffring] - [WMS] Buffering Started 2008-09-25 12:43:38,921 INFO ; [ExpandView.OnBuffring] - *** Not Gui thread using this.Invoke *** 2008-09-25 12:43:38,921 INFO ; [AudioBufferBase.Seek] - Seek m_seekPosition=9250, m_state=Working 2008-09-25 12:43:38,921 INFO ; [CompressedInBuffer.Seek] - Seek : 9250 2008-09-25 12:43:38,921 DEBUG; [VoiceServer.InternalJumpToPos] - JumpToPos Token 191 startTime 25/09/2008 09:45:13 404 StopTime 25/09/2008 09:47:04 403
Chapter 3: Playback Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
105
Sample Player Logs
2008-09-25 12:43:38,937 INFO ; [AudioBuffer.GetPacket] - Poped:Packet: Media 1, Time 25/09/2008 09:45:22.120-25/09/2008 09:45:26.210, Position 8716-12806 (=4090ms), Data 4090by (=4090ms), Active 2008-09-25 12:43:38,937 ERROR; [LoggerPBAPI880.StartPlayback] AUD_AudioOverLAN_Ex failed res = -1 2008-09-25 12:43:38,937 INFO ; [VoiceServer.InternalJumpToPos] - AUD_AudioOverLAN_Ex failed at NiceApplications.Playback.MediaSourceAPI.LoggerPBAPI880.StartPlayback(Int32 token, DateTime startTime, DateTime endTime) at NiceApplications.Playback.Streaming.Nice.VoiceServer.InternalJumpToPos(DateTime startTime) 2008-09-25 12:43:38,937 ERROR; [VoiceServer.InternalJumpToPos] JumpToPos error VOICE_SERVER_START_OVER_LAN_FAILED 2008-09-25 12:43:38,937 DEBUG; [AudioProcessing.DecodePacket] - DecodePacket, startPos=9250, endPos=110999, packetStartPos=8716, packetEndPos=12806 2008-09-25 12:43:38,937 INFO ; [RenderVoiceMediaBase.Seek] - AUD_AudioOverLAN_Ex failed at NiceApplications.Playback.Streaming.Nice.VoiceServer.InternalJumpToPos(DateTime startTime) at NiceApplications.Playback.Streaming.Nice.MediaDataServerBase.JumpToPos(Int64 ticks) at NiceApplications.Playback.MediaServices.Logic.AudioFetcher.MediaServerSeek(Int64 position) at NiceApplications.Playback.MediaServices.Logic.AudioFetcher.StartFetch(Int64 position) at NiceApplications.Playback.MediaServices.Logic.CompressedInBuffer.SetPosition(Int64 position, Boolean& isWorking) at NiceApplications.Playback.MediaServices.Logic.CompressedInBuffer.Seek(Int64 position) at NiceApplications.Playback.MediaServices.Logic.CompressedBuffer.Seek(Int64 position) at NiceApplications.Playback.MediaServices.Logic.AudioBufferBase.Seek(Int64 position) at NiceApplications.Playback.MediaServices.Logic.RenderVoiceMediaBase.Seek(Int64 position) 2008-09-25 12:43:38,937 INFO ; [OutputDS.RenderVoiceSeekCallback] RenderVoiceSeekCallback stat = VOICE_SERVER_START_OVER_LAN_FAILED 2008-09-25 12:43:38,937 INFO ; [RenderVoiceMediaBase.WriteAudioToDSBuffer] - MediaId 1 writeSize 16000 clentBufferLength 16000 2008-09-25 12:43:38,953 INFO ; [AudioBuffer.GetPacket] - Poped:Packet: Media 1, Time 25/09/2008 09:45:26.210-25/09/2008 09:45:30.300, Position 12806-16896 (=4090ms), Data 4090by (=4090ms), Active
Chapter 3: Playback Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
106
Sample Player Logs
Checking the NiceScreen Logs Settings for your NiceScreen Logger are stored in the MMLogger.cfg configuration file located in the NiceScreen Logger installation directory. LogFile_Path indicates the location of the NiceScreen log files. The logs are located in the installation folder, by default in: C:\Programs Files\Nice Systems\Multimedia Logger. To check the NiceScreen Logs:
Review the NiceScreen Logger logs (logfileXX.log).
Chapter 3: Playback Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
107
Example of Troubleshooting using the Logs
Example of Troubleshooting using the Logs This section includes sample logs useful for troubleshooting playing back an interaction. For further information on the playback process, see Understanding the Playback Flow on page 79. To Troubleshoot using the Logs: 1. In PlaybackAdministration.txt (see PlaybackAdministration.txt on page 108), search for: a.
Start PreparePlayListForInteractions: This indicates the start of the playback process.
b. Locate Results: Check if the locate process succeeded. In this example two parts of the
call are located. c.
RecordingId:XXX: This indicates the recording ID for each part of the call.
d. Build media per participant: This indicates the start of the media building process (per
media). e.
New Voice Media start or New Screen Media start: This indicates that the interaction
has been located and that the media has been built. If the locate succeeded, but the media is missing, it could indicate a timing problem on the client. 2. If the locate failed, open LocateLog.txt (see LocateLog.txt on page 110) and search for: a.
AddRequest … recording Id: XXX: This indicates that a locate request for a recording
was initiated. b. StorageCenterLocator.LocateRecordingCallback … Operation Result = 0: A zero
(0) indicates that the recording was located on the Storage Center. A value other than zero (0) indicates that the part of the call is missing from the Storage Center. c.
with result Found: This indicates that the locate process was successful.
d. with result NotFound: This indicates that the locate process was not successful. This
may indicate a time difference between the Interactions Center and Logger, or the Interactions Center and Applications Server. Refer to TN0752: Windows Network Time Protocol Configuration. This note provides procedures for configuring the Windows Network Time Protocol on NICE servers. e.
If there are Storage Center errors, see the Storage Center Logs to establish the reason for the error. Review the Storage Center Logs. By default, the log files are stored in the same directory where the Storage Center is running. The default log file names are:
•
StorageCenterLog.log
•
StorageCenterLog1.log through StorageCenterLog100.log
PlaybackAdministration.txt 2008-08-24 09:59:04,479 DEBUG; [PlayListManager.BuildPlayItemsForInteractions] - : Start PreparePlayListForInteractions 2008-08-24 09:59:04,479 DEBUG; [PlayListManager.BuildPlayItemsForInteractions] - : Create play list items Chapter 3: Playback Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
108
Example of Troubleshooting using the Logs
2008-08-24 09:59:04,479 DEBUG; [PlayListManager.BuildPlayItemsForInteractions] - : Bring segments from db for requested calls 2008-08-24 09:59:04,479 DEBUG; [PlayListManager.BuildPlayItemsForInteractions] - query url: zandberg:62074 2008-08-24 09:59:04,620 DEBUG; [PlayListManager.BuildPlayItemsForInteractions] - : Build stages for interactions using Analyzer class 2008-08-24 09:59:04,620 DEBUG; [PlayListManager.BuildPlayItemsForInteractions] - : Create play media 2008-08-24 09:59:04,620 DEBUG; [PlayListManager.CreatePlayMedia] - : prepare locate requests 2008-08-24 09:59:04,620 DEBUG; [PlayListManager.CreatePlayMedia] - Add locate requests for call 0 2008-08-24 09:59:04,620 INFO ; [Participant.BuildRecordings] - Added another recordings: 417 for the participant ID 11 2008-08-24 09:59:04,635 DEBUG; [PBAdminServer.GetServerIdOfLogger] GetServerIdOfLogger took:00:00:00 2008-08-24 09:59:04,635 INFO ; [Participant.BuildRecordings] - Added another recordings: 416 for the participant ID 13 2008-08-24 09:59:04,651 DEBUG; [PBAdminServer.GetServerIdOfLogger] GetServerIdOfLogger took:00:00:00.0156250 2008-08-24 09:59:04,651 DEBUG; [PlayListManager.CreatePlayMedia] - : run Locate 2008-08-24 09:59:04,651 DEBUG; [PBAdminServer.GetLoggersOnSite] - GetLoggersOnSite took: 00:00:00 2008-08-24 09:59:04,651 DEBUG; [PBAdminServer.GetLoggersOnSite] - GetLoggersOnSite took: 00:00:00 2008-08-24 09:59:04,667 DEBUG; [PBAdminServer.GetLoggersOnSite] - GetLoggersOnSite took: 00:00:00 2008-08-24 09:59:04,667 DEBUG; [PBAdminServer.GetLoggersOnSite] - GetLoggersOnSite took: 00:00:00 2008-08-24 09:59:04,698 DEBUG; [LocateManager.LocateFinished] - Locate callback recieved 0 2008-08-24 09:59:04,698 INFO ; [LocateManager.LocateFinished] - Locate Callback Request 0 Times: 633518804008930000, 633518804238930000 2008-08-24 09:59:04,729 DEBUG; [LocateManager.LocateFinished] - Locate callback recieved 1 2008-08-24 09:59:04,729 INFO ; [LocateManager.LocateFinished] - Locate Callback Request 0 Times: 633518804008930000, 633518804238930000 2008-08-24 09:59:04,729 DEBUG; [PlayListManager.CreatePlayMedia] - Locate Request took: 00:00:00.0781250
Chapter 3: Playback Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
109
Example of Troubleshooting using the Logs
2008-08-24 09:59:04,729 DEBUG; [PLAYLISTMANAGER.WRITELOCATERESULTSTOTRACE] :LOCATE RESULTS 2008-08-24 09:59:04,729 DEBUG; [PLAYLISTMANAGER.WRITELOCATERESULTSTOTRACE] - LOGGERID :1722315 CHANNELID:49 TOKEN:0 OFFLINEMEDIANUMBER:0 DEVICENUMBER:0 ISPARTIAL:FALSE RECORDINGS: INTERACTIONID:326924 RECORDINGID:417 SERVERID:30 SITEID:1/N 2008-08-24 09:59:04,729 DEBUG; [PLAYLISTMANAGER.WRITELOCATERESULTSTOTRACE] LOGGERID :1722315 CHANNELID:19 TOKEN:0 OFFLINEMEDIANUMBER:0 DEVICENUMBER:0 ISPARTIAL:FALSE RECORDINGS: INTERACTIONID:326924 RECORDINGID:416 SERVERID:30 SITEID:1/N
2008-08-24 09:59:04,729 DEBUG; [PlayListManager.CreatePlayMedia] - : BUILD MEDIA PER PARTICIPANT
2008-08-24 09:59:04,729 INFO ; [PlayListManager.BuildMediaForParticipant] - Building Media: Stage Times : 7/17/2008 8:33:20 AM, 7/17/2008 8:33:43 AM Locate Times: 633518804008930000,633518804238930000 2008-08-24 09:59:04,729 DEBUG; [PlayListManager.BuildMediaForParticipant] - : NEW VOICE MEDIA START
2008-08-24 09:59:04,729 INFO ; [PlayListManager.BuildMediaForParticipant] - Building Media: Stage Times : 7/17/2008 8:33:20 AM, 7/17/2008 8:33:43 AM Locate Times: 633518804008930000,633518804238930000 2008-08-24 09:59:04,729 DEBUG; [PlayListManager.BuildMediaForParticipant] - : NEW VOICE MEDIA START
2008-08-24 09:59:04,729 DEBUG; [PlayListManager.CreatePlayMedia] - : Finish CreatePlayMedia 2008-08-24 09:59:04,729 DEBUG; [PlayListManager.PreparePlayListForInteractions] - : Build xml document 2008-08-24 09:59:04,729 DEBUG; [PlayListManager.PreparePlayListForInteractions] - : Return PlayList
LocateLog.txt 2008-08-24 09:59:04,651 [20] DEBUG; [LocateServiceServer.AddRequest] - ADDREQUEST: Request id:23 LocateOrder = Custom start time: 7/17/2008 8:33:20 AM Milli: 893 stop time: 7/17/2008 8:33:43 AM Milli: 893 for RECORDING ID: 417 interactionId: 326924 serverId:30 siteId: 1 2008-08-24 09:59:04,651 [20] DEBUG; [LocateServiceServer.EnterCheckLoggerState] - The request 23 for recording Id: 417 interactionId: 326924 serverId:30 siteId: 1 moved to CheckLogger state 2008-08-24 09:59:04,667 [20] DEBUG; [LocateServiceServer.AddRequest] - ADDREQUEST: Request id:24 LocateOrder = Custom start time: 7/17/2008 8:33:20 AM Milli: 893 stop time: 7/17/2008 8:33:43 AM Milli: 893 for RECORDING ID: 416 interactionId: 326924 serverId:30 siteId: 1 2008-08-24 09:59:04,667 [20] DEBUG; [LocateServiceServer.EnterCheckLoggerState] - The request 24 for recording Id: 416 interactionId: 326924 serverId:30 siteId: 1 moved to CheckLogger state Chapter 3: Playback Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
110
Example of Troubleshooting using the Logs
2008-08-24 09:59:04,667 [STP SmartThreadPool Thread #31] DEBUG; [StorageCenterLocator.OriginalLocate] - Locate calls Storage Center: zandberg:30003 recordings: Id: 417 interactionId: 326924 serverId:30 siteId: 1 2008-08-24 09:59:04,667 [STP SmartThreadPool Thread #31] INFO ; [StorageCenterLocator.OriginalLocate] - Call LocateRecording for Storage Center: zandberg:30003 recordings: Id: 417 interactionId: 326924 serverId:30 siteId: 1 2008-08-24 09:59:04,667 [STP SmartThreadPool Thread #32] DEBUG; [StorageCenterLocator.OriginalLocate] - Locate calls Storage Center: zandberg:30003 recordings: Id: 416 interactionId: 326924 serverId:30 siteId: 1 2008-08-24 09:59:04,698 [14] INFO ; [STORAGECENTERLOCATOR.LOCATERECORDINGCALLBACK] - GOT LOCATE CALLBACK FOR RECORDING 1;30;326924;417 OPERATION R ESULT = 0
2008-08-24 09:59:04,698 [STP SmartThreadPool Thread #31] DEBUG; [StorageCenterLocator.OriginalLocate] - Locate for recording 1;30;326924;417 returned with 1 results 2008-08-24 09:59:04,698 [STP SmartThreadPool Thread #31] DEBUG; [LocateServiceServer.HandleState] - The request 23 for recording Id: 417 interactionId: 326924 serverId:30 siteId: 1 finished state CheckLogger with result Found 2008-08-24 09:59:04,698 [STP SmartThreadPool Thread #31] DEBUG; [LocateServiceServer.SendLocateFinishedCallback] - FinishLocate for recording Id: 417 interactionId: 326924 serverId:30 siteId: 1 Results: Request id:23 results number: 1 results: source: SC start time: 7/17/2008 8:33:20 AM Milli: 893 stop time: 7/17/2008 8:33:43 AM Milli: 893 2008-08-24 09:59:04,698 [STP SmartThreadPool Thread #31] DEBUG; [LocateServiceServer.SendLocateFinishedCallback] - Locate callback called with times: 633518804008930000, 633518804238930000 2008-08-24 09:59:04,698 [STP SmartThreadPool Thread #31] DEBUG; [LocateServiceServer.SendLocateFinishedCallback] - Finish request Avail: 499 Active999 2008-08-24 09:59:04,698 [STP SmartThreadPool Thread #32] INFO ; [StorageCenterLocator.OriginalLocate] - Call LocateRecording for Storage Center: zandberg:30003 recordings: Id: 416 interactionId: 326924 serverId:30 siteId: 1 2008-08-24 09:59:04,714 [14] INFO ; [STORAGECENTERLOCATOR.LOCATERECORDINGCALLBACK] - GOT LOCATE CALLBACK FOR RECORDING 1;30;326924;416 OPERATION R ESULT = 0
2008-08-24 09:59:04,729 [STP SmartThreadPool Thread #32] DEBUG; [StorageCenterLocator.OriginalLocate] - Locate for recording 1;30;326924;416 returned with 1 results 2008-08-24 09:59:04,729 [STP SmartThreadPool Thread #32] DEBUG; [LocateServiceServer.HandleState] - The request 24 for recording Id: 416 interactionId: 326924 serverId:30 siteId: 1 finished state CheckLogger WITH RESULT FOUND
Chapter 3: Playback Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
111
Example of Troubleshooting using the Logs
2008-08-24 09:59:04,729 [STP SmartThreadPool Thread #32] DEBUG; [LocateServiceServer.SendLocateFinishedCallback] - FinishLocate for recording Id: 416 interactionId: 326924 serverId:30 siteId: 1 Results: Request id:24 results number: 1 results: source: SC start time: 7/17/2008 8:33:20 AM Milli: 893 stop time: 7/17/2008 8:33:43 AM Milli: 893 2008-08-24 09:59:04,729 [STP SmartThreadPool Thread #32] DEBUG; [LocateServiceServer.SendLocateFinishedCallback] - Locate callback called with times: 633518804008930000, 633518804238930000 2008-08-24 09:59:04,729 [STP SmartThreadPool Thread #32] DEBUG; [LocateServiceServer.SendLocateFinishedCallback] - Finish request Avail: 499 Active999 2008-08-20 19:02:56,620 [8860] INFO ; [StorageCenterLocator.LocateRecordingCallback] - Got locate callback for recording 1;26;239481090;4323375 Operation Result = 300008 2008-08-20 19:02:56,620 [8860] INFO ; [STORAGECENTERLOCATOR.LOCATERECORDINGCALLBACK] - ERROR WHEN LOCATE RECORDING 1;26;239481090;4323375 OPERATION RESULT = 300008
2008-08-20 19:02:56,620 [STP SmartThreadPool Thread #7] DEBUG; [StorageCenterLocator.OriginalLocate] - Locate for recording 1;26;239481090;4323375 returned with 0 results 2008-08-20 19:02:56,620 [STP SmartThreadPool Thread #7] DEBUG; [LocateServiceServer.HandleState] - The request 7 for recording Id: 4323375 interactionId: 239481090 serverId:26 siteId: 1 finished state StorageCenterLocate with result NotFound 2008-08-20 19:05:52,077 [8860] INFO ; [StorageCenterLocator.LocateRecordingCallback] - Got locate callback for recording 1;26;253423018;4569139 Operation Result = 300007 2008-08-20 19:05:52,077 [8860] INFO ; [StorageCenterLocator.LocateRecordingCallback] Error when locate recording 1;26;253423018;4569139 Operation Result = 300007 2008-08-20 19:05:52,077 [STP SmartThreadPool Thread #86] DEBUG; [StorageCenterLocator.OriginalLocate] - Locate for recording 1;26;253423018;4569139 returned with 0 results 2008-08-20 19:05:52,077 [STP SmartThreadPool Thread #86] DEBUG; [LocateServiceServer.HandleState] - The request 70 for recording Id: 4569139 interactionId: 253423018 serverId:26 siteId: 1 finished state StorageCenterLocate with result NotFound
Chapter 3: Playback Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
112
Collecting the Server Logs using the NICE Perform Logs Collector Tool
Collecting the Server Logs using the NICE Perform Logs Collector Tool The Log Collector Tool collects all relevant playback information at one click of a button. This information includes all configuration parameters and all playback related log files. This information can be saved and used when opening or escalating cases. For more information, see Collecting Information Using the Log Collector on page 537.
Collecting the Client Logs using NICE Perform Logs Collector Tool Use the NICE Perform Logs Collector to collect the Player and ScreenAgent Log files. For more information, see Collecting Information Using the Log Collector on page 537. Player:
•
All *.txt; *.comments; *config; *.xml files from %APP_DATA%\NicePlayer\Release3 directory.
ScreenAgent:
•
All *.* files from directory specified by the Common AppData registry key. NOTE: Collecting all the necessary information can speed up resolution of a support case.
Chapter 3: Playback Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
113
Copying the Playlist
Copying the Playlist The playlist is built each time an interaction is played, and the previous playlist is overwritten. The playlist.xml comprises several elements.
Element
Description
Item
Part of an interaction. There maybe multiple items in an interaction.
Stage
Typically an item comprises one stage. An additional stage may be added by, for example, a conference call.
Time
The Start and End time of the stage (Year, Month, Day Hour, Minute, Second, Millisecond).
Participant
A participant in the interaction stage. There will usually be multiple participants. This includes information:
ContactKey
•
Media information including the media type (voice or screen), the media original location (Logger ID and Channel ID) ,and the Logger token.
•
Participant information, including Agent ID, User ID, and Device ID. No participant information will appear for customers.
Included only if this is a complete call.
To copy the playlist: 1. If you have a problem playing back an interaction, copy the following file immediately and
submit it to NICE Customer Support: \NICE Player\temp\Playlist.xml 2. For further information on troubleshooting using the playlist, see Troubleshooting Player
Errors on page 96.
Chapter 3: Playback Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
114
Dumping the Client Memory
Dumping the Client Memory If requested to by NICE Customer Support, you may need to dump the client memory. Do this using ADPlus, a Windows debugging tool that starts the CDB debugger and executes a set of debugger commands to gather information. For more information regarding ADPlus, see http://support.microsoft.com/kb/286350. To Dump the Client Memory: 1. Copy the debug tools directory onto the client machine (C:\Program Files\Debugging Tools for Windows\). Alternatively, connect a USB Flash drive to the problematic machine
and run the dump command from the USB Flash drive. 2. Identify the process ID: a.
Run Internet Explorer. You should only have one Internet Explorer window open.
b. In Windows Task Manager, click the Processes tab. c.
From View > Select Columns.
d. Select PID (Process Identifier) and click OK. e.
Note the Internet Explorer Process ID.
3. From the Debugging Tools for Windows directory or USB Flash drive, run the following
command: adplus -hang -p XXXX -quiet
•
The dump is created in the Debugging Tools for Windows directory. To specify an output path use the -o option.
•
If the process throws a fatal exception and then terminates, replace the -hang option with the -crash option. This takes the dump before the process terminates.
•
If any question dialogs appear during the dump process, click OK.
4. The cdb.exe command window appears, and the dump process starts. 5. The command window closes when the dump is finished and the results are saved in a file with the format <Mode>_Date_ _Time_<Time>, for example: Hang_Mode__Date_12-27-2005__Time_17-37-1919. 6. Verify that this is the date and time that you took the dump, ZIP the entire directory (take into
account that the file can be large), and mail or FTP it to NICE Customer Support.
Chapter 3: Playback Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
115
Understanding Playback Retrieval
Understanding Playback Retrieval When you cannot play back a recording from the Business Analyzer application and receive the error There are no recordings to retrieve, you need to verify that the Media Library and Storage Center are installed. If they are not installed, you can retrieve the entire storage media that includes the specific recording from the Backup Manager.
Other Troubleshooting A No recording was found message could be related to activation of CTI Analysis in the integration. Check the Playback Administration service log file as well as the Business Analyzer service log file.
Chapter 3: Playback Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
116
4 Archiving Troubleshooting Flow This chapter describes common troubleshooting issues for the various archiving methods used with NICE Perform Release 3.1.
Contents General Archiving Troubleshooting Flow .................................................................. 118 Tape/DVD Archiving Troubleshooting ........................................................................ 119 Media Library Troubleshooting ...................................................................................121 Backup Server Device Detection Troubleshooting ...................................................123 Other Backup Server Troubleshooting.......................................................................125 Storage Center Troubleshooting.................................................................................126 Setting the Log Level to DEBUG..............................................................................128 Checking for Archived Calls .....................................................................................129 Gathering Storage Center Information .....................................................................132 Checking the Storage Retention and Deletion Values .............................................133 Verifying that the Storage Center Database is Selected ..........................................133 Verifying the Used and Free Space..........................................................................133 Verifying the Storage Unit Threshold........................................................................134 Checking that the Archiving Rule Inserts New Calls ................................................135 Troubleshooting the Archiving Rule .........................................................................135 Checking the Storage Center Archiving Tables........................................................137 Checking the Storage Center Logs ..........................................................................138 Checking the Nice Storage Streaming Service Logs................................................139 Checking the Player Logs for Storage Center Errors ...............................................139 Other Troubleshooting Issues ..................................................................................140
Chapter 4: Archiving Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
117
General Archiving Troubleshooting Flow
General Archiving Troubleshooting Flow 1. Which archiving activity are you troubleshooting?
•
Online archiving on a Logger.
•
Archiving on tape/DVD.
•
Organizing the archiving media at a site with Media Library. The Media Library assigns sequential numbers to the cassettes, and the Media Library printer prints labels automatically.
•
Archiving on Storage Center.
2. If your problem is archiving online to a Logger, see the High Density Logger
Troubleshooting Flow on page 303 and VoIP Troubleshooting Flow on page 329. 3. If your problem is archiving to tape, see Tape/DVD Archiving Troubleshooting on page 119. 4. If your problem is with the Media Library, see Media Library Troubleshooting on page 121. 5. If your problem is with the Backup Server, see Backup Server Device Detection
Troubleshooting on page 123. 6. If your problem is archiving to Storage Center, see Storage Center Troubleshooting
on page 126.
Chapter 4: Archiving Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
118
Tape/DVD Archiving Troubleshooting
Tape/DVD Archiving Troubleshooting See the Backup Configuration Guide and Archiving Devices and Media Reference Guide for further information. To troubleshoot tape/DVD archiving: 1. View the status of the drive: a.
Log in to the NICE Perform Applications Suite.
b. From the Accessories menu, select Backup Manager. c.
Double-click the NiceLog icon
.
d. Select a Logger. Logger details (Server Name, Logger ID, Host/IP Address and Backup Mode) appear in the Logger Details area; device information (if media is loaded) appears in the Devices Information area including:
•
Device Assignment shows the current operation taking place: Automatic Archiving, Retrieval, Retrieve while Archiving, Manual Archiving, Erase.
•
Device Status shows the current status: Loading, Empty, Ejecting, Retrieving, Suspended.
e.
In the Device Information area, select a row representing a device.
f.
Click the Device Info
button on the Backup Manager toolbar.
The Device Info window appears. Figure 4-1
Device Info Window
2. Ensure that there is a media in the drive. 3. Check that the retention period of the media has not expired yet.. 4. Check that the media was closed properly and is valid for usage in append mode. 5. Ensure that the media is not full or write protected. 6. Ensure that the media version is supported. For a list of the supported media, see the Archiving
Devices and Media Reference Guide. 7. Verify that the archiving device can be viewed as a drive. 8. Verify that the correct driver is installed for the archiving device. See Archiving Devices
Driver Installation in the Archiving Devices and Media Reference Guide. Chapter 4: Archiving Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
119
Tape/DVD Archiving Troubleshooting
9. If necessary, use a cleaning media to clean the archiving drive. For a list of supported cleaning
media and how to clean the archiving device, see the Archiving Devices and Media Reference Guide. WARNING If an archiving drive is accidentally cleaned with a spent cleaning cassette, the used-up cassette will slowly contaminate the drive. The contamination can spread to the archiving medium. The archiving medium can then contaminate a new archiving drive and cause it to fail. 10. Test the archiving drive using the NICE Test Kit. a.
If you do not have the NICE Test Kit you can download it from ExtraNICE > Services > Diagnostic Utilities.
b. See the General High Density Logger Troubleshooting Flow on page 304 and the
NICE Test & Recovery Kit User Manual. c.
Use the following tests that run under Windows: Test/Operation
Test Description
Log File Location
Backup Sony AIT & DDS Tape Tool
Checking SONY AIT IDE Backup Devices.
Root:\Programs\SONY Tape Tool for Windows\S.N.dat
Firmware Upgrade for SONY IDE & SCSI
S.N indicates the tape serial number
Backup Test DVD Creator Application
Creating a Test DVD
No log file.
Backup DVD Read & Verify
Checking DVD Drives
Root:\Programs\BurnInTest\BIT_lo g*.log
d. Use the following tests that run under DOS: Test/Operation
Test Description
Log File Location
Backup SONY AIT & DDS Test (SCSI)
Checking SONY AIT & DDS SCSI Backup Devices
Root:\Programs\SONY\NTAU\*.ana Root:\Programs\SONY\Ship\*.abc
Chapter 4: Archiving Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
120
Media Library Troubleshooting
Media Library Troubleshooting See the Backup Configuration Guide. To troubleshoot Media Library: 1. Verify that all Media Library Loggers appear in the Media Library window as Active =1. Figure 4-2
Media Library Window
2. If automatic archiving is not working, try to archive manually. 3. If cannot print lables because of archiving issues using the Backup Server, see Tape/DVD
Archiving Troubleshooting on page 119. 4. Check that all the required updates are installed. 5. Check if the Media Library is not printing labels for one or more of the Loggers. 6. Check the Media Library printer status and which Loggers are associated to a specific Media
Library printer: a.
In System Administrator, in the Organization tree, select Media Library Servers and select the required Media Library Server.
b. Selected the required Media Library printer. c.
Check the Printer Status in the General tab.
d. Click the Loggers tab. e.
Check which Loggers are associated to the Media Library Printer.
7. Check the network connection between the Media Server and the affected Loggers. 8. Check that the printer is plugged in and online. Chapter 4: Archiving Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
121
Media Library Troubleshooting
9. Check the location of the printer sensors and check the label cartridge in the printer. 10. Use the test button on the printer to print a test label. 11. The following are the relevant Media Library tables:
•
Nice_admin.tblMLibLoggers
•
nice._admin.tblSystemResources includes all defined Media Library Loggers with iLoggerId (iLoggerid must be = iResouseId from tblSystemResources)
•
Nice_interaction.tblMLCassestes describes cassettes that Media Library created labels
for.
•
Nice_interaction.tblMLChannels describes for a given tape the channels and times.
•
Nice_interaction.tblMLLog is the Media Library log.
12. In System Administrator, navigate to Applications > Configurations > Locate Settings
and check that Storage Center is configured for locating and check the Locate Order. For more information, see Checking the Locate Order on page 94. 13. Increase the reporting level for the NICE Playback Administration service, which includes the
Locate service and Storage Center service. 14. Check the Locate and Storage Center log files at the following paths:
•
Locate service log file: NiceSystem\Applications\ServerBin\Locate.log
•
Storage Center log file: NICE Systems\Storage Center\StorageCenterLog.log
a.
Check that the StorageCenterLog.log does not contain error messages related to the Storage Center startup process.
b. Check that the Locate.log reflects the correct locate process for off-line call playback:
•
A locate request for a specific InteractionID is received.
•
The NICE Playback Administration service (locate) checks the source Logger and returns Result NotFound.
•
The NICE Playback Administration service (locate) checks the rest of the Loggers and returns Result NotFound.
•
The NICE Playback Administration service (locate) checks the Storage Center and returns Result OfflineFound.
•
The NICE Playback Administration service (locate) activates RetrieveServiceSystem and gets a list of tape drives.
•
If one or several interactions are not online, the Retrieval Wizard appears. See “Retrieving Recordings of Interactions” in the Business Analyzer Guide.
15. Ensure that the NICE Storage Center Service and Nice Storage Streaming Service (PlaybackServiceSC) are running (Note: A site using the Media Library must have the NICE
Storage Center to enable retrieval of recordings.)
Chapter 4: Archiving Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
122
Backup Server Device Detection Troubleshooting
Backup Server Device Detection Troubleshooting The Backup Server uses DVD-RAM devices, USB tapes, or SCSI tapes to back up data. This section troubleshoots how to enable detection of each of these backup methods:
•
DVD-RAM Devices Not Detected on page 123
•
USB Tapes Not Detected on page 124
•
SCSI Tapes Not Detected on page 124
DVD-RAM Devices Not Detected Follow the procedure below when the Backup Server does not detect DVD-RAM devices. To troubleshoot DVD-RAM devices not detected by the Backup Server: 1. Check that the hardware connections of the DVD-RAM devices are correctly attached. 2. Navigate to Control Panel > System , and click the Hardware tab. Click the Device Manager button to open the Device Manager. Check that the DVD-RAM devices are listed. 3. Navigate to Control Panel > Add or Remove Programs and check that the WriteDVD!
application appears in the list of installed programs. 4. Restart the computer. 5. From My Computer, check that a removable device appears for each DVD-RAM device
installed on the Backup Server. 6. From the Device Manager (Control Panel > System > Hardware > Device Manager),
check that the name of each DVD drive includes a specific identifier. Figure 4-3
DVD Drive Name with Identifier
Correct
Wrong
7. If the Logger is installed as a service: a.
From the Start menu of the Backup Server machine, select Run and enter gpedit.msc. The Group Policy window appears. Chapter 4: Archiving Troubleshooting Flow
NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
123
Backup Server Device Detection Troubleshooting
b. Navigate to Windows Settings > Security Settings > Local Policies > Security Options, and check that the following Policy is disabled: Devices: Restrict CD-ROM access to locally logged on user only.
USB Tapes Not Detected Follow the procedure below when the Backup Server does not detect USB tapes. To troubleshoot USB tapes not detected by the Backup Server: 1. Check that certified drivers are installed for the USB tapes. 2. Additional installed backup devices prevent the detection of USB tapes. Check that no
DVD-RAM devices or SCSI/IDE tapes are connected to the Backup Server.
SCSI Tapes Not Detected Follow the procedure below when the Backup Server does not detect SCSI tapes. To troubleshoot SCSI tapes not detected by the Backup Server:
•
Installed DVD-RAM devices prevent the detection of SCSI tapes. Check that no DVD-RAM devices are connected to the Backup Server.
Chapter 4: Archiving Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
124
Other Backup Server Troubleshooting
Other Backup Server Troubleshooting •
If an error occurs in the DVD, do not restart the Logger before ejecting the archiving media from all DVD drives in the system.
•
Before performing a DVD device assignment change, such as from auto-archiving to retrieval or vice versa, ensure that the Logger's front panel doors are fully open because the DVD tray will automatically eject after the assignment change. Failing to do so can cause errors in the Logger.
•
When browsing in the My Computer folder, do not browse in the removable disk drives or CD drives as this might cause an interruption in the Logger archiving operation.
Chapter 4: Archiving Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
125
Storage Center Troubleshooting
Storage Center Troubleshooting The diagram below describes the workflow for troubleshooting NICE Storage Center.
Chapter 4: Archiving Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
126
Storage Center Troubleshooting
NOTE:
•
In the Storage Center troubleshooting flow on page 126, SC refers to Storage Center, as a matter of convenience.
•
The main steps for troubleshooting the NICE Storage Center are detailed below. The numbers (
) in the flow on page 126 refer to these steps.
1. Set the log level to debug. See Setting the Log Level to DEBUG on page 128. 2. Check if the system is recording. 3. Check if the Storage Center is archiving any calls (See Checking for Archived Calls
on page 129):
•
In Business Analyzer run a query and check if the calls are archived.
•
Physically check on the Storage Center where the files are archived.
•
Use Storage Center Investigator to check the archiving.
•
Use Reporter and run the Archiving Coverage Report.
4. If the system is not archiving any calls: a.
Check the connectivity between the Storage Center and Loggers, the Storage Center and Database Server and the Storage Center and Rule Engine (located on the Applications Server).
b. Check that the Storage Center ports are open. See the Server Hardening Guide. c.
Check the relevant services on the (for further information on service, see the Maintenance Guide):
•
Check the Storage Center services: LogService, Nice Storage Streaming Service, and Nice Storage Center Service.
•
Check the rule services on the Applications Server: Nice Rule Engine service and Nice RulesManager Service.
•
Check the Nice Playback Administration service.
•
Check the SQL Services on the Database Server.
d. Check that active storage rules are defined. In Rules Manager, click Storage and check the
storage rules. e.
Check the storage deletion and retention values. See Checking the Storage Retention and Deletion Values on page 133.
f.
Verify that the Storage Center Database is selected. See Verifying that the Storage Center Database is Selected on page 133.
g. Check if there is a space problem. See Verifying the Used and Free Space on page 133. h. Check the Storage Unit maximum threshold configured in System Administrator. See
Verifying the Storage Unit Threshold on page 134.
Chapter 4: Archiving Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
127
Storage Center Troubleshooting
i.
If you need to define more storage space, add another quota unit (under the quota group) and restart the Storage Center. The Storage Center swaps to the next quota. There is no need to create a new rule, as this can lead to calls not being archived (from the previous rule).
j.
Check that all the required updates are installed.
k.
Check the Locate Order. See Checking the Locate Order on page 94.
l.
Check the Playback logs. See Example of Troubleshooting using the Logs on page 108.
m. For playback problems, see the General Troubleshooting Flow on page 81. 5. If the system is not archiving specific calls, check for a pattern and establish which calls are
not being archived: a.
Calls from a specific Logger. See the General High Density Logger Troubleshooting Flow on page 304 or the VoIP Troubleshooting Flow on page 329.
b. Calls which meet certain criteria, for example, long calls (these may be regarded as
personal calls and excluded via a rule). In this case, update the rule. c.
Calls from new agents. In this case, update the rule.
d. Calls from new extensions. In this case, update the rule. e.
Calls from extensions that have been changed. In this case, update the rule.
6. Check that the storage rule inserts calls. See Checking that the Archiving Rule Inserts New
Calls on page 135. 7. Troubleshoot the storage rules. See Troubleshooting the Archiving Rule on page 135. 8. Check the archiving table in the database. See Checking the Storage Center Archiving
Tables on page 137. 9. Check the relevant logs. See Checking the Storage Center Logs on page 138, Checking the
Nice Storage Streaming Service Logs on page 139, and Checking the Player Logs for Storage Center Errors on page 139.
Setting the Log Level to DEBUG You change the level of the Storage Center log files and of the Nice Storage Streaming Service log files to DEBUG. To set the log level to DEBUG: 1. Set the Storage Center log level from LOW to DEBUG by changing the entry in the registry: a.
Click Start > Run.
b. Enter Regedit. c.
From the Edit menu, select Find, and enter DEBUGCATEGORYMASK.
d. Set the value to 255. e.
Restart the Storage Center service. Chapter 4: Archiving Troubleshooting Flow
NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
128
Storage Center Troubleshooting
2. Change the log level in Nice Storage Streaming Service log files to DEBUG. a.
Navigate to the Nice Storage Streaming Service storage directory and open Nice.Storage.StorageStreaming.StorageStreamingHost.exe.
b. In the Nice Storage Streaming Service configuration file, change the log Filter attribute to DEBUG. c.
Save the Nice Storage Streaming Service configuration file.
Checking for Archived Calls You can check if calls are archived by running a query in Business Analyzer, checking physically in the folders on the Storage Center or using the Storage Center Investigator. To check for archived calls: 1. Run a query in Business Analyzer to check which segments were archived: a.
Run a query, for example, Segment - Last 7 days.
b. Click Preferences c.
.
In the Table View area, move the Voice Archive Status and Screen Archive Status columns from the Available Columns to the Selected Columns list box.
Chapter 4: Archiving Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
129
Storage Center Troubleshooting
d. Click
Save. The icon
indicates a segment that is not archived.
2. Check the folders on the Storage Center to see if files have been written: a.
The Storage Center contains Storage Groups which are a logical grouping of one or more Storage Units. Each time you create an archiving rule (via the Rules Manager), you associate the rule with a Storage Group. The a Storage Unit contains either:
•
Directory Per Day: Creates subdirectories under the defined directory path (for each
day of the year).
•
Multiple Directories: Creates subdirectories under the defined directory path (based
on the destination type). Files are written to the subdirectories in a cyclic manner: When one subdirectory reaches full capacity, files are written to the next subdirectory. b. In the example below, recordings are archived in Directory Per Day storage units,
organized by server name\task name\archiving date.
Chapter 4: Archiving Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
130
Storage Center Troubleshooting
3. Use Storage Center Investigator to check the archiving. See Using the Storage Center
Investigator on page 109. a.
Double-click the
icon on the desktop to activate the application.
b. Click Start Storage Center Investigator in automatic mode. c.
In the Storage Center Installation directory field, verify that the Storage Center path is correct.
d. In the Available Databases area, select the checkbox of the database you want to query. e.
f.
In the Automatic Reporting Settings fields:
•
Enter the desired time interval to generate automatic reports.
•
Enter the desired number of days backwards to include in the reports.
Click Next to view the results.
g. In the NICE Storage Center Investigator Results tab, click the Database overview
sub-tab to view query results.
Chapter 4: Archiving Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
131
Storage Center Troubleshooting
4. Check the archiving to the Storage Center. In Reporter, run the Archiving Coverage Report, and check the Archiving Coverage per Storage Center view.
Gathering Storage Center Information Use the Storage Center Investigator to verify that all the required files are archived on the file system. The Storage Center Investigator is installed together with the Storage Center in the same installation directory. The default path is: C:\Program files\NICE Systems\Storage Center. See Using the Storage Center Investigator on page 107 for detailed instructions on how to use the Storage Center Investigator.
Chapter 4: Archiving Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
132
Storage Center Troubleshooting
Checking the Storage Retention and Deletion Values The Storage retention value is the number of retention days for the files in the Storage Group, meaning the number of days that the recordings must remain in the Storage Group. By default, there is no mandatory retention. The retention value is usually set to the amount of days that the site is obligated to save recordings. The maximum retention period is seven years (based on the NICE database design). When you define a storage rule, you can define a value for Force Deletion, meaning that files archived by a specific rule will be deleted after a certain number of days. What happens if a conflict occurs between the group's Retention value and a rule's Force Deletion value (such as the group's Retention value is set to 100 days)? The Force Deletion value of the rule takes preference! The group Retention value is a more general definition; the archiving rule Force Deletion value is specific to a group of archived files.
Verifying that the Storage Center Database is Selected In the System Administrator, check that the Storage Center database is selected.
•
In System Administrator, in the Organization tree, under Master Site > Database Servers, General tab, check that the nice_storage_center database is selected in the Databases on this SQL Server area.
Verifying the Used and Free Space Check the used and free space on the Storage Center servers and storage areas. 1. In System Administrator, in the Organization tree, under System Monitoring > Storage Center Servers branch, expand the hierarchy to view a list of all Storage Centers. 2. Expand a Storage Center to view its storage areas.
Chapter 4: Archiving Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
133
Storage Center Troubleshooting
3. Select a Storage Center or one of its storage areas. The Status window appears.
Verifying the Storage Unit Threshold Check the storage unit threshold on the Storage Center server. To verify the storage unit threshold: 1. Log in to the NICE Perform Application Suite. In the navigation bar, click Accessories, and select System Administrator.
2. From the Settings menu, select Technician Mode.
3. In the Organization Tree, select the Storage Center Servers branch. 4. Expand a Storage Center to view its storage groups and units (SCGroup > SCUnit), and
check the Maximum Threshold.
Chapter 4: Archiving Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
134
Storage Center Troubleshooting
Figure 4-4
Storage Details
Maximum Threshold
Checking that the Archiving Rule Inserts New Calls Archiving to Storage Center starts with the archiving rule. The rule determines which calls needs to be archived and then adds the required information to the tblStorageCenter and tblSCTasks tables in the database. 1. Check the Storage Center tables, tblStorageCenterXX in the nice_interactions of the
relevant calls to verify that the Rule Engine inserts new calls. From Storage Center Investigator, query iArchiveID to see if the ID number increases, and check the Start and Stop times of the last interactions. When you know the time of the last interaction, you can verify whether new interactions were inserted. 2. Check whether new tasks are added to the tblSCTasks table in the nice_admin database. If
tasks are added, check that they match the rule filters. If the Rule Engine inserts new calls into tblStorageCenterXX, the archiving rule is probably correct and the problem is in the Storage Center. In this case, continue with Checking the Storage Center Archiving Tables on page 137. If there are no new calls, continue with Troubleshooting the Archiving Rule on page 135.
Troubleshooting the Archiving Rule Troubleshooting the archiving rule comprises the following steps: 1. Check the Rule Engine Log file for errors. 2. Check that there is an Active Rule in the Rule Manager, and check its schedule. 3. Check the Storage Center rule schedule. See the procedure below. 4. Check that the Rule Engine generates actions. See page 136. To check the Storage Center rule schedule: 1. In the nice_rule database, check the next scheduled time of the Storage Center rule by
running the following query: SELECT tblREStorageCenterRule.nvcRuleName, tblRELocalRuleSchedule.iScheduleID, tblRELocalRuleSchedule.dtNextScheduleTime, Chapter 4: Archiving Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
135
Storage Center Troubleshooting
tblRELocalRuleSchedule.dtLastRunTime FROM tblRELocalRuleSchedule INNER JOIN tblREStorageCenterRule ON tblRELocalRuleSchedule.iScheduleID = tblREStorageCenterRule.iScheduleID WHERE (tblREStorageCenterRule.iStatus = 1)
A section of the tblREStorageCenterRule table appears below, showing the format of the date fields.
dtNextScheduleTime is the GMT time. dtLastRunTime is the local time of the Applications Server. 2. Check that the Rule Engine database is not full. To check that the Rule Engine generates actions: 1. Check that the Rule Engine generates actions for the Storage Center Rules by running the
following query: SELECT * FROM tblREStorageCenterAction 2. In the query results, check the start time, stop time and interaction IDs in the results of the
query. 3. If the table does not fill up, check that the Event Providers collect the interactions by running
the query: SELECT * FROM tblRECallServerEvent 4. Check that the results contain calls that are not archived.
•
If the table does not contain the calls, the error can be in the Event Collection step. Call NICE Customer Services.
•
If the interaction is in the Event table, the problem can be in the Generation step. The reason could be one of the following:
•
The Generation Query takes a lot of time and does not complete because of timeout. Maybe there are missing indexes. See the Database Administrator.
•
The rule has problems with the file name mask and the post process of the Storage Center rule. The database function spREStorageCenterRulePostProcessing failed. Check that the File name mask is not empty by running the following query:
Chapter 4: Archiving Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
136
Storage Center Troubleshooting
SELECT nvcRuleName, iRuleId, nvcDescription, iStatus, nvcFileNameMask FROM tblREStorageCenterRule
Check that nvcFileNameMask is not empty for the required rule.
•
The problem could be in the Execution step. In a Multi Data Hub environment, the secondary Rule Engine Log file can contain the following message: [StorageCenterExecuter.OnProcessWorkItem] Failed to execute Storage Center action System.Data.SqlClient.SqlException: The operation could not be performed because the OLE DB provider 'SQLOLEDB' was unable to begin a distributed transaction. [OLE/DB provider returned message: New transaction cannot enlist in the specified transaction coordinator.] OLE DB error trace [OLE/DB Provider 'SQLOLEDB' ITransactionJoin:JoinTransaction returned 0x8004d00a].
•
It may be that the Microsoft Distribute Transaction Coordinator (MSDTC) is not configured properly. Storage Center rules require that the MSDTC works correctly. This can be checked by performing the following command: BEGIN TRANSACTION SELECT * FROM nice_rule_link.nice_admin.dbo.tblSCTasks COMMIT
If the command fails, MSDTC must be fixed.
•
In a Multi Data Hub environment, the problem could be in the Rules Manager behavior. Rules defined by the Rules Manager are relevant for all sites. The Rules definitions are written to the Master site and are replicated to all Secondary Data Hubs.
Checking the Storage Center Archiving Tables 1. Check the iFSArchiveClass column in the tblStorageCenterXX table. The values represent
the following: ID
Status
Action
0
Not archived yet to Storage Center file system.
Check if the Storage Center service is up and check the Storage Center log files.
1
Archiving in progress to Storage Center file system.
Check the Storage Center log files.
2
Archived to Storage Center file system.
Everything is OK.
3
Call is being deleted from Storage Center file system.
Chapter 4: Archiving Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
137
Storage Center Troubleshooting
ID
Status
Action
4
Call deleted from Storage Center file system.
5
Archiving failed to Storage Center file system.
Check the Storage Center log files.
6
Call deletion from Storage Center file system failed.
Check the Storage Center log files.
7
Call was not found in the Logger.
Check why the call is not found in the Logger
2. Check the iEsmArchiveClass column in the tblStorageCenterXX table. The values
represent the following: ID
Status
Action
0
Not archived yet to ESM.
Check if the Storage Center service is up and check the Storage Center log files.
1
Archiving in progress to ESM.
Check the Storage Center log files.
2
Archived to ESM.
Everything is OK.
3
Call is being deleted from ESM.
4
Call deleted from ESM.
Check the Storage Center log files.
5
Archiving failed to ESM
Check the Storage Center log files.
6
Call was not found in the Logger.
Check the Storage Center log filesr
Checking the Storage Center Logs 1. Review the Storage Center logs. By default, the log files are stored in the same directory
where the Storage Center is running. The default log file names are:
•
StorageCenterLog.log
•
StorageCenterLog1.log through StorageCenterLog100.log Time
Event Category
Owner
Description
Thread ID Chapter 4: Archiving Troubleshooting Flow
NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
138
Storage Center Troubleshooting
2. Track the task snapshots in the log files. Each task snapshot details the current tasks and the
number of elements distributed among Loggers. Task Name Task ID
Logger ID
Rule ID
Number of Elements
3. Ensure that the logs do not contain errors relating to the Storage Center startup process. 4. To collect the logs, use the Log Collector. See the Collecting Information Using the Log
Collector on page 537 for further information.
Checking the Nice Storage Streaming Service Logs You check the Nice Storage Streaming Service logs to verify that playback performed from Storage Center is successful. By default, the log files are stored in the following directory:...\Program Files\Nice Systems\Storage Center\Nice Playback Service SC\StorageStreamingLogs. The Process ID is a value that changes each time the system is restarted. The following procedure lists the text strings you need to find in the Nice Storage Streaming Service logs in order to verify that playback from the Storage Center was successful. To check the Nice Storage Streaming Service logs: 1. Search for the text ERR:, and verify that it does not appear in the log file. 2. Search for the text StartOutgoingSessionEvent, and verify that it appears. 3. Search for the text OpenSession, and verify that it appears. 4. In the same row as the OpenSession text, verify that Result=OK appears. 5. Search for the text StartStreamingPackets, and verify that it appears. 6. Search for the text StopOutgoingSessionEvent, and verify that it appears. 7. Search for the text StopStreamingPackets, and verify that it appears.
Checking the Player Logs for Storage Center Errors The player errors are recorded in the Player log files, in *.txt format, in %APP_DATA%\NicePlayer\Release3. The following Storage Center errors may appear in the Player logs.
Chapter 4: Archiving Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
139
Storage Center Troubleshooting
Error No.
Description
Action Required
300007
The Storage Center failed to connect to a backup device on a Logger. This should not be considered an error if there are no backup devices in the site.
Check the Logger where the backup device is located.
300008
The Storage Center did not archive or it had an error in the locate flow.
See the Playback Troubleshooting Flow on page 77.
Other Troubleshooting Issues •
If the Storage Center is not communicating with the NMS, see the Storage Center Installation Guide. Perform the following verification tests to ensure that all the objects are configured properly:
•
Verify Nice Storage Center Service User
•
Verify the SnmpAgent Object
•
Verify the SNMP Service Settings
•
If interactions are not being archived on a Storage Center server with multiple active network interface cards, check that the Storage Center IP address defined in the System Administrator is the primary network connection. For more details, see the Storage Center Installation Guide.
•
If there are non-archived calls in the Storage Center tables, use the Storage Center Investigator to check if the iRuleID of the calls is still active. Storage Center will not archive calls that belong to an inactive rule. In this case, change the iRuleID to an active one. See Using the Storage Center Investigator on page 107.
Chapter 4: Archiving Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
140
5 Multi Data Hub Troubleshooting Flow This chapter describes common troubleshooting issues when employing a Multi Data Hub with NICE Perform Release 3.1.
Contents Gathering Information ..................................................................................................142 Multi Data Hub Implications.........................................................................................143 Checking Connectivity .................................................................................................144 Checking Replication ...................................................................................................145 Troubleshooting Replication .......................................................................................147 Troubleshooting Saving in a Multi Data Hub Environment ......................................148 Changing Passwords ...................................................................................................148 Enabling MSDTC to Function in Multi Site Environments ........................................148 Other Troubleshooting .................................................................................................149
Chapter 5: Multi Data Hub Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
141
Gathering Information
Gathering Information Important issues to consider are:
•
Is this a new installation? If yes, when was the installation performed?
•
Were there any changes to the environment that might have influenced the system? If yes, check these changes first.
•
How often does the problem occur?
•
Is the problem reproducible? If yes, what were the steps?
•
Were the Servers restarted? If yes, did this resolve the problem?
Chapter 5: Multi Data Hub Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
142
Multi Data Hub Implications
Multi Data Hub Implications A Multi Data Hub system has site implications that affects the different system components. Changes in the components are all done automatically with no user intervention or configuration.
Database •
The Admin, Interaction, Audit and Rule databases are mandatory on all Data Hubs.
•
The nice_interactions databases are not replicated across Data Hubs.
System Administrator •
Administration updates are blocked when the Primary Data Hub is not available.
•
The login mechanism will check login token on all Data Hubs. Users will belong to a specific Data Hub.
Rule Engine •
All rules are replicated from the Master Data Hub to the Secondary Data Hubs.
•
Each local rule engine will handle the local rules.
•
Storage Center rules are an exception as they are all stored only on the Master Data Hub. This means that all Data Hubs operate using the same Storage Center rules.
Business Analyzer •
Public, private and saved items are replicated between all Data Hubs.
•
User preferences are local.
•
Running a query on Multi Data Hubs:
•
•
The Unified Query engine queries the local Data Hub.
•
Each remote Data Hub with agents from the query filter will be also queried.
•
All results will be combined into one result.
Locate
•
The Unified Query service will connect all relevant Locate services in all Data Hubs.
•
The Locate process looks only in the local Data Hub according to the locate order.
Playback •
Each Data Hub has its own Media Server.
•
One or more Playback Servers can be configured in each Data Hub.
Chapter 5: Multi Data Hub Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
143
Checking Connectivity
Monitor •
User preferences are saved on the local Data Hub.
•
Monitor will subscribe to events to the local CLS or remote Monitor service.
Quality Management •
All QM objects (Evaluations, Coaching Packages, Calibrations and Clips) will be saved in the user's local Data Hub.
•
Evaluations on remote calls will be replicated to the remote Data Hubs.
•
Form definitions are replicated to all remote servers.
My Universe •
Preferences are stored locally.
Interaction Server •
All recording rules are replicated.
•
Only handles local Integrations.
Storage Center •
Tasks are read from the Master Data Hub only.
•
Archiving can be done from all Data Hubs.
Checking Connectivity Check connectivity between sites (PING using IP address and hostname).
Chapter 5: Multi Data Hub Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
144
Checking Replication
Checking Replication Use the following methods to check replication in Multi Data Hub environments. To check replication in multi data hub environments: 1. In the NICE System Administrator, verify that the replication is defined using only the
machine names. 2. Check the connectivity between the Primary and Secondary Data Hubs. 3. Connect to the SQL Server Management Studio. In the Object Explorer, navigate to Replication and check if there are components with an error. 4. In the NICE System Administrator: a.
Navigate to Master Site > Database Servers > Databases and click the Replication tab.
b. Verify that no errors appear in the Status column. 5. If the Primary and Secondary Data Hubs have different database structures, the replication will
fail at those specific points. To ensure that the replicated data is identical, run tests on all Data Hubs to ensure that additional information (business data, any external additions such as Professional Services solutions) can be accessed from different sites. 6. Verify that the same version of the NICE Perform Database and the same updates are installed
on all Data Hubs. Check the following tables in the nice_admin database: a.
The database version appears in the tblDatabaseDetails table. Figure 5-1
tblDatabaseDetailsTable
b. The update versions appear in the tblDatabaseHotFix table. Figure 5-2
tblDatabaseHotFixTable
NOTE: Only those updates that were installed using an Update Pack appear in the tblDatabaseHotFix table; updates that were installed manually do not appear.
Chapter 5: Multi Data Hub Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
145
Checking Replication
7. When a Multi Data Hub environment is upgraded, replication should be removed during the
procedures. After the upgrade has been completed, create the replication again. Then test that the replication is functioning properly.
Chapter 5: Multi Data Hub Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
146
Troubleshooting Replication
Troubleshooting Replication In most problems that deal with replication, the Primary Data Hub is usually updated with information. If there is a problem updating the Primary Data Hub, the replication itself will not begin and the problem lies elsewhere. It is therefore important to first verify that the Primary Data Hub has been updated and only then start troubleshooting replication problems.
Troubleshooting Communication Problems between Primary and Secondary Data Hubs Secondary Data Hubs are read only. All updates are written to the Master Data Hub and replicated to the Secondary Data Hub. To troubleshoot communication problems: 1. If there is a communication problem between the Primary and Secondary Data Hubs, data
updates on the Secondary Data Hub will not take effect. These data updates will not be stored locally and will have to be completed once the communication problems have been resolved. 2. If a Secondary Data Hub is currently disconnected from the Master Data Hub, updates from
other Data Hubs will not be seen. Since all information is stored in the distribution database on the Master Data Hub, these data updates will be completed once the Data Hub is connected again.
Troubleshooting Database Structure Differences The database structure can be changed in different ways, for example, names, number of fields, field type, field length, additional or fewer tables. If a Master Data Hub and Secondary Data Hub have different database structures the replication will fail at those specific points. NOTE: It is very important not to make "manual" changes to the Secondary Data Hubs as these will be overwritten when the data is replicated from the Primary Data Hub. To check the database structure: 1. To ensure that the data is identical, run tests on all Data Hubs to ensure that additional
information (business data or any external additions such as Professional Services solutions) can be accessed from the different Data Hubs. 2. For example, an error from the Nice DW Population Schedule job may indicate a problem in the tblReservedXX tables (containing the business data fields).
Stopping Replication When Upgrading All Data Hubs Upgrades must be applied to all Data Hubs simultaneously. An upgrade can be an Update or Service Pack. Since upgrades may cause a change within the SQL structure, it is important to remove the replication before implementing the upgrade.
Chapter 5: Multi Data Hub Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
147
Troubleshooting Saving in a Multi Data Hub Environment
NOTE: You must bring down communication between all the Data Hubs when upgrading. To stop replication when upgrading all data hubs: 1. Remove the replication prior to the upgrade. The replication should be stopped to all Data
Hubs, not just the Data Hub that is being upgraded 2. Once the upgrade is complete, create the replication again. 3. Test the site operation.
Other Troubleshooting Issues •
Replication works with the Machine Name only.
•
Any change to the database scheme of replicated or un-replicated tables requires that the replication be shut down for that site.
Troubleshooting Saving in a Multi Data Hub Environment In a Multi Data Hub environment, some of the information in nice_admin is replicated. If replication is down, you will be unable to save in System Administrator.
Changing Passwords If the nicesa user password is changed, the replication agents will indicate that the password is not valid and the replication will not work and must be recreated. To change nicesa password in Multi Data Hub environment: 1. Remove the replications. 2. Change the nicesa password. 3. Recreate the replications.
Enabling MSDTC to Function in Multi Site Environments The SQL Server uses Microsoft Distribute Transaction Coordinator (MSDTC) to execute distributed transactions. MSDTC will not function properly in an environment where the two servers are using the same Windows OS image, for example where the 'Ghost' program is used to copy the Windows image to the disk. The reason for the problem is that both servers that are installed from the same image, have an identical key in the MSTDC section in the SQL registry. If the distributed transaction fails and the registry keys are different, there may be other reasons for the failure. See TN0676 Enabling Microsoft Distribute TransactionCoordinator (MSDTC) to Function in Multi Site Environments. This note describes how to fix the problem without re-installing the entire system.
Chapter 5: Multi Data Hub Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
148
Other Troubleshooting
Other Troubleshooting •
A replication error Cannot insert explicit value for identity column in table 'tblColumnInfo' appears. This occurs if the subscription table tblColumnInfo.iColumnId identity is set to true. To solve the problem set the identity value to False.
•
If there is a problem with replication, submit the following log file to NICE Customer Support: Program Files\NICE Systems\ Applications\ServerBin\NiceApplications.SystemAdministrator.Host.exe.log
Chapter 5: Multi Data Hub Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
149
Blank page for double-sided printing.
6 Media Encryption Troubleshooting Flow This chapter includes tables of trap messages and ScreenAgent Management System (SAMS) log messages as well as common troubleshooting issues.
Contents Troubleshooting Workflow ..........................................................................................152 Trap Messages ..............................................................................................................159 SAMS Log Messages ...................................................................................................163 Verifying the IIS User Permissions for the KSM Configuration................................164 Using a Fully Qualified Domain Name instead of a Host Name ...............................166 Data Not Entered in the nice_crypto Database ..........................................................169 Renewing Certificates ..................................................................................................171 ScreenAgent Not Working Properly............................................................................172 Ensuring Connection between SQL Server and Applications Server .....................175 Installing the CA certificate on Vista Workstations ...................................................175
Chapter 6: Media Encryption Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
151
Troubleshooting Workflow
Troubleshooting Workflow If you experience a problem with Media Encryption follow the steps below to isolate the problem. 1. Change the logging level to ALL. See Changing the Logging Level on page 153. 2. On the Key Storage Manager (KSM), check the connectivity between the Applications Server
and the Database. See Checking Server Connections on page 153. 3. Verify the connectivity between the Logger/ScreenAgent and the KSM on the Applications
Server by pinging the Applications Server. 4. Verify the connectivity between the KSM on the Applications Server and the Cryptographic
Database by pinging the SQL Server which has the NICE Cryptographic Database (nice_crypto). 5. Use Telnet to check that the Logger/ScreenAgent can reach the Applications Server on port
443. 6. Check the Logger/ScreenAgent. See Checking the Logger/ScreenAgent on page 153. 7. Check the KSM status. See Checking the KSM Status on page 154. 8. If the KSM status is FAILED, verify that the KSM is functioning properly. See Verifying the
KSM is Functioning on page 154. 9. Check the CIM trap messages. See Checking the SNMP Traps on page 154. 10. Check the KSM trap messages. See Checking the SNMP Traps on page 154. 11. Check the client-side logs. See Checking the Client-side Logs on page 156. 12. Check the server-side logs. See Checking the Server-side Logs on page 156. 13. Check for certificate problems. See Checking for Certificate Problems on page 157.
Gathering Information Important issues to consider are: 1. Is this a new installation? If yes, when was the installation performed? 2. Were there any changes to the environment that might have influenced the system? If yes,
check these changes first. 3. How often does the problem occur? 4. Is the problem reproducible? If yes, what were the steps? 5. Were the Servers restarted? If yes, did this resolve the problem?
Chapter 6: Media Encryption Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
152
Troubleshooting Workflow
Changing the Logging Level To generate more detailed logs, you change the logging level in the Logger and the ScreenAgent. To change the logging level:
•
On the Logger/ScreenAgent, in ConfigPlugin, set the debug level, by changing the values in both Level and Threshold to ALL:
<param name="Level" value="ALL" /> <param name="Threshold" value="ALL" />
Checking Server Connections You can check on the KSM whether the files were sent to the SQL Server database. If the folder contains files, this indicates a connection problem between the Applications Server and the SQL Server database. To check the Applications Server to Database connection:
•
On the KSM, check the EncryptionBackupFiles folder located at the following path: ...\\KeyStoreManager\WebService\EncryptionBackupFiles.
If this folder contains files, this indicates a problem with the connection between the Applications Server and the SQL Server database.
Checking the Logger/ScreenAgent You need to verify that the encryption services are running properly on the Logger/ScreenAgent. To check the Logger/ScreenAgent: 1. On the Logger/ScreenAgent, verify that the LogService located in C:\Program Files\Common
Files\Nice\LogService\Bin\LogSysSrvr.exe is running. 2. Verify that the CIMService.exe service is running. 3. Verify that CIMService.exe service has Local System Account privileges. 4. Verify that the service has privileges to write to the disk. 5. Verify that there is enough space on the disk.
Chapter 6: Media Encryption Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
153
Troubleshooting Workflow
6. Check if the CimClinet, CimHistory and CimKSM files exist in the C:\document and settings\all users\application data\Nice\CryptoInfo folder on the Logger/ScreenAgent.
The size of the CimKSM file is 0 KB. If the CimKSM file is not 0 KB, then there is a problem with the connection between the Logger and the KSM.
Checking the SNMP Traps In the SNMP Console, view the events generated by the CIM and KSM. See the NMS User's Guide or Trap Messages on page 159 for a list of the trap messages and recommended actions.
Checking the KSM Status To check the KSM status: 1. In the navigation bar, click Accessories, and select System Administrator. 2. Verify that you are working in Technician mode: From the Settings menu, select Technician Mode.
3. In the Organization tree, select Security > Media Encryption.
4. Click Test to check the KSM status. If the status is Failed, verify that the KSM is functioning
properly. See Verifying the KSM is Functioning on page 154.
Verifying the KSM is Functioning When you cannot connect to the KSM, you do not know if the problem lies with the KSM or with the client machine. In this procedure, you verify that the KSM is functioning properly.
Chapter 6: Media Encryption Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
154
Troubleshooting Workflow
To verify if KSM functions properly: 1. On the KSM server, navigate to IIS > Web Sites > Default Web Site > KSMWebService. 2. In the right pane, right-click KSMService.asmx and select Browse.
The Internet Explorer opens. 3. In the Address field, change the URL address to: https:///KSMWebService/KSMService.asmx.
If the following screen appears, then the problem is probably with the client workstation.
Chapter 6: Media Encryption Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
155
Troubleshooting Workflow
NOTE: If you did not replace the name of the local host with the KSM machine name, the following window appears:
Make sure that you click No. (If you click Yes, the verification process is skipped.)
Checking the Client-side Logs Check the following client-side logs:
•
Nice Player :\Documents and Settings\Administrator\Local Settings\Application Data\NicePlayer\Release3\NicePlayer.txt
•
Logger/ScreenAgent :\Documents and Settings\All Users\Application Data\Nice\Log\CryptoInfoManagerLogExceptions.log
•
Encryption :\Documents and Settings\Administrator\Local Settings\Application Data\NICE\EncryptionPluginClientLog.txt
Checking the Server-side Logs Check the following server-side logs:
•
KSM C:\Program Files\NICE Systems\Applications\KeyStorageManager\WebService\KeyStorageManager.txt
•
Encryption C:\ Program Files\NICE Systems\Applications\ServerBin\NiceApplications.EncryptionPlugin.log
Chapter 6: Media Encryption Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
156
Troubleshooting Workflow
Checking for Certificate Problems The certificates might have expired. To check for certificate problems: 1. Check with your certification authority if your certificates have expired. 2. On the Applications Server, check in the IIS log files (KeyStorageManager.txt.x) for the
following type of error: ERROR: [SNMP.RegistrateToTheSNMPConnector] SnmpConnector: New SNMPConnectorObject was added to the connector.
This indicates a problem with the site certificate. Check for other errors as well. 3. On the Logger/ScreenAgent, check in the Event Viewer if there are any CIMService error
messages. In the Source EFS and Description areas, the following error indicates a problem with the site certificate. An error with Source: EFS and Description: EFS recovery policy contains invalid recovery certificate.
4. On the Logger/ScreenAgent, check the properties of the CryptoInfo folder: a.
Go to C:\Documents and Settings\All Users\Application Data\CryptoInfo, right-click the CryptoInfo folder and select Properties. Chapter 6: Media Encryption Troubleshooting Flow
NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
157
Troubleshooting Workflow
b. Select Advanced. The Advanced Attributes window appears.
c.
Select Encrypt contents to secure data. If you receive the following error, this indicates a problem with the site certificate.
5. Issue new certificates for the SQL Server and IIS.
Chapter 6: Media Encryption Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
158
Trap Messages
Trap Messages Trap Messages for CIM Module Trap No.
Trap
Cause
Action
6141
Cryptographic Information Manager cannot connect to KSM.
There is no certificate. (This trap will be sent with the no certificate trap).
Install CA root on the machine.
There is a problem in the KSM (not running).
Check the DB of the SysAdmin for the KSM URL – see that its valid.
The KSM URL obtained from the System Administrator is invalid. 6142
Check the KSM.
Cryptographic Information Manager cannot connect to System Administrator.
There is no connection to the System Administrator.
Check that the SysAdmin is up.
System Administrator URL is invalid.
Install the client machine once again and change the SysAdmin URL.
6143
Cryptographic Information Manager cannot save encryption data to a persistent file.
The file attribute is read only.
Change the attribute of the files to read/write. The path is C:\Documents and Settings\All Users\Application Data\Nice\CryptoInfo.
6144
Cryptographic Information Manager is not initialized.
There is probably a problem in the initialization (such as encryption not configured). This trap will be sent together with another trap.
6145
Encryption is not configured in the system.
The encryption is not yet configured in the system. The KSM is not yet configured.
Check the entire encryption components and check that the KSM is configured properly.
6146
The Customer has no valid certificate for encryption.
There is no certificate installed on the client machine.
Install the CA root in client machine.
SystAdmin URL registry key is empty.
There is no SysAdmin in the registry – it was not provided in the client installation.
6147
(Make sure that the CA is installed both on current user and on the local machine.) Run the client installation and set the SysAdmin URL.
Chapter 6: Media Encryption Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
159
Trap Messages
Trap No.
Trap
Cause
Action
6148
KSM URL does not exist in the System Administrator.
The SysAdmin URL is not aware of the KSM URL, therefore when asked for the KSM URL it returned null.
Check the SysAdmin database for the KSM URL.
6149
There is no license for encryption in the system.
The encryption feature does not exist in the client site.
Check that there is license for encryption for the client.
6150
The Logger login token is not valid.
The login token against SysAdmin is invalid probably because of mismatch between DLLs in the CIM server and in the SysAdmin.
Check the version of the SysAdmin DLLs in the CIM Server - check that this is the correct version.
Trap Messages for KSM Module Trap No.
Trap
Cause
Action
5200
Cannot establish a secure connection between Key Storage Manager and the database.
a. Database is down.
a. Start the database.
b. Connection problem to the database.
b. Verify that the connection to the database is OK (using the Query Analyzer).
c. Secure connection problem to the database: no CA root installed on the KSM machine, problem with the SQL certificate.
5201
Secure connection between Key Storage Manager and database was established successfully.
Fixes the above trap.
5202
The database certificate is missing/corrupted: data cannot be encrypted/decrypted.
The database certificate is missing. Therefore, no encryption or decryption of the encrypted data is possible.
5203
The database certificate is valid: data can be encrypted/decrypted.
Fixes the above trap.
c. Check that there is a CA root installed on the KSM machine. Verify that there is a certificate assigned to the SQL service and that it is valid.
Call NICE Support Services.
Chapter 6: Media Encryption Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
160
Trap Messages
Trap No.
Trap
Cause
Action
5204
Cannot establish a secure connection to remote Key Storage Manager.
a. Remote KSM machine is down.
a. Start the machine.
b. No connection to the remote KSM machine.
b. Verify that the connection to the remote machine is OK.
c. Remote KSM web service is unavailable. d. There is no CA root installed on the local KSM machine.
5205
Secure connection to remote Key Storage Manager was established successfully.
Fixes the above trap.
5206
Database master key cannot be opened: data cannot be encrypted/decrypted.
Restore procedure of the database was incomplete: the database master key was not restored correctly.
c. Use the Media Encryption branch in order to test the connection to the remote KSM. d. Verify that there is a CA root installed on the local KSM machine. Also, use the Media Encryption branch from the local KSM machine in order to test the connection to the remote KSM.
Check in the nice_crypto database, security, certificate folder, that the certificate exists. Notify NICE Support Services.
5207
Database master key opened successfully.
Fixes the above trap.
5208
Duplicated GUID was stored to the database.
A GUID that already exists.
KSM module is up.
The KSM web service is up.
Failed while writing Cryptographic Information to file.
The KSM failed to write to the backup file. This could happen if the KSM application pool user does not have privileges to the EncryptionBackUpFiles folder.
5209
Notify NICE Support Services.
Use the Media Encryption branch in order to test the KSM status. KSM also tries to create a file in the EncryptionBackUpFiles folder.
Chapter 6: Media Encryption Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
161
Trap Messages
Trap No.
Trap
Cause
5210
Succeeding in writing Cryptographic Information to file.
Fixes the above trap.
5211
Failed while reading Cryptographic Information from file.
a. The KSM failed reading the backup file. This could happen if the KSM application pool user does not have privileges to the EncryptionBackUpFiles folder. b. Can also happen if the file was written and encrypted by a different user (The KSM application pool user was replaced).
5212
Succeeded in reading Cryptographic Information from file.
Action
a. Verify that the KSM application pool user has read privileges to the EncryptionBackUpFiles folder. b. Check which user encrypted the file, by right-clicking the file > Properties. Then in the General tab, click Advanced > Details. If the user is not the current KSM application pool user, change it to the user who encrypted the file, and wait for the file’s content to be written to the database.
Fixes the above trap.
Chapter 6: Media Encryption Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
162
SAMS Log Messages
SAMS Log Messages ScreenAgent Module Messages Message.
Cause
Action
ScreenAgent Behavior
Problem to connect to server
CIM Error
ScreenAgent will retry until the operation succeeds.
Server not initialized
CIM Error
ScreenAgent will retry until the operation succeeds
Problem connecting to KSM
Network, Server is down, wrong IP, etc.
Verify server is up, IP is correct, etc.
ScreenAgent will record with encryption.
Problem connecting to SysAdmin
Network, Server is down, wrong IP, etc.
Verify server is up, IP is correct, etc.
ScreenAgent will retry until the operation succeeds.
Encryption not configured
Encryption is not configured in SysAdmin
Configure Encryption.
ScreenAgent will retry until the operation succeeds
Client is not connected
CIM Error
-
ScreenAgent will retry until the operation succeeds
Problem to create client connection
CIM Error
-
ScreenAgent will retry until the operation succeeds.
Problem in client connection
CIM Error
-
ScreenAgent will retry until the operation succeeds.
SysAdmin URL does not exist
Enter the SysAdmin URL in the registry.
ScreenAgent will retry until the operation succeeds.
KSM URL does not exist
Enter the KSM URL in the registry.
ScreenAgent will retry until the operation succeeds.
Problem saving Crypto to file
CIM Error
-
ScreenAgent will retry until the operation succeeds.
No license for encryption in site
No encryption license in the System.
If the site is to be an encryption site, install a license. If the site is not an encryption site, reinstall the ScreenAgent with no encryption.
ScreenAgent will record without encryption.
No certificate on the machine
Certificate was not installed on the workstation.
Install certificate.
ScreenAgent will record with encryption.
Chapter 6: Media Encryption Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
163
Verifying the IIS User Permissions for the KSM Configuration
Verifying the IIS User Permissions for the KSM Configuration If there is an error when configuring the KSM, you need to verify that the appropriate permissions are granted to the IIS user for the following folders:
•
Temp folder
•
EncryptionBackupFiles folder
•
WebService folder
IMPORTANT : Make sure that all permissions are also inherited by sub-folders.
To verify IIS user identity permissions: 1. Open the C:\Windows\Temp folder. 2. Click the Security Tab.
The following window appears.
3. Select the group or user names of the current IIS user. 4. In the Permissions for Administrators area, verify that the following permissions are selected in the Allow column: Chapter 6: Media Encryption Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
164
Verifying the IIS User Permissions for the KSM Configuration
•
Read & Execute
•
Read
•
Write
5. Click OK. 6. Navigate to the
...\\KeyStoreManager\WebService\EncryptionBackupFiles folder and repeat Step 2 and Step 3. 7. In the Permissions for Administrators area, verify that the Full Control permission is selected in the Allow column. 8. Navigate to the ...\\KeyStoreManager\WebService\WebService
folder and repeat Step 2 and Step 3. 9. In the Permissions for Administrators area, verify that the Full Control permission is selected in the Allow column.
Chapter 6: Media Encryption Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
165
Using a Fully Qualified Domain Name instead of a Host Name
Using a Fully Qualified Domain Name instead of a Host Name If the IIS and SQL certificates require a Fully Qualified Domain Name (FQDN), you need to ensure that all calls to the KSM Web service and connections to the Database Server are made through the FQDN, as opposed to through the host name. IMPORTANT : Ping the Database Server using the FQDN to verify that it can connect in this manner before changing the host name. The FQDN is displayed in the Full computer name field of the Computer Name tab in the System Properties window. (Right-click My Computer and select Properties to open the System Properties window.)
To use an FQDN for the KSM Web service and SQL Server: 1. Ping the Database Server using the FQDN to verify that it can connect in this way. 2. From the System Administrator application, expand the Master Site (Primary Data Hub) branch and select Applications > Application Servers >
Storage Manager is installed>.
Chapter 6: Media Encryption Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
166
Using a Fully Qualified Domain Name instead of a Host Name
The Application Server Services screen appears.
3. In the Host Name/IP Address field of the Key Storage Manager (KSM), enter the FQDN
instead of the host name. 4. Restart the CIM service on all Loggers and ScreenAgents. 5. Select Database Servers.
The Database Server screen appears.
Chapter 6: Media Encryption Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
167
Using a Fully Qualified Domain Name instead of a Host Name
Figure 6-1
Database Server Screen
6. In the Host Name/IP Address field of the SQL database, enter the FQDN instead of the host
name. 7. Restart the SQL Server.
Chapter 6: Media Encryption Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
168
Data Not Entered in the nice_crypto Database
Data Not Entered in the nice_crypto Database Sometimes the data in the CryptoInfoData.1.txt file is not entered as expected into the nice_crypto database. This happens if the current KSM user is not the user who originally encrypted the file. In this procedure, we do the following:
•
Verify the identify of the original user who encrypted the file
•
Change the KSM user back to the original user in order to access the file
•
Change the original KSM user back to the current KSM user
To enter data from CryptoInfoData.1.txt file into the nice_crypto database: 1. Right-click the CryptoInfoData.1.txt file, and select Properties. NOTE: By default, the CryptoInfoData.1.txt file is located at the following path: ...\Program Files\NICE Systems\Applications\KeyStorageManager\WebService\ EncryptionBackupFiles 2. On the General tab, click Advanced.
The Advanced Attributes window appears.
3. Click Details.
Chapter 6: Media Encryption Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
169
Data Not Entered in the nice_crypto Database
The Encryption Details window appears.
4. The user who originally encrypted the file appears in the Data Recovery Agents For This File As Defined By Recovery Policy area. 5. If the original user is different than the current user, configure the Key Storage Manager server
with the original user. 6. Restart IIS. 7. Wait for the nice_crypto database to access the CryptoInfoData.1.txt file. 8. Change the original user back to the current user.
Chapter 6: Media Encryption Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
170
Renewing Certificates
Renewing Certificates Instead of renewing certificates, you need to issue new certificates for the SQL Server and IIS.
Screens Not Recorded At times, screens are not recorded when encryption is enabled, and the following error message appears: CCimHelper: GenerateCryptoInfo failed. Retrying...: Problem saving Crypto to file. state = -4>.
In this case, the Antivirus software is interfering with screen recording.
Chapter 6: Media Encryption Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
171
ScreenAgent Not Working Properly
ScreenAgent Not Working Properly ScreenAgent is not working properly, and the following appears in the ScreenAgent log file: EVENT: 28/03/07 14:07:08:276 CCimHelper:GenerateCryptoInfo failed. Retrying...: Problem saving Crypto to file. state = -4; Context = (3464).
This problem may occur if the CimClinet, CimHistory and CimKSM files are missing from the C:\document and settings\all users\application data\Nice\CryptoInfo folder on the workstation. This problem occurs as a result of a Windows EFS (Encrypting File System) configuration issue. The certificate must be renewed. To check ScreenAgent: 1. Check if the CIMService.exe service is running. 2. Verify that CIMService.exe service has Local System Account privileges. 3. Verify that the service has privileges to write to the disk. 4. Verify that there is enough space on the disk. 5. Check if the CimClinet, CimHistory and CimKSM files exist in the C:\document and settings\all users\application data\Nice\CryptoInfo folder on the workstation. The size of the CimKSM file is 0 KB.
Chapter 6: Media Encryption Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
172
ScreenAgent Not Working Properly
6. Check in the Event Viewer if there are any CIMService error messages. An error with Source: EFS and Description: EFS recovery policy contains invalid recovery certification indicates a problem with the site certificate.
7. Check if there are any error messages in the SAMS log. 8. Check the CryptoInfo folder properties: a.
Right-click on the CryptoInfo folder and select Properties.
b. Click Advanced.
Chapter 6: Media Encryption Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
173
ScreenAgent Not Working Properly
c.
Select Encrypt contents to secure data. If you get the following error this indicates a problem with the site certificate.
9. Renew the certificate as described in Renewing Certificates on page 171.
Chapter 6: Media Encryption Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
174
Ensuring Connection between SQL Server and Applications Server
Ensuring Connection between SQL Server and Applications Server Sometimes the FQDN of the SQL Server in the SQL certificate differs from the FQDN configured in the Applications Server. When this is the case, the following error message appears in the log file: The certificate's CN name does not match the passed value.
To correct this problem, you need to make sure that the FQDN of the SQL Server and its assigned FQDN in the System Administrator are the same. To ensure a connection between the SQL Server and the Applications Server: 1. Ping the SQL Server from the Applications Server to get its FQDN in the Applications Server. 2. If the FQDN in the Applications Server differs from the FQDN in the SQL certificate, add the
FQDN of the SQL Server to the DNS. The connection between the SQL Server and the Applications Server is ensured.
Installing the CA certificate on Vista Workstations See the Microsoft Vista Workstation Configuration Guide for information regarding installing the CA certificate on Vista workstations.
Chapter 6: Media Encryption Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
175
Blank page for double-sided printing.
7 Channel Mapping Troubleshooting Flow This section provides quick solutions for common problems you face when defining Channel Mapping. The procedures in this section are designed for the seasoned user who is familiar with the NICE Perform system and the System Administrator application. The procedures in this section are designed to be directly solve the problem at hand. For complete instructions on using the System Administrator, see the System Administrator’s Guide.
Contents Channel Mapping Troubleshooting Workflow ..........................................................179 Gathering Information ..............................................................................................179 Identifying a Channel Mapping Problem ..................................................................179 Verifying the Channel Mapping Configuration..........................................................181 Troubleshooting an Out of Memory Error.................................................................183 Troubleshooting Two Users Updating Channel Mapping Concurrently....................184 Troubleshooting Saving in a Multi Data Hub Environment.......................................184 Adding a New Agent ................................................................................................184 Other Troubleshooting Issues ..................................................................................185 Open Case Procedures ................................................................................................186 Open Case Procedures for System Administrator Parameters (CLS) .....................186 Open Case Procedures for the RCM .......................................................................187 Troubleshooting Scenarios .........................................................................................190 Error Message appears when Updating Configuration ............................................190 A New or Modified Mapping Definition Does Not Work............................................191 New Channel Mapping Definitions are not Saved....................................................191 Channel Mapping does not Work! ............................................................................191 Channel Mapping in System Administrator Differs from what is in Effect ................192 Channels Definitions: Logger does not Appear........................................................192 Channels Definitions: Input Channels do not Appear ..............................................193 Source Definitions: Cannot define Virtual Extensions ..............................................193 Source Definitions: Physical Switch does not Appear..............................................193 Chapter 7: Channel Mapping Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
177
Dynamic Mapping: Screen Logger does not Appear ...............................................194 Dynamic Mapping: Virtual Extensions......................................................................194 Deleting or Changing Logger, Switch, or CLS Definitions........................................194
Chapter 7: Channel Mapping Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
178
Channel Mapping Troubleshooting Workflow
Channel Mapping Troubleshooting Workflow If you experience a problem with Channel Mapping follow the steps below to isolate the problem.
•
Gathering Information
•
Identifying a Channel Mapping Problem
•
Verifying the Channel Mapping Configuration
•
Troubleshooting an Out of Memory Error
•
Troubleshooting Two Users Updating Channel Mapping Concurrently
•
Troubleshooting Saving in a Multi Data Hub Environment
•
Adding a New Agent
•
Other Troubleshooting Issues
•
Open Case Procedures for System Administrator Parameters (CLS)
•
Open Case Procedures for the RCM
Gathering Information Important issues to consider are:
•
Is this a new installation? If yes, when was the installation performed?
•
Were there any changes to the environment that might have influenced the system? If yes, check these changes first.
•
How often does the problem occur?
•
Is the problem reproducible? If yes, what were the steps?
•
Were the Servers restarted? If yes, did this resolve the problem?
Identifying a Channel Mapping Problem Check whether the problem is a Channel Mapping problem using one of the following methods. To identify a Channel Mapping problem: 1.
Check for a Channel Mapping problem using ROD: a. Generate a simple test call. b. Verify that you can see the test call in the Monitor application. If the call does not appear
in the Monitor application, see the Recording Troubleshooting Flow and the Interactions Center Installation Guide. c. If the test call appears in Monitor, try to ROD the test call. If the test call is not recorded,
check the mapping.
Chapter 7: Channel Mapping Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
179
Channel Mapping Troubleshooting Workflow
For a trunk-side recording environment, add the participant trunk group and participant trunk number columns in Business Analyzer, and verify which trunk groups and trunk numbers were assigned to the interactions. Check the mapping for these trunks. 2.
Check for a Channel Mapping problem in the logs: a. Generate a simple test call. b. Verify that you can see the call in the Interactions Center RCM log files. See the
Interactions Center Installation Guide 3.
Check for a Channel Mapping problem by querying the database for Channel Mapping exceptions: a. Query the tblInteractionCatalog table to check over what period the calls occur. b. Get all the exceptions relating to interactions: SELECT * FROM tblExceptionXX, tblInteractionXX WHERE tblExceptionXX.iInteractionID = tblInteractionXX.iInteractionID c. Get all exceptions relating to recordings: SELECT * FROM tblExceptionXX, tblRecordingXX WHERE tblExceptionXX.iRecordingID = tblRecordingXX.iRecordingID d. Check the vcExceptionDetail field in tblException.
4.
Check for a Channel Mapping problem by checking for exceptions in Business Analyzer. Select the interaction and click on the Exceptions tab in the Details pane.
No
Exception
Description
Possible Cause
Recommended Action
15
Unmapped voice recording.
Unmapped voice recording.
Problem in the voice channel configuration.
Switch the logs to DEBUG. Check channels configuration. See if there is a mapping configured for this call. Check the RCM logs. Check what the RCM received in the Start request.
16
Unmapped screen recording.
Unmapped screen recording.
The recording request was received with empty Station or IP address (depends on the screen allocation mode).
If the allocation mode is by IP address, then check if the agent logged in. Check ports. Verify in RCM logs that the start request for recording the agent's screen contained the screen agent's IP.
Chapter 7: Channel Mapping Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
180
Channel Mapping Troubleshooting Workflow
Verifying the Channel Mapping Configuration The Channel Mapping configuration is dependent on:
•
Recording Mode: Total or Interaction based.
•
Recording Type: Plain, Time Selective, Dynamic Matrix, Passive VoIP or Active VoIP
•
Mapping Type: Static or Dynamic
•
Recording Method: Extension or Trunk
To verify the Channel Mapping configuration: 1.
2.
Check if changes were made to the site without making the necessary changes to the Channel Mapping configuration, for example:
•
Logger was reinstalled.
•
Channels were changed.
•
Capture type was changed.
•
Changes were made to the monitored devices is System Administrator without reflecting the changes in Channel Mapping.
•
For Avaya DMCC, changes in virtual extensions were made without remapping all sources.
•
For N+1, any change in the chain, for example, an addition of a Logger to a chain, converting all Loggers to 1+1, and so on, without reflecting the changes in Channel Mapping.
For extension-side recording, verify the following: a. Channel definition (Recording type = Plain or Time Selective, Summation Support and
Inserter Support). b. Source definition (extensions). c. Static Channel Mapping definition. 3.
For trunk recording, if the call was recorded but with the wrong voice or silence, the trunk may be mapped to the wrong Logger channel. Verify that the mapping matches the physical connection to the Loggers.
4.
For trunk-side recording, verify the following: a. Channel definition (Recording type = Plain, Time Selective or Digital Matrix, Summation
Support and Inserter Support). b. Source definition (trunks). c. For Plain or Time Selective channels, static Channel Mapping definition. d. For Digital Matrix channels, static Channel Mapping or dynamic Channel Mapping
definition.
Chapter 7: Channel Mapping Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
181
Channel Mapping Troubleshooting Workflow
5.
If you are using dynamic Channel Mapping a recording may fail if there is an attempt to record more calls simultaneously that there are assigned channels. Depending on your system you can:
•
Buy more channels.
•
Ensure in your dynamic mappings that all sources are mapped to more than one Logger.
•
For QA recording, try to change your rule percentage/timing.
6.
Verify that all ACDs (Hunt Groups), IVRs, and Pickup Groups participating in the call are mapped to Logger channels.
7.
Check that the mapping details match the Logger configuration in the Logger summary file (\NTLOGGER\CONFIG\summary.txt), for example, total versus interaction-based recording, the Logger summation support definition (Summed or Co-PABX/Stereo), and so on.
8.
Use the Functional Tester to verify that the channel definitions on the Logger match the devices that were mapped to the channels, for example, summation and in the mapping and in the Logger Channel Mapping import file. See the High Density Logger Troubleshooting Flow and the Channel Mapping Guide for Importing Mapping Definitions.
9.
Depending on the integration, check that the mapping is according to the extension number, MAC #, Device Unique Identifier (DUI) or IP address.
10. Verify that the mapping configuration was updated to the CLS. See the Channel Mapping
Guide. NOTE: It may take up to 10 minutes for the RCM to apply the configuration when there are many mappings and Logger channels. Chapter 7: Channel Mapping Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
182
Channel Mapping Troubleshooting Workflow
11. Check the RCM logs to verify that the mapping configuration was successfully updated to the
Loggers:
•
Check that the mappings have been applied, and that all the relevant change commands have been sent to the Loggers.
•
Search for ERROR/WARN/Exception messages during the execution of the configuration changes. Otherwise, the only way you will find the errors is when the mapping does not work, or specific changes are not applied, and you may lose recordings as a result.
Troubleshooting an Out of Memory Error An Out of Memory error appears in System Administrator while defining Channel Mapping (that is, Internet Explorer is connected to the Applications Server). The System Administrator’s memory consumption increases after many configuration changes. NOTE: The out of memory error does not necessarily indicate a problem with Channel Mapping and can also be caused by other parts of System Administrator.
To troubleshoot an Out of Memory error: 1.
On the Applications Server, in Task Manager, click the Processes tab, and check the memory usage for NICE System Administrator.
2.
On the client workstation, in Task Manager, click the processes tab, and check the memory usage for iexplore.exe.
3.
Check the performance counters: a. In the Run window enter perfmon. b. Click Add. c. For each counter (Processor, Memory, Private Bytes, Threads, Handles,% CPU, Page
Faults / Sec), do the following:
•
From the Performance Object list, select the Object.
•
If enabled, select All instances.
•
Select Select counters from list.
•
Select a counter and then click Add.
d. Click Close. e. Click View Report and monitor the results and check for irregularities. 4.
On the Applications Server, create a memory dump file: a. Navigate to \NICE Applications\Tools. b. Double-click NICE Services Configuration Manager.exe.
Chapter 7: Channel Mapping Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
183
Channel Mapping Troubleshooting Workflow
c. Click Dump
.
d. Click Yes. The Browse For Folder window appears. e. Select the folder that will contain the memory dump files and click OK. Memory dump
files are created. 5.
On the client workstation, create a dump file using the Windows ADPlus utility. For more information regarding ADPlus, see http://support.microsoft.com/kb/286350.
6.
Restart the Applications Server.
7.
Re-launch Internet Explorer.
Troubleshooting Two Users Updating Channel Mapping Concurrently Prior to NICE Perform Release 3.1, in the unlikely event of two users simultaneously updating the Channel Mapping configuration, the changes saved by the second user did not take into account the first user’s changes, resulting in a potential conflict. In NICE Perform Release 3.1, a message appears informing the second user that their changes will be discarded. The Channel Mapping configuration will reload, and the second user may then make the required Channel Mapping configuration changes. This is called an atomic save.
Troubleshooting Saving in a Multi Data Hub Environment In a Multi Data Hub environment, some of the information in nice_admin is replicated. If replication is down, you will be unable to save in System Administrator. NOTE: This affects all the branches in System Administrator, not only Channel Mapping.
The atomic save (described in Troubleshooting Two Users Updating Channel Mapping Concurrently on page 184) in a Multi Data Hub environment relies on the Primary Data Hub being up.
Adding a New Agent You may need to add a new agent. To add a new agent: 1.
If required, for TDM Loggers, connect the audio input. For further information, see the NiceLog High Density Logger Installation and Configuration Guide. Chapter 7: Channel Mapping Troubleshooting Flow
NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
184
Channel Mapping Troubleshooting Workflow
2.
In System Administrator > Channel Mapping (for further information, see the relevant Integration Guide): a. Define Logger channels b. Define sources c. Map channels to sources d. Update the Interactions Center
3.
In the Users Administrator: a. Add the new agent. See Adding a New User in the Users Administrator Guide. b. Add the agent to the relevant group. See Adding Group Members in the Users
Administrator Guide. c. If required, give the user privileges. See Profile Privileges in the Users Administrator
Guide. 4.
In the Rules Manager, ensure that the new agent is covered by any relevant rules, for example, if the agent will be recorded according to a Scheduler rule, or whether the interactions of the agent’s group will be archived according to Storage rule.
5.
Make a test call: a. Monitor the test call. b. If the test call does not appear in the Monitor, there may be a CTI Server to Driver
problem, Driver to CLS problem or Interactions Center process problem. See the Recording Troubleshooting Flow. c. If the test call appears in Monitor, try to ROD the test call. d. If the test call is not recorded, check the mapping.
Other Troubleshooting Issues If you edit the configuration files:
•
Ensure that you use a plain text editor such as Notepad without any options.
•
DO NOT use text wrap as this will corrupt your configuration file.
•
DO NOT save the files in Unicode as this will cause the System Administrator to crash on
startup.
•
In any event, any change of configuration files should be done ONLY in consultation with NICE Customer Support!!!!
Chapter 7: Channel Mapping Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
185
Open Case Procedures
Open Case Procedures This section tells you what information to provide if you need to send a problem to NICE Customer Support for the following:
•
Open Case Procedures for System Administrator Parameters (CLS)
•
Open Case Procedures for the RCM
Open Case Procedures for System Administrator Parameters (CLS) Data to be provided in Interactions Center Parameters (CLS Plug-ins) case reports, when contacting Customer Support:
•
•
Details regarding System configuration:
•
Exact Interactions Center release, service pack, and installed Updates
•
Any special site configuration, for example, Multi Data Hubs
Problem description:
•
Expected behavior
•
Actual behavior
•
Actions taken
•
Is the problem related to a specific configuration (such as a specific integration or mapping type) or does it effect all configurations?
•
Does the problem reoccur after restarting the System Administrator or browser?
•
Is there a work around?
•
Are there also problems with additional parameters, such as Integrations parameters, Loggers parameters, DB Server parameters.
•
Provide specific examples and details
•
Complete log files, in debug level from the server (located on the computer where the Interactions Center sits, under NICE Systems > Applications> ServerBin)
•
NiceApplications.CLSPlugin.log.*.txt
•
A backup copy of the nice_admin database (located on the computer where the database is installed)
•
Please point to the relevant times in the logs when the problem can be seen in the logs. Quote the lines if you can.
•
Provide a screenshot of an error if one is displayed.
•
Provide Performance Counters of SysAdmin process:
•
Private Bytes
•
Threads Chapter 7: Channel Mapping Troubleshooting Flow
NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
186
Open Case Procedures
•
Handles
•
% CPU
•
Page Faults/Sec
•
All performance counters under the .Net Memory Category.
•
In case of Out Of Memory issues, please provide dump of SysAdmin process.
•
In case of Online Configuration Update issues, please also provide full Interactions Center Logs.
•
Provide any other relevant data, for example, memory usage or screen shots.
Open Case Procedures for the RCM Data to be provided in RCM case reports, when contacting customer support:
•
Details regarding System configuration:
•
Exact Interactions Center release, service pack and NICE Updates (Update Packs) installed
•
Details of all Loggers configured to each Interactions Center: version, capture type, number of channels, recording method, GMT and local time of the Logger.
•
Mapping details for each Logger attached to the Interactions Center:
•
Total/Interaction Based
•
Recording method of Logger channels (for example, Passive VoIP, DMCC, Nortel DMS, Digital Matrix)
•
Type of Sources and their number
•
Static/Dynamic Mapping
•
Connection between Logger and Interactions Center (LAN/WAN). If WAN, please indicate Bandwidth and Latency.
•
N+1 chain details (when relevant)
•
Details from CTI plug-in on the drivers and the type of interfaces (for example, Avaya TSAP, Genesys, CISCO ICM etc.).
•
Site's SBC/Gateway topology (when relevant).
•
Number of ScreenAgents reporting to the Interactions Center (when relevant)
•
Call Rate on the Interactions Center Machine
•
Work hours of agents mapped to the Interactions Center, including peak hours (for example, hours of shifts, 24/7)
•
Problem description:
•
Expected behavior Chapter 7: Channel Mapping Troubleshooting Flow
NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
187
Open Case Procedures
•
Actual behavior
•
Actions taken
•
Provide specific examples and call details (for example, Interaction ID, Segment ID, extension number, Agent number, time of the call, relevant DB Records for the call).
•
If the issue occurs in specific call scenarios, describe the scenario.
•
Does the issue occur in a specific type of call? (for example, only internal calls)
•
Complete log files, in debug level (located on the Interactions Center computer, under NICE Systems > Interactions Center > Log):
•
Call Server.log.*.txt
•
RCM.log.*.txt
•
Complete log files, in debug level, of drivers attached to the Interactions Center.
•
Complete log files of Loggers (including capture logs) attached to the Interactions Center.
•
All RCM configuration backup files found under NICE Systems > Interactions Center > Loggers
•
Files found under NICE Systems > Interactions Center > bin
•
RCM.log.Persistency.soap
•
RCM.exe.config
•
Please point to the relevant times in the logs when the problem can be seen in the logs. Quote the lines if you can.
•
Provide the following database tables in CSV format (query from before the time the problem occurred, preferably overlapping with the time in the provided logs):
•
•
nice_interactions
•
tblInteraction
•
tblRecording
•
tblParticipant
•
tblException
nice_admin
• •
Channel Mapping tables
The following Performance Counters of the RCM.Exe process:
•
Private Bytes
•
Threads
•
Handles
•
% CPU Chapter 7: Channel Mapping Troubleshooting Flow
NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
188
Open Case Procedures
•
•
Page Faults / Sec
•
All Performance Counters under the .Net Memory Category.
Provide any other relevant data, for example, the users included, memory usage, screen shots.
Chapter 7: Channel Mapping Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
189
Troubleshooting Scenarios
Troubleshooting Scenarios Troubleshooting techniques are provided for the following issues which can occur when your system is up and running. Error Message appears when Updating Configuration A New or Modified Mapping Definition Does Not Work New Channel Mapping Definitions are not Saved Channel Mapping does not Work! Channel Mapping in System Administrator Differs from what is in Effect Channels Definitions: Logger does not Appear Channels Definitions: Input Channels do not Appear Source Definitions: Cannot define Virtual Extensions Source Definitions: Physical Switch does not Appear Dynamic Mapping: Screen Logger does not Appear Dynamic Mapping: Virtual Extensions Deleting or Changing Logger, Switch, or CLS Definitions Procedure for Reassigning a Logger to a Different CLS
Error Message appears when Updating Configuration Scenario:
You click Update Configuration and receive an error message, similar to the one below, that update was unsuccessful.
Solution 1: 1.
Check the Network connection between the Applications Server and the Interactions Center.
2.
Check the status of the CLS Monitor service on the Interactions Center.
3.
Click Update Configuration again.
Solution 2:
Only if Solution 1 fails, do the following: Chapter 7: Channel Mapping Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
190
Troubleshooting Scenarios
1.
2.
Restart the following:
•
RCM
•
TRS Service
•
In Media Provider Controller Manager (MPCM - FLM) environments, restart the Call Server
Click Update Configuration again.
A New or Modified Mapping Definition Does Not Work Scenario:
You add and/or modify Channel Mapping definitions. Then you click Update Configurations. One or more of the new or modified mapping definitions does not work. IMPORTANT Some configurations require that you restart the RCM to complete the update process. For a complete list, see the Channel Mapping Guide. Solution: 1.
Stop the CLS service.
2.
Delete all RCM persistency files located under the Interactions Center/Loggers folder.
3.
Restart the CLS service.
New Channel Mapping Definitions are not Saved If you click Save and your changes do not appear to be saved, the following occurred: Another user was updating Channel Mapping definitions at the same time that you were and clicked Save first. What to do: 1.
Refresh the screen to see the new changes.
2.
Recreate your changes.
See also Channel Mapping does not Work! on page 191.
Channel Mapping does not Work! When you define Channel Mapping and save your definitions, they are saved only in the interface. To apply them to the system, you must update the CLS. For instructions, see the Channel Mapping Guide. See also New Channel Mapping Definitions are not Saved on page 191. Chapter 7: Channel Mapping Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
191
Troubleshooting Scenarios
Channel Mapping in System Administrator Differs from what is in Effect Scenario:
You open the System Administrator application and view the Channel Mapping definitions. They appear to be different from what you know to be currently running. Cause:
Changes were made to the Channel Mapping definitions in the System Administrator and saved. However, Update Configuration was not applied, so therefore the Interactions Center was not updated and the new Channel Mapping definitions were not applied.
Channels Definitions: Logger does not Appear Scenario 1
Loggers in the Channels Definition branch are identified by their Logger ID, not by the Logger name that appears in the Organization tree. Verify that you are searching for the Logger ID. Logger IDs are defined during Logger installation. To determine the Logger ID: 1.
In the System Administrator, expand the Loggers branch and select the Logger.
2.
The Logger ID appears in the General tab of the Logger definition.
Scenario 2
If the Logger is attached to more than one CLS, then it will not appear in the Channel Definition window. If the Logger is not attached to a CLS, then attach it now. To attach a Logger to a CLS: 1.
In the System Administrator, click the Loggers branch and define the Logger.
2.
Click the Channels tab to check that input channels are defined.
3.
Click the CLS branch and select a CLS. In the Logger tab, select the new Logger.
4.
Go back to the Channel Mapping branch and click the Channels definition branch.
5.
Select the new Logger. The Update Channels Details window opens. Loggers are identified here by the Logger ID, not by the Logger name you supplied.
6.
For Screen Loggers, you cannot edit channel details. However, you must open their definition and click Save.
Chapter 7: Channel Mapping Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
192
Troubleshooting Scenarios
Channels Definitions: Input Channels do not Appear To display input channels: 1.
In the System Administrator, expand the Loggers branch and select the Logger.
2.
Click the Channels tab and move the input channels to the Configured Input Channels list.
3.
Save the Logger definition.
4.
Open the CLS servers branch, detach the Logger. (All mapping in the Channel Mappings branch will be removed.)
5.
Click the Channel Mapping branch and expand one of the branches. This step is required to update the database.
6.
Open the CLS servers branch and reattach the Logger.
7.
Return to the Channel Mapping branch and continue.
Source Definitions: Cannot define Virtual Extensions When working with Virtual Extensions, the Location of both the Logger and the Connection Manager must be identified using the same method: either by IP Address or by Host Name.
Source Definitions: Physical Switch does not Appear Scenario:
A Physical Switch is present in the CTI Integrations, but does not appear in the Sources definition branch. Solution: To attach a Physical Switch: 1.
Define the Physical Switch on a CTI Interface: a. Expand the CTI Integrations branch and click the CTI Interfaces branch. b. From the Actions menu, select New CTI Interface. c. Define the new Interface with the Physical Switch.
2.
Attach the Interface to a Connection Manager: a. Expand the CTI Integrations branch and then expand the Connection Manager branch.
Select a Connection Manager. b. Click the Interfaces tab and attach the Interface (physical switch). 3.
Attach the CLS to the Driver. a. Expand the CTI Integrations branch and then expand the Drivers branch. Select a Driver. b. In the General tab, expand the Attach CLS section. Chapter 7: Channel Mapping Troubleshooting Flow
NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
193
Troubleshooting Scenarios
c. Attach the CLS to the Driver. 4.
Attach the Driver to the Connection Manager. a. Expand the CTI Integrations branch and then expand the Drivers branch. Select a Driver. b. Click the Interfaces tab and select the Physical Switch/Connection Manager.
Dynamic Mapping: Screen Logger does not Appear Scenario:
A Screen Logger in your Data Hub does not appear in the Dynamic Mapping branch. Solution:
Screen Loggers do not need to be defined in the Sources branch. However, you must open their definition and click Save. To add a Screen Logger: 1.
Expand the Channel Mapping branch and click the Channels definition branch.
2.
Click the Screen Logger to expand its definition.
3.
Click Save.
4.
Return to the Dynamic Mapping branch.
Dynamic Mapping: Virtual Extensions If you receive an error message when attaching channels pools, verify that the number of Virtual Extensions is greater than or equal to the number of Channels in the Channels pool. Confirm that the Location of both the Logger and the Connection Manager are identified using the same method: either by IP Address or by Host Name.
Deleting or Changing Logger, Switch, or CLS Definitions Channel Mappings are dependant upon the Logger, Switch, and CLS definitions made in the System Administrator. Changing or deleting attachments from a switch to a CLS or from a CLS to a Logger, affect the recordings once the RCM is restarted, in the following ways. If you...
Then...
Delete a Logger definition
All channel definitions and mapping configurations for that Logger are deleted.
-orDetach a Logger from a CLS
Source definitions remain.
Delete a switch definition
All Source definitions and mapping configurations for that switch are deleted.
-orDetach a CLS from a driver
Channel definitions remain. Chapter 7: Channel Mapping Troubleshooting Flow
NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
194
Troubleshooting Scenarios
If you...
Then...
Delete a CLS definition
• •
All mapping configurations for that CLS are deleted.
•
All Source definitions for switches to which the CLS was attached remain.
Attach a Logger to a second CLS after the Logger’s channels have been defined and mapped.
All Channel definitions for Loggers that were attached to the CLS are deleted.
The Logger’s Channel Mapping channel definitions and mapping configurations are deleted.
To update Channel Mapping definitions, see the Channel Mapping Guide.
Procedure for Reassigning a Logger to a Different CLS If you move a Logger from one Interactions Center to another, you must change its assignment in the System Administrator. After removing the Logger from its current CLS, you must update the Channel Mapping configuration and wait 10 minutes for the update to be complete. Then you can attach the Logger to the new CLS and continue. These steps are outlined in the following procedure. To attach a Logger to a different CLS: 1.
In the System Administrator Organization tree, expand a Data Hub and select CLS Definitions > CLS Server.
2.
Select the CLS Server to which the Logger is currently attached and click the Loggers tab.
3.
Move the Logger from the Attached Loggers list to the Available Loggers list. The following message appears.
4.
Click Yes. Then click Save
5.
Select CLS Definitions > Channel Mapping. Then click Update Configurations.
6.
Wait 10 minutes for the CLS to be completely updated!
.
IMPORTANT Wait 10 minutes before continuing with this procedure! Chapter 7: Channel Mapping Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
195
Troubleshooting Scenarios
7.
Select CLS Definitions > CLS Server. Then select the new CLS server and click the Loggers tab.
8.
Move the Logger from the Available Loggers list to the Attached Loggers list. Then click Save
9.
.
Select CLS Definitions > Channel Mapping.
10. Define Channel Mapping for the Logger following the regular Channel Mapping procedures.
Chapter 7: Channel Mapping Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
196
8 Applications Troubleshooting Flow This section contains troubleshooting workflow for the NICE Perform Application Server, including solving Desktop problems and troubleshooting Browser problems using Logs. A list of the services, config and log files is also provided.
Contents Workflow........................................................................................................................198 Gathering Information ..................................................................................................199 Checking the Basics ....................................................................................................200 Troubleshooting a Communications Failure..............................................................202 Troubleshooting Communications Failure with Importer or Reporter ....................203 Checking the Applications Server Services...............................................................205 Checking the Installation Files ....................................................................................208 Checking the Log Files ................................................................................................209 Solving Desktop Problems ..........................................................................................217 Troubleshooting Browser Problems Using Logs ......................................................237 Solving ActiveX Control Error on the Applications Server ......................................241 Optimizing Client Server Performance .......................................................................242 Optimizing Client Server Performance on Computers without Internet..................244 Solving Email Failure ...................................................................................................246 Troubleshooting Crystal Reports ................................................................................250 Using the Process Explorer Utility..............................................................................251 Using the Desktop Tester.............................................................................................252
Chapter 8: Applications Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
197
Workflow
Workflow If you experience a problem with the Applications Server follow the steps below to isolate the problem.
Chapter 8: Applications Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
198
Gathering Information
Gathering Information Important issues to consider are:
•
Is this a new installation? If yes, when was the installation performed?
•
Were there any changes to the environment that might have influenced the system? If yes, check these changes first.
•
How often does the problem occur?
•
Is the problem reproducible? If yes, what were the steps?
•
Were the Servers restarted? If yes, did this resolve the problem?
Chapter 8: Applications Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
199
Checking the Basics
Checking the Basics For Applications Server problems, check the following: 1. Troubleshoot a Communication Failure (see Troubleshooting a Communications Failure
on page 202) 2. Troubleshoot a Communication Failure between the Applications Server and the Importer
and/or Reporter Packages when using Language Pack (see Troubleshooting Communications Failure with Importer or Reporter on page 203) 3. Check the Applications Server Services (see Checking the Applications Server Services
on page 205)
•
Use NICE Services Configuration Manager (see Using NICE Services Configuration Manager on page 206)
•
Solve Application Services User Rights Problem (see Solving Application Services User Rights Problem on page 207)
4. Check the Installation Files (see Checking the Installation Files on page 208) 5. Check the Logs (see Checking the Log Files on page 209) 6. Solve Desktop Problems
•
Verify the .NET Framework Versions (see Verifying the .NET Framework Versions on page 217)
•
Solve Set Security Wizard Continues to Appear After Installation (see Set Security Wizard Continues to Appear After Installation on page 218)
•
Solve Set Security Wizard Hangs or Displays an Error (see Solving Set Security Wizard Hangs or Displays an Error on page 221)
•
Set Security Manually (see Set Security Manually on page 222)
•
Set Security Manually Using Caspol (see Set Security Manually Using Caspol on page 224)
•
Solve Post Installation HTTP Error (see Solving Post Installation HTTP Error on page 224)
•
Verify the Applications are Not Loaded under the Internet Zone (see Verifying the Applications are Not Loaded under the Internet Zone on page 228)
•
Configure the NICE Perform as a Trusted Web Site (see Configuring NICE Perform as a Trusted Web Site on page 228)
•
Check NICE Perform Initialization Errors (see Checking NICE Perform Initialization Errors on page 229)
•
Solve Service Unavailable Error (see Solving Service Unavailable Error on page 230)
•
Disable Custom Errors (see Disabling Custom Errors on page 233)
•
Solve ActiveX Control Error (see Solving ActiveX Control Error on page 233) Chapter 8: Applications Troubleshooting Flow
NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
200
Checking the Basics
•
Troubleshoot Application Access (Power Management Issue) (see Troubleshooting Application Access (Power Management Issue) on page 234)
7. Troubleshoot Browser Problems Using Logs
•
Explorer Log (on the Client Machine) (see Explorer Log (on the Client Machine) on page 237)
•
Desktop Client Logs (on the Client Machine) (see Desktop Client Logs (on the Client Machine) on page 237)
•
IIS - WWW Log File (on the Applications Server) (see IIS - WWW Log File (on the Applications Server) on page 238)
•
IIS - HTTPERR Log File (on the Applications Server) (see IIS - HTTPERR Log File (on the Applications Server) on page 239)
•
Application Event Log File (on the Applications Server) (see Application Event Viewer (on the Applications Server) on page 239)
8. Solve ActiveX Control Error on the Applications Server (see Solving ActiveX Control Error
on the Applications Server on page 241) 9. Solving Email Failure (see Solving Email Failure on page 246) 10. Troubleshoot Crystal Reports (see Troubleshooting Crystal Reports on page 250) 11. Optimizing Client Server Performance (see Optimizing Client Server Performance
on page 242) 12. Optimizing Client Server Performance on Computers without Internet (see Optimizing Client
Server Performance on Computers without Internet on page 244) WARNING Manually modifying registry settings can cause your system to malfunction. This procedure should only be done by qualified personnel.
IMPORTANT Back up your config files before editing them. If you edit the configuration files, ensure that you use a plain text editor such as Notepad without any options. DO NOT use text wrap as this will corrupt your configuration file. In any event, any change of configuration files should be done ONLY in consultation with NICE Support!
Chapter 8: Applications Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
201
Troubleshooting a Communications Failure
Troubleshooting a Communications Failure If you ping the server and it responds, but the communications failure persists, you may have a problem with the fully qualified domain name. To identify the problem:
•
Ping the server, using its fully qualified name: <ServerName>.. In order to resolve this, the FQDN should either be resolved by the DNS (requires assistance of IT) or added to the hosts file (C:\WINDOWS\system32\drivers\etc\hosts).
Chapter 8: Applications Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
202
Troubleshooting Communications Failure with Importer or Reporter
Troubleshooting Communications Failure with Importer or Reporter NOTE: This procedure applies only to NICE Perform systems that use the Language Pack.
If your NICE Perform system uses a Language Pack, there may be a conflict between the Applications Server and the Importer and/or Reporter Packages features. To prevent such conflicts, you must configure the Applications Server to apply the language settings to the default user profile. To configure the language settings: 1. In the Start menu, navigate to Settings > Control Panel > Regional and Language Options.
The Regional and Language Options window appears. 2. Click the Advanced tab. Figure 8-1
Regional and Language Options Window - Advanced Tab
3. In the Default user account settings area, select Apply all settings to the current user account and to the default user profile.
When you select the checkbox, the Change Default User Settings window appears.
Chapter 8: Applications Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
203
Troubleshooting Communications Failure with Importer or Reporter
Figure 8-2
Change Default User Settings Window
4. Click OK to accept the changes.
The Regional and Language Options window reappears. 5. Click OK. NOTE: If, after changing the language settings, you open the Regional and Language Options window again, the checkbox appears not selected. However, the language settings have been changed and are in effect despite the fact that the checkbox is cleared. 6. Restart your computer.
Chapter 8: Applications Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
204
Checking the Applications Server Services
Checking the Applications Server Services The following services run on the Applications Server:
•
NICE System Administrator: The main Applications Server service that controls the login
process to NICE applications. If this service is not running, users will be unable to login.
•
NICE Business Analyzer: Runs the Business Analyzer application (the main query
application for searching and playing back interactions) and the queries in the DB. There are some sub-services under this service.
•
NICE Locate Service: Relevant to SP3 and SP4 only. Runs under the NICE Business
Analyzer service, and finds the actual location of the audio in the preparation for playback (Loggers, Storage Center, or Media Library).
•
NICE Rule Engine: Reads all defined rules from the database, ensures that the rules run at the defined times, and notifies the relevant unit that needs to run the rule (Storage Center, Audio Analysis, and soon) with details of what to do.
•
NICE Rules Manager Host: Supports the Rule Manager application where users create rules.
•
NICE Monitor Server: Handles all ROD and monitor requests, and communicates with the
Interactions Center (directly to the Call Server).
•
NICE Audit Trail Service: Runs the Audit Trail, logging all user and server operations and
functions to the relevant database (nice_audit).
•
NICE Coaching Server: Manages all Coaching activity that is created by supervisors and evaluators that is delivered to Agents (via "My Universe").
•
NICE MyUniverse: Manages the My Universe application, and is responsible for collecting
the required information and displaying the relevant components, based on settings defined by them, and/or their managers.
•
NICE Media Server: Relevant to SP3 only. Fetches media from the Loggers and Storage
Center. See the Storage Center Troubleshooting Flow.
•
NICE Playback Resource Manager: Relevant to SP3 and SP4 only. Manages the state of
each of the ICB (Instant Callback) resources, for example, ports configured for ICB, and so on, and displays an error if no resources available. See the Playback Telephony Server Troubleshooting Flow.
•
NICE Playback Administration: Relevant to SP4 and 3.1. Builds the play list. In SP4, replaces the NICE Media Server. In 3.1 this service incorporates the NICE Playback Resource Manager service and NICE Locate service functionality.
•
NICE Playback Streaming: Relevant to SP4 and 3.1 only. Ensures that the media is delivered to the NICE Player (according to the configuration in the System Administrator).
•
NICE Evaluations Form Server: Creates evaluation forms and the evaluations themselves,
as well as runs the Form Designer and Evaluator. If the NICE Reporter is running on the Applications Server, the following services will also be running:
Chapter 8: Applications Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
205
Checking the Applications Server Services
•
NICE Reporter Engine: Executes reports and allows the Reporter application to run.
•
NICE Reporter Scheduler: Searches the DB for any active report schedules and, at the
scheduled times, sends a request to the Reporter Engine to run the report(s).
•
Crystal Report Application Server: Must be running on the Reporter server. This is a third-party program that NICE Perform uses as its underlying reporting technology. NOTE: The NICE Reporter usually runs on the Data Mart.
To solve services problems:
•
Use NICE Services Configuration Manager (see Using NICE Services Configuration Manager on page 206)
•
Solve Application Services User Rights Problem (see Solving Application Services User Rights Problem on page 207)
Using NICE Services Configuration Manager The NICE Services Configuration Manager enables you to manage the services on the Applications Server.
To configure the NICE Services Configuration Manager to run on startup: 1. Copy a shortcut from Applications\Tools\Nice Services Configuration Manage\NICE Services Configuration Manager.exe to C:\Documents and Settings\Administrator\Start Menu\Programs\Startup. 2. Right-click the Shortcut to Nice Services Configuration Manager shortcut link and select
Properties. 3. From the Run drop-down list, select Minimize. 4. Click OK.
Chapter 8: Applications Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
206
Checking the Applications Server Services
Solving Application Services User Rights Problem A service may not start if the user does not have Log on as a service rights. To solve the problem: 1. To check which user is running the service, use the NICE Services Configuration Manager or right-click My Computer > Manage > Services and Applications > Services, select Properties and click the Log On tab.
2. Open Control Panel > Administrative Tools > Local Security Policy. 3. Expand Local Policies > User Rights Assignment > Log on as a service. 4. Verify that user used to run the service has this permission.
Chapter 8: Applications Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
207
Checking the Installation Files
Checking the Installation Files The files are typically installed on the D:\ partition of the Applications Server, located under D:\Program Files\Nice Systems\Applications. Sub-folders include:
•
ServerBin: Contains binaries and DLL files for server-based applications.
•
ClientBin: Contains the binaries and DLL files needed by clients.
•
Assemblies: Contains COM wrappers for many applications.
•
Desktop: Contains, among other things, ASPX files for the login page.
•
NICE Database Suite: Contains SQL scripts used by the applications.
•
NICE_Help: Contains help files for the web applications suite.
•
Tools: Contains tools used by installers and NICE customer support.
•
Client Side Applications: Contains all installations files for client side applications (Player,
ROD, setsecurity.exe and so on).
•
KeyStorageManager: Contains the Key Storage Manager Web service assemblies and log
files.
Chapter 8: Applications Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
208
Checking the Log Files
Checking the Log Files The configuration and log files for each application are found in either (or both) the ServerBin, and ClientBin folder, together with the binary files.
•
Set the Log Level in the Config File (see Setting the Log Level in the Config File on page 210)
•
Set the Log Level in System Administrator (Setting the Log Level in System Administrator on page 209)
•
Check the Logs (see Checking the Logs on page 214)
Setting the Log Level in System Administrator To set the log level in System Administrator: 1. In System Administrator, verify that you are working in Technician mode: In the Organization tree, click Organization. Then select the Technician Mode checkbox and click Save. 2. In the Organization tree, select Applications > Configurations > General. Click the Logs Level tab.
3. Set the Master Level and File Level to Debug. 4. After troubleshooting ensure that you reset these values.
WARNING Failure to do so may cause system problems due to a performance overhead.
Chapter 8: Applications Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
209
Checking the Log Files
Setting the Log Level in the Config File Each of the components on the Applications Server has config files in the following folder: <Applications Server Installation Path>\ServerBin\.
Each config file includes a section with several loggers:
Each Logger includes several appendices, that is, the types of log file output:
Component
SA*
Configuration File
Application Messages
Y
NiceApplications.SystemAdministrator.Host.exe.config
Audio Analysis branch (System Administrator)
Y
Audio Analysis Search Controller
Y
NiceApplications.AASearchController.Host.exe.config
Audit Trail
Y
NiceApplications.AuditTrail.Host.exe.config
Branches
Y
NiceApplications.SystemAdministrator.Host.exe.config
NiceApplications.SystemAdministrator.Host.exe.config
Business Analyzer Coaching
NiceApplication.UnifiedQuery.host.exe.config Y
NiceApplications.CoachingTool.Host.exe.config NiceApplications.CoachingTool.Host.vshost.exe.config
Chapter 8: Applications Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
210
Checking the Log Files
Component
SA*
Configuration File
Encryption Plugin
Y
NiceApplications.SystemAdministrator.Host.exe.config
Evaluation Forms
Y
NiceApplications.EvaluationForms.Host.exe.config
FTF Query
Y
NiceApplications.FTFQuery.Host.exe.config
Importer
Y
NiceApplications.SystemAdministrator.Host.exe.config
Insight Manager (Investigations)
Y
Interactions Server Plugin
NiceApplications.Investigations.Host.exe.config NiceApplications.SystemAdministrator.Host.exe.config
IP Phone Tester
Y
IPPhoneTester.exe.config
License Manager
Y
NiceApplications.SystemAdministrator.Host.exe.config
Lists Editor
Y
NiceApplications.SystemAdministrator.Host.exe.config
Locate Service
NiceApplications.UnifiedQuery.LocateService.Startup.exe.config
Relevant to SP3 and SP4 only
Location Service
NiceApplications.SystemAdministrator.Host.exe.config
Locator
NiceApplications.UnifiedQuery.LocateService.LocateServiceHost. exe.config
Loggers Plugin
NiceApplications.SystemAdministrator.Host.exe.config
Media Server
NiceApplications.Playback.MediaServer.Startup.exe.config
Relevant to SP3 only
Monitor
NiceApplications.Monitor.Server.ServerStartup.exe.config NiceApplications.Monitor.Server.ServerStartup.exe.config
MyUniverse
Y
NiceApplications.MyUniverse.Host.exe.config NiceApplications.MyUniverse.Server.ServerStartup.exe.config
Playback Administration
NiceApplications.Playback.Administration.Startup.exe.config
Relevant to SP4 and 3.1 only
Chapter 8: Applications Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
211
Checking the Log Files
Component
SA*
Playback Resource Manager
Configuration File NiceApplications.Playback.ResourceManager.RMServerStartup.e xe.config
Relevant to SP3 and SP4 only
Playback Streaming
NiceApplications.Playback.Streaming.Encoder.exe.config NiceApplications.Playback.Streaming.Startup.exe.config
Relevant to SP4 and 3.1 only. Playback branch (System Administrator)
NiceApplications.SystemAdministrator.Host.exe.config
Retention Service
Y
NiceApplications.RetentionService.Host.exe.config
Rule Engine
Y
NiceApplications.RuleEngine.Service.exe.config
Rules Manager
Y
NiceApplications.RulesManager.Host.exe.config
Storage Center branch (System Administrator) Summary Page
NiceApplications.SystemAdministrator.StorageCenterPlugins. config Y
NiceApplications.SystemAdministrator.Host.exe.config
System Administrator
Y
System Administrator SNMP
Y
Users Administrator
Y
NiceApplications.SystemAdministrator.Host.exe.config NiceApplications.SystemAdministrator.Host.exe.config NiceApplications.SystemAdministrator.Host.exe.config
IMPORTANT Backup your config files before editing them. If you edit the configuration files, ensure that you use a plain text editor such as Notepad without any options. DO NOT use text wrap as this will corrupt your configuration file. In any event, any change of configuration files should be done ONLY in consultation with NICE Support!!!!.
Chapter 8: Applications Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
212
Checking the Log Files
To set the log level in the config file: 1. Navigate to <Applications Server Installation Path>\ServerBin\. 2. In Notepad, open the relevant .config file. See Log File Config Files (see Setting the Log
Level in System Administrator on page 209). 3. Expand the section. 4. Expand the required logger name section.
5. Set the debug level, by changing the values on both Level and Threshold to be defined as ALL. 6. Save and close the config file. There is no need to restart the Applications Server. 7. After troubleshooting ensure that you reset these values.
WARNING Failure to do so may cause system problems due to performance overhead.
Chapter 8: Applications Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
213
Checking the Log Files
Checking the Logs The server log files are found in the <Applications Server Installation Path>\ServerBin\ folder unless specified otherwise. Component
Log File
Analytics
NiceApplications.ClearSightApp.txt
Application Messages
NiceApplications.ApplicationMessages.log
Audio Analysis branch (System Administrator)
NiceApplications.AAPlugin.log
Audio Analysis Search Controller
AASearchController.log
Audit Trail
AuditTrailLog.txt AuditTrailDefaultLog.txt
Branches
NiceApplications.SystemAdministrator.Branches.log
Business Analyzer
NBA.log
Coaching
CoachingHostLog.txt
Encryption branch (System Administrator)
NiceApplications.EncryptionPlugin.log
Evaluation Forms
EvaluationFormsLog.txt
FTF Query
NiceApplications.FTFOpenCalls.Log
NBADefault.log
NiceAppliactions.FTFQuery.Log Importer
NiceApplications.Importer.log
Insight Manager (Investigations)
NiceApplications.InvestigationsApp.txt
Interactions Center - CLS branch (System Administrator)
NiceApplications.CLSPlugin.log
IP Phone Tester
XmlAppsHost.log XmlAppDefaultLog.txt Chapter 8: Applications Troubleshooting Flow
NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
214
Checking the Log Files
Component
Log File
License Manager
NiceApplications.LicenseManager.log
List Editor
NiceApplications.ListsEditor.log
Locate Service
LocateDefaultLog.txt
Relevant to SP3 and SP4 only.
LocateLog.txt
Location Service
NiceApplications.LocationService.log
Locator
Locator.log
Loggers branch (System Administrator)
LoggersPlugin.log
Media Server
MediaServer.txt
Relevant to SP3 only
Monitor
MonitorLog.txt monitorDefaultLog.txt
MyUniverse
MyUniverseApp.txt
MyUniverse
MyUniverseBasicPlugin.txt
Playback Administration
PlaybackAdministration.txt
Relevant to SP4 and 3.1 only
PlaybackRM.txt (3.1 only)
Playback Resource Manager
ResourceManagerDefaultLog.txt
NiceApplications.PlaybackAdministration.SystemFramework.log LocateLog.txt (3.1 only) ResourceManagerLog.txt
Relevant to SP3 and SP4 only
Playback Organizer
NiceApplications.Workflow.Log
Playback Streaming
NiceApplications.PlaybackStreaming.SystemFramework.log PlaybackStreaming.txt
Relevant to SP4 and 3.1 only
Playback branch (System Administrator)
NiceApplications.PlaybackPlugin.log
Chapter 8: Applications Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
215
Checking the Log Files
Component
Log File
Reporter
ReporterBin\ReporterManager.txt ReporterBin\ReporterEngine.txt ReporterBin\ReporterSchedulerLog.txt The Reporter application also includes logs within the application itself. The log provides information regarding the execution of reports. You can also export it to csv format. To view the Reporter logs: 1. In Reporter, click Administration. 2. Click Log. 3. Click Export to save the log files as a csv file.
Retention Service
RetentionService.log
Rule Engine
RuleEngine.ActionQueue.txt RuleEngine.SystemFramework.txt RuleEngine.txt
Rules Manager
NiceApplications.RulesManager.Host.exe.log
Summary Page
NiceApplications.SummaryPage.log
System Administrator
NiceApplications.SystemAdministrator.Host.exe.log
System Administrator SNMP
NiceApplications.SystemAdministrator.SNMP.log
Users Admin
NiceApplications.UsersAdmin.log
NiceApplications.SystemAdministrator.SystemFramework.log
Chapter 8: Applications Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
216
Solving Desktop Problems
Solving Desktop Problems There are several types of Desktop problems:
•
Verify the .NET Framework Versions (see Verifying the .NET Framework Versions on page 217)
•
Solve Set Security Wizard Continues to Appear After Installation (see Set Security Wizard Continues to Appear After Installation on page 218)
•
Solve Set Security Wizard Hangs or Displays an Error (Solving Set Security Wizard Hangs or Displays an Error on page 221)
•
Set Security Manually (see Set Security Manually on page 222)
•
Set Security Manually Using Caspol (see Set Security Manually Using Caspol on page 224)
•
Solve Post Installation HTTP Error (see Solving Post Installation HTTP Error on page 224)
•
Verify the Applications are Not Loaded under the Internet Zone (see Verifying the Applications are Not Loaded under the Internet Zone on page 228)
•
Configure the NICE Perform as a Trusted Web Site (see Configuring NICE Perform as a Trusted Web Site on page 228)
•
Check NICE Perform Initialization Errors (see Checking NICE Perform Initialization Errors on page 229)
•
Solve Service Unavailable Error (see Solving Service Unavailable Error on page 230)
•
Enable Custom Errors (see Disabling Custom Errors on page 233)
•
Solve ActiveX Control Error (see Solving ActiveX Control Error on page 233)
•
Troubleshoot Application Access (Power Management Issue) (see Troubleshooting Application Access (Power Management Issue) on page 234)
Verifying the .NET Framework Versions Verify that the correct version of .NET Framework is installed:
•
SP 3: .NET 1.1 or .NET 2.0 or .NET 3.0 or .NET 3.5
•
SP 4 and Release 3.1: .NET 2.0 or .NET 3.0 or .NET 3.5
Ensure that the following Updates are installed:
•
•
•
.NET 1.1:
•
.NET Framework 1.1 dotnetfx.exe, version: 1.1.4322.573
•
.NET Framework 1.1 hotfix NDP1.1sp1-KB867460-X86.exe, version 1.0.871.2738
.NET 2.0 and SP1:
•
.NET Framework 2.0 dotnetfx.exe, version: 2.0.50727.42
•
.NET Framework 2.0 hotfix NDP20-KB914460-X86.exe, version 1.0.36.79
.NET 3.0 and 3.5 Chapter 8: Applications Troubleshooting Flow
NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
217
Solving Desktop Problems
Set Security Wizard Continues to Appear After Installation After installation, when opening applications in the browser the Set Security page appears even though the SetSecurity.exe application was run.
Problem Occurs on One Workstation If this happens on one workstation this may be because:
•
.NET Framework is not installed.
•
Applications are running under the internet zone
If .NET framework is installed, check if the Applications work when running Desktop Tester (this runs the Applications without using the browser). For more information, see Using the Desktop Tester on page 252. If the Applications work when running Desktop Tester but do not work in the browser, possible causes include:
•
SetSecurity.exe was run with the wrong URL
•
There is a problem with the browser download cache not been refreshed.
•
The browser is running with .NET 1.1. (from Release 3.1)
To solve the problem for one workstation: 1. If you see the following error when running SetSeurity.exe, install the .NET framework by typing http://ServerName/Nice in Internet Explorer. You will be prompted to download
.NET.
Chapter 8: Applications Troubleshooting Flow NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
218
Solving Desktop Problems
2. Check if the Applications are running under the internet zone. See Verifying the Applications
are Not Loaded under the Internet Zone on page 228. 3. On the Workstation, run SetSecurity.exe with the correct URL. Verify that the URL you
entered is the same one used in the browser. 4. Clear the browser download cache: a.
Close all open Internet Explorer Browser windows. If these are left open, an error may occur when clearing the cache.
b. Right-click on the Internet Explorer icon, click Properties (in Internet Explorer 7.0 click Internet Properties and then click Delete), click Delete Files and then click OK. -or-
Open the Run window and run gacutil.exe -cdl. c.
Verify that Download cache deleted successfully appears. If an error occurs, ensure that all browser windows are closed, and repeat Step b.
TIP: gacutil.exe can be found in either of the following locations: C:\WINDOWS\Microsoft.NET\Framework\<.NET version> C:\Program Files\NICE Systems\Applications\Tools You can drag-and-drop the file into the Run window. d. In Internet Explorer enter the URL again. Verify that you are entering the server name correctly. (URL format is: http://ServerName/Nice). 5. Verify/set the security manually. See Set Security Manually on page 222 or Set Security
Manually Using Caspol on page 224. 6. From Release 3.1: a.
Verify that .NET 2.0 is installed on the machine. Chapter 8: Applications Troubleshooting Flow
NICE Perform ® 3.1: Troubleshooting Guide (Rev. A2)
219
Solving Desktop Problems
b. Open C:\Program Files\Internet Explorer\iexplore.exe.config and check that the <supportedRuntime version="v2.0.50727" /> is not commented out. <startup>