Readiness Check Version4

  • Uploaded by: Pooja
  • 0
  • 0
  • May 2020
  • PDF

This document was uploaded by user and they confirmed that they have the permission to share it. If you are author or own the copyright of this book, please report to us by using this DMCA report form. Report DMCA


Overview

Download & View Readiness Check Version4 as PDF for free.

More details

  • Words: 4,606
  • Pages: 30
Dr. Julia Hambrock

Updated by Karsten Blankenstein Solution Operation Support

Readiness Check A compilation of simple and advanced checks for the Process Integration of Netweaver 2004 (XI 3.0) and Process Integration Netweaver 2004s (PI 7.0) Version 4 05. Dezember 2007

Readiness Check

Content: Introduction........................................................................................................................3 1 Basic Checks ..................................................................................................................4 1.1 SLDCHECK ............................................................................................................................................4 1.2 Testing the HTTP services on the Integration Server ..........................................................................5 1.3 RFC Connections ..................................................................................................................................7 1.4 Testing the connection to the Integration Builder ...............................................................................8 1.5 Java Component Check ......................................................................................................................10 1.6 Netweaver Administrator ....................................................................................................................12 1.7 Testing the availability of the PI-JAVA applications..........................................................................14 1.8 Testing the connectivity between Integration Repository and SLD..................................................16 1.9 Testing the connectivity between Integration Directory and Integration Repository.......................16 1.10 Caches in PI .................................................................................................................................17 1.10.1 Cache Connectivity Test ............................................................................................................17 1.10.2 SXI_CACHE ..............................................................................................................................18 1.10.3 Adapter Engine Cache ...............................................................................................................19 1.10.4 Testing the CPA Cache..............................................................................................................20 1.11

Runtime Workbench (RWB).........................................................................................................22

1.12 Adapter Monitoring ......................................................................................................................23 1.12.1 Adapter Monitoring via Communication Channel Monitoring .......................................................23 1.12.2 XI 3.0 <SPS 17 / PI 7.0 <SPS 8: Adapter Monitoring on the central and decentral Adapter Framework........................................................................................................................................25 1.13 Backgroundjob Monitoring..........................................................................................................26 1.13.1 Background jobs running on the ABAP.......................................................................................26 1.13.2 Background jobs running on the J2EE engine ............................................................................26

2 Advanced Checks.........................................................................................................27 2.1 SLD Associations................................................................................................................................27 2.2 Testing cache notifications of the Integration Directory ...................................................................28 2.3 Small Scenario.....................................................................................................................................29

3 Additional tests for existing Scenarios ......................................................................30 3.1 Configuration Test...............................................................................................................................30

Page 2

Readiness Check

Introduction To simplify things we will use for both releases Exchange Infrastructure 3.0 and Process Integration 7.0 the name Process Integration (PI). From an architectural point both releases does not show significant differences (beside of HA specific facts). The following graphic shows the components that make up the SAP Process Integration. The SAP Process Integration consists of the following functional components: Integration Builder, Integration Repository, Integration Directory, Integration Server, Integration Engine, Business Process Engine, Adapter Engine, Runtime Workbench (RWB) and System Landscape Directory (SLD)

Most of these components have to exchange information during runtime of the Process Integration. Thus, it is essential that each component can be accessed and that the communication between them is set up correctly. However, major changes of the PI infrastructure, especially when applying patches or when changing hostnames in the context of a High Availability setup, are potentially critical for the correct collaboration of the PI components. Therefore we recommend to carry out a set of simple checks that test the basic communication paths between the PI components and strive to ensure a smooth runtime after a software patch or HA setup. These checks are divided into 2 groups: the first group is meant to be carried out by everyone and requires little to no knowledge about PI. The second group of checks is addressed to PI experts, e. g. developers or PI system administrators, and require a basic understanding of the Process Integration.

Page 3

Readiness Check

1 Basic Checks This group of checks can be carried out easily, with a small expenditure of time and does not require extended PI know how.

1.1 SLDCHECK The SLDCHECK is a useful transaction that tests several things: Do the RFC connections SAPSLDAPI and LCRSAPRFC work? Are the server access settings in transaction SLDAPICUST correct? Is it possible to read data from the SLD and the Exchange Profile? Does the Integration Server have a Business System defined?

Procedure Log into the client of the Integration Server and call transaction SLDCHECK. This will open up a browser window which allows you to log into the SLD and thereby check if the SLD can be accessed by a dialog user. After verifying that you can logon to the SLD, go back to the R/3 window and check for the following successful checks: In the section “Properties of RFC destination SAPSLDAPI” there should be the statement “RFC Ping successful” In the section “Calling function LCR_LIST_BUSINESS_SYSTEMS” there should be the statement “Function call terminated successfully” and you should be able to see a list of a few business systems In the section “Calling function LCR_GET_OWN_BUSINESS_SYSTEM” there should be a statement “Function call terminated successfully” and you should see the business system of the integration server. In the section “Calling function LCR_GET_BS_DETAILS” there should be a statement “Function call terminated successfully”. Check if the URL given in this section corresponds to http://:80<sysnr>/sap/xi/engine?type=entry In the section “Properties of RFC destination LCRSAPRFC” there should be the statement “RFC Ping successful” In the section “Calling function EXCHANGE_PROFILE_GET_PARAMETER” there should be the statement “Function call terminated successfully”.

Troubleshooting If you should experience errors while carrying out this check, use the online help for the SLD which provides a detailed problem analysis scenario. Use the link: Access To SLD From ABAP Fails.

Page 4

Readiness Check

1.2 Testing the HTTP services on the Integration Server If an HTTP service is not running on the Integration Server, this can cause connection problems, which in turn can cause the notification of the cache on the Integration Server to fail or messages which not arrive in the Integration Engine. This check controls the availability of important PI specific services for the cache and for the pipeline.

Procedure: 1.

Start transaction SICF. Choose default_host sap xi. All services should be active, particularly the services: - cache - cache_ssl engine (for pipeline processing) - adapter_plain (for HTTP Adapter) You can activate a service, by choosing the corresponding option in context menu.

2.

Start transaction SICF. Choose default_host sap bc ping (connection test) and then choose Test service in the context menu of the entry.

Page 5

Readiness Check

A browser should then open, prompting a logon. If the browser does not open, this indicates a possible problem with the HTTP service of the SAP system and needs to be examined.

Page 6

1.3 RFC Connections Only two RFC connections are tested during the SLDCHECK, that is SAPSLDAPI and LCRSAPRFC. The remaining RFC connections can be tested via transaction SM59. The AI_RUNTIME_JCOSERVER connection is used to connect to the Java Mapping Runtime. The connection Integration_Directory_HMI is used to get data from the specific cache tables which contain the configuration data.

Procedure Log into the client of the Integration Server and call transaction SM59. Test the following connections one at a time by double clicking on them and then pressing the button “Test connection”: AI_RUNTIME_JCOSERVER (TCP/IP connection) Integration_Directory_HMI A successful test of the connections should look like this:

INTEGRATION_DIRECTORY_HMI (HTTP connection, use the XIISUSER with XI3.0 and the PIISUSER with PI 7.0 to authenticate). Is the Path Prefix set to “/dir/CacheRefresh” as recommended since SP1 of XI 3.0? Note that a HTTP 500 response is a successful test for this connection. The connections SAPSLDAPI and LCRSAPRFC have already been checked with transaction SLDCHECK.

Readiness Check

1.4 Testing the connection to the Integration Builder Transaction SPROXY (a transaction usually used for the creation of proxies) provides 4 checks that can be used even if no proxies have been created in the PI landscape. The following aspects are tested: The address of the Integration Builder The HTTP connection of the Integration Server The Integration Builder is running The data of the Integration Builder is understood

Procedure Log into the client of the Integration Server and call transaction SPROXY. Start the checks by going to GoTo -> Connection Test. This will give you a pop-up in which the tests can be started by clicking on the program names highlighted in red. The test SPROX_CHECK_IFR_ADDRESS should give you the status: OK: address maintained.

The screenshot for Release 2004s (PI) looks similar:

The test SPROX_CHECK_HTTP_COMMUNICATION should give you the information that the communication is working.

Page 8

Readiness Check

The test SPROX_CHECK_IFR_RESPONSE should give you a valid XML document. For higher SPS of XI 3.0 and also for release 2004s (PI) you should receive a confirmation that the communication is working:

The test SPROX_CHECK_IFR_CONNECTION should give you the information that the communication is working.

Page 9

1.5 Java Component Check The purpose of the check is to ensure that the java part is up and running from a technical perspective. If this check fails most of the subsequent checks will also fail.

Procedure: Open a Browser Window and type in the URL: http://<j2ee_host>:<j2ee_port>/index.html

If you can enter the page then this is the first hint that the dispatcher an the at least one server node is up and running and this specific application is started. On this page you will find important application. Among others a link to the “System Information” page. Enter the page by clicking on the link. A LogOn window appears. To LogOn you need j2ee_admin rights. When you enter the page you will get an overview about the SAP Web AS Java and specific information to every component. Further below you find information to the dispatcher and server nodes. Green indicates that the component is up and running. Yellow and Grey should be investigated.

Readiness Check

Troubleshooting For error troubleshooting you find further information in SAP Note Analysis of errors within the startup of J2EE Engine.

Page 11

Readiness Check

1.6 Netweaver Administrator The Netweaver Administrator becomes more and more important for the administration of an SAP double stack system. Up to XI 3.0 SP 20 and PI 7.0 SP 13 a lot of functionalities can also be found in other tools but with upcoming releases the Netweaver Administrator will become the only administration tool. The purpose of this check is to verify if the Netweaver Administrator is ready to use.

Procedure: Open a Browser Window and type in the URL: http://<j2ee_host>:<j2ee_port>/index.html Open the link SAP NetWeaver Administrator. A LogOn window appears. To LogOn you need j2ee_admin rights.

You can choose via the select box which system you want to administer/monitor. When you access the Administration page system.

System you should be able to see the status of the connected

Page 12

Readiness Check

Troubleshooting In case of problems check the Monitoring Set Up Guide for SAP NetWeaver in the SDN or under http://service.sap.com/operationsnw70 Monitoring and http://service.sap.com/nwa.

Page 13

1.7 Testing the availability of the PI-JAVA applications Procedure: Log into the client of the Integration Server and call transaction SXMB_IFR. This will bring up a browser window with 4 links: Integration Directory, Integration Repository, Runtime Workbench and System Landscape Directory (SLD). The SLD has already been tested in the first check, so only the first three components have to be tested at this point.

The following check cover the readiness and availability of the Integration Repository/ Integration Directory and the Mapping Runtime.

Procedure: Start a web browser and type in the following URL: Integration Repository: http://:<port>/rep/hmidiag/ext?method=info Integration Directory: http://:<port>/dir/hmidiag/ext?method=info Mapping Runtime: http://:<port>/run/hmidiag/ext?method=info

(Host name and port can be determined from the SAP Process Integration start page.)

A logon dialog should be displayed. If it is not displayed, this indicates that the Integration Repository (or even J2EE) is not running and needs to be started.

Readiness Check

1.8 Testing the connectivity between Integration Repository and SLD Procedure: Start the Integration Repository as described in the previous chapter. Go to ‘Tools’ -> ‘Transfer from System Landscape Directory’ -> ‘Import Software Component Versions’ This test is successful if you can see a list of Software Component Versions.

1.9 Testing the connectivity between Integration Directory and Integration Repository Procedure: Start the Integration Directory as described in the previous chapter. Go to ‘Tools’ -> ‘Transfer Integration Scenario from Integration Repository’. Then use the F4-Help for the Field ‘Name’. This test is successful if you can see a list of Integration Scenarios from the Integration Repository. Cancel the Transfer once you have made sure that the list can be displayed properly.

Page 16

Readiness Check

1.10 Caches in PI In SAP NetWeaver Process Integration information is often cached to speed up the input help of the tools or improve performance of the runtime. Here different caches are used during design and during runtime. Important runtime caches are: Integration Server ABAP (SXI Cache) Integration Server Java CPA Cache (Adapter Engine) SLD Cache (covered by SLD Check) Adapter Engine Cache

1.10.1 Cache Connectivity Test The Runtime Workbench offers a Cache Connectivity Test with a comprehensive view on the most important runtime caches. You use this function to check whether both steps for updating the runtime cache are working correctly: 1) Notification of the components with cache data (consumers) using a cache refresh and 2) Retrieval and update of cache data by the consumer. In this procedure, a change list with a test object is created in the Integration Repository and released, and the Integration Directory is notified. The Integration Directory in turn notifies the relevant consumers, who then retrieve the test data and update their runtime cache accordingly.

Procedure Log into the client of the Integration Server and call transaction SXMB_IFR. This will bring up a browser window with a link to the RWB. Once logged in, go to the ‘Component Monitoring’ choose ‘All’ from the dropdown menu of the field “Component with Status” and hit the “Display” button. Navigate to the Test by using the “Cache Connectivity Test” Button. To start the test hit the Button “Start Test”. Please note that you will at first see yellow triangles that indicate that the test is still ongoing (the cache notifications and updates need some time) and that you will have to hit “Refresh Display”. Check if all Cache Refreshes could be carried out, that is all components with cache data show a green light.

In the above screenshot two Adapter Engines show a red light. These Adapter Engines are decentral Adapter Engines and were not online during the Cache Connectivity Test. Clicking on them enables troubleshooting of the problem, in this case “Unable to establish http connection”:

Page 17

Readiness Check

Troubleshooting: In case of any error check the following documents by clicking on the link.

- “How to handle Caches with XPI Caches in SAP Netweaver 7.0” - “How to handle Caches with in SAP XI 3.0”

1.10.2 SXI_CACHE Transaction SXI_CACHE lists most of the content of the Integration Repository and Integration Directory. This information is used by the Integration Server during runtime and it is thus critical that this information is always up to date. A cache refresh is usually carried out automatically, but for testing purposes it can also be started manually. By carrying out a manual cache refresh several things can be tested, the most important being: Connection to the SLD and the Integration Builder Correct entries in the SLD

Procedure Log into the client of the Integration Server and call transaction SXI_CACHE. Start a manual cache refresh via XI Runtime Cache Start Complete Cache Refresh (Shift + F6). This will trigger a background job that can be seen in transaction SM58 (its name is SAI_CACHE3_REFRESH_BACKGROUND). Once this is finished check transaction SXI_CACHE for (be aware that a full cache refresh can take a while until be finished). Instead of a full cache refresh you can also trigger a delta cache refresh: a green light for the PI Runtime Cache status, saying “cache contents are up-to date”

Page 18

Readiness Check

Troubleshooting If you should observe a red light or a warning, navigate to the reported problems/errors (this is available for support packages 4).

For more information check regarding caches check the following documentation: - “How to handle Caches with XPI Caches in SAP Netweaver 7.0” - “How to handle Caches with in SAP XI 3.0” In case the exception does not make sense to you, open an OSS ticket on component BC-XI.

1.10.3 Adapter Engine Cache The Adapter Engine Cache contains the URL of the Adapter Engine. The information is used when a message is sent from the Integration Engine to the Adpater Engine.

Procedure: Log into the client of the Integration Server and call transaction SXI_CACHE. Navigate to: GoTo -> AE Cache

The test is successful if you find the URL of your Adapter Engine. Bear in mind that the cache is filled when a first message has been processed and sent to the Adapter Engine.

Troubleshooting The Adapter Engine URL is taken from the SLD where the Adapter Engine is registered. If you find the wrong URL listed then check the information in the SLD. For more information refer to the Troubleshooting Guide.

Page 19

1.10.4 Testing the CPA Cache The cache of the Adapter Engine is called CPA Cache. It contains information about all adapters that run on the J2EE Engine. Thus, it is important that the refresh of this cache is working correctly.

Procedure: Open a browser window and call the URL: http://:5<sysnr>00/CPACache/refresh?mode=delta | full. Use the XIDIRUSER (PIDIRUSER) to authenticate yourself, it is the only user that is allowed to carry out a CPA Cache Refresh. The test is successful if you get the message that the cache refresh has been carried out in xyz milliseconds. Please note that any additional text. e. g. “The XML page cannot be displayed” or “invalid at the top level of the document”, points at an error.

You can also see the content/ history of cache refreshes in a browser window by calling the following URL: http://:<port>/CPACache Choose Display CPA Cache Content or View Cache Update History.

Readiness Check

Troubleshooting: 1) Note 741214 - Troublesh. during cache update of the J2EE CPACache service gives valuable hints about possible reasons for errors. 2) In case of the error message “invalid at the top level of the document” the viewing of the source code of the respective Explorer page gives detailed information about the error.

Page 21

Readiness Check

1.11 Runtime Workbench (RWB) The Runtime Workbench is the central monitoring tool of the Process Integration. It offers self and status tests for most of the PI components: Integration Server, SLD; RWB, Adapter Framework, J2SE Adapter Engines, integrated Business Systems, Proxy Runtimes and Mapping Runtime.

Procedure: Log into the client of the Integration Server and call transaction SXMB_IFR. This will bring up a browser window with a link to the RWB. Once logged in, go to the ‘Component Monitoring’ choose ‘All’ from the dropdown menu of the field “Component with Status” and hit the “Display” button. In the list of components mark the radio button of the entries one by one and check if you get green lights displayed at the bottom of the page

Click on every link in the Runtime Workbench and check if they open up without error messages. The links are: ‘Message Monitoring’, ‘End-to-End Monitoring’, ‘Performance Monitoring’, ‘Index Administration’, ‘Configuration’, ‘Alert Configuration’, ‘Alert Inbox’ and ‘Cache Monitoring’. For each link, carry out the specific action, e. g. try to display existing mapping programs in the Cache Monitoring. This test ensures that the connectivity between the RWB and ABAP as well as the connectivity between the RWB and other JAVA applications works properly. In addition, authorizations of the XIRWBUSER (PIRWBUSER) are being checked.

Page 22

Readiness Check

1.12 Adapter Monitoring With very few exceptions messages enter and leave the Process Integration via adapters. Some of them run on the Adapter Framework (typically JDBC, File, JMS, SOAP Adapter), some of them run on the integration server (IDoc Adapter, HTTP plain adapter). You may also use the J2SE standalone adapter engine or decentral adapter engines. Starting with XI 3.0 SPS 17 / PI 7.0 SPS 08 a new Communication Channel monitor was introduced. Unlike the existing adapter monitor, the communication channel monitor also provides current runtime information about the communication channels of individual adapters, as long as they are registered for communication channel monitoring. To monitor adapters that do not provide runtime information to communication channel monitoring, continue to use the adapter monitor. For all standard adapters the Communication Channel monitor is the tool to be used and no information will be displayed in the Adapter Monitor.

1.12.1 Adapter Monitoring via Communication Channel Monitoring Procedure: Logon to the Runtime Workbench start page (e. g. by running transaction SXMB_IFR and using the link in the browser window) and choose component monitoring. A first indicator about the status of the communication channels is already given by the result of the self-test in the Component Monitoring of the RWB for a selected Adapter Engine. If you see a read light in the Communication Channel (CC) monitoring this already indicates that there are CC in error. When clicking on the Details link you will see an overview about the status of the Communication Channels as shown in the screenshot below.

Check that the Communication Channels that are in use, display a green light. If you observe a red light, have a look at the Error messages displayed on the bottom of the page and inform the respective Application People/ Administrator/ Developer..

Page 23

Readiness Check

Page 24

Readiness Check

1.12.2 XI 3.0 <SPS 17 / PI 7.0 <SPS 8: Adapter Monitoring on the central and decentral Adapter Framework The adapter monitoring has to be used, if adapters do not provide runtime information to communication channel monitoring.

Procedure Log on to the RWB start page and navigate to “Component Monitoring”. Choose display of all components and select the appropriate Adapter Engine. Click on the button “Adapter Monitoring”. Check that the Adapters that are in use, display a green light. If you observe a red light, have a look at the Error messages and inform the respective Application People. Note that the content of this Adapter Monitor varies with the support package of your PI.

Page 25

1.13 Backgroundjob Monitoring The housekeeping jobs for PI ensure a smoothly running Process Integration. They do not only carry out the archiving and deletion, but also care for performance data and retry mechanisms. Even you should have not scheduled any jobs up to now there are some standard jobs which are scheduled during installation.

1.13.1 Background jobs running on the ABAP When you have set up a PI System you will not find PI specific jobs running. On the ABAP part the background jobs need to be scheduled manually. You can find a good overview which jobs might need to scheduled under the note: Note 884865 - PI/XI Admin Check -> Job Monitoring of Housekeeping Jobs.

1.13.2 Background jobs running on the J2EE engine In the J2EE engine a number of background jobs are scheduled during the installation process. Thez should be checked if they run smoothly.

Procedure Log in to the RWB and call your central or decentral Adapter Engine in the Component Monitoring. Choose the button “Background Processing”. A new window opens displaying the background processes being executed in your Java environment. Check if a default deletion job is scheduled and in status green Check if a default Recover Job is scheduled and in status green Additional jobs to be monitored here could be the automatic message restart job or the archiving job if scheduled.

Readiness Check

2 Advanced Checks The following tests take up more time and require a certain degree of insight into the architecture of the Process Integration, especially for the last check of this chapter.

2.1 SLD Associations For several application processes it is required that the PI components all belong to the same ‘domain’. This domain and the associations of this domain are set during start-up of the system. However, when changing the hostname of the Integration Server the associations may become inconsistent and it is necessary to recheck / recreate them.

Procedure: Log into the client of the Integration Server and call transaction SXMB_IFR. This will bring up a browser window with a link to the SLD. Once logged in go to the link “Technical Infrastructure” and choose “Exchange Infrastructure” from the drop down menu. Check if the Integration Server has a Domain assigned (in the screenshot below the Domain of the XI 3.0 Integration Server YIS is pwdf2031). Check if for the above Domain there are all components registered as well: Adapter Engine, Integration Directory, Integration Repository and RWB. Check if there are other Domains that are no longer used.

Troubleshooting: See note Note 764176 - Error in XI due to inconsistent SLD contents if you find any inconsistencies.

Page 27

Readiness Check

2.2 Testing cache notifications of the Integration Directory For the development of scenarios it has to be ensured that the newly created information about objects (interface determinations, mappings etc.) is transmitted to the two components that need the information at runtime: the Adapter Framework and the Integration Server. The Integration Directory provides you with a Notification Service that is available via the menu entry “Environment” at the top of the user interface. This notification shows green/red status lights of cache notifications to the Integration Server as well as to the Adapter Framework. Since the creation or the change of an existing object is required to carry out this test, it is NOT recommended for productive systems.

Procedure: Log into the Integration Directory and open an existing object. You may also create a new one if you prefer. If you choose to open an existing object, go into the edit mode afterwards and make a small change (like removing and subsequently adding a character in the description of the object). This small change is needed to be able to save and activate the object again. After saving and activating the cache notifications are started automatically. Check if both notifications are executed successfully, that is they show a green light.

Page 28

Readiness Check

2.3 Small Scenario The ultimate test to ensure that the PI landscape is working correctly is to create a little scenario and carry it out. In this way all development tools as well as the runtime is tested. This does not require as much time as the development of scenarios for your business processes since a very easy set-up can be chosen and since this procedure could be documented within your company. A suggestion would be to create a File Adapter for the Integration Server. This File Adapter would poll a *.txt file from a test folder, convert it into an XML and send it to the PI Integration Server. Create a simple mapping, for example rename the XML tags of your test file, that is then carried out in the Integration Server. The endpoint can again be a simple File Adapter that would store the mapped file into an archive folder. If you need a more detailed description about how to set up a scenario, we recommend to implement the demo scenario for which SAP provides a detailed documentation as well as delivers the relevant objects in the Integration Repository (Software Component Version SAP Basis 640). The demo scenario can be found at help.sap.com under following link: Demo Examples (NW 2004) or Demo Examples (NW 2004s)

Page 29

Readiness Check

3 Additional tests for existing Scenarios 3.1 Configuration Test For PI there is the opportunity to test the configuration of your scenarios. The test is called “configuration test” and is part of the Integration Builder Directory. During the configuration test, the individual pipeline services of the Integration Server are called, and the change to the message in each case is simulated in the individual processing steps. The advantage of this test is the detailed test report after having executed the test. If there are errors, it is possible to navigate to the relevant configuration objects and to see the errors on the “Error Log” tab page. Detailed information can be found in the online documentation.

Procedure: Log on to the Integration Builder Directory and go to Tools -> Test Configuration. Define the input parameters like header data and a payload if required. Then start the test for the either the whole pipeline (”Run”) or for single pipeline steps (“Step Over”). Check if all pipeline steps are executed successfully, that is they show a green light for a single step and that the status is the black/white flag for a successful processing of the whole pipeline.

Page 30

Related Documents

Readiness
November 2019 18
Ipo Readiness
May 2020 7
College Readiness
May 2020 8
Readiness File
November 2019 12
College Readiness
June 2020 9

More Documents from ""

Gaurav_project.docx
November 2019 30
Poster.docx
November 2019 21
Documents For Evidence
December 2019 21
E Discovery
November 2019 29
Mutimedia Final Report.docx
November 2019 21