Six Sigma

Patch Mana gement usin g S ix S ig ma ISSA Chapter PatchPhoenix Management July 11, 2006 Process using the Six Sigma Methodology

Phoenix ISSA Chapter meeting – July 11, 2006 Dee Ramon, CISSP LIRM

CSC Proprietary 09/17/09 05:11 AM 3115_FMT 1

Agenda 

CompanyA, CompanyB and CSC



Six Sigma



Six Sigma Patch Management - Case Study



Measured Results – CompanyB only



Summary comparisons - before & after



Conclusion



Questions?



Real life challenges interspersed throughout 2

CompanyA, CompanyB and CSC 

CompanyA Inc. spun off One Sector business into a completely separate publicly traded company.



CompanyB has • 22,000 Employees worldwide in 30 countries CompanyA outsourced infrastructure Business to CSC in 2003 in a 10 year agreement. CompanyB signed a similar agreement Variety of players, new roles, significant change of people and roles





3

Timeline March 2004

13 Jan, 2005

CompanyB name announced Network Separation

20 Jan, 2005

Project ends 15 May, 2004

6 Sigma Project Initiation

May 2004 - January 2005

6 Sigma Project

February 2005 - March 2006

CompanyB Only Patch Management

January 2005 March 2004

January 2006 March 2006

13 July 2004

• Competing activities made priorities interesting

4

What is Six Sigma? Six Sigma is a process improvement methodology using data and statistical analysis to identify and fix problem/opportunity areas. Six Sigma also refers to a deployment model that aligns employees with a series of high-impact projects. Over the past ten years, Six Sigma has delivered a variety of benefits to companies, e.g., • • • • •

reducing costs increasing revenue improving process speed raising quality levels deepening customer relationships

In addition, Six Sigma has been used across a variety of industries and business models, from manufacturing to services. Companies using Six Sigma include: • • • • • • •

General Electric AlliedSignal Dow DuPont Ford CompAor Company Merrill Lynch Toshiba

Six Sigma has provided billions of dollars of top-line growth and bottom-line earnings improvement. 5

Real life  Six

Sigma was very ingrained in CompanyA culture  Six Sigma was new to CSC  However……

6



Sigma and it’s practical use in the real world

CompanyA pioneered sigma use in 1986 to improve product quality by driving variance out of the manufacturing processes



1 sigma equates to 68% of values being within 1 standard deviation



Original goal was six sigma product quality

• •

Under 3.4 defects per million opportunities (practically 99.999%) 100x quality improvement in five years



Methodology was later applied to business processes



CompanyA has saved $16 billion to date

Units:

1,000 circuit boards

Units:

9.9 million airline flights in 2004

Opportunities:

58

Opportunities:

1

Defects:

place graphic 25 crashes resulted in fatalities in this area

Sigma:

6.52

(1 board + 13 resistors + 4 capacitors + 2 diodes + 38 solder points)

Defects:

place graphic in this area 18 boards

Sigma:

4.92

7

Digital Six Sigma Process Roadmaps DMADDD

•

Define Measure Analyze

DMAIC

place graphic in this area

DMADV

Improve

Design

Design

Control

Digitize

Verify Product Or Process Design

Process Improvement

Draw Down

• • • •

Alignment & mobilization Clarity & CompAivation Profound understanding Breaking tradition Institutionalizing change

Productivity Increase Slide 8 TM

CompanyB™ and the CompanyB logo are trademarks of CompanyB Semiconductor, Inc. All other product or service names are the property of their respective owners. © CompanyB Semiconductor, Inc. 2005.

D

Patch Management Process

D

Wh at is im por ta nt ?

M

Ho w a re we doin g?

A

Wh at is wro ng ?

I

M

A

I

C

C

Wh at ne ed s Ho w d o we to be Guar an te e do ne? pe rf orm an ce?

9

D Phase/Activity

Target Date

M

A

Comp Date

Define 

Schematic (Yyx Alignment)



Team Charter



SIPOC



“AS-IS” Process Map



Voice Of Business/Voice of Customer to CTQ’s



Cause-n-Effect Diagram



Quick Wins Identified

Measure 

Data Collection Plan



Operational Definition



Source of Variation Study



Sigma Analysis



Process Capability

I

10

C

D Phase/Activity

Target Date

M

A

Comp Date

Analyze 

Pareto Analysis & Stratification



Regression Analysis



Root Cause Analysis

Improve 

Cost Benefit Plan



Alternative Solutions Identified



“SHOULD BE” Process Map



Change Plan



Pilot Plan and Results

Control 

Digitization Plan



Standardization/Adoption Plan



Lessons Learned and Feedback

I

11

C

“Define” phase Schematic (Yyx alignment) D

M

A

I

C

What i s the Big Goal? Reduce Threat to CompanyA Business through Patch Management (The Bi g Y )

What are the inputs? (The litt le y’ s)

What driv es the y’s?

Cycle Time 7 days

Automation

Patch Compliance

Impact

100%

0

Quality

Standards & Policy

Process Efficiency Team

Scope (Asset Database)

Downtime

(Th e vita l x’ s)

Communications Timing Customer’s or Business’ Schedule (system shutdowns, closures)

Environmental Capability (“patchability”) (SMS failure root causes and criteria for alternative)

IT Resources (bandwidth, vacation backups)

Shortly after we finished this, the six sigma trained project leader was reassigned…..

12

D

M

A

“Define” phase Opportunity Statement Team Charter

I

C

Business Case

The rate and frequency of security vulnerabilities and the exploitation of those vulnerabilities that have disrupted CompanyA business operations in the past year has been increasing steadily. Delays in patching vulnerable systems continue to represent significant risk and cost to CompanyA.

Risk to CompanyA’s business assets, productivity and reliability can be decreased by reducing the cycle time of the patch management process and increasing the compliance rate. Cost of remediation can be decreased as a result of patch management cycle time reduction as less CompanyA resources are diverted from their jobs to address issues resulting from vulnerable systems.

Goal Statement

Project Scope

Reduce

patch management cycle time and impact and increase compliance.

Targets Total cycle time from Patch Availability to Implementation: Impact : 0 Compliance: 100%

7 days

Project Plan One End-to-End Project Digital Six Sigma DMAIC process methodology will be used Joint team of CompanyA, CSC and Foundstone representatives Identification of sub-project dependent upon acceleration of program (TBD) DEFINE stage completion: MEASURE stage completion: ANALYZE stage completion: IMPROVE stage completion:

June 11th, 2004 * July 9th, 2004 August 13th, 2004 September 1, 2004

Dependent on resources being allocated appropriately.

Each step in the process from the CompA/CSC agreement to patch… … to……100% of systems identified as vulnerable being patched. Project Governance: Sponsors: CISO VP Champions: CSC VP

GIS VP CSC Ops Manager

Project Steering Committee: CSC Security Ops Director, CompA Security Ops Director Process Owners: TBD after SIPOC

Team Selection Project Leader: CSC Co-Leader: CompanyA Master Black Belt: Finance: TBD

CompA IT Security Black belt CSC SMS Manager GIS VP Assistant

Team: SMS: package lead Servers: CSC Server Manager Field Services: CSC regional manager Help Desk: TBD Non-Managed: MIPS specialist Exceptions: MIPS Sector manager CSC Security: CSC Ops manager, CSC Europe 13security manager CompA Security: MIPS Microsoft specialist

D

M

A

End Boundary: “Define” phase - SIPOC

SUPPLIERS:

4

INPUTS:

MCERT Team (CSC, CompA, MS)

MCERT Discussion

Microsoft

.cab file

MCERT Team

Deployment Plan/ Certified Package

3

PROCESS

1

OUTPUTS:

C

Patching Completed

Start Boundary: Decision to Patch 5

I

2

CUSTOMERS:

Dep loyment Plan /

SMS / System Admins

Create / Certify package

Certified Package

SMS / System Admins

Create / Send Email

Communication Test

All CompanyA / CSC / Contractors

Patch result

Desktop Owners / Field Service / MIPS

Create Techni cal Co mmun icati on deployment plan

CSC / SMS Team

Communication / Certified package

Deployment

MIPS Specialist / CSC Scan administrator

Foundscan / HINV / Various DBs / Art Jr.

Identification of Vuln systems

MIPS Sector Managers

Failed Patch Process/ Updated Tracking DB

MIPS Sector Managers

CIO Approval / MIPS Sector mangers

MI PS/ CSC / Sect or CIOs Tracking DB / Sy s Ad mi ns / MIPS Sect MI PS /CSCor /Man Sectagorer CIO s Upda ted T rack ing DB Alternate / Sys Ad min s / & Man ua ll y patc h s ys tems remediation MIPS Sect or Man ag er Desk top Own ers / La st di tc h effort MI PS Sec tor Mang ers / on ) s / App Closure (Segreg ati on / Disc onne ctiCIO licatio n O wne rs

This did help define high level process and

14

D

“Define” phase AS-IS process map

This was a lot of work, but valuable as few people knew the entire process; most people understood just their piece

M

A

15

I

C

M

D

A

I

C

“Define” phase - Quick Wins DSS Solution Kit Project: Equipment Returns Rate Reduction Answer yes or no as to whether the condition applies to the potential quick win opportunity.

Potential Quick Win Opportunity

Easy to Fast to Implement Implement

Cheap to Implement

Within the Team’s

Benefits will be Reversible Implement

Increase Penetration of SMS tool Create standard SMS document for SMS admins

YES YES

NO YES

YES YES

NO YES

YES YES

YES YES

NO YES

Define Parameters of a heathly SMS client Publish the policy for Infrastructure compliance Weekly compliance reporting status Note: we can do this easily from the central server web reporting.

YES YES YES

YES YES YES

YES YES YES

YES NO YES

YES YES YES

YES YES YES

YES NO YES

Weekly Healthy Client Compliance. Note: we can generate the reports today, Is this just reporting?

YES

YES

YES

YES

YES

YES

YES

Define a standard communications policy Standard user FAQ area Publish policy for patching - cycle time Make available all bundles in one place

YES YES NO YES

YES YES NO YES

YES YES YES YES

NO YES NO YES

YES YES YES YES

YES YES YES YES

YES YES NO YES

List of workstations to be spoonfeed into sms - this will not include remediation of any issue

YES

YES

YES

YES

YES

YES

YES

Pilot announcement to include specific directions on how to open tickets - specific subject like MS04-028 pilot issue

YES

YES

YES

YES

YES

YES

YES

16

Hard to get priorities to do quick wins – but most got done

D

M

A

I

C

“Measure” phase - Data collection plan Performance Measure

Data Source and Operational Definition Location Continuous Describe Defect

Who Will Collect the Data?

When Will Data Be Collected?

Other Data That Should Be Collected How Will Data Be at the Same Collected? Time

Impact Number of formal pilot users who have installed Number of pilot users patch during pilot period Number of unique issues reportedThis includes both the during deployment number of distinct issues reported about the patch during the deployment period and the number of total issues reported about the patch during the deployment period.

SMS

Number of users with SMS clients who have installed at the middle Via SMS query at SMS deployment and the end of the pilot middle and end of team period pilot

view of Monet tickets opened during deployment period daily for first week of that were reported deployment; after that against the patch weekly deployment

Monet

Comments

Due date

Historical available Yes/No

This won't reflect people who have installed on their own and are not in SMS Dee needs to check if we can do this historically 8-Oct

N

Consistent Monet profile Try to see if we can get history on this and provide number of issues published via FAQ 8-Oct

N

Cycle-time Pilot cycle-time

Time measured in days sms. from pilot start date to pilot end date ( F - E)

Patch deployment cycle-time

Time measured in days from package deployment to start to when it passed. A approx 1,000 ( for all systems) are vulnerable approx 98%

SMS team

VirusUpdate MIPS (scanning data), Update expert records, SMS central server

end of pilot

End of deployment period

via SMS.CompA.com database for Nov 2003 to current if available in daily/weekly scan process for Nov 2003 to current if available

30-Sep

Y

30-Sep

Y

Compliance Unhealthy SMS clients - SMS cannot patch these systems.

Total number of machines SMS for machines SMS Team that don't have healthy that have SMS SMS clients (has SMS butinstalled and do not not returned inventory have a healthy client within X days)

snapshot from last week and this week reports

Total Windows machines on Total Windows machines Foundscan, CSC = Foundscan,snapshot CompA network that should be on CompA network that CompCheck, Browse CompCheck SMS managed should be SMS managed -lIst, BDNA includes CompA CompA IT = Browse owned/leased not lIst personaly owned / short term contractor machines CompA IT = BDNA

Total number of reachable SMS managed machines that have not reported inventory in X days ( X = 7, 14, 21, 30 not reporting in inventory). Superset of: Foundscan, BDNA, CompCheck and browse list. Will determine what ideal set of fields to collect.

Don't think we can collect8-Oct historically

????

Machine name and if Oct 8 for list fo fields necessary domain or Oct 15 have data workgroup - minus Oct 20 have data ready duplicates and servers for review we are not going to do historically as the data doesn't change that much; should start on this next week

Fair amount of work, but useful as it defines what data if available

17

next week to start completion by Oct 8, review Oct 1

Real life - Measure 



 

What we selected and originally started to measure was based from Vulnerability scanning Accurate, but this wasn’t available for every vulnerability; also point in time versus continuous Ended up using SMS, not as good coverage, but could be used consistently Had to do ‘measure’ twice

18

“Analyze” phase Pareto and root cause 

This shows the best areas to focus on are:

D

M

A

I

Pareto Diagram 50%

1.2000

45% 1.0000

40%

•

Unmanaged systems Broken / unhealthy clients

35%

0.8000

30%

Rel. Freq.

•

25%

0.6000

20% 0.4000

15% 10%

Failure Mode Effect Analysis identified high risk areas

0%

0.0000

Br ok M S en Sc m / he un an du he ag l in al ed th g y ex cl ce ie nt pt s io ns Pr /d el eay re U q ns f ai up lu po re s rte d R O eb S/ SP oo t pe Pa nd tc h in D fa g i sk ile Pe d( sp nd ve ac in nd e g or ap er pl ro ic at r)* io n U cl nk os ur no SM SM e w n S S s ta Pe pr tu oc rm s es is si s on d D el s/ is ay ad ab m le in d/ rig de C ht c an om s 't m In p is at ve si ch nt on or ex y ce re pt po io rti n ng fa i lu re



Scheduling exception

0.2000

5%

N ot S

•

C

Measuring this challenged some long held assumptions

19

M

D 1. Standard change control window for servers 2. Standard change window for labs/factories 3. Login script to communicate patch status 4. Group policy to enforce SMS client

16. Standard disconnection policy 17. Server ownership process 18. Network team to provide network list

7. SMS auto-discovery tools for machines in AD domains

19. Report on both pkg. success and applicable High reports for pending 20. Standard reboot 16 1

Benefit

10. Add SMS installer to image

2

9 21

11

20

8. Predictable Reboot delay 9. Allow SMS pull during communications time

19 14

Low

8 10

18

Low

3 4

22

6

24 13

12

7

5 17 23

15

Effort

High

11. Increase hardware for Foundstone scanning 12. Foundstone Enterprise

I

C

21. CSC Patchmanager 22. Standard procedures for Field Services 23. Replace machines for which automated housekeeping 24. Automate housekeeping of machines with low disk space

5. Move all machines into AD domain 6. Ongoing SMS client health monitoring

A

20

“Improve” phase

D

Long Term recommendations 1. Network Admission Control 2. Patch Management Tool vs. SMS

M

A

I

13. Notification via e-mail on patch success

3. Group policy to prevent login when not patched 4. Group policy to enforce standards

High

1

5. Forced reboot if patched and pending reboot 6. Open tickets for any machines that don’t have SMS client 7. Consistent means of detection of all Windows machine on network

10 3

Benefit

8. Select business users be part of Microsoft pilot program 9. Internal Windows update infrastructure 10. IPS 11. Proactive enforcement of OS/SP standards 12. Proactive enforcement of IE standards

2

9 5

13 8

12 11

4 6

10

7

Low Low

Effort

High

21

C

“Improve” phase Compliance projection after 7 days

D

M

A

I

C

Sigma

Compliance Projection 5 4 3 2 1 0

Ms03-043

Ms04-007

Ms04-011

Ms04-022

Ms04-022*

ms04-028*

Ms04-032*

Ms04-040*

Short term changes

Long term w ithout netw ork admission control

Long term -only netw ork admission control

Deployment 





Sigma compliance by the number of systems not patched 7 days after deployment. 6 sigma is achievable if network admission control keeps non-compliant systems off the network. CompanyA estimated a cost of poor quality reduction by $800K 22 per year.

M

D

A

I

“Improve” phase Cycle time projections 



Projection of cycle time reductions by implementing changes: • Short term • Long term - without network admission • Long term – just network admission Next steps: • Quantifiable benefits • ROI calculations • Expected results

Estimated cycle time changes

66 14 0

10

60

47

20

30

40

50

60

70

Cycle time in days

Present Short term changes Long term changes without network admission Long term changes - just network admission control

Network admission control was put off to a separate project for CompanyA

23

C

Measured results – CompanyB only

Emergency Patch Compliance - 2005 Approx. 17,500 systems

100% 90%

J anuary(HTML)

80%

February(SMB) February(Multiple)

% Patched

70%

J une (SMB)

60%

J uly(Multiple) August (PnP)

50%

October (MSDTC)

40%

November (Graphics)

30%

December (IE)

20% 10%

Days

0% 1

workdays only 90%3 can 5be achieved within 7 days during accelerated scheduling 7 9 11 13 15 17 19 21 23

24



Measured results – CompanyB only Improved High & Critical patch deployments

• •

High risk patch achieves 90%

•

90% = 2.78 sigma

24 days in 2005 -> 12 days in 2006

Critical risk patch achieves 90%

•

12 days in 2005 -> 7 days in 2006

Emergency Patch Compliance - Jan2006 Approx. 17,500 systems

100% 90%

(5) 2005 High

% Patched

80%

(4) 2005 Critical

70% 60%

January (WMF)

50%

January (TNEF)

40% 30% 20% 10%

Days

0% 1

3

5

7

9

11

13

15

25

workdays only

Measured results – CompanyB only CompanyB Microsoft Patch Compliance - 2005 / 06 (approx. 17,500 SMS managed workstations)

100

Average # of patches per workstation

97.3% 80

98.2%

97.9%

95.4% 2 3

60

98.4%

98.0%

97.5% 2

100.0%

2

1

98.5%

1 95.0%

2

2

% Compliance

9

90.0%

11 40

20

49

42

85.8% 59

64

72

73

81

82

83

87

85.0% Missing

0

80.9% March





80.0% May

June

July

August

September October

November December

January

1 in 5 security patches missing prior to Patch Manager implementation in May 2005. 26 1 in 87 today, 17x improvement

Installed Compliance

CompanyB Patch Management Process Internal Source





 



Monthly patch process follows Microsoft release schedule Reboot delay options • 3 times, up to 60 minutes each Limited use of disconnections Incorporated High risk patches into the monthly regular schedule • Process is standardized • Only one single restart per month • Includes prior security patches if missing Critical risk patching follows an accelerated process • Change control bypass approval for Servers • Single patch only

Notification of Security Update is now available

FIS Analyst categorizes

Matches High or Critical criteria?

Risk Assessment meeting.

Report with deployment plan options.

Vendor release, CERT, etc

Medium / Low High Regular patch bundle

Patch certification and Lab test

Pilot Test

Analyze cause with vendor for resolution.

Fail

Report if problems

CISO / CIO change control bypass approval based on the assessment.

Patch bundle

Special advisory

Patch certification and Lab test

Advisory updated with single EXE patch.

Pilot Test

Report if problems

Pass

Accelerated certification, lab and pilot initiated

Mandatory Deployment commences

Pass

Patch deployed as advertised updates for 2 days.

Daily patch reports

Deploy

Daily patch reports

Patch is readvertised 3 days after the initial mandatory deployment.

Deployment changed to mandatory after 2 days.

IT Management informed where compliance is low.

Compliance reports changed to weekly after 10 daily reports.

Schedule deployment or Exception?

Process stops after 30 days

Exception request.

Fail Retest

Regular monthly deployment

Critical

Analyze cause with vendor for resolution.

FIS Decision

27

Conclusion 

Start with client needs – “voice of the customer”



Get management support by following the Digital Six Sigma methodology (or another company backed program) • Define, Measure, Analyze, Improve, Control • http://www.isixsigma.com • http://sixsigmatutorial.com



Follow consistent processes • Reduce variability is key



Use tools that provide fast and accurate data (correct tools for the job)



Baseline, improvement trends and compliance • If you can’t measure it, you can’t manage it.



Develop solutions with a clear business case

28

Real Life Conclusions  Measurement,

process and tools are key to getting improvements; We implemented all three ( with or without six sigma)  Teams add a lot; but are also a challenge to keep focused  Change can be interesting

29

Questions?

30