Using Predictive Analytics to Calibrate FMEDA Why FMEDA gives the best failure rate results

Transcription

1 Using Predictive Analytics to Calibrate FMEDA Why FMEDA gives the best failure rate results ifea March 8, 2012 Dr. William M. Goble ex Sellersville, PA USA Copyright ex

2 IEC Fundamental Concepts IEC61508 Safety Life Cycle detailed engineering process Systematic Faults Design SOFTWARE Mistakes RELIABILITY Probabilistic performance based system design Random Failures HARDWARE RELIABILITY Copyright ex Copyright ex

3 SIF Verification Steps Safety Requirements Specification SIF Functional Description, Target SIL, RRF, Mitigated Hazards, Process Parameters, Logic, Bypass/Maintenance Requirements, Response Time, Proof Test Targets, etc. Manufacturer s Safety Manual Application Standards Manufacturer s Failure Data Failure Data Database Manufacturer s Safety Manual Application Standards 8. SIF Conceptual Design Select Technology No IEC Clause SIF Conceptual Design Select Architecture IEC Clause SIF Conceptual Design Determine Test Plan IEC Clause SIF Conceptual Design Reliability/Safety Calculation IEC Clause 11 RRF, SIL Achieved? Yes 12. Detailed Design IEC Clause 11, 12 Equipment Justification Report H/W & S/W Design Safety Requirements - Technology Chosen, Voting Logic, Proof Test Requirements, Automatic Diagnostic Logic, Bypass Logic, Repair Time Requirements, SIL achieved, etc. Detailed Design Documentation Loop Diagrams, Wiring Diagrams, Logic Diagrams, Panel Layout, PLC Programming, PLC Program Testing, FAT Test Plan, Installation Requirements, Commissioning Requirements, Proof Test Plans, etc. Detailed Safety Lifecycle Drawings, Copyright ex 2008, used with permission. Copyright ex

4 Field Failure Studies Pressure Transmitter Source Product Type Failure Rate per hour Comment NRPD95 Pressure Transducer 8.31E-06 Highest Number Refinery Data [Shel00] Analog Pressure Transducer 2.71E-06 Refinery Data [Shel00] Microprocessor Based Pressur 7.19E-06 DOW Plant Study [Skwe08] Pressure Transmitter 4.96E-07 Manufacturer Study [Moor98] Microprocessor Based Pressur 3.57E-07 Lowest Number When researching failure data it can be seen that results from various studies vary by an order of magnitude! This has caused many to conclude that the whole probabilistic approach to design cannot work! Copyright ex 2009 Copyright ex 4

5 Field Failure Studies Source Product Type Failure Rate per hour Comment CCPS89 Diff. Pressure Transmitter, low 1.01E-06 CCPS89 Diff. Pressure Transmitter, mean 6.56E-05 CCPS89 Diff. Pressure Transmitter, high 2.54E-04 Highest Number NRPD95 Pressure Transducer 8.31E-06 Refinery Data [Shel00] Analog Pressure Transducer 2.71E-06 Refinery Data [Shel00] Microprocessor Based Pressure Transmit 7.19E-06 DOW Plant Study [Skwe08] Pressure Transmitter 4.96E-07 Manufacturer Study [Moor98] Microprocessor Based Pressure Transmit 3.57E-07 Lowest Number In a letter to the Editor, IEEE Spectrum Magazine, August 4, 2000, Dr. Patrick D. T. O Connor, a well known reliabiliity engineer and author wrote: IEC61508 introduces a threat greater than inefficiency and cost, since it relates to safety.... In particular, it requires the quantification of risk probabilities... Copyright ex 2009 Copyright ex 5

6 Field Failure Studies Solenoid Valve Source Product Type Failure Rate per hour Comment NRPD95 Valve, Pneumatic Solenoid 1.67E-05 Highest Nu Refinery Data [Shel00] Solenoid Valve Not Available DOW Plant Study [Skwe08] Solenoid Manufacturer s Valve 7.02E-07 Manufacturer Study [AEAT05] Solenoid Valve 1.70E-08 Lowest Num Warranty Data Which number is "correct"? Why the different results? Copyright ex 2009 Copyright ex 6

7 Field Failure Studies Owner/Operator Field Failure Studies Variations of amount of data collected Different definitions of FAILURE Wear-out failures included? Categorizing and Merging Technologies Fault Isolation often not complete Manufacturer Field Return Data Studies Data Analysis methods vary widely Rarely is it known what percentage of actual failures are returned to manufacturer Different definitions of FAILURE (Not a problem scenario) Copyright ex 2009 Copyright ex 7

8 Owner/Operator Field Failure Studies Variations of amount of data collected Different definitions of FAILURE Wear-out failures included? Categorizing and Merging Technologies Lack of fault isolation After performing dozens of studies ex experience recognized that the data collection process varies by an order of magnitude or more! The process must be understood in order to analyze the data. Copyright ex 2009 Copyright ex 8

9 Owner/Operator Field Failure Studies The process must be understood in order to analyze the data. Questions: When is a failure report written? What is the definition of failure? Are proof test results included? Are "as found" conditions recorded for a proof test? What were the operating conditions? Copyright ex 2009 Copyright ex 9

10 Owner/Operator Field Failure Studies An extensive set of test results from a test shop indicated that a manufacturer had an exceptionally low failure rate when compared to the ex Predictive Analytics benchmark. An onsite visit to the test shop showed that when each instrument was returned for testing it was "cleaned up" then tested. Cleaning included disassembly and replacement of seals and o-rings. It is surprising that any units failed the test after that refurbishment activity. The test, repair and data collection process must be understood to analyze data. As Found conditions must be recorded. Copyright ex 2009 Copyright ex 10

11 Owner/Operator Field Failure Studies Data from an owner/operator repair shop is entered into maintenance management system (MMS) where equipment failure rates are automatically analyzed. The MMS report showed that the failure rates were much lower than expected. An instrument engineer visited the test shop. It was discovered that failure reports were entered into the MMS only when a device was sent out for repair. That was how the repair purchase order was generated. All units repaired internally were not recorded as no purchase order was needed. The data in the MMS was very optimistic. The proof test interval was not extended. The test, repair and data collection process must be understood to analyze data. As Found conditions for all failures and tests must be recorded. Copyright ex 2009 Copyright ex 11

12 Owner/Operator Field Failure Studies A detailed study of a valve showed a low failure rate when compared with the ex Predictive Analytics benchmark. A conclusion was that this valve would be an excellent choice for safety applications. The valve was designed and manufactured for control applications. Although not stated in the analysis report, it may be possible that the data was gathered for the dynamic operating conditions of a control valve. The failure rates for the same product in a static application would be much larger as many other failure modes exist in static applications. Application conditions must be recorded and listed in any analysis report. Copyright ex 2009 Copyright ex 12

13 Proof Test Data Collection The repair shop is not the only source of failure data. Proof test records may be the only good source of data for good as found indication of fail-danger mode failures. 2 Proof test records must be carefully analyzed with an understanding of the application and the proof test procedures. What is the PROOF TEST COVERAGE FACTOR? Copyright ex Copyright ex

14 Manufacturer Field Return Studies Manufacturer Field Return Data Studies Calculation methods vary widely- how many operational hours? Cannot know what percentage of actual failures are returned Different definitions of FAILURE (Not a problem scenario) Many manufacturers classify returned items as a failure only if a manufacturing defect is found. Many returned items are marked no problem found. I have seen calculations where operational hours are estimated based on shipping records and it is assumed that all failures are returned. However, root cause analysis is typically done providing valuable information. Copyright ex 2009 Copyright ex 14

15 Manufacturer Field Return Studies Manufacturer Field Return Data Studies Calculation methods vary widely Cannot know what percentage of actual failures are returned Different definitions of FAILURE (Not a problem scenario) Use warranty period only: Count failures during warranty period. Count operational hours only during warranty period. Assume a percentage of units returned: 10%, 50%, 70% depending on cost of device Lower Bound Lambda Some products have a built-in power on hours counter. For those products use the TTF number given by the product to calculate upper bound on failure rate Upper Bound Lambda. Copyright ex 2009 Copyright ex 15

16 Failure Modes, Effects, & Diagnostics Analysis (FMEDA) Concept CONCEPT: Break problem to smaller pieces Perform a detailed study of each component and how the component failure will affect the instrument failure Copyright ex Copyright ex

17 Failure Modes, Effects and Diagnostics Analysis (FMEDA) Extension of FMEA Technique developed in the early 1990s by ex engineers Adds quantitative failure rate column Add diagnostic and diagnostic coverage (DC) columns Documents how each component failure will impact operation of the system in terms of system failure mode Results in a failure rate for each failure mode of the system Copyright ex Copyright ex

18 FMEDA COMPONENT DATABASE Product λ Component λ s FMEDA Product Failure Modes Failure Mode Distribution Diagnostic Coverage Using a component database, failure rates and failure modes for a product (transmitter, I/O module, solenoid, actuator, valve) can be determined far more accurately than with only manufacturer s field failure data Copyright ex Copyright ex

19 Fault Injection Testing Simulate component failures and test that diagnostics perform as expected. F.I.T. test suites determined from FMEDA to test each diagnostic and a sample of other failures. Copyright ex Copyright ex

20 FMEDA Biggest Negative COMPONENT DATABASE Component λ s Failure Mode Distribution FMEDA Product λ Product Failure Modes Diagnostic Coverage The accuracy of the FMEDA depends of the accuracy of the component database. It must include failure rates and failure mode distributions of each component as a function of operating profile. The useful life of each component should also be listed as a function of operating profile. Copyright ex Copyright ex

21 Therefore the component database must be based on and calibrated by FIELD FAILURE DATA. Detail Design Information Components Used Stress Factors Application Environment Used In Field Failure Data Calibrates Component Failure Database Used In FMEDA Analysis Root Cause Analysis Copyright ex 2009 Explain Differences Adjust as needed Compare Product Level FMEDA Results Failure Rates Failure Modes Useful Life 21

22 FMEDA Based Failure Model Whenever an FMEDA is done, the results are compared with field failure data. The comparison often shows new failure modes of components, new failure rates due primarily to operational profile. Field Failure Data Product λ Compare FMEDA Product λ ELEC./MECH. COMPONENT DATABASE Industry Database Significant Difference? YES Update Component Database NO Copyright ex 2009 Finish Copyright ex 22

23 Ten Billion Unit Operating Hours After over one hundred field failure studies of both end user and manufacturer data sets that represent more than ten billion unit operating hours, ex has: Updated the component failure database to constantly improve the model. Identified and updated the model when differences between the model and the results are explained Field Failure Data Product λ Compare FMEDA Product λ ELEC./MECH. COMPONENT DATABASE Industry Database Significant Difference? YES Update Component Database NO Copyright ex 2009 Finish Copyright ex 23

24 FMEDA Based Failure Model A predictive failure rate / failure mode model for some components can be constructed from a hierarchical set of FMEDAs. The component database is the repository of the data. Product λ Product Failure Modes Diagnostic Coverage FMEDA Environment ELEC./MECH. COMPONENT DATABASE Component FMEDA Copyright ex 2009 Copyright ex 24

25 FMEDA Based Failure Model FMEDA Examples of component FMEDA include: Failure Relays Modes Microprocessors MEDA Product λ Product Diagnostic Coverage Environment Microprocessor designed for functional safety: Duplicate memory w/comparison Dual CPU with diverse substrate Automatic self-diagnostics on Registers, Instruction decoder, Cache, I/O, etc. ldu < 2E-9 failures per hour ELEC./MECH. COMPONENT DATABASE Component FMEDA Copyright ex 2009 Copyright ex 25

26 Comparing FMEDAs Some FMEA results based on manufacturer warranty return data analysis has produced very low failure rate numbers. Source Product Type Ball Valve, Close to trip, ex FMEDA [exid09b] Full Stroke Ball Valve, Close to trip, ex Manufacturer s FMEDA [exid09b] Tight Shutoff FMEA/Manufacturer Data (TUVRh10)Ball Valve Warranty Data Actuator, Rack and Pinion ex FMEDA [exid09a] Spring Return Actuator, Rack and Pinion ex Manufacturer s FMEDA [exid09a] Double Acting FMEA/Manufacturer Data (TUVRh11)Actuator, Rack and Pinion Warranty Data Lambda D - failures per hour 4.83E-07 Comment 1.35E-06 Highest Numb 6.55E-08 Lowest Numb 4.29E E-07 Highest Numb 8.54E-09 Lowest Numb Copyright ex 2009 Copyright ex 26

27 Comparing FMEDA and Field Failure Results Failure Rate DU Failure Source Product Type per hour Rate Comment Refinery Data [Shel00] Analog Pressure Transducer 2.71E-06 Refinery Data [Shel00] Microprocessor Based Pressure Transm 7.19E-06 DOW Plant Study [Skwe08] Pressure Transmitter 4.96E-07 OLF-070 Pressure Transmitter 3.00E-07 FMEDA Analog 1151 Analog Pressure Transducer 3.53E E-07 No Remote Seals FMEDA Analog 1152 Analog Pressure Transducer 8.13E E-07 No Remote Seals FMEDA Microprocessor 1151 Microprocessor Based Pressure Transm 5.64E E-07 No Remote Seals FMEDA Microprocessor 3051 Microprocessor Based Pressure Transm 5.43E E-08 No Remote Seals FMEDA Safety Certified Pressure Transmitter 5.36E E-08 No Remote Seals FMEDA Safety EJX Certified Pressure Transmitter 5.01E E-08 No Remote Seals FMEDA 3051 w Remote Seal Microprocessor Based Pressure Transm 7.04E E-07 Include Remote S Field Failure Data ranged from 4.96E-07 to 7.19E-06 including impulse lines, remote seals? No Info available. FMEDA Data ranged from 3.53E-7 to 8.13E-7 Copyright ex

28 l Total Failure Rate e Comparing FMEDA and Field Failure Results NAMUR Study, 2009: Dupont, Daniel, "Merging Bottom-up and Top-down Availability for realistic Analysis of Safetyrelated Loops, ISBN , Shaker Verlag GmbH, Aachen, Final Element Series Sensors Logic Solver FMEDA NAMUR FMEDA NAMUR FMEDA NAMUR Copyright ex

29 FMEDA Results Do Not Include Maintenance Induced Failures: These are site specific and not product specific. However as they are real, the exsilentia tool has a Maintenance Capability parameter that adjusts probability of successful repair, probability of failures, etc. Systematic Failures- Failures due to design or procedure: Systematic failures should be strongly reduced by careful root cause analysis of failures with corrective action. However, the Maintenance Capability parameter of exsilentia will adjust for site specific problems. Wearout Failures: Instead a useful life parameter is published. Copyright ex

30 FMEDA Results Environmental Specific Operational Parameters are specified. Control Room Cabinet Field Equipment - 2 wire Field Equipment 4 wire Automotive Subsea Copyright ex

31 Predictive Analytic Analysis USE DESIGN KNOWLEDGE By developing a detail understanding of dozens of different designs for a particular product, ranges of expected failure rates, ratios of failure rates and expected product level failure modes are known. FMEDA 1 Results Failure Rates Failure Modes Useful Life 28 FMEDA n Results Failure Rates Failure Modes Useful Life Predictive Analytic Model Failure Rate Range Failure Modes Useful Life Range 28 Field Failure Data Set Statistical Signature Analysis Compare to Benchmark Find Source of Differences Copyright ex 2009 Copyright ex 31

32 Predictive Analytic Analysis EVALUATE FIELD DATA QUALITY By comparing field data analysis results with the Predictive Analytic Benchmark, data quality can be judged and issues in the data collection process can investigated. FMEDA 1 Results Failure Rates Failure Modes Useful Life 28 FMEDA n Results Failure Rates Failure Modes Useful Life Predictive Analytic Model Failure Rate Range Failure Modes Useful Life Range 28 Field Failure Data Set Statistical Signature Analysis Compare to Benchmark Find Source of Differences Copyright ex 2009 Copyright ex 32

33 Future End User companies are beginning to see value in keeping good records of failure and proof test events. If a good product database is combined with a good data event capture tool, future field failure data will improve. This will be used to improve accuracy of the Predictive Analytic Models and FMEDA results. We do have work to do. SILStat Copyright ex

34 Questions? Comments? More Information: 1. Free Wednesday Web Seminars see 2. White Papers 3. Safety Automation Equipment List 4. Books: e Copyright ex

35 Field Data Collection Tool SILStat Failure Database creation and maintenance Import data from other exsilentia applications including the SILver calculation tool Event Recorder available for ipad/ipad2, Windows Laptop, Windows Desktop Built-in failure taxonomies Pick list simplified selection Data Analysis service using Predictive Analytics Data output reports compatible with PERD SILStat Copyright ex Copyright ex 2011