Preparations for Installation, Testing and Commissioning based on Experience at CERN, SNS and Siemens Eugène Tanke FRIB / MSU ESS Seminar, Lund, 6 March 2013
Outline Project Goal for the Accelerator Path towards the Goal Technical obstructions on the Path and some Remedies pertaining to installation, testing and commissioning Summary and conclusion 2
Project Goal and typical path towards it For the accelerator the Project Goal translates as: Deliver a (fully) integrated and (fully) functional accelerator on time and within budget Reach specified beam performance at the output Given the design, reaching the goal will require Scheduling each of the installation, testing and commissioning phases and doing the actual installation, testing and commissioning and tracking that scope is implemented on schedule and within budget and success is guaranteed... 3
Questionable Path towards the Goal 4
Preparations for Installation Receiving, Assembly, Testing and Storage area is crucial SNS had RATS facility of more than 5500 m2 Installation will be further aided by Installing tested equipment (FAT*) Service Building: Detailed rack layout and rack installation plan Tunnel: Lattice file, containing names and locations in global coordinates of beam line devices Good estimate and coordination of work involved SNS "Field Coordination" 119 FTEy *Factory Acceptance Test 5
Obstructions on the Path towards the Goal (1) Ultimately during beam commissioning (if not sooner) lack of performance may show up Examples of root causes for lack of performance are poor documentation, poor design, poor quality materials, lack of integration, machining errors, cabling errors, software errors, alignment errors, inadequate testing etc Such deficiencies are not uncommon (most if not all labs have their examples) and... can be very costly and time consuming to fix 6
Obstructions on the Path towards the Goal Design phase: LHC magnets from FNAL* (2) * http://fnal.gov/directorate/oqbp/index/oqbp_misc/final_lhc_root_cause_analysis_report_rev2_19sep07.pdf 7
Obstructions on the Path towards the Goal Construction/assembly phase: SNS DTL (3) Shortly before mounting drift tubes in the first SNS DTL tank (#3), vacuum leaks were detected on the drift tubes Leaks were traced to e-beam weld (water/vacuum) E-beam was deflected due to insufficient shunting of the PMQ field, missing the intended joint E-beam penetrated the Cu body and damaged the PMQs 8
Obstructions on the Path towards the Goal Testing phase: CERN RFQ2a (4) Performance of the CERN RFQ2a was hampered by oil pollution from a roughing pump due to an improperly functioning vacuum interlock system An additional RFQ (RFQ2b) had to be built CERN RFQ2b (200 ma p) with pulsed ion source on the right and linac2 on the left 9
Can one reduce the risk of such Obstructions? Proper testing and Q/A procedures combined with full integration will reduce or even eliminate such risk But what exactly should one test and how? In industry the so-called V-model is widely used (see next slide) How can one improve systems integration? 10
V-model as defined in industry DVM=Design Verification Method DVP=Design Verification Plan 11
V-model applied in an accelerator: Requirements Definition Define and document System Requirements, and the Sub-System, Device requirements and specifications driven by these, e.g. Magnet requirements (45 deg dipole, 5 cm aperture 1T, 1 m, db<0.1%) drive specifications (e.g. RT, n windings of diameter x, 100 A, yoke shape etc) Avoid requirements/specifications that cannot be tested Do not list the same requirements in multiple documents Once device/system built, test against these requirements 12
Now we may have a handle on this problem 13
V-model applied in an accelerator: Requirements Verification Define and document test plans Define appropriate and comprehensive tests e.g. Leak test a water cooling system with water or with helium? Verify in/out control of diagnostics Test against requirements/specifications and document results Use a Design Verification Matrix (DVM) to track which requirements have been verified 14
V-model applied in an accelerator: Siemens Particle Therapy systems Layout for Kiel facility (now dismantled) Raster scan Synchrotron and Linac 15
Test at vendor where appropriate Test equipment before installation and where appropriate, have as much tested as possible at the vendor (FAT) Avoids time loss due to shipping equipment back Witness construction and tests at vendor Due to schedule pressure, one may be tempted to skimp on testing because after installation and/or during beam commissioning proper functioning of equipment will be confirmed, but... 16
Consequences of insufficient / delayed testing (as apposed to thorough / early) If proper functioning of equipment is not confirmed after installation and/or during commissioning, one may suffer substantial delays Commissioning is typically on the critical path increased costs Example: how much does it cost to run the cryoplant during 4 hours of debugging and repair of a magnet problem? (additional) rework "standing army" 17
Can one reduce the risk of such Obstructions (to reaching the Goal)? Proper testing and Q/A procedures combined with full integration will reduce or even eliminate such risk But what exactly should one test and how? In industry the so-called V-model is widely used How can one improve systems integration? Preparation: Set up a framework for integration, that allows for tracking of requirements. Industry uses tools such as DOORS or Caliber to assist in the DVM function Preparation: Sign off on interface definition by system owners on both sides of each interface during design Clearly define installation, testing and commissioning phases (see following slide) 18
Typical installation, testing and commissioning sequence Given the accelerator design, reaching this sequence will consist of the following phases Building/buying accelerator equipment Equipment testing prior to installation (FAT) Equipment installation (phase A) Equipment testing (stand alone) (phase B) Equipment integration or V test (phase C) Equipment testing with beam (phase pre-d) Equipment beam commissioning (phase D) Project is helped by clearly defining handover points from one phase to the next 19
Systems Integration on Site (V tests) "Vertical" Tests (Phase C) are end to end tests across multiple systems When planning these, take installation and commissioning into account To be performed after successful FATs/stand alone tests of all systems concerned Cables should also undergo stand alone testing (Phase B in preparation for C) The cables used to test a device in the FAT are usually not the same cables as installed on the facility Cables may be hooked up to the wrong device/terminals 20
Preparations for Beam Commissioning (1) The goal of Beam Commissioning (Phase D) is to achieve the required beam parameters at the output under required conditions In preparation for beam commissioning, beam diagnostics should be tested with a (pilot) beam (Phase pre-d) Testing should include a full V test Subsequently, the beam itself can be commissioned 21
Preparations for Beam Commissioning (2) Beam Commissioning is aided by a Beam Commissioning plan, delineating Commissioning team Commissioning sequences, e.g. Source to RFQ input, Source to DTL input Commissioning sequence details Detailed description of measurements to be made and their durations, backed up by beam dynamics simulations, including those away from the nominal setting 22
Preparations for Beam Commissioning (3) SNS example of commissioning sequence Example for SNS DTL Tank 1 TASK 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 Ion Source and RFQ Startup Beam transport to MEBT beam stop Beam transport through DTL 1 to Energy Degrader/Faraday Cup Beam transport through DTL to D-plate Beamstop Commission D-plate Diagnostics and MPS Perform Fault Studies RF System Checkout/Verification with Beam Establish Phase and Amplitude Set-point Establish Transverse Matching Conditions Establish Long-Pulse Operation Perform Aperture Scan Establish Nominal 38 ma beam conditions at DTL output Characterize DTL 1 nominal output beam Commission Chopper Systems Characterize DTL 1 chopped output beam High Duty Factor Measurements Other Measurements Duration (shifts) 4.5 10 6 7 17 4 4 23 14 8 7 13 16 15 9 18 Peak Current (ma) 20 20 5 20 20 10 20 20 20 20 5 38 38 38 38 38 Pulse Width (usec) 50 50 20 10 50 20 100 50 50 680 10 50 50 50 50 680 Rep Rate (Hz) 3 3 2 2 2 2 3 3 3 3 2 3 3 3 3 30 Beam Power (kw) 0.0075 0.0075 0.0005 0.003 0.015 0.003 0.045 0.0225 0.0225 0.306 0.00075 0.04275 0.04275 0.04275 0.04275 5.814 Beam stop MEBT MEBT ED/FC D-plate D-plate D-plate D-plate D-plate D-plate D-plate D-plate D-plate D-plate MEBT D-plate D-plate 23 Priority high high high high high high high high high high low high high med med high low
Preparations for Beam Commissioning (4) SNS example of commissioning form Have "sanity check" alternatives In this case, check RF tank amplitude on cavity 24
Commissioning is aided by support from beam dynamics tools Various tools: Off-line model(s) On-line model(s) Virtual accelerator RFQ (foto) and linac originally foreseen for SSC being used for production of medical isotopes Loss in beam transport to target was found Debugged with off-line code DYNAC 25
Use of multiple beam dynamics codes aids understanding of measured beam During the MEBT commissioning of CERN Linac3, measured and simulated beam emittances and Twiss parameters seemed to be similar, but... double checking with another emittance measurement gear and with another code (DYNAC*) demonstrated flaws in both the simulation and measurement *DYNAC source code available at: http://dynac.web.cern.ch/dynac/dynac.html 26
Success of Commissioning helped by well planned use of temporary diagnostics Apart from using "inline" diagnostics, use strategically placed temporary ones Use emittance scanner to measure beam at the input plane of the RFQ Explore other settings than the nominal one Could condition RFQ in parallel "parking" position Use bunch length, energy and energy dispersion measurements as well as an emittance scanner to characterize the beam at the input plane of the linac Compare measurement results from inline and temporary diagnostics with each other and with beam dynamics code(s) results 27
Temporary diagnostics used at the RFQ input and the IH DTL input at Siemens RFQ and IH placed in "parking position", allowing for simultaneous beam commissioning and RF conditioning Siemens AG 2011, Healthcare Sector, Particle Therapy. All rights reserved. 28
Examples of temporary Diagnostics Emittance scanner with fast Faraday cup and end cup at location of Siemens RFQ X and Y grids can be read out simultaneously for real time isometric profile plot "D-Plate" used behind SNS DTL tank 1 29
Managing Installation, Testing and Commissioning Optimize sequence duration by planning simultaneous installation, testing and commissioning (macroscopic scale) Base durations on bottom-up estimates from system/task owners Have on-site day to day planning (microscopic scale) Define clear handover from one phase to the next e.g. final alignment is part of installation include handover parameters as needed 30
SNS CCL during installation and... Photo courtesy of Oak Ridge National Laboratory 8 CCL segments linked by bridge couplers; awaiting installation of inter-segment quadrupoles 31
..simultaneous DTL1 Commissioning PPS Access Ion source & LEBT RFQ DTL tank 1 MEBT D-plate Towards remainder of DTL and CCL D-plate 32
Summary highlights Installation, testing and beam commissioning will profit from well defined scope and plans for each of these as well as durations (based on bottom-up estimates from system/task owners) well defined handover from one phase to the next Equipment/system tests are crucial to successful and timely project completion There are methods to reduce the risk and impact of "system failures" 33
Conclusion There are methods that will....successfully aid in reaching the end goal and will help avoid undesirable outcome(s) 34
Thank you for your attention! 35