Improving EPICS IOC Application (EPICS user experience)

Improving EPICS IOC Application (EPICS user experience) Shantha Condamoor Instrumentation and Controls Division 1

to overcome some Software Design limitations A specific use case will be taken as an example and discussed. Usefulness of Benchmarking EPICS IOC applications. Illustrate how the Software Design limitation on SLAC's LCLS LLRF PAD EPICS IOC software was overcome by adopting a completely new Design strategy, how it was successfully implemented and released in production. Discussion on how perceived hardware limitations can be overcome with actual test data gathering to upgrade live systems. 2

The IOC under scrutiny Phase and Amplitude Detector (PAD) is used by Low Level RF Systems for digitizing RF signals at SLAC s LCLS. The PAD CPU is a 64 MHz coldfire processor (PowerPC) running on RTEMS and EPICS framework. The initial LLRF SW design (4 years back) of the IOC took into account the LCLS commissioning beam rate of 30 Hz. Thus the LLRF ioc, EPICS database and device support were all designed with a trigger rate of 30 Hz in mind. The same PAD board was also used in a different beam diagnostics application the BPMs but with a very different software design. 3

Who is to be blamed? Down the road, as SLAC s LCLS became successful, the push for higher trigger rate necessitated that the LLRF PAD IOCs keep up with a beam rate of 120 Hz. Several LLRF PADs had trouble keeping up with this trigger rate and sampling sizes had to be reduced drastically. It was blamed as always on the limitations of the hardware specifically the cpu and the bus speed. Was it really? 4

Take stock of the situation Analysis of the PAD IOC software and its design found that: Existing PAD IOC acquires data, digitizes, processes and finally ships results over CA or udpcomm (a custom protocol for fast data transfers) to a Master VME IOC for Phase/Amplitude and feedback controls. Coldfire CPU loads max out with increases in CA clients. PVs start disconnecting when window sample sizes are close to 512 for all four channels of the PADs at high trigger rates. Snapshots of all running PAD statistics during a typical Operations day showed average loads above 80% for several PADs even with total sample size (offset + ROI window) much less than 2048 words (16-bits). 120 Hz BSA (beam synchronous acquisitions) had issues (even for scalar values). Dynamic increases of ADC sample window size and offset adjustments at high trigger rates can bring down a PAD operators are not aware of the limits for each PAD. HOPR and LOPR and other such protections for Phase/Amplitude and other critical parameters missing / not implemented for several records resulting in unexpected, undesirable behavior. 5

Can LLRF PAD software be improved? PAD hardware design provides good SNR. 16-bit ADC-based PAD specifically designed for LCLS RF signals. Functional or operational limitations are imposed due to software implementation only. With the same hardware design but efficient software design (by T.Straumann), PAD BPM software could handle high trigger rates up to 360 Hz as validated by the BPM PADs. Time between ISR and DAQ Task processing start (readout of FIFOs) much less than 1 ms as observed on the scope for 512 samples (128 * 4). So theoretically, should be able to handle 120 Hz (8.3 ms) trigger rates easily. Limitations of Coldfire processing power can be overcome by offloading most of the processing to the VME Master IOC. Keep the PAD software simplistic and ship all ADC data over the dedicated Second Ethernet Port. 6

Benchmarking efforts Coldfire Hardware timers used to benchmark different sections of the existing code. Snapshots of all the PADs statistics running in LCLS production during a nominal operations day. Most PADs showed > 80% CPU loads teetering towards 100% with a few increase in CA clients. It is easy to push a PAD over the brink @ 120 Hz. Benchmark BPM PAD FIFO read out times using scope. Readout times were < 1 ms at 100 Hz. 7

LLRF PAD Trigger and Timing When a trigger occurs, the PAD immediately ships the data packed in a special format to the VME IOC as UDP packets using a custom protocol. LLRF PAD Input signals are typically locked to the external triggers. LLRF PAD ADC Trig Beam Cycle ---- ----- ----- ----- (120Hz)...*... *... *...*... (120Hz) data, data data, data 8

VME IOC and PAD EIOC Processing Modules and drivers Channel Access Dedicated Ethernet for fast data transfers via UDP VME IOC Application Device Support Layer PAD EIOC Application drvllrfpad Driver Layer Device Support Layer drvllrf (generic helper library for LLRF PADs) udpcomm drvpadudpcomm drvpadadcstream drvpad (ADC, FIFO) 9

LLRF VME IOC Processing with new SW design Number crunching in a Real Time task Outcome of data processing deposited in 4 * 5 = 20 ai records (max): Phase (_PACT), Amplitude (_AACT), _IACT, _QACT and Power (_WACT) for each of the 4 ADC channels. RF Signals can be interlaced or non-interlaced or simple waveforms. Interlaced or non-interlaced Phase, Amplitude, I, Q and Power of down-mixed signals or Actual Beam Voltage (_SACT) for simple channels. SCAN I/O Intr, HIGH PRIO This is the Head of the processing chain. One lockset for each ADC channel consists of (_PACT,_AACT,_IACT,_QACT,_WACT) or (_SACT) for a total of 4 locksets with five (or 1 record) for each ADC channel. Processed 4 times per trigger @ 120 Hz. Last record in each lockset fans out to BSA record processing. 4 waveform records : _RAW_WF for each ADC Channel s waveform Each Raw Waveform in its own lockset not linked to any other record SCAN I/O Intr set to HIGH to end up in high-priority scan list in the udpcomm protocol. TSE = -2 to timestamp the waveform for BSA 4 * 5 = 20 ai records (max): (_S_PA, _S_AA, _S_IACT, _S_QACT, _S_WA) Slowly scanned waveforms for each of the HIGH PRIO ai records for display purposes. First record in lockset s SCAN 1 second, LOW PRIO All other linked records in lockset SCAN Passive. 4 slowly scanned waveform records : _S_R_WF for each ADC Channel s waveform SCAN 1 second, LOW PRIO for display purposes. Several I/O parameters of various record types: Input parameters for each ADC channel: ROI Window Offset (_OFST), Sampling Points (_SIZE), Phase Offset (_POC), Voltage Scale Factor (_AVC), Power Scale Factor (_APC) Output: PAD Stream Switch to start the streams from PAD (ACCESS) 10

VME IOC Callbacks while (1) { wait for streamed PAD ADC data -1 udp packet = 1 ADC CH data perform basic validity checks call user 'cook' callback on the waveform; user processes data callbacks in drvllrf /* cook' callback puts the processed data in llrf data structures or arrays where a lightweight, fast record (Phas, Ampl etc. ai records) can pick it up. Data goes to BSA processing from here @ beam rate */ if 'cook' callback is successful then post waveform to message buffer area (High Priority) depending on 'cook' return code Use the public function drvpadudpcommpostreply() to let cook post the same reply to a second slot on the message board for slowly scanned waveforms. } 11

LLRF PAD-specific signal types Each LLRF PAD ADC channel can process any one of 3 input signal types. Each of the following three functions handles these 3 types. Old Functions were processed by ColdFire PAD in old design. Now, equivalent new functions process them in VME IOC function to choose determined by unique combination of PADSLOT and Channel Number. Interlaced signals padgenreadsignal() ->drvpadcalcphasampl() Input signal comes in at 4 * 25.5 MHz clock rate. Interlaced I/Q samples. Data in quadruples (I0,Q1,I2,Q3). Multiply sample by appropriate scale factor. Calculate avgi, avgq, Ampl, Phas, Power. Not implemented for Klystron L1S Diagnostics PAD Non-Interlaced signals padifqreadsignal() -> drvpadnoninterlaced() Input signal does not come in at 4 * 25.5 MHz rate (eg. 119 MHz Ref) I/Q for each point in WF. Samples in multiples of 6. Multiply sample by appropriate scale factor. Calculate avgi, avgq, Ampl, Phas, Power. Simple signals padsimpreadsignal() ->drvpadcalcsimpavg() Input signals do not need I/Q processing. Data can be any length (max 512) generally multiples of 4. Multiply sample by a constant. 12

LLRF VME IOC and PAD EIOC Data Flow LLRF PAD EIOC Data Flow VME IOC Data Flow init_record() Start the padthread() task Start padudplistener() task Setup and enable Trigger Interrupt Start or Stop udpcomm callbacks() PAD Stream Start/Stop Command for all 4 channels Setup DrvPadUdpcommPrefs() Setup drvpadudpcommcallbacks() padinit() switches all PADs to singlechannel mode padthread() Wait for event flag set in isr() For channel=3,2,1,0: If streaming data can be engaged { getdata() udpcomm callback reads in PAD FIFO data in RowMajor format for each channel. padstreamsend() sends PAD stream data of 512 words for each channel in each UDP packet.} else { Engage ordinary record support } PAD Stream data 1 UDP packet per ADC channel @ 120 Hz rate padcook() Wait for one UDP packet from udpcomm callback. For channel=3,2,1,0: If datakind == 2 { drvpadcalcsimavg() drvpostsimpavg() /* waveform */} else { drvpadcalcphasampl() drvpostphasampl() /* waveform */} 13

LCLS Production SW Release: CH0: PAC Out CH1: Klystron Drive CH2: Klystron Beam Voltage Ch3: Klystron Forward EVR trigger @ 120 Hz supplied to PAD 512 samples per ADC channel * 4 = 2048 words per trigger. Phas /Ampl, Iavg/Qavg for all 4 channels time-stamped at 120 Hz rate. Waveforms delivery tested at 120 Hz. (display waveforms at 1 Hz). 14

LLRF VME IOC and PAD EIOC CPU Loads 15

Release in production Software Design completed based on the new generation of udpcomm. Commissioning of new hardware and release of Initial version of IOCs in production completed successfully within 3 months. Verified that this new SW design supports 120 Hz delivery of calculated I,Q,Phase,Amplitude, Voltage, Power etc. Additionally, it can deliver 512 sample waveforms for 4 channels at 120 Hz BSA rate. Benchmarking done after release showed that PAD load is around 30% irrespective of the number of CA clients connected to it. VME IOC load less than 5% Migrated all existing PVs to new VME IOC and PAD IOC. Worked with Operations to announce new design/implementation. 16

Acknowledgements T.Straumann: for releasing new udpcomm SW, code and design reviews R.Akre: for providing RF lab facilities and test setup E.Williams: for answering several EPICS questions D.Rogind: for supporting the project All SLAC physicists who gave positive feedback about the improvements. SLAC operators and all co-workers who gave support to this little project from start to finish. 17

Thank You! Questions 18