CMS Pixel Detector Performance, Operations,Calibrations & Software Split, 8/10/2012 Content: 1) Introduction 2) Performance 3) Operations 4) Problems (tomorrow) 5) Pixel DAQ hardware 6) Calibrations 7) Software 1
Operations 2
Fraction of detector alive Operations 3
Barrel pixel occupancy map Operations Layer 1 Layer 2 OL D Mod ul es Layer 3 10/9/12 G. Bolla, CMS PX workshop, Grindelwald 4 4
Forward pixel occupancy maps Operations S l ow c h annel s B r ok en l as er d r i v er 5 5
Not Everything Is Perfect Operations An overlay of all modules (CMS) showing the bad bump-bonds. 7300 bad pixels in 48M 1.5E-4 (CMS) Negligible inefficiency. Lay1 Lay2 Number of masked pixels 600 -> 1.2E-5 CMS Lay3 6
Operations Radiation Effects At the accumulated luminosity of 20 1/fb the total charge particle fluence until now is about 50 *1012 particles/cm2. This is still low when compared to the total expected fluences of about 5*1014 particle/cm2. 1) Sensor damage increased leakage currents Seen from the beginning (see next slide). 2) Sensor damage partial depletion, charge trapping, need to increase the bias voltage. Not yet observed. But we clearly see the full depletion voltage change. 3) Single Event Upsets (SEUs) change of state 0<->1 in flip-flops due to the passage of a heavy ionizing particle. First observations at instant luminosities of 1033 (2.5*106 particles/cm2/sec). The effect manifests itself by a sudden change in some readout parameter. Detector reconfiguration brings it back to normal. 4) ROC (Readout chip) damage change of the internal voltages due to irradiation. 7
Operations Leakage current increase with radiation dose 8
Bias scans Operations Bias voltage scan -> find the full depletion voltage Full efficiency -> Full depletion 9
Bias scans Operations Layer 1 bias scans versus time (radiation dose) Full depletion voltage versus time 10
Single Event Upsets (SEUs) Operations From Wikipedia Single event upsets were first described during above ground nuclear testing, from 1954 to 1957, when many anomalies were observed in electronic monitoring equipment. Further problems were observed in space electronics during the 1960s, although it was difficult to separate soft-fails from other forms of interference. In 1978, the first evidence of soft errors from alpha particles in packaging materials was described by Timothy C. May and M.H. Woods. In 1979, James Ziegler of IBM, joined with W. Lanford of Yale, first described the mechanism whereby a sea level cosmic ray could cause a single event upset in electronics. In practice, in our circuits, an SEU can change the state of a flip-flop from one state to another 1 <-> 0, usually with bad consequences. The effect is very circuit specific (design, layout, voltage,...). For out read chips it was measured in test beams to be about 3 * 10-16 (cm2/ sec/transistor). 11
Single Event Upsets (SEUs) Operations Number of readout chips which do not work. It lowers efficiency but does disturb data taking. Much worse are SEUs in TBMs (readout control chips) -> CMS DAQ stops. time reprogram Occupancy of selected readout chips SEUs can be only fixed by reprogramming. In CMS a complex system is used to detect SEUs and react to them by reprogramming all front-end electronics time reprogram 12
Radiation effects - Voltage Drifts CMS pixel ROCs use two low voltage power supplies: Digital adjusted by Vd, the current is called Idigi, changes with luminosity Analog adjusted with Vana, the current is called Iana, was expected to be constant Analog current versus radiation dose What is wrong with Iana drift? Power supply limit! Threshold changed! 13
Pixel Thresholds changes with radiation dose 2012 Thresholds drift up with time (radiation) 2011 Thresholds increase with radiation. This is bad -> position resolution. Needs regular recalibration. Done by Jelena Luetic. 14
Operations BPix Threshold after Vana and Vcthr change Run 203391 Threshold status after the last recalibration: 40 vcal -> 2200elec. black L1, red L2, blue - L3 15
Problems 16
Problems Dynamical Data Losses The pixel readout chip (ROC) has finite data buffer sizes. These buffers can sometimes overflow. The ROC design was optimized to minimize such effects, but they cannot be avoided completely. Buffer overflows lead lost pixel hits -> loss of efficiency. We call the dynamical because a pixel can be dead in one event but will be fine in the next. The magnitude of the effect depends on the data rate (LHC luminosity and the distance from the collision point) and on the CMS trigger rate. In out pixel detector it was estimated to be 3-4% for layer 1 pixels at full LHC luminosity (1034) and full 100 khz trigger rate. 17
Problems Dynamical Data Losses Example from 2011 data Pixel efficiency in the barrel layers versus data rate (LHC luminosity). 18
LHC beam-gas events example event in pixel layer 1 Problems Event display of a background event (PKAM) PKAM = Previously Known As Monsters phi zoomed z Streams (showers) of particles coming along the z axis. Very phi asymmetric for each event but symmetric if integrated over many events. 19
Background Origin Problems Consistent with beam background. Not halo muons but rather beam-gas interactions. phi Later confirmed by LHC simulations. Such events create very large events, many pixels in single readout channels. zz They are much larger than planned collision events, and take very long time to read. particle All other channels have to wait for the bad one! 20
Problems More Details About the Pixel Readout Readout sequence Channel 1 normal case READOUT Trig Chanel 2 with an ultra PKAM READOUT READOUT Trig READOUT of a PKAM event timeouts Trig X READOUT Time -> X READOUT Many lost readouts. No gap! Channel cannot resynchronize itself FED channel cannot resynchronize itself, an external action is needed - a resync. 21
Pixel DAQ Hardware 22
Pixel Readout & Control Clock,T1 ROCs detector (UXC) control room (USC) I2C PI2C Data Pix FED Pix FEC Clock & L1A & Ctrl CAEN Interface Pixel PCs VME Clock & T1 Reset, Cal, Sync, BCO,... S-link Ready, Busy, Error, OutOfSync, Warning TTCci CMS fast signals Pix fast signals TTS Pixel readout & control scheme 11/04 TBM + HUB 23
Pixel Front End Pixel Readout & Control, d.k. 11/03 ROC ROC ROC ROC MODULE/ PANEL L1A Token In High Density Interconnect (HDI) CLK TBM PI2C Token Out Analog Out PI2C HUB A single chip CLK PI2C CLK L1A Analog Out READOUT CONTROL 24
Pixel Control Hardware PSI2C CLK L1A BARREL ENDFLANG E LCDS drivers CLK L1A(trig, cal, sync, reset) Gate-Keeper Slow I2C SERVIC Hard E Reset TUBE CCU LVDSmux PI2C PLL FEC Slow I2C mdoh CLK PI2C HV Slow I2C CLK DATA ALT Slow I2C Delay25 DOH USC5 5 Analog out Slow I2C AOH Analog Out CLK PixFEC LV*2 Power Supplies PixFED 25
Important construction steps: the supply tubes T Many electrical components are not directly on the pixel modules but in the so called supply tube. which also brings all the electric power and cooling. 26
Read-out Chain Analog Out of Barrel Module, no Hits TBM Header Event Counter Front-End Driver Read-out Unit Builder Unit TBM Trailer Status and Error Flags ALT Analog Level Translator Amplification of Analog Out Signal AO H Analog Optical Hybrid Conversion of electrical to optical Signals Front-End Read-out Link Event Builder 16 ROC Header Filter Unit High Level Trigger 27
What can we do to fix the problem Our readout electronics (FEDs) We cannot modify the detector. So the fix has to be done in the readout electronics. Find a more robust readout algorithm in the FEDs -> change the FPGAs firmware. 28
Pixel control electronics Control is also done through VME boards (9U). VME: VME is an industrial crate/board standard established sometime in the early 1980-ties. It became popular in physics (particle & nuclear) during the LEP area. Replaced the older CAMAC and Fastbus (HEP specific) standards. At some point used much in the industry for control & communication, usually the 3Uand 6U version. Was very related to the Motorola 68k processors. For example older SUN workstation used it as the main communication bus. More recently it has been replaced by utca bus standard (in telecommunication). 29
Pixel - FED 9U VME board ADC card s VME interface FPGAs Optical receivers (only 1 shown here) TTC SLink output Designed and built at the HEPHY Institute Vienna (M.Pernicka, H.Steininger). 36 channels per board. 48 fi nal modules built, 40 needed in P5. 30
Detector Control System - Architecture www ssh Tunnel Configuration CMS Technical Network PSX Conditions 31
Calibrations 32
Pixel Calibration The pixel detector has many parameter to program. How many? 1) Trim & mask bits per each pixel - ~ 62M parameters 2) ROC & TBM registers 15k * 28 + 960 * 10 = 0.43M parameters 3) Slow I2C devices 64 * (60 + 24) = 5.4k parameters 4) FED parameters 110k parameters The value of all these parameters have to be determined -> calibrations. 33
Introduction to Pixel ThresholdsCalibrations comparator threshold We measure thresholds using S-Curves. An S-Curve is the hit efficiency as a function of injected charge (VCal). This procedure has to be applied for all pixels (66M). 34 34
More complex example - Threshold Measure pixel thresholds by charge injection using the SCurve method. Scan the injected signal amplitude (VCAL) and measure efficiency. Efficiency Absolute threshold measured in 2 clocks (2 WBCs) In-time threshold measured in 1 clock (1 WBC) VCAL 35
Threshold calibration Usually one iterates: - adjust VcThrComp to move the threshold closer to the desired value - measure it with an SCurve calibration Mean ROC threshold distribution from February 2012 black bpix red - fpix VCAL units (65 electrons) 36
Online Software The online CMS software framework is called XDAQ. It is a set of libraries which define communication protocols and state definitions (FSM). The main software components are called supervisors. The supervisor control is done through web interfaces (accessed from Firefox). For histograming & visualization ROOT is used. Offline Software The offline software is CMSSW, the CMS reconstruction framework. All reconstruction, simulation and offline calibrations are run within CMSW. 37
Pixel Control System PC11 Clock Orbit Pixel Supervisor Global trigger TTC PC-RCMS RunCtrl TTC PC-TTC Supervisor V M E L T C T T C ci T T C ex DCS PC-PSX PSX-Server PC PixFEC 15,16 Supervisor PC PixFED 12-14 Supervisor TTC V M E F E C F E C To the front end (Fast commands&pi2c) F E D F E D From the front ends (Data packets) TTS FMM VME S-Link out 38
Software A typical calibration loop: 1) If needed send SOAP to FECSupervisor, reconfigure front-ends 2) Send SOAP to TTCSupervisor, generate 1 trigger 3) Send SOAP to FEDSupervisor, readout the event Go back to (1). All calls are blocking, that is the response waits until the operation is done. The number of pixels calibrated in one step is limited by the size of the SpyFIFO3 in the FED 2k pixels. This results in the maximum number of 3 pixels per ROC. The limit is given by the FPix (21/24 ROCs per channel). FECs have no memory (only input&output FIFOs) so all information has to be always transferred from the PC (VME traffic!). Bigger FIFOs in the FED and memories in the FEC would help a lot. A single SOAP is about ~1 msec, therefore combining more activity into a single message would help. 39
Calibration Clock phase calibration Clock phase scan to optimize the efficiency 40
Other changes Lorenz angle Plot from Jelena Luetic 41
Software Overview Raw Data Raw2Digi data from Slink Digis hit pixels per module SimHits Position Estimator Clusterizer Clusters RecHits clusters of pixels 2D positions in local module coordinates track finding Digitizer simulated hits from Geant 4 42
Raw Data Software Pixel raw data is in the container: FEDRawDataCollection (the label is different for data11, data12, MC and lumi stream). Data format: 64 bit header 2 * 32 bits 2 pixel words (each word = 1 pixel)... 64 bit trailer 1 pixel : pulse height (8 bits) + pixel id (8 bits) + dcol (5 bits) + roc (5 bits) + fed channel (6 bits) The FED number is in the header. Very convenient, 1 pixel fits into a 32 bit word. Note that status and error codes are coded as non-existing rocs and channels. One can look at raw data with the following analyzer (program) /CMSSW/EventFilter/SiPixelRawToDigi/test/SiPixelRawDumper.cc 43
DIGIS Software A list of hit pixels grouped according to the module number (for fpix a plaquette). Each pixel (PixelDigi) is an unsigned integer (32 bits): row (8 bits), col (9 bits), adc (11 bits), time (4 bits). Row & column are indexes within a module, adc is uncalibrated, time only for MC. Stored in the container which is a special vector (DetSetVector) of PixelDigi. The usual name is something like: PixelDigiedmDetSetVector_siPixelDigi RECO Digis are not persistent, they are usually not stored in RECO. There is a simple test program to inspect digis in: CMSSW/SimTracker/SiPixelDigitizer/test/PixelDigisTest.cc & testpxdigi_cfg.py 44
Digi-To-Clusters: Clusterizer Software List of clusters Hit pixels Clusters Digis Clusterizer Needs the gain calibration from DB Steps: 1) Convert ADC to electrons using the gain calibration ADC VCAL (per pixel) and the common VCAL calibration (e = 65.5 * vcallow 414). In reconstruction we presently use a linear calibration. For offline offsets are used per pixel and gains are averaged over ROC columns. In HLT the offsets are also averaged over columns to reduce the DB payload. ADC Electrons 45
Digi-To-Clusters: Clusterizer Software 2) Check if a pixel is above a threshold (usually 1000-2000 electrons). Start building a cluster: Any touching pixels form a cluster A missing pixel breaks a cluster into 2 separate clusters. Some attempt was done to use dead pixel information but was dropped. 3) Add charge of all pixels, check that the total cluster charge is above a threshold (usually around 4 kelec.). 4) Store all pixels which belong to a cluster. The software to do this is in: CMSSW/src/RecoLocalTracker/SiPixelClusterizer There are several input parameters: cluster threshold, pixel threshold, 46
CLUSTERS Software 47
Clusters-To-RecHits : Position Estimator Clusters RecHits Software (also called RecHits, RecHitConverter) Position Estimators Needs from DB: Lorentz angle Error estimates Templates Magnetic field RecHits are based on clusters but have a more precise position evaluation. The reason these two are split is that clusters are build once in RECO, rechits however can be recalculated many times depending on the needed & possible precision. The software to do that is in: CMSSW/RecoLocalTracker/SiPixelRecHits/ scripts to run it are in /test. 48
Software Clusters-To-RecHits : Position Estimator To estimate better the cluster position an estimate of the expected charge sharing width is needed. d q1 q2 p If d not known : position = pixel border + ( q2 * p/2 q2 * p/2) / (q1+q2) This can lead to large shifts if q2>>q1 (or q2<<q1). If d known: position = pixel border + ( q2 * d/2 = q1 * d/2) / (q1+q2) The shift is limited to the physical region. q1&q2 can vary a lot due to charge fluctuations ( landau fluctuations for thin layers). These fluctuations are the dominant effect in degrading position resolution. They act as 20-30% noise added to the analog signal. 49
Spare 50
Position of the cluster charge peak (Landau peak) Plot from Jelena Luetic 2012 51
Position of the cluster charge peak (Landau peak) Plots from Viktor Veszpremi & Janos Karancsi BPix 2011 FPix 2012 Looks like the worse is over 52
Gain calibration This calibration establishes the relation between ADC counts and the amplitude of the injected signal (VCAL). It is then parametrized by a linear function. 53
Analog Coded Pixel Addresses Good case (rms ~2.5) 54
Calibration - Examples Baseline calibration: Set the black (base signal) at 450 (middle of the ADC). The FED (ADC) has to see the full signal range. SDA-RDA calibration: Set the phases of the input and output clocks for the 40MHz I2C protocol. Without this step we cannot talk to ROCs & TBMs. 55
RecHits Position resolution insoftware Y from simulations with track angle without 56
Calibrations - A summary Calibration Name/Type Device Duration Frequency Upgraded-Detector --------------------------------------------------------------------------------------------------------------Programming clock phases Delay25 ~2 hours yearly YES ADC clock phase FED ~2 hours yearly NO? Signal Baseline FED 5 min weekly YES (or similar) AOHbias/gain AOH ~2 hours yearly YES ROC/TBM-UB ROC/TBM ~1 hour yearly NO CalDel ROC ~2 hours yearly YES AddressLevels FED ~1 hour monthly NO Threshold (VcThrComp) ROC SCurves (measure thresholds) Trimming ROC HoldDelay Analog signal range Gain calibration ROC ROC - ~1 hour 2-3 per year ~1 hour weekly a few hours yearly (fpix) YES YES YES ~2 hours ~2 hours ~3 hours yearly yearly monthly YES? YES a few hours a few hours yearly 3-4 per year YES YES Vana (presently does not exist) ROC Timing scan (LHC) Bias scan (LHC) Delay25 HV 57
Analog signal changes gain calibration This calibration establishes the relation between ADC counts and the amplitude of the injected signal (VCAL). It is then parametrized by a linear function. Slope Offset 58