The Readout Architecture of the ATLAS Pixel System

The Readout Architecture of the ATLAS Pixel System Roberto Beccherle / INFN - Genova E-mail: Roberto.Beccherle@ge.infn.it Copy of This Talk: http://www.ge.infn.it/atlas/electronics/home.html R. Beccherle - INFN / Genova

ATLAS Pixel System Main features: 1 sensor with 46,080 50x400 µm pixels; 16 analog Front End chips directly bump bonded to the detector; FE-Chip 1 Flex Hybrid kapton circuit glued on top of the sensor; MCC sensor FE-Chip 1 digital Module Controller Chip (MCC) wire bonded to the flex hybrid circuit; 2 VCSEL driver chips; 1 Pin diode receiver chip (DORIC). The chip amplifies the PIN diode signal and regenerates the 40 MHz Clock and Data/Cmd signals. Carmel,CA (USA) 10 Sep 2002 2

System Architecture 2880 FE chips 1 2 DORIC 1 PIN CTRL Sensor 16 chips MCC Opto Package 3 fibres/module Opto Card ~30 ROD ROB 2880 FE chips 1 1 1 VDC 1 1 VCSEL VCSEL TIM 4 fast + 4 slow Power supplies module 1 Sensor 16Front End chips (FE) 1 Module Controller Chip (MCC) 2 VCSEL Driver Chips (VDC) 1 PIN diode receiver(doric) control room Optical Receivers Readout Drivers (ROD) Readout Buffers (ROB) Timing Control (TIM) Slow Control, Supplies Carmel,CA (USA) 10 Sep 2002 3

FE Main Characteristics Analog FE: DC feedback preamplifier, differential second stage amplifier, differential discriminator, digital output to control logic. Digital Readout: Column pair based readout. 8 bit Grey coded 40 MHz differential Timestamp in each pixel, measure leading and trailing edge, local RAM, shared bus to transfer hits at 20 MHz, EoC buffers for storage during Lev1 latency (up to 6.4 µs). Read out sequencer (up to 16 pending events), Lev1 signal requests all associated hits that are sent out. Control Logic: 20 bit Command Register, 166 bit Global Register, Pixel shift register to access all configuration bits of single pixels (14 bit per pixel). Error handling: (EoC overflows, disable single pixels, ) IBM 0.25 µm rad-tolerant design die size is 7.2 x 10.8 mm 2 50 x 400 µm 2 pixel cells 2880 cells 18x160 colums = 9 column pairs Carmel,CA (USA) 10 Sep 2002 4

MCC Main Characteristics System startup and initialization. Decode data/command signals (from the ROD): A simple serial protocol is used for all communication between the ROD and the MCC and also between the MCC and the FE chips (Slow, Fast and Trigger commands). Trigger, Timing and Control: the MCC has to provide Triggers to all FE chips and keep event synchronization. Receive serial data from 16 FE chips, accumulate data in local FIFO's. Event building: complete module events are reconstructed with some data compression. Scoreboard mechanism allows to start event building as soon as all enabled FE chips finish sending the data of one complete event. Send event to DAQ (via VDC) Error handling: (FIFO overflows, misalignment of data from FE chips with BCID information, disable defective or noisy FE chips,...) IBM 0.25 µm rad-tolerant design Die size is 6.38 x 3.98 mm 2 16 FIFO s (128 21bit words) 1 analog delay line 650.000 transistors Carmel,CA (USA) 10 Sep 2002 5

ROD or TPLL + TPCC Optical components (VDC, DORIC and optical fibers) are still in a development stage and are therefore not jet fully integrated in the system. The system works properly and integration will occur soon. The Read Out Driver (ROD) for the whole system is a joint effort between the Pixel and SCT community and is still in an early prototyping stage. Most of the operational tests performed so far on the Pixel system are performed with an ad hoc hardware: Turbo Pixel Control Card (TPCC): Controls power distribution to the module, regeneration of clock and data signals, up to 4 modules can be operated with one card (data commands and clock sent to all modules, one multiplexed output line), support for temperature measurement. Turbo Pixel Low Level card (TPLL): VME interface, clock generation and synchronization, data FIFO, trigger FIFO, 16 MByte on board SRAM supports module level histogramming, FPGA for encoding/decoding the MCC serial data protocol, support for all 4 MCC output modes. This year full radiation characterization of the electronic was performed. Test beam operation were successfully performed this year for the first time within the DAQ-1 framework. Carmel,CA (USA) 10 Sep 2002 6

System Initialization and Configuration At power up the whole system has to be correctly initialized. FE chips and the MCC are connected by means of a star topology in which each FE has dedicated parallel connections with the MCC. Each FE has one 40 MHz LVDS data and clock line and 3 slower (5 MHz) common configuration lines. FE chips can be addressed by the MCC either in broadcast mode or one by one using their geographical address. In order to reduce the number of electrical connections to the module our system does not provide a pin reset signal. The MCC Command Decoder is designed so that at power up, after a finite amount of time, it returns to the idle state in order to be able to accept a Global Reset command that puts the chip in it s default state. At this point global configuration data (number of enabled FE chips, desired output mode, ) can be stored in the MCC register bank. The next phase implies a Global reset to all FE chips of the module. Once the whole system is initialized configuration of the single FE chips can begin. Carmel,CA (USA) 10 Sep 2002 7

Configuration: MCC The MCC Command Decoder allows 3 type of serial commands; Slow Commands, used during module (FE and MCC) configuration; Fast Commands (data syncronization) that can be issued without exiting data taking mode; Trigger commands. There are 8 16-bit wide configuration registers inside the MCC that allow to configure the chip for all its features, like: Enabling a certain number of FE chips on the module; Output data mode: 40 Mbit/s, 80 Mbit/s and 160 Mbit/s on one or two output lines are supported; Selecting the level of error checking/reporting; There is the ability to self test most of the chip circuitry. One can,for example, send data that simulate data coming from certain FE chips and perform real event building on these data. Calibration of all FE pixels is possible using an analog delay line that provides a signal sent to a chopper circuit inside the FE chip that allows charge to be injected in each single pixel cell (2 different charge ranges may be selected). FE configuration data is sent on a dedicated line and validated by a load signal. Carmel,CA (USA) 10 Sep 2002 8

Configuration: FE Vcal High Low Vreplica 5-bit TDAC 5-bit FDAC Shift Reg Select Mask Preamp Second amp Differential Discriminator PixClk HitBus Kill HitBus OR PixData Each pixel of the FE chip has 14 distinct configuration bits; Preamp kill bit, preamp mask bit, Hit Bus enable, Calibration enable; HitBus 5 bit threshold adjust capability, 5 bit feedback current trim (trimming of ToT). A global shift register that sneaks through the whole pixel array allows access for reading and writing the 14 control bits. A 20 bit Command Register and a ~200 bit Global Register, located in the bottom part of the chip, control all the configuration phase of the chip. 11 8 bit DAC s control all critical bias currents and voltages needed on the chip and 1 DAC is used for charge injection. Configuration loading of the whole module takes ~ 130 ms. Carmel,CA (USA) 10 Sep 2002 9

Single chip before and after tuning Th. = 3130 e - σ = 955 e - Threshold Noise = 275 e - Noise Th.=3140 e - σ = 88 e - Threshold Noise = 250 e - Noise Carmel,CA (USA) 10 Sep 2002 10

Module with AMS bumps after tuning Here we see that, on a fully loaded module, the electrical performance of the system is not degraded with respect of single chips. Threshold tuning performed in concurrent mode (all FE chips enabled at once). Th. = 3078 e DVDD shorts in both FE Noise ~290 e 3 missing column pairs and 2 non working chips on this module. Carmel,CA (USA) 10 Sep 2002 11

FE: Self Triggering Mode A self Trigger generator can use the internal Fast OR signal (active each time there is a hit somewhere in the whole pixel array) to generate its own Lev1 signal with a programmable latency. This allows the chip to be read out even with a source. When this mode is active the MCC sends a Trigger command to the enabled Fe chip which will produce output data once the Fast OR fires. The MCC is able to collect data in this way just for one FE chip at a time and therefore, in order to image a whole module, one has to perform a Source Scan stepping through all 16 FE chips in a row. Am 241 Am 241 Cd 109 Am241 source plot Carmel,CA (USA) 10 Sep 2002 12

Data-Taking Data Taking phase can occur immediately after system initialization. The system has to be set in Run Mode and after that only Trigger and Fast commands are allowed, any Slow command will return to Configuration Mode. The architecture of the module is Data Push, i.e. as soon as the MCC receives a Trigger command it is immediately sent to the FE chips for data collecting. Up to 16 pending Triggers are allowed in the system. Trigger commands are 5-bit commands that allow automatic correction for eventual bit flips inside the command preserving correct timing information. A Pending Event FIFO keeps track of how many Events have still to be processed. In case more than 16 Triggers are received before an event is fully reconstructed they are simply dropped and the information will be propagated to the ROD inserting a Warning word in the corresponding Event. This mechanism allows the ROD to insert empty events to maintain synchronization with the data flow. Event Counter Reset and BCID Reset commands (Fast commands) can be issued to keep correct event synchronization. Carmel,CA (USA) 10 Sep 2002 13

A 8 bit time stamp is distributed to all pixels in the column. After a hit the time stamps for leading and falling edge of the discriminator are stored in a local RAM. Hits are transferred from the pixel column logic as soon as the trailing edge has been detected and the hit data is stored in EoC buffers. 20 MHz transfer rates are possible. The hit pixel is cleared. After the trigger latency, the data is either cleared from the EoC buffers or sent to the MCC if a trigger occured. Data are sent out as soon as the Trigger signal is detected (data push). 64 EoC buffers are used on the FE for hit storage during Lev1 latency. Data-Taking: FE HitL Trigger Pixel pair (2x160) 8-bit Grey Counter Readout Controller Sense amplifier ToT subtraction Serializer Carmel,CA (USA) 10 Sep 2002 14 Hit logic Sparse Scan Trigger FIFO RAM RAM 64 EoC Buffers ROM RAM RAM Sparse Scan Hit logic Bottom of chip digital logic Serial Data Output HitR

Data-Taking: MCC Only enabled FE chips participate to Event building. 16 parallel data streams are received and stored in 16 independent FIFO s. Each Event is identified by an EoE word. EoE information is stored in the Scoreboard. As soon as all 16 EoE words of an Event are collected event building is performed. 8 bit BCID and 4 bit Lev1 information is stored with each incoming Trigger. Event building collects data from all 16 FIFO s and formats the output data stream according to the selected output mode. Up to two output lines (each sampling data on both clock edges) can be used in order to provide a data throughput up to 160 Mbit/s. Data consistency checking between hits from the same FE chip and between EoE words from different FE s is performed. FIFO data overflow, which produces loss of hits, may occur and is signaled in the data flow. Carmel,CA (USA) 10 Sep 2002 15

Architecture Simulations Detailed simulation of the architecture has been performed in order to validate its performance, and eventual safety margins. Main concern parameters were: FIFO size in words to avoid data loss; Output bandwidth occupation; Safety margins with respect of number of minimum bias events and Trigger rate. Main features of SimPix are: Independent simulation of all read-out electronic modules; Simulated physics data can be used as input; Time correlation between simulated events; Electronic description that is more detailed than a parametric model and faster than a low-level one; Entire detector simulation. Carmel,CA (USA) 10 Sep 2002 16

Nominal conditions: Analyzed sample: 1500 b-jet events produced by Higgs boson decay (m H = 100GeV/c 2 ); Average number of pile-up events: 24; Average Lev1 trigger selection rate: 100 khz; Electronic noise occupancy: 10-5 Hit/Pixel/BCO; Selected module: B-layer, η=0. Analyzed conditions Front-End: Ideal model used, which introduces no inefficiencies (due to 64 EoC buffers). MCC: Receiver FIFO size: 32 128 words. Output link: 40, 80, 160 Mbit/s. Real model used (Verilog, C ++ ) in order to simulate all possible inefficiencies. Carmel,CA (USA) 10 Sep 2002 17 Pixel occupancy (%) x 10-3 0.6 0.4 0.2 0 B Layer 2nd Layer EndCaps 3rd Layer -3-2 -1 0 1 2 3 η

Simulation Results Output link occupancy: Link occupancy does not strongly depend on FIFO size; Even with only 1 link available the occupancy is acceptable. Receiver FIFO occupancy: Below 5% with 80 or 160 Mbit/s. Skipped Lev1 triggers: Sharp correlation with Number of lost Hits. Number of lost Hits: Absolutely no problem with higher output modes; Conclusions: Results show a big safety margin operating with 128 word FIFO s and 160 Mbit/s output links; Occupancy (%) Level1 triggers (%) 100 80 60 40 20 15 10 5 0 (a) output link occupancy 40 Mbit/s 80 Mbit/s 160 Mbit/s 16 32 48 64 80 96 112 128 144 Receiver FIFO size (c) skipped Level1 triggers 40 Mbit/s 80 Mbit/s 160 Mbit/s 16 32 48 64 80 96 112 128 144 Receiver FIFO size Occupancy (%) Hits (%) 40 20 (b) Receiver FIFO occupancy 40 Mbit/s 80 Mbit/s 160 Mbit/s 0 16 32 48 64 80 96 112 128 144 Receiver FIFO size 20 10 (d) lost hits 40 Mbit/s 80 Mbit/s 160 Mbit/s 0 16 32 48 64 80 96 112 128 144 Receiver FIFO size Carmel,CA (USA) 10 Sep 2002 18

Safety margins Simulations done with a FIFO depth of 128 words and 160 Mbit/s Occupancy (%) 40 30 20 Number of pile-up events Input links occupancy Output link occupancy Occupancy (%) 100 80 60 40 20 40 60 80 100 120 140 20 40 60 80 100 120 140 Number of pile-up events Number of pile-up events Occupancy (%) 25 20 15 Input line occupancy Trigger rate Occupancy (%) 60 50 40 30 100 125 150 175 200 Trigger Frequency Output line occupancy 100 125 150 175 200 Trigger Frequency Pending Level1 FIFO occupancy Receiver FIFO occupancy Pending Level1 FIFO occupancy Receiver FIFO Occupancy Occupancy (%) 60 40 20 0 20 40 60 80 100 120 140 Number of pile-up events Occupancy (%) 40 20 0 20 40 60 80 100 120 140 Number of pile-up events Occupancy (%) 8 6 4 100 125 150 175 200 Trigger Frequency Occupancy (%) 3 2.5 2 1.5 100 125 150 175 200 Trigger Frequency Level1 (%) 2 1 0 Skipped LEVEL1 20 40 60 80 100 120 140 Number of pile-up events Hits (%) 20 10 0 Lost hits 20 40 60 80 100 120 140 Number of pile-up events Triggers (%) 1 0.5 0-0.5-1 Skipper Triggers 100 125 150 175 200 Trigger Frequency Hits (%) 1 0.5 0-0.5-1 Lost Hits 100 125 150 175 200 Trigger Frequency Carmel,CA (USA) 10 Sep 2002 19

Test Beam: Hit map of a module Extensive measurements of 3 modules were performed this summer at the CERN SPS proton facility. The VME based system was run in the DAQ-1 framework. Data acquisition was performed at almost 7 khz (9,000 events per spill). No system failures were observed during the whole test period. Hit map (log scale) for a complete module. Carmel,CA (USA) 10 Sep 2002 20

Ganged and Long Pixels In the region between different FE chips pixels are either longer (600µm) or ganged (see layout). The effect of this can be observed both in pixel counts (below) and in correlations (for the ganged pixels) with extrapolated tracks (lower left picture). Carmel,CA (USA) 10 Sep 2002 21

Irradiation at CERN PS Both Single FE chip assemblies and the MCC were successfully irradiated at the PS facility at CERN (24 GeV protons) during this year. Main goals of the MCC irradiation program was to verify correct operation of the chip during irradiation and study Single Event Upset effects. More than 65 MRad of accumulated dose were collected. We irradiated 7 MCC chips powered at 2.2 V and 1.8 V. Before each particle spill, configuration data was written to 6 chips (both Registers and FIFO s being written). After the spill data was read back and compared with written data in order to understand Static Bit Flip probability. The remaining MCC was operated synchronizing data taking with the particle spill in order to study possible problems during normal chip operation. During one week of continuous operation we never observed once a state transition in the chip causing an unrecoverable error that needed a hard reset, confirming the robustness of the Command Decoder architecture. At 1.8V we had twice the errors than at 2.2V. 0 -> 1 flips occur at almost at a double rate than 1 -> 0 ones in FF s. Errors in the full custom FIFO s are 2 to 4 times higher than in Std Cell FF s. All chips were fully functional after the irradiation period!!! Carmel,CA (USA) 10 Sep 2002 22

MCC: SEU induced system errors There are 13.4x10 6 FIFO bits and 0.6x10 6 Flip Flops in all the B-layer. B-layer errors Proton fluence (p/year/cm2) 3.00E+14 Seconds in a year run (100 D) 8,640,000 MCC/B-layer 312 FIFO bit / B-layer 13,418,496 FF / B-layer 624,000 Average EoE words in FIFO 8 Average No. of hits in FIFO 12 For the FIFO s only an End-of-Event (EoE) word corruption by a bit flip ( event over run ) represents a real problem. EoE words are tagged by only 3 bits out of 21 forming a FIFO word. This causes data corruption on all subsequent events requires a data path reset. This happens every 5 10 s. Not all FF have the same importance; Probably only <10% of the total have an important effect at the system level. Triple logic with the addition of a majority logic on the output may cure the problem. Run number run176 run184 run191 run204 run226 EoE -> hit / MCC@2.2V x-sect (cm2) 4.3E-12 5.2E-12 7.4E-12 6.4E-12 6.6E-12 EoE -> hit / MCC@1.8V x-sect (cm2) 8.1E-12 9.3E-12 1.3E-11 1.1E-11 1.1E-11 Hit -> EoE / MCC@2.2V x-sect (cm2) 1.3E-12 1.6E-12 2.2E-12 1.9E-12 2.0E-12 Hit -> EoE / MCC@1.8V x-sect (cm2) 2.4E-12 2.8E-12 3.8E-12 3.4E-12 3.4E-12 No. of s to event over run (MCC@2.2V) 5200-10400 4300-8600 3000-5900 3500-7000 3400-6700 No. of s to event over run (MCC@1.8V) 2800-5500 2400-4800 1800-3500 2000-4000 2000-3900 No. of s to event over run (B-layer@2.2V) 15-30 14-28 10-19 11-22 11-21 No. of s to event over run (B-layer@1.8V) 9-18 8-15 6-11 6-12 6-13 No. of s to a flip in any FF (MCC@2.2V) 4000-8000 3300-6500 2300-4600 2600-5200 2600-5200 No. of s to a flip in any FF (MCC@1.8V) 2000-4000 1700-3400 1300-2500 1400-2700 1500-2900 No. of s to a flip in any FF (B-layer@2.2V) 13-26 11-21 7-15 8-17 8-17 No. of s to a flip in any FF (B-layer@1.8V) 6-13 5-11 4-8 4-9 5-9 Carmel,CA (USA) 10 Sep 2002 23

Conclusions & Outlook Detailed system simulation show a big safety margin on all main parameters. Extensive tests of the ATLAS Pixel readout architecture have been performed this year. Results shown prove the effectiveness and the robustness of the chosen architecture. The system is very reliable. Full module irradiation. SEU improvements of our electronics. Both the MCC and the FE chips will be resubmitted at the end of this year and will address this issue. Integration of the optical components in the system. ROD testing and characterization for Pixel needs. Build and test a complete bi-stave (13 + 13 modules). Carmel,CA (USA) 10 Sep 2002 24