Signal Transport And Networks

Transcription

1 Signal Transport And Networks Chris Shenton Jodrell Bank Observatory University of Manchester STaN CoDR June

2 Custom or Proprietary Solution What does that really mean? Well of course we all know that a custom solution is always the most expensive... Actually a custom implementation built from commercial standard building blocks No new esoteric technology involved Aiming for the removal of unnecessary cost Full engagement with Industrial Partners 2

3 System Topology In SKA Phase 1 the majority of elements are within AA lo stations; 250 dish Rx's vs 560,000 AA RX Elements Majority of elements within 5Km of centre Majority of elements separated by only a few metres Maximum of 100Km reach required in Phase 1 Dishes grouped in small 'clumps' with AA station close Phase 1 deployment

4 System Characteristics Asymmetric data flows Tolerant of bit errors within data Synchronous data transfer required Tolerant of high latency Many 'end points' in the networks Fine Grained at element level (10G) 4

5 What Kind of Networks Terminate at an End Point Data transmission (Digital, Proprietary) Reference Phase transfer (Analogue, Proprietary) Synchronisation (Digital, Proprietary) Monitoring & Control (Digital, COTS, Ethernet) 5

6 What Are We Aiming For? Something that looks a bit like this will dominate the system cost... 6

7 Physical Realisation Concept High Volume Manufacture > 0.5 Million units required... Design for low Cost LRU for simple maintenance and installation Reducing number of connectors improves reliability Not high tech Good enough... not as good as possible. It s a Mini not a Rolls Royce Same design works for dishes 7

8 Integrated Digital Receiver Concept 4x 4b Fsample/4 4x 4b Deserialiser 1:6 Fdeser = 600MHz 4x Fdeser/6 Flogic Flogic Flogic = 100MHz MHz XGMII MHz Interfaces I IF in dc - 1GHz 800mV n n+1 ADC 1:4 n+2 9.6Gb/s R Ch DSP Framer 10Gb/s PCS 64b66b Gb/s Gb/s 1x 1b GHz LOGIC Core Interface ADC Interface PCS Interface PHY Interface n+3 Metadata Q IF in dc - 1GHz 800mV ADC 1:4 n n+1 n+2 n+3 9.6Gb/s R Ch DSP Framer PCS 64b66b Gb/s 4 Channel Parallel Optic Transceiver VCSEL Array 850nm 300m MPO Fibre Cable 4 x OM3 Data Aggregation Fsample = 2.4GHz 4 bit sample word Frame Sync M&C Subsystem uc 10/100 Eth PCS 64b66b Gb/s Timing Offset Subsystem Low Cost XTAL Clock Synthesis Subsystem PLL TxRx 50MHz Analogue Digital Receiver Boad Xilinx ARTIX Class FPGA 8

9 How to Reduce Costs Electrical & Logical Aggregation maximises the utilisation of available bandwidth Sharing of high cost Physical Resources such as interconnect and FPGA/ASIC platform Optical Aggregation via WDM maximises utilisation of physical fibre Standards based technology allows interception of commoditised technology Reduce the number of high cost terminations Reduce data volume by 'upstream' processing (this also has the happy side effect of simplifying the station and core processing) Reduce number of cables at Station and Core processing (1120 4x,600 12x) 9

10 System With Aggregation 10G-BASE-LR (10Km 1550 SMF) 10G-BASE-SR (100m 850nm OM3) Aggregating or 'Tile' Processing ~ 1:10 Data Reduction 10G-BASE-SR (10m 850nm OM3) x 10 Digital Receiver Digital Receiver 760Ms/s 4 bits per sample x 2 polarisations 6.06Gb/s 12 x 100Gb/s 1.2Tb/s Station Processing ~ 1:10 Data Reduction 1,120 x 10Gb/s Aggregating or 'Tile' Processing ~ 1:10 Data Reduction x 10 Digital Receiver Digital Receiver x 1, x 100Gb/s 1.2Tb/s Aggregating or 'Tile' Processing ~ 1:10 Data Reduction x 10 Digital Receiver Digital Receiver Core Processing x x 100Gb/s 1.2Tb/s From other Station Processors 10

11 System Without Aggregation 760Ms/s 4 bits per sample x 2 polarisations 6.06Gb/s Digital Receiver 10G-BASE-LR (10Km 1550 SMF) 10G-BASE-SR (100m 850nm OM3) Digital Receiver 12 x 100Gb/s 1.2Tb/s Station Processing ~ 1:100 Data Reduction 10G x x Digital Receiver 12 x 100Gb/s 1.2Tb/s Core Processing x x 100Gb/s 1.2Tb/s From other Station Processors 11

12 Strategy for getting to a Cost Effective System Solution Opportunities for resource sharing across multiple logical networks Use of appropriate technology for the application Obtain reduced costs by economies of scale Highest tech solution not necessarily appropriate Targeted use of high cost resources 12

13 Aggregation Processor Concept Aggregation Module Board Xilinx VIRTEX 7 Class FPGA SPF Rx Block SPF Rx Block SPF Rx Block 4 Channel Parallel Optic Transceiver VCSEL/PIN Array 4 Channel Parallel Optic Transceiver VCSEL/PIN Array 4 Channel Parallel Optic Transceiver VCSEL/PIN Array MPO Fibre Cable 4 Way MPO Fibre Cable 4 Way MPO Fibre Cable 4 Way MPO Fibre Cable 4 Way 4 Channel Parallel Optic Transceiver VCSEL/PIN Array Agg Block M&C Embedded Proc M&C 10/100 Ethernet Hub Signal Processing & Beamforming SPF Rx Block SPF Rx Block SPF Rx Block 4 Channel Parallel Optic Transceiver VCSEL/PIN Array 4 Channel Parallel Optic Transceiver VCSEL/PIN Array 4 Channel Parallel Optic Transceiver VCSEL/PIN Array MPO Fibre Cable 4 Way MPO Fibre Cable 4 Way MPO Fibre Cable 4 Way SPF Rx Block 4 Channel Parallel Optic Transceiver VCSEL/PIN Array MPO Fibre Cable 4 Way GPS Module Digital Offset + GPS timecode SPF Rx Block SPF Rx Block 4 Channel Parallel Optic Transceiver VCSEL/PIN Array 4 Channel Parallel Optic Transceiver VCSEL/PIN Array MPO Fibre Cable 4 Way MPO Fibre Cable 4 Way Digital Phase Processor Analogue Phase Lock and 1:10 Fanout SPF Rx Block 4 Channel Parallel Optic Transceiver VCSEL/PIN Array MPO Fibre Cable 4 Way REF RTN 1KHz x 10 Digital Phase Pairs REF 50MHz Analogue Phase Ref Analogue Phase Signal Conditioning Board RTN 50MHz Analogue Phase Return Analogue Conditioning Board + Optical Modules High Quality LO 13

14 Electrical Aggregation and Tile Processing Majority of terminations short hop (560,000 connections less than 10m) Creation of sub groups or tiles allows electrical aggregation using low cost silicon ($10 class FPGA not $1000 class FPGA) Element Volumes make a simple ASIC a further cost reduction opportunity (cheap and cheerful 90nm technology will do this job... no need for esoteric 22nm...) Distribution of the processing into the end points and aggregators enables the maximum utilisation of logic resources. Major cost driver is IO in the form of the network termination Appropriate but simple processing at the element or tile level becomes effectively 'for free' as part of FPGA or ASIC logic Channelisation and forming of partial beams reduces upstream data rate 14

15 Physical Layer 4x/12x/24x 10G MPO Structured Cabling System VCSEL Driver Array 40G/120G/240G Capacity over fibre pairs 25Gb/s VCSEL TXCR s in development pushing capacity to 100G/300G/600G 850nm MMF (1310nm and 1550nm SMF products in development) OM3 Cable for SR Links IEEE G-BASE-SR/LR PHY specification Driven by data centre patch interconnect market Target $0.25 per gigabit per termination ($20 per 4x) Power Consumption 700mW per end 40% Tx/60% Rx per 4x (4 x 10G) MSA (SFP+) plug-able interfaces high cost and not getting cheaper quickly 15

16 TE Low Cost 10G-BASE-SR Solution real & demonstrated - not slideware 10G Now 25G Target 16

17 Transport Layer Proprietary Simple Synchronous (Plesiochronous) framing required Ethernet UDP/TCP/IP Could be used for framing... No real benefit Unidirectional Costs Tx power Minimal transport overhead required probably just a CRC Only really required for M&C network. No retransmission, use it or lose it M&C Network over TCP/IP can easily be carried within synchronous frames as a VC Half as many driven Tx nodes in the data backhaul Could implement partial Layer 2-4 stack (ARP, Ping etc) at the endpoint... but why bother when it isn t a switched network? 17

18 Can We Use 100Gb DP-QPSK Technology to Reduce Costs Depends on cost of transceiver technology - 10G SFP+ still expensive and cost reduction curve looks shallow Could reduce number of connections from Station to Core processing Still immature technology Probably required in SKA 2 for dense AA 18

19 Use of DWDM Optical Aggregation sub assemblies are expensive Optical MUX s and DeMUX s at 50GHz channel spacing very expensive Relationship between number of EDFA s and no of channels per fibre needs analysing for best cost tradeoff Point to point many core fibre is becoming more widespread Driven by Fibre to the Home Use of DWDM could be cost effective in long reach links with many stations Deploy where physical fibre costs become significant proportion of system cost Cost Tradeoff analysis needs to be done to find optimum solution Potentially more significant as the number of remote stations increases in Phase 2 19

20 Interface Standardisation ADC Logic LVDS (or low power derivative) Lower power solution required for dense arrays Electrical TXCVR Optical Module XGMII (or CGMII if 100G arrives) Plug-able Module? SFP+ is still very expensive SFP+ Cost includes high percentage of housing and mechanical assembly Integrate the E/O and O/E into the Digital Receiver subassembly Reduces cost Improves Reliability 20

21 Cost Breakdown Aggregated Based on 10:1 Aggregation A 10 Element Group Consists of; 10 x DR FPGA/ASIC ($10) 10 x 10m Terminated Optical 4x MPO Cable Assembly ($30) 1 x AGG FPGA/ASIC ($100) 1 x 100m Terminated Optical 4x MPO Cable Assembly ($120) Total Cost per Group = $620 per group of 10 elements A Element Station Consists of; 1120 x 10 Element Groups ($620) 12 x 100m Terminated Optical 4x MPO Cable Assembly ($120) Total Cost per Station (at station inputs) 1120 x 620 = $ per station Station to Core Connectivity Cost 12x MPO Termination $60 Fibre 5Km SMF in 192 per metre 24 Cores $3125 per 12x Connection Fibre termination (Shared Fibre Cable) $800 Total is; $60 + $ $800 = ~$4000 per connection Total SKA 1 Cost 50 x $700K x $4K = $37.4M 600 connections at the core; (12 x 50) 600 x $4000 = $2.4M 21

22 Cost Breakdown Direct Connection to Station No Aggregation A Single Element DR Consists of; 1 x DR FPGA ($10) 1 x 100m Terminated Optical 4x MPO Cable Assembly ($120) Total Cost per Element = $130 per element A Element Station Consists of; x Single Elements ($130) Total Cost per Station (at station inputs) x 130 = $1.456M per station Station to Core Connectivity Cost 12x MPO Termination $60 Fibre 5Km SMF in 192 per metre 24 Cores $3125 per 12x Connection Fibre termination (Shared Fibre Cable) $800 Total is; $60 + $ $800 = ~$4000 per connection Total SKA 1 Cost 50 x $1.456M x $4K = $75.2M 600 connections at the core; (12 x 50) 600 x $4000 = $2.4M 22

23 Power Requirements Aggregated x (1.4W + 5W) = 72KW 1120 x (1.4W +20W) = 31KW Total 104KW per station Non Aggregated x (1.4W + 5W) = 72KW Total 72KW per station 23

24 Risks Compromises are required in order to simplify and cost reduce the system Hard to pin down spec from scientists, let alone impose constraints Cost constraint now hard to ignore Data Rate estimates are questionable for AA technology LOFAR has data growth in beamforming to maintain S/N performance is the required station data rate actually bigger than current spec states... Proposal based on emerging technologies - pricing is fluid We really need a cheap 10G-BASE-LR (10Km) option, 1550nm VCSEL based technology not currently mature Work with technology providers, use their expertise to get real world solutions Subject to wider market forces Fallback implementation schemes look expensive At the mercy of interconnect technology futures market 24

25 Scaling up For Phase 2 All of the above... but much worse! Apply same principles to dense array and focal plane array solutions Its a scalability and affordability problem not a technical one Dense Array Tile processor for AA-Mid or PAF will look something like this Need to find the right level of integration to remove cost and maintain adequate performance 25

26 Multi Channel Integrated Digital Receiver Concept aka Tile Processor 8 x I IF in dc - 1GHz 800mV Fsample = 2.4GHz 4 bit sample word ADC 8 channels 2.4Gs/s 4bit word 8 x 9.6Gb/s 8 x 1:32 Gearbox 8 x 32bit 300MHz 8 samples every 3.33ns 8 x 9.6Gb/s Aggregator 4 x 19.2Gb/s 4x 64b 4x 64b Framer 4 x 25Gb/s 4x 64b PCS 4x 66b 8 x I IF in dc - 1GHz 800mV ADC 8 channels 2.4Gs/s 4bit word 8 x 9.6Gb/s 8 x 1:32 Gearbox 8 x 9.6Gb/s Aggregator 4 x 19.2Gb/s 4x 64b 4x 64b Framer 4 x 25Gb/s 4x 64b PCS 4x 66b 8 x Q IF in dc - 1GHz 800mV 8 x Q IF in dc - 1GHz 800mV ADC 8 channels 2.4Gs/s 4bit word ADC 8 channels 2.4Gs/s 4bit word 8 x 9.6Gb/s 8 x 9.6Gb/s 8 x 1:32 Gearbox 8 x 1:32 Gearbox 8 x 9.6Gb/s 8 x 9.6Gb/s Aggregator 4 x 19.2Gb/s Aggregator 4 x 19.2Gb/s 4x 64b 4x 64b DSP 4x 64b 4x 64b Framer 4 x 25Gb/s Framer 4 x 25Gb/s 4x 64b 4x 64b PCS PCS 4x 66b 4x 66b 24 Channel VCSEL Array 28Gb/s 1550 SMF MPO Fibre Cable 24 x SMF 28nm class ASIC Sampling Clock Distribution M&C Subsystem uc 10/100 Eth Frame Sync Subsystem Timing Offset Subsystem TxRx 1.25Gb/s TxRx 100Mb/s TxRx 100Mb/s 16 Element DP Digital Receiver Module PCA or MCM Logic Clock Distribution PLL TxRx 50MHz Analogue Clock Synthesis Subsystem LO Local Clock Subsystem 26

27 What Happens Next... Continue Design work on integrated solution based system reference design Continue work on enclosure designs and RFI containment Architectural proving using 2-PAD and other Array based systems Update documentation Update the design concept with Station and Core processor Line Interfaces Concepts understood and partially developed but additional work required Costing of these interfaces Establish a real world reference design and a component level design database under way Explore cost implications of Analogue Transport to Integrated Tile Processor Conventional Analogue over Fibre Continue to build relationships with key technology suppliers and potential delivery partners; Interconnect Vendors current work is supported by TE (Formerly Tyco) We are way past the stage where generic costings are adequate FPGA Vendors (Altera, Xilinx) 27

28 Conclusions We need to put constraints on the system performance in order to meet the cost targets Moving raw data samples over expensive links is expensive and unnecessary This is not a datacomms network, we don t have to deliver unprocessed data We need to establish detailed reference designs for major options The scientists need to support these engineering activities by analysing the impact of the constrained design and buying into it... Working at a theoretical level is now not adequate Can we build phase 1? Yes, but not without compromises Can we build phase 2? Yes, but we probably need to move to a high level of integration to make it affordable We need to continue to develop real engineering relationships with technology partners 28