Precision Time Protocol (IEEE 1588) profile

Transcription

1 INTERNATIONAL ELECTROTECHNICAL COMMISSION IEC Annexes IEC/IEEE Precision Time Protocol (IEEE 1588) profile for clock synchronization in Industrial Automation networks Fault-tolerant clocks attached to redundant local area networks, especially PRP and HSR picture from shawnhallwachs' photostream Prof. Dr. Hubert Kirrmann, Solutil, Switzerland IEC SC65C WG15; IEC 57 WG10; IEEE P1588 Architecture 2017 IEC SC65C WG Baden, Switzerland

2 Abstract Two standards developed by IEC SC65C WG15 were published in 2016 that provide microsecond precision clock synchronization for all Ethernet-based industrial networks They have the same base specifications: IEC / IEEE 1588 [1588 in the sequel]: IEC :2016 Annexes A-E Industrial Communication Networks (high-availability) IEC :2016 Communication Networks for Power Utility Automation There standards open the way to precision time-stamping and deterministic data transmission. 2 IEC SC65C WG15 / IEC 57 WG HK Solutil

3 Contents 1. Application domain operating principles 3. Link delay measurement in IEC Clock model 5. RedBoxes as Three-way Boundary Clocks 6. RedBoxes as Doubly-Attached Boundary Clocks 7. RedBoxes as Doubly-Attached Transparent Clocks 8. RedBoxes as Stateless Transparent Clocks 9. MIB 3 IEC SC65C WG15 / IEC 57 WG HK Solutil

4 PRECISION TIME APPLICATION DOMAINS 4 IEC SC65C WG15 / IEC 57 WG HK Solutil

5 Applications that require sub-micro second synchronization Electrical substations: differential protection: 10 µs (absolute time) Electrical grids: wide area protection: 1 µs (absolute time) Motion control: newspaper printing : 4 µs (relative time) Drive (GTO, IGBT firing): 1 µs (relative time) Typical time synchronization protocols: SNTP: µs accuracy PTP: 1 µs GPS / Galileo: 0,1 µs accuracy 5 IEC SC65C WG15 / IEC 57 WG HK Solutil

6 Synchrophasor network PDCs (Phasor Data Concentrators) detect grid instabilities by comparing the phase of current and voltage measured by PMUs (Phasor Measurement Units) at strategic locations with 4 µs accuracy. 6 IEC SC65C WG15 / IEC 57 WG HK Solutil

7 1588 operating principles 7 IEC SC65C WG15 / IEC 57 WG HK Solutil

8 clockaccuracy Definitions: precision and accuracy of a clock accuracy 3 99,73% 68,28% reference mean 68,28% 3 99,73% precision time inaccuracy time error: deviation from the time reference used for measurement or synchronization, evaluated over a short span. in this example: the red curve representing a histogram of the time error. accuracy: mean of the error on time or frequency between the clock under test and a perfect reference clock, over an ensemble of measurements.[1588]: in this example: -60 ns precision: deviation from the mean error on time or frequency between the clock under test and a perfect reference clock [1588] in this example: 120 ns with a variance of 3 time inaccuracy: time error not exceeded by 99.7% of the measurements, evaluated over a series of 1000 measurements (about 20 minutes) in steady state [IEC ]. in this example: 180 ns clock accuracy: time inaccuracy guaranteed by the manufacturer in this example: 200 ns clockaccuracy: clock accuracy enumeration transmitted in the Announce message [1588] 8 IEC SC65C WG15 / IEC 57 WG HK Solutil time error [ns]

9 Notions: Synchronization & Syntonization + - Synchronization = adjust the time Syntonization = adjust the frequency 9 IEC SC65C WG15 / IEC 57 WG HK Solutil

10 Precise synchronization: Inaccuracy sources Influence of the quartz Influence of the medium Usual quartz have high precision typical: 50 ppm ( 50 μs per second) precision Usual quartz has a temperature dependence of 1 ppm/c o (use of oven is costly) Shock and vibration are a greater inaccuracy source Cable: about 500 ns delay per 100 m (CAT5 cable) cable asymmetry is nominally 25-50ns/100m Wireless: 300 ns / 100 m, but reflections can change the path length Hub / Media converter: delay 500 ns, jitter about 50 ns, independent of frame length Bridge (Switch) cut-through bridges: minimum delay of 1,12 μs, max 124,0 μs if switch supports prioritization, unlimited otherwise 10 IEC SC65C WG15 / IEC 57 WG HK Solutil store and forward bridges: minimal delay 6,7 μs, max 124,0 μs (with prioritization), asymmetry depending on traffic.

11 TIME SCALES 11 IEC SC65C WG15 / IEC 57 WG HK Solutil

12 U vs TAI TAI (Temps Atomique International) is the international time base maintained by a network of some 400 atomic clocks worldwide synchronized by GPS satellites, it bases on the second definition of the Cesium atom. U is the legal time, it increases at the same rate as TAI, but is corrected about every 1.5 year by a leap second to compensate the slowdown of the Earth rotation. High-precision clocks have a problem with U since the handling of leap seconds is tricky. At the same time, it is not possible to calculate time differences without a table of leap seconds. 12 IEC SC65C WG15 / IEC 57 WG HK Solutil

13 Calculating time differences: a problem Two events were retrieved from the archive. They were time-stamped in the Unix format (no fractions): Timestamp value A: < > Timestamp value B: < > Question: what is the time interval between these two events? Answer: 1) if the timestamp is TAI: ( ) = s = 24:00:00 hours 2) If the timestamp is U: it depends (it was 24:00:01 since there was a leap second on , but to know this you need to compute the absolute time and look-up the actual leap second table) 13 IEC SC65C WG15 / IEC 57 WG HK Solutil

14 A recommendation Industrial control systems should not rely on U, but only use TAI. This applies to OPC, etc U is a human-readable scale. We propose to make this a general recommendation in SC65C and IEC SC65C WG15 / IEC 57 WG HK Solutil

15 NTP (internet time protocol) only estimates path delays NTP distributes U only request path a) symmetrical network delay time client network server response t 1 request t response t 3 estimate of network delay b) asymmetrical network delay time t 1 t 4 1 request t 2 ( t4 t1 3 t2 ) ( t 2 ) 15 IEC SC65C WG15 / IEC 57 WG HK Solutil 2 t 3 response t 4 distance NTP estimates the path delay end to end, assuming same delay in both directions. This is far from being the case due to packet delay version (network congestion, route)

16 1588 elements Reference signal GPS The master broadcasts TAI time e.g. every 1 s Grand Master Clock GC Sync Sync (corrected) = transparent clock GC = grandmaster clock link delay transparent clock residence delays OC = ordinary clock link delay residence delays Sync OC OC OC OC Each bridging device relays the Sync message from the GC and adds to it a time correction to compensate for its own residence time and the delay on the link from which the frame came 16 IEC SC65C WG15 / IEC 57 WG HK Solutil

17 1588 time correction The time received by the slave clock is corrected by the time that elapsed between the master and the slave, called the path delay. Path delay includes all delays between the master and the slave(s) clock, divided into: 1) Residence delays (~100 µs) measured using the local clock (possibly syntonized) of the network elements 2) Link delays (~5 ns/m) measured using a ping-pong exchange with the partner, assuming that the link delay is the same in both directions, with two methods: a) end-to-end b) peer-to-peer 17 IEC SC65C WG15 / IEC 57 WG HK Solutil

18 Precision time correction principle GPS MC Master Clock link delay 2µs residence delay 50µs t 0 0 Transparent Clock (switch, bridge) 5µs t µs Transparent Clock (switch, bridge) 4µs t OC t = t Ordinary Clock (slave) 18 IEC SC65C WG15 / IEC 57 WG HK Solutil

19 Precise time-stamping by hardware link delay residence delay link delay Transparent Clock PHY link (e.g. cable) PHY bridging logic PHY link (e.g. cable) PHY Xtal reference plane synceventingresstime pdelay_reqeventingresstime pdelay_respeventingresstime reference plane synceventeggresstime pdelay_reqeventeggresstime pdelay_respeventeggresstime Each clock transition introduces a jitter and a constant delay due to the synchronizer. To keep track of the time-stamps: - at reception: subtract the ingress timestamp (and add the peer delay) from the correction field, - at sending: add the residence delay and the egress timestamp to the correction field. 19 IEC SC65C WG15 / IEC 57 WG HK Solutil

20 grandmaster (role): top level clock of the time domain master of the top region, defines grandmaster identity master port sends Sync & Announce: sets sourceportidentity for this region Transparent Clock forwards and corrects PTP messages Boundary Clock has a slave port slave port in the upper region and a master (or passive) port BC in the lower region subdomain B slave port OC GC PTP elements top subdomain master-enabled OC (currently slave) ingress port egress ports OC BC Rb DC OC slave port master port sends Sync & Announce subdomain C designated master (back-up clock of domain) first to become master (after BMCA) if the current master stops sending Announce slave (or passive) port ordinary clock can take the role of master or slave Sync & Announce carry the MAC address of the master, (not that of the grandmaster) and the sourceportidentity of the master if this link is established, the two regions merge and one boundary clock s master port becomes either slave or passive, according to the BMCA OC HC HC bridging nodes Hybrid clocks combine a and an OC, HC have two or more ports 20 IEC SC65C WG15 / IEC 57 WG HK Solutil

21 1588 module with /OC functionality OC bridge switching CPU MII MII PHY management workstation port 1 MAC MAC port 4 TS time-stamping TS OC TS port 2 medium-independent interface port A TS MAC MAC port 3 MII MII PHY PHY port B MC OC 21 IEC SC65C WG15 / IEC 57 WG HK Solutil

22 Time distribution in 1588 (1-step correction, transparent clocks) path bridge bridge master clock link transparent clock link transparent clock link ordinary (slave) clock time Sync contains t 1 Sync contains t 1,, ( ) residence delay t 1 t 1 link delay residence delay Sync contains t 1, ms = i, ( i ) t 2 every transparent clock along the path estimates the delay it introduces and adds it to the correction field 22 IEC SC65C WG15 / IEC 57 WG HK Solutil

23 Accuracy degradation PTP path SC grand master clock transparent clock transparent clock slave clock local clock error (pdf) distance total error (pdf) slave clock 23 IEC SC65C WG15 / IEC 57 WG HK Solutil

24 Intermediate clocks in has two different intermediate clocks: BC: boundary clocks, that are slave clocks in a region acting as master for another region (used mainly in wide area network routers) : transparent clocks, that are relaying the synchronization signal within a region (used mainly in local area networks). 24 IEC SC65C WG15 / IEC 57 WG HK Solutil

25 Comparison: degradation in and BC chains The mean at the slave clock is the sum of the mean errors The variance at the slave clock is the sum of the variances Transparent clocks SC grand master clock transparent clock transparent clock transparent clock slave clock pdf(time error) Boundary clocks Transparent clocks introduce a small mean error and a small variance (50ns) 2 The contribution of the master clock dominates BC BC BC SC grand master clock boundary clock boundary clock boundary clock slave clock pdf(time error) Boundary clocks introduce both a mean error and a large variance (200ns) 2 Their contribution dominate the grand master s variance, since there form a chain of control loops. 25 IEC SC65C WG15 / IEC 57 WG HK Solutil

26 Why time domains? Plant networks are connected for: Unified Operations / Common Platform Common communication infrastructure (e.g. bridges) Power Management e.g. load shedding Common Network Management Reuse devices and designs Use same clock reference for all segments BDS Workplaces OC GC GPS Rb OC Controllers Control Network IEC PROFINET Field Network Web HMI AIS HV GIS Power trafo MV Switchgear MV Drives LV Switchgear Distribution trafo Drives LV Products Remote I/O Proxies to other buses Profibus others Valves Instrumentation Where different time distribution systems are used, they work in different domains 26 IEC SC65C WG15 / IEC 57 WG HK Solutil

27 1588 options and profiles 27 IEC SC65C WG15 / IEC 57 WG HK Solutil

28 1588 options and IEC profiles These concepts apply to the major options in IEEE 1588: Time correction using 1-step or 2-step Link delay measured by end-to-end or peer-to-peer Communication takes place on Layer 2 (Ethernet) or Layer 3 (IP) Boundary clocks and transparent clocks layer2 layer3 end-to-end L3E2E drives peer-to-peer L2PTP utilities 1-step 2-step 28 IEC SC65C WG15 / IEC 57 WG HK Solutil

29 The IEC Annex C profiles Medium Transmission Master clock accuracy Transparent clock inaccuracy Boundary clock inaccuracy Sync correction method Sync message interval Pdelay message interval Redundancy method SNMP MIB common Ethernet multicast 250 ns 50 ns 200 ns L3E2E Profile identifier 00-0C-CD z Delay measurement end-to-end Announce interval (default) 2 s (*) 1-step and 2-step (mixed) 1 s 1 s PRP slave choses master IEC L2P2P 00-0C-CD z peer-to-peer 1 s (*): there is no technical reason for the 2s Announce interval, it is a legacy value. 29 IEC SC65C WG15 / IEC 57 WG HK Solutil

30 Compatibility L3E2E L2P2P The L3E2E and L2P2P protocols cannot be used in the same time domain simultaneously because they use different delay measurement mechanisms. The 1588 standard currently prohibits transparent clocks that support both the E2E and the P2P delay calculation, even in different time domains. This is overspecified, since manufacturers of equipment can build such devices and they will be interoperable. It can be expected that L2PTP will become the industry standard. 30 IEC SC65C WG15 / IEC 57 WG HK Solutil

31 The L3E2E Profile end-to-end link delay measurement, layer 3 communication, 1-step and 2-step PTP attribute Default value Range portds.logannounceinterval 0-3 to +1 log seconds portds.logsyncinterval 0-3 to +1 log seconds portds.announcereceipttimeout 2 for preferred grandmasters 2 to10 sync intervals 3 for all other grandmasters portds.logminpdelay_reqinterval 0 0 to 5 log seconds defaultds.priority1 defaultds.priority2 128 for not slave-only clocks 255 for slave-only clocks 128 for not slave-only clocks 255 for slave-only clocks 0 to for slave-only clocks for slave-only clocks defaultds.domainnumber 0 as specified in Table 2 of IEC transparentclockdefaultds. primarydomain 0 as specified in Table 2 of IEC IEC SC65C WG15 / IEC 57 WG HK Solutil

32 The L2P2P Profile peer-to-peer link delay measurement, layer 2 communication, 1-step and 2-step PTP attribute Default value Range portds.logannounceinterval 0-3 to +1 log seconds portds.logsyncinterval 0-3 to +1 log seconds portds.announcereceipttimeout 2 for preferred grandmasters 2 to10 sync intervals 3 for all other grandmasters portds.logminpdelay_reqinterval 0 0 to 5 log seconds defaultds.priority1 defaultds.priority2 128 for not slave-only clocks 255 for slave-only clocks 128 for not slave-only clocks 255 for slave-only clocks 0 to for slave-only clocks for slave-only clocks defaultds.domainnumber 0 as specified in Table 2 of IEC transparentclockdefaultds. primarydomain 0 as specified in Table 2 of IEC IEC SC65C WG15 / IEC 57 WG HK Solutil

33 Standardisation and experience 33 IEC SC65C WG15 / IEC 57 WG HK Solutil

34 State of standardization To avoid the emergence of a parallel standard in IEEE, IEC and IEEE moved the Power Utility Profile of IEC under the umbrella of the Joint Development IEC/IEEE The text in IEC/IEEE is essentially the same as in IEC , profile L2P2P, with the addition that the range was further restricted, that time domain 93 was recommended and that double attachment is not mandatory. Both use exactly the same network management, IEC Annex E IEC was published on 2016 April 1 st. IEC/IEEE was published on 2016 May 1 st. IEC SC65C WG15 is responsible for keeping the two specification aligned, so a device that claims conformance to IEC will be conformant to IEC/IEEE The reverse is not true since a IEC/IEEE device is not obliged to implement redundant attachment according to PRP or HSR. 34 IEC SC65C WG15 / IEC 57 WG HK Solutil

35 Plug-fest and interoperability test (IOT), San Francisco companies deployed IEC devices, among them two IED manufacturers (ABB - HSR/PRP master-capable) and SEL (slave-only end device). 35 IEC SC65C WG15 / IEC 57 WG HK Solutil

36 Brussels interop test In November 2015, an interop test for IEC/IEEE was conducted in Brussels with a number of companies and observers. No flaws were encountered with the clock synchronization, while some improvements of the PRP and HSR specifications made their way into the standard released in IEC SC65C WG15 / IEC 57 WG HK Solutil

37 37 IEC SC65C WG15 / IEC 57 WG HK Solutil

38 IEEE 1588 Correction Transmission 1-step <> 2-step 38 IEC SC65C WG15 / IEC 57 WG HK Solutil

39 1-step and 2-step correction Each transparent clock corrects the time by the amount: (received correction + egress timestamp ingress timestamp (+ path delay 1 )) IEEE 1588 foresees two correction methods for 1) only in peer-to-peer 1-step correction: The sender of a Sync message inserts the correction while transmission is in progress. 1-step requires hardware support to read the egress timestamp and insert the correction on-the-fly. 2-step correction The sender of the Sync forwards the Sync it received and reads its egress timestamp. It sends in a subsequent Follow_Up message the correction. 2-step correction can be implemented by software, but needs hardware timestamps. This solution has a weakness since the path taken by the Follow_Up is not necessarily the one taken by the Sync. 39 IEC SC65C WG15 / IEC 57 WG HK Solutil

40 1588 Sync Message (P2P) timestamp point preamble destination (6 octets) preamble source (6 octets) 0: default 1: 802.1AS Announce: 0 Pdelay_Resp: t3-t2 Pdelay_Req: 0 64-bit clockidentity + 16-bit portnumber 0-3: 125 us 0: 1s 2: 4s ETPID = x8100 (2) prio CFI VID (12 bits) (2) HTPID = x897f (2) path size (2) sequencenr (2) PTP = 0x88F7 (2) transportspec messagetype reserved versionptp messagelength (2) domainnumber (1) reserved (1) flagfield (2) correctionfield (8) reserved (4) sourceportidentity (10) sequenceid (2) controlfield (1) logmessageinterval (1) timestamp (10) 802.1Q tag (optional) HSR Tag (optional) 1588 EtherType 0 alternatemasterflag 1 twostepflag 2 unicastflag 5 profilespecific 1 6 profilespecific2 7 security 0 leap61, 1 leap59 2 currentutcoffsetvalid 3 ptptimescale, 4 timetraceable 5 frequencytraceable incremented per message type and destination, except for responses and follow-ups 0: Sync 1: Delay_Req 2: Pdelay_Req 3: Pdelay_Resp 8: Follow_Up 9: Delay_Resp A: Pdelay_Resp_Follow_Up B: Announce C: Signalling D: Management padding (10) 40 IEC SC65C WG15 / IEC 57 WG HK Solutil FCS (4)

41 1-step correction in a Sync frame subtract ingress time-stamp from egress time-stamp, add link delay and add to former correction timestamp point correct checksum preamble header correction body 100 Mbit/s >1760 ns > 2240 ns 1-step correction requires on-the-fly modification of a frame while it is being sent. 41 IEC SC65C WG15 / IEC 57 WG HK Solutil

42 IEEE 1588 Path delay measurement End-to-end <> peer-to-peer 42 IEC SC65C WG15 / IEC 57 WG HK Solutil

43 Path delay calculation methods Path delay consists of the sum of residence delay in the transparent clocks and of the link delays Each transparent clock evaluates its own residence delay based on its local clock For the link delay measurement, there are two methods 1. E2E end-to-end (former IEEE 1588v1): each slave evaluates the delay to the master (with the help of the master) 2. P2P peer-to-peer (IEEE ): each transparent clock evaluates the delay to its peer 43 IEC SC65C WG15 / IEC 57 WG HK Solutil

44 End-to-end link delay measurement 1-step correction ordinary (slave) clock bridge 1 bridge 2 link 1-step link 1-step link transparent transparent clock clock master clock t 3 Delay_Req (0,0) residence delay t 10 sm1 t 21 t 11 Delay_Req (0, sm2 ) t 22 sm Delay_Req (0, sm ) t 4 link delay calculation Master responds with Delay_Resp (t4, sm ) Slave receives t 4 and sm Delay_Resp (t4, sm ) Delay_Resp (t4, sm ) Delay_Resp (t4, sm ) ( t1 t4) ( t2 t3) ms 2 sm residence delay link delay t 1 t 1 Add ( ) t 2 Sync (t 1, ms) ms residence delay Sync (t 1, ms2 ) ms1 Sync (t 1, 0) In this figure, Delay_Req is sent before Sync to outline that a Sync cannot be evaluated without a previous Delay_Req. I time distance 44 IEC SC65C WG15 / IEC 57 WG HK Solutil

45 End-to-end link delay measurement 2-step correction bridge 1 bridge ordinary (slave) clock t 3 link 2-step link 2-step link transparent transparent clock clock link delay Delay_Req residence delay t 10 residence delay master clock t t Delay_Req ( 1 ) t 22 t 4 Master responds with ( ms) link delay calculation Slave receives Delay_Resp contains ( ms), t 4 Delay_Resp Delay_Resp t 4 and sm t 4 t 1 t 1 Sync Sync Follow_Up t 2 Sync (t 1 ) Follow_Up Follow_Up* contains = ( ms ), t 1 Sync + Follow_Up contain = ( i + i ) time 45 IEC SC65C WG15 / IEC 57 WG HK Solutil distance

46 Peer-to-peer link delay measurement 1-step correction ordinary (slave) clock bridge bridge link 1-step link 1-step link transparent transparent clock clock master clock link delay t 11 t 14 Pdelay_Req Pdelay_Resp t 12 t 13 t 11 t 14 Pdelay_Req Pdelay_Resp t 12 t 13 t 11 t 14 Pdelay_Req Pdelay_Resp t 12 t 13 path delay calculation ( t4 t1 3 t2 ) ( t 2 ) t 1 residence time residence time Sync (contains + ) Sync time distribution t 2 Sync (contains ms + ) distance 46 IEC SC65C WG15 / IEC 57 WG HK Solutil

47 Peer-to-peer link delay measurement 1-step correction (both directions) bridge bridge ordinary (slave) clock t 11 t 14 link 1-step link 1-step link transparent transparent clock clock Pdelay_Req Pdelay_Resp (contains t 3 -t 2 ) t 12 t 13 master clock peer delay calculation t 12 Pdelay_Req t 11 t Pdelay_Resp 13 t 14 t 11 t 14 Pdelay_Req Pdelay_Resp t 12 t 13 t 2 t 3 Pdelay_Req Pdelay_Resp t 1 t 4 t 11 t 14 Pdelay_Req Pdelay_Resp t 12 t 13 residence delay link delay t 1 t 1 t 2 t 22 t 21 Sync contains = ( i + i ) residence delay t 11 Sync contains t1 + + t 11 Sync contains t 1 distance 47 IEC SC65C WG15 / IEC 57 WG HK Solutil

48 Peer-to-peer delay measurement 2-step correction ordinary (slave) clock bridge bridge link 2-step link 2-step link transparent transparent clock clock master clock t 11 t 14 Pdelay_Req Pdelay_Resp t 12 t 13 Pdelay_Resp_Follow_Up (contains t 3 -t 2 ) t 11 Pdelay_Req Pdelay_Resp t 12 t 13 t 11 Pdelay_Req Pdelay_Resp t 12 t 13 link delay t 14 t 14 Pdelay_Resp_Follow_Up Pdelay_Resp_Follow_Up t 1 Sync Sync Follow_Up t 2 Sync Follow_Up Follow_Up* contains = ( i + i ) time Sync + Follow_Up contain = ( i + i ) distance 48 IEC SC65C WG15 / IEC 57 WG HK Solutil

49 Considering redundant paths 49 IEC SC65C WG15 / IEC 57 WG HK Solutil

50 Redundant attachment of clocks The present 1588 standard do not consider clocks synchronized over redundant, simultaneously active paths. This extension of 1588 is needed for high availability automation networks as specified in IEC (PRP/HSR), but also others. The duplicate discard method of IEC cannot be applied to PTP messages, since the correction field depends on the path taken. Also, the Pdelay_Req messages are link-specific and therefore have no duplicates. The PTP messages have no RCT trailer in PRP and cannot use the HSR header to reject duplicates in HSR. Since the same master can appear over redundant paths, the Best Master Clock Algorithm needs extension with a time quality selection, and the clock model needs extension for doubly attached clocks. The transition between duplicated and single paths, especially using RedBoxes as transparent clocks and boundary clocks need specification. These changes are specifically reflected in the clock object model and in the MIB. 50 IEC SC65C WG15 / IEC 57 WG HK Solutil

51 Connection of a Master to a Slave clock over PRP Announce A Sync A Follow_Up A MC Follow_Up B Sync B LAN_A and LAN_B are independent Announce B residence delay link delay residence delay link delay LAN_A LAN_B link delay residence delay RedBox link delay link delay (Follow_Up A) Sync A Announce A Announce B Sync B (Follow_Up B) OC SAN OC 51 IEC SC65C WG15 / IEC 57 WG HK Solutil

52 Doubly-attached clock as Boundary Clock OC1 LAN_A LAN_B (master) (slave) port A Port B BC This model exists already in IEEE 1588 and could be used for doubly attached clocks. For this, the BMCA must be adapted to handle the same master seen over two ports. An application-dependent clock quality shall select the port rather than the port identity. This allows to select the best clock signal and also to switch regularly to test the other path. 52 IEC SC65C WG15 / IEC 57 WG HK Solutil

53 Doubly attached clock as Boundary Clock provide additional redundancy singly-attached OC4 can be synchronized over OC2 if port A of OC1 fails. grand master OC1 port A Port B master master OC4 SAN LAN_A LAN_B master (passive) master port A Port B slave Issues: can fault-independence of A and B still be guaranteed? slave OC2 Should simple sensors with limited processing power be able to host a boundary clock? 53 IEC SC65C WG15 / IEC 57 WG HK Solutil

54 Doubly-attached clock To fulfill the requirements of L3E2E, a new clock type was introduced, doubly attached clock, which behaves like a boundary clock, excepts that only one port synchronizes the slave clock, and the other port cannot become master. This same model can also be used by L2P2P It is described as a common solution in IEC Annex A. 54 IEC SC65C WG15 / IEC 57 WG HK Solutil

55 Doubly Attached Clock as Transparent Clock variant A doubly attached clock consists of an ordinary clock with two ports (in IEEE 1588, an ordinary clock has only one port) Conceptually, only one port synchronizes the slave clock. The slave can use for this the BMCA, but information from both ports can also be used. This is the general case. PRP is a particular case, when both MCA and MVB are the same MCA MCB region A region B portds.state = SLAVE port A Port B portds.state = PASSIVE_SLAVE OC slave 55 IEC SC65C WG15 / IEC 57 WG HK Solutil

56 Doubly Attached Clocks with same master one ordinary clock with two ports grandmaster MC portds.state = MASTER port A Port B portds.state = MASTER LAN_A LAN_B portds.state = SLAVE port A Port B portds.state = PASSIVE_SLAVE OC ordinary clock (slave) 56 IEC SC65C WG15 / IEC 57 WG HK Solutil

57 PRP-HSR RedBoxes as Boundary Clocks proven operation GMC Sync A Follow_Up A Follow_Up B Sync B LAN A LAN B interlink A interlink B RedBox A BC HC BC RedBox B Sync A HSR Ring Sync BA Sync BB B A B A B A B A HC HC HC HC 57 IEC SC65C WG15 / IEC 57 WG HK Solutil

58 Change with respect to 1588 The 2 OC plus HW clock and the two-port OC models do not exist in foresees that ordinary clocks have one port only. Boundary clocks have several ports, and for each port a state machine. Annex A adapted a two-port model for an ordinary clock. A doubly attached clock is therefore different from a boundary clock, although it is possible to achieve redundancy with a boundary clock. The only addition to the 1588 is the PASSIVE_SLAVE state, which is similar to the PASSIVE state, with the difference that the port in PASSIVE_SLAVE is supervising the path it is attached to. 58 IEC SC65C WG15 / IEC 57 WG HK Solutil

59 Changes to the BMCA and port model 59 IEC SC65C WG15 / IEC 57 WG HK Solutil

60 BMCA for BC with same master on both ports The application-specific criterion assess the clock quality before the port identity is used as tie-breaker. It can consider for instance the magnitude of the correction field, but should include a hysteresis to avoid frequent change of side. return B better than A Receiver < Sender compare identities of Receiver of A and Sender of A Receiver < Sender compare Steps A > B+1 Removed of B > A+1 A and B A > B A > B A within 1 of B compare Steps Removed of A and B A = B compare quality of A and B if available A = B B > A B > A return A better than B Receiver < Sender compare identities of Receiver of B and Sender of B Receiver < Sender Infrequent change is recommended to test the inactive path return B better by topology than A A > B A > B compare identifiers of senders A = B of A and B A = B B > A return A better by topology than B B > A IEEE does not have a time-out on Sync or Pdelay_Resp. compare Port Numbers of Receivers of A and B A = B 60 IEC SC65C WG15 / IEC 57 WG HK Solutil error-1 error-2 error-1

61 any FAULT_DETECTED POWER_UP any INITIALIZE any DESIGNATED_DISABLED FAULTY FAULT_CLEARED INITIALIZING READY DISABLED DESIGNATED_ENABLE D state machine extended by: PASSIVE_SLAVE state BMC_SLAVE BMC_PSLAVE LISTENING ANNOUNCE_TO BMC_MASTER PRE_MASTER BMC_SLAVE BMC_PSLAVE QUALIFICATION_TIM EOUT_EXPIRES UNCALIBRATED MASTER BMC_SLAVE && new_master!= old_master ANNOUNCE_TO MASTER_CLOCK_SELECTED BMC_PSLAVE BMC_SLAVE BMC_PASSIVE SLAVE SYNC_FAULT BMC_PSLAV E ANNOUNCE_TO PASSIVE_SLAVE BMC_SLAVE PASSIVE_MASTER BMC_SLAVE && new_master!= old_master BMC_SLAVE && new_master == old_master ANNOUNCE_TO BMC_SLAVE 61 IEC SC65C WG15 / IEC 57 WG HK Solutil

62 Why a change to the BMCA? The default BMCA in 1588 does not consider the case when two masters are the same, seen over different paths (since it assumes RSTP or IP would cut loops). The extended BMCA applies to the slave clock to chose one master over the other by considering its clock quality and correction field, all other fields being equal. The extended BMCA also applies to the master in end-to-end delay measurement. Indeed, only one port may be in the master state, responding to Delay_Req. The other is in the PASSIVE_MASTER state and does not respond to Delay_Req. Therefore, IEC introduces an additional port state and extends the comparison algorithm. This method is identical to IEC in the non-redundant case. IEC explicitly allows to define an alternate BMCA in a profile. 62 IEC SC65C WG15 / IEC 57 WG HK Solutil

63 REDBOXES Three models of Redboxes are considered: 1) Three-way Boundary Clocks 2) Doubly attached Boundary Clocks 3) Doubly attached Transparent Clocks 4) Stateless Transparent Clocks 63 IEC SC65C WG15 / IEC 57 WG HK Solutil

64 General naming of components 1) OC1: DAC connected to both LANs, as master; 2) OC2: DAC connected to both LANs, as slave; 3) OC3: SAC in one of the PRP LANs, as master; 4) OC4: SAC in one of the PRP LANs, as slave; 5) OC5: SAC outside of the PRP LAN, as master; 6) OC6: SAC outside of the PRP LANs, as slave; 7) BC7: RedBox M connected to the singly attached master clock OC5, as DABC; 8) BC8: RedBox S connected to the singly attached slave clock OC6, as DABC; 9) 9: RedBox M connected to the singly attached master clock OC5, as DA or SL; 10) 10: RedBox S connected to the singly attached slave clock OC6 as DA or SL. BC 7 9 LAN A 1 LAN B 3 OC 4 OC1 OC3 SAN master SAN master MASTER SLAVE port A 5 LAN C RedBox M DAC master 2 OC5 port C OC2 DAC slave port B MASTER PASSIVE_SLAVE OC4 SAN slave port A port B BC 8 RedBox S 10 port C LAN D 64 IEC SC65C WG15 / IEC 57 WG HK Solutil SAN slave 6 OC6

65 RedBoxes as Three-Way Boundary Clock that execute the BMCA and forward in all directions 65 IEC SC65C WG15 / IEC 57 WG HK Solutil

66 RedBoxes as Three-Way Boundary Clock - Principle SAN master OC5 PASSIVE OC3 is best master LAN C RedBox port C MASTER port A BC7 port B LAN A SLAVE DAC master OC OC1 MASTER LAN B OC3 MASTER SAN master SLAVE OC2 DAC slave PASSIVE SLAVE OC4 SAN slave port A port B RedBox S BC8 port C LAN D 66 IEC SC65C WG15 / IEC 57 WG HK Solutil SAN slave OC6

67 RedBox as Three-way Boundary Clock - The Redboxes behave as a three-way Boundary Clock as defined in IEC 61588:2009, with the addition of the consideration of the clock quality in case the same Master emerges on two ports. The RedBox at the same time can bridge LAN_A and LAN B and add to resiliency. However, this can introduce a single mode of failure. 67 IEC SC65C WG15 / IEC 57 WG HK Solutil

68 Redboxes as Doubly-Attached Boundary Clock that executes BMCA to select port A or port B 68 IEC SC65C WG15 / IEC 57 WG HK Solutil

69 RedBoxes as Doubly Attached Boundary Clock End-to-End LAN A and LAN B operate independently, the OCs chose the side, but Delay_Req and Delay_Resp are sent on both LANs The Boundary Clock decouples the PRP networks from the SANs This works for both peer-to-peer and end-to-end 69 IEC SC65C WG15 / IEC 57 WG HK Solutil Sync A Delay_Req A SAN master 7 Delay_Resp A MASTER RedBox M port A port C BC7 DAC master LAN A 1 LAN B 3 OC 4 OC1 2 OC3 OC2 OC4 SAN SAN master DAC slave SLAVE PASSIVE_SLAVE slave 8 port A SAN slave 5 LAN C RedBox S LAN D 6 OC5 BC8 port C OC6 port B port B Sync C Delay_Req C Delay_Resp C Delay_Req B Delay_Resp B Sync B MASTER Pdelay_Req D Pdelay_Resp D Sync D Pdelay_Req D Pdelay_Resp D

70 IEC RedBoxes Annexes as Doubly Attached Boundary Clock, Peer-to-Peer, 1-step LAN A and LAN B operate independently, the OCs chose the side. Pdelay_Req and Pdelay_Resp are sent on all links LAN A 1 LAN B 3 OC 4 OC1 OC3 SAN master Sync A SAN master Sync_C 7 MASTER SLAVE LAN C port A 5 RedBox M DAC master 2 OC5 port C BC7 OC2 DAC slave port B MASTER PASSIVE_SLAVE Pdelay_Req Pdelay_Resp Pdelay_Req Pdelay_Resp Sync B OC4 SAN slave port A port B 8 RedBox S BC8 port C 70 IEC SC65C WG15 / IEC 57 WG HK Solutil LAN D SAN slave 6 OC6 Sync_D Delay_Req Delay_Resp

71 RedBoxes as Doubly Attached Boundary Clock Why not use always Boundary Clocks? Some claim that boundary clocks in series cause loop instabilities. The theoretical limit seems to lie by 16 BCs in series. Evidence for this is scarce. However, in the worst case, there are only two boundary clocks in series. Although there is no strict necessity for this, it is attempted to model RedBoxes with transparent clocks also as a means to generalize the solution. A number of problems though exist. 71 IEC SC65C WG15 / IEC 57 WG HK Solutil

72 RedBoxes as Doubly-Attached Transparent Clocks that execute the BMCA to select A or B 72 IEC SC65C WG15 / IEC 57 WG HK Solutil

73 RedBoxes as Doubly-Attached Transparent Clocks General model 5 SAN master s provide a more robust connection of clocks than BCs OC 5 LAN C DA choose the best port according to the BMCA 9 RedBox M port A port C 9 port B DAC master 1 3 OC 4 OC1 LAN LAN A 2 B OC3 OC2 OC4 SAN master DAC slave SAN slave port A port B 10 RedBox S 10 port C LAN D 73 IEC SC65C WG15 / IEC 57 WG HK Solutil SAN slave OC6

74 RedBoxes as Doubly-Attached Transparent Clocks End-to-End, 1-step 5 SAN master Difficulty: keep the information about which path was taken in the redundant LAN for end-toend link delay measurement Delay_Req (src=10) Delay_Resp (src=00,req=10) 9 OC 5 port C LAN C Delay_Req (src=11) Delay_Resp (src=00,req=11) Trick: use the source port identity The four most significant bits of the source port identity are reserved, the number of ports per device reduced to 4096 Delay_Resp (src=00, req=00) DAC master 1 3 OC 4 LAN OC LAN A 1 2 B OC 3 SAN master Delay_Req (src=00) RedBox M port A port A 9 OC DAC 2 slave port B port B Delay_Resp (src=00,req=00) OC 4 SAN slave Delay_Req (src=00) Delay_Req (src=00) 10 1 port 0 C RedBox S Delay_Resp (src=00, req=00) Sync LAN D 74 IEC SC65C WG15 / IEC 57 WG HK Solutil SAN slave OC 6

75 RedBoxes as Doubly Attached Transparent Clock Stateful RedBox (DA) The RedBox operates as Transparent Clock with three ports: PRP Port A & Port B (paired), and SAN Port C. It does not transfer between Port A and Port B. Master side: When the RedBox receives a Sync or a Delay_Resp on port C, it duplicates it (after individual correction) to port A and port B. When a RedBox (master side) receives a Delay_Req from port A or port B, it forwards it over port C (there are twice as many Delay_Req on LAN C as in the non-redundant case). It however tags the Delay_Req as coming from A or B by setting the two most significant bits to 10 resp. 11. When the RedBox receives a Delay_Resp from port C, it sends it over the port from which it received the Delay_Req. Slave side: When the RedBox receives a Sync from its best port A or B, it forwards it (corrected) to port C. When a RedBox (slave side) receives a Delay_Resp on its best port A or B, it forwards it (after correction) to port C. It discards the Delay_Resp that comes over the other port, but nevertheless registers its arrival to check the redundancy. 75 IEC SC65C WG15 / IEC 57 WG HK Solutil

76 RedBoxes as Doubly Attached Transparent Clock End-to-End (1-step or 2-step) ordinary (slave) clock D link D D RedBox S AD, BD DA, DB A B A B link A link B RedBox M AC, BC CA, CB C link C C master clock Selection of Sync and Delay_Resp done by BMCA at reception (RedBox S) t 2 t 3 D3 Sync D Delay_Req t Ams D B A Sync A Sync B B A C Sync C (Delay_Req and Delay_Resp tagged on LAN C) t 1 Delay_Req does not indicate over which LAN RedBox S accepted Sync t Asm Delay_Req (10) Delay_Req Delay_Req (11) t 4 Delay_Resp B Delay_Resp (00, 11) Delay_Resp A (t 4, sm ) Delay_Resp A Delay_Resp (00, 10) (source, request) 76 IEC SC65C WG15 / IEC 57 WG HK Solutil

77 RedBoxes as Doubly Attached Transparent Clock Peer-to-Peer, 1-step Sync Pdelay_Resp Pdelay_Req SAN master 9 LAN C 5 RedBox M OC5 port C Pdelay_Req Pdelay_Resp Sync C Pdelay_Req Pdelay_Resp 9 port A port B The forwards only the Sync that it received from its best master port (DAC master) LAN A 1 LAN B 3 OC OC1 4 OC3 (SAN master) 2 OC2 DAC slave OC4 SAN slave Sync A port A port B Sync B 10 RedBox S 10 port C 77 IEC SC65C WG15 / IEC 57 WG HK Solutil Sync C SAN slave 6 OC6 LAN D Pdelay_Req Pdelay_Resp Pdelay_Req Pdelay_Resp

78 RedBoxes as Doubly Attached Transparent Clock Peer-to-Peer (1-step) ordinary (slave) clock D link D D RedBox S AD, BD DA, DB A B A B link A link B RedBox M AC, BC CA, CB C link C C master clock D B A B A Sync A (t 1, Ams + Ams ) C Sync C (t 1, 0) CA t 1 Sync D (t 1, ms + Ams ) t Ams CB t 2 Sync B (t 1, Bms + Bms ) Pdelay_Req A Pdelay_Req D Pdelay_Req C Pdelay_Resp D Pdelay_Resp A Pdelay_Req B Pdelay_Resp C Pdelay_Resp B 78 IEC SC65C WG15 / IEC 57 WG HK Solutil

79 REDBOXES - Stateless Transparent Clock 79 IEC SC65C WG15 / IEC 57 WG HK Solutil

80 RedBoxes as Stateless Transparent Clock (SL) The RedBox operates as Transparent Clock Fork with three ports: PRP Port A and Port B (paired), SAN Port C. It does not transfer between Port A and Port B. Master side: When the RedBox receives a Sync or a Delay_Resp on port C, it duplicates it (after individual correction) to port A and port B. When a RedBox (master side) receives a Delay_Req from port A or port B, it forwards it over port C (there are twice as many Delay_Req on LAN C as in the non-redundant case). It however tags the Delay_Req as coming from A or B by setting the two most significant bits to 10 resp. 11. When the RedBox receives a Delay_Resp from port C, it sends it over the port from which it received the Delay_Req. Slave side: When the RedBox receives a Sync from its best port A or B, it forwards it (corrected) to port C. When a RedBox (slave side) receives a Delay_Resp on its best port A or B, it forwards it (after correction) to port C. It discards the Delay_Resp that comes over the other port, but nevertheless registers its arrival to check the redundancy. 80 IEC SC65C WG15 / IEC 57 WG HK Solutil

81 RedBox as Stateless Transparent Clock, End-to-end Sync (00) Delay_Req (10) Delay_Resp (00,10) 5 OC5 SAN master LAN C Delay_Req (11) Delay_Resp (00,11) 9 port C RedBox M 9 port A port B MASTER MASTE DAC master R 1 3 OC 4 LAN OC1 LAN A 2 B OC3 OC4 SAN master SLAVE 10 port A RedBox S OC2 DAC slave 10 port C port B PASSIVE_SLAVE SAN slave Delay_Req (10) Delay_Req (11) Delay_Req (00) Delay_Resp (tagged as A ) Sync (tagged as A ) LAN D Delay_Resp (tagged as B ) Sync (tagged as B ) 81 IEC SC65C WG15 / IEC 57 WG HK Solutil SAN slave OC6

82 RedBox as Stateless, End-to-End ordinary (slave) clock D link D D RedBox S AD, BD DA, DB A B A B link A link B RedBox M AC, BC CA, CB C link C C master clock D B A B A C Sync C (00) t 1 tagged as A Sync B (00) Sync (11) t 2 Sync A 00) t D2 t 3 D3 Sync (10) Delay_Req (00) Delay_Req (src=10) t Asm does not indicate over which LAN Sync was accepted Delay_Req (src=11) Delay_Req (src=10) Delay_Req (src=11) t 4 Delay_Resp (11, 00) Delay_Resp (10, 00) Delay_Resp (src=00, req=11) Delay_Resp (00, 11) Delay_Resp (00, 10) Delay_Resp (src=00, req=10) Delay_Resp (00, 11) Delay_Resp (src=00, req=10) (source, request) 82 IEC SC65C WG15 / IEC 57 WG HK Solutil

83 Stateless RedBox When transparent clocks operate in cut-through, it is not possible to chose the port over which transmission should take place. once transmission began. Therefore, stateless RedBoxes cannot rely on the BMCA to chose the best port, selection must be done at reception 83 IEC SC65C WG15 / IEC 57 WG HK Solutil

84 RedBox for HSR 84 IEC SC65C WG15 / IEC 57 WG HK Solutil

85 RedBox as Transparent Clock To HSR GMC Follow_Up A Sync 2-step Announce RedBox Sync1Step A HSR Sync1Step B Redbox does the transition from 2-step to 1-step. works, but not used 85 IEC SC65C WG15 / IEC 57 WG HK Solutil

86 RedBoxes as Transparent Clocks, PRP-HSR: not any more considered! no implementation yet issue: four Syncs in the ring, GMC Sync A Follow_Up A Follow_Up B Sync B LAN A LAN B interlink A interlink B RedBox A HC RedBox B Sync A Sync AA HSR Ring Sync AB Sync BA Sync BB B A B A B A B A HC HC HC HC 86 IEC SC65C WG15 / IEC 57 WG HK Solutil

87 Changes to the clock model of 1588 and MIB 87 IEC SC65C WG15 / IEC 57 WG HK Solutil

88 Doubly-attached clock model state machine BMCA state machine timestamp timestamp 88 IEC SC65C WG15 / IEC 57 WG HK Solutil

89 IEEE 1588 clock model: datasets for Boundary Clock and doubly-attached clock defaultds about this clock (quality, etc..) per clock timepropertiesds about received time information (U,..) parentds about whom the time is received from 1588 defines the state machine per port. The port in the SLAVE state controls the local clock per port foreignmasterds about all discovered masters currentds quality of received time INITIALIZING FAULTY DISABLED LISTENING PRE_MASTER MASTER PASSIVE UNCALIBRATED SLAVE Port A INITIALIZING FAULTY DISABLED LISTENING PRE_MASTER MASTER PASSIVE UNCALIBRATED SLAVE Port B 89 IEC SC65C WG15 / IEC 57 WG HK Solutil

90 Alternative It was proposed to introduce the concept of using two ordinary clocks with each one port, both controlling in an application-dependent way the local clock. Although one can implement the doubly-attached clock this way, it is not practicable for modelling since the dependencies between the ports cannot be expressed. In particular, the application cannot see from which master the clock is taken and over which path it is synchronized. This introduces a layer of redundancy which is different from application to application. Also, such a double OC model does not cover the simple model of a boundary clock. 90 IEC SC65C WG15 / IEC 57 WG HK Solutil

91 IEEE 1588 Datasets details defaultds: /* ordinary clock */ - twostepflag BOOL - clockidentity: Octet [8] - numberports: UInt 16 - clockquality.clockclass ENUM8 /* 6 = synch ed to ref */.clockaccuracy UINT8 /* 25ns, 100 ns */.offsetscaledlogvariance Uint16 - priority1 UINT8 - priority2 UINT8 - domainnumber UINT8 - slaveonly BOOL parentds /* master and grandmaster */ parentportidentity Octet [8] - parentstats Bool /* below are valid */ - observedparentoffsetscaledlogvariance /* */ observedparentclockphasechangerate /* */ - grandmasteridentity Octet [8] - grandmasterclockquality Strct32 - grandmasterpriority1 Uint8 - grandmasterpriority2 Uint8 currentds /* synchronization state */ - clockstepsremoved - offsetfrommaster - meanpathdelay timepropertiesds currentutcoffset /* offset between U and TAI in s */ currentutcoffsetvalid Bool /* currentutcoffset valid */ leap59 Bool leap61 Bool timetraceable Bool frequencytraceable Bool ptptimescale Bool timesource Enum8 /* atomic, GPS, */ portds /* per port */ - portidentity Octet [10] - portstate Enum8 - logmindelay_reqinterval Int8 - peermeanpathdelay Int64 - logannounceinterval Int8 - announcereceipttimeout Int8 - logsyncinterval Int8 - delaymechanism Bool - logminpdelay_reqinterval Int8 - versionnumber Uint8 foreignmasterds /* for each <= 5 master */ - foreignmasterportidentity - foreignmasterannouncemessages 91 IEC SC65C WG15 / IEC 57 WG HK Solutil

92 MIB The MIB for IEC doubly attached clocks are derived from the IEEE 1588 object model Three additions were necessary: 1) Indicate the profile 2) indicate which other port is paired for PRP 3) introduce the PASSIVE_SLAVE in the port status enumeration These changes are already inserted in IEC object model or will be amended when moving to IEC IEC SC65C WG15 / IEC 57 WG HK Solutil

93 MIB basic structure ieee1588base OcBc Tc TLVs DefaultDs CurrentDs ParentDs TimePropDs ports PortDS tcdefaultds tcports PortDS additional objects: net Protocol, profile ID, additional objects: Paired, DelayAsymm, PTP enabled, additional objects: profile ID, missing objects: Paired, DelayAsymm, PTP enabled, new TLV1: MANAGEMENT TLV9: ALTERNATE_TIME_OFFSET_INDICATOR 93 IEC SC65C WG15 / IEC 57 WG HK Solutil

94 MIB tree 1588 PortDS addition for power profile addition for redundancy 94 IEC SC65C WG15 / IEC 57 WG HK Solutil

95 Other changes This MIB can be used by all IEC x variants. Therefore, it is proposed to append the MIB not to IEC , but as a general MIB as IEC IEC SC65C WG15 / IEC 57 WG HK Solutil