Understanding FICON Channel Path Metrics

Transcription

1 Understanding FICON Channel Path Metrics Dr.H.PatArtis Performance Associates, Inc. PAI/O Driver is a registered trademark of Performance Associates, Inc. Performance Associates, Inc., 2003.

2 Topics Warning notice Key references The problem What do FICON channel and bus busy mean? Correlation analysis Estimating the number of active open exchanges FICON channel operations What happens at saturation? FICON channel steam gauge Questions

3 Warning Notice This session includes discussions of both queueing theory and multivariate statistical analysis techniques. The Surgeon General has warned that sessions which contain these topics, for no apparent reason, may cause spatial disorientation, confusion, and nausea in the audience.

4 Key References Cronin, C., FICON and FICON Express Channel Performance Version 1, February 2002, ftp://ftp.software.ibm.com/software/mktsupport/techdocs/ficonpe rfv1_0a.pdf Cronin, C., FICON and FICON Express Channel Performance Version 2, November 2003, 1.ibm.com/servers/eserver/zseries/library/techpapers/pdf/gm pdf Tukey, J. W., Exploratory Data Analysis, Addison Wesley, SAS/Stat User s Guide, Version 6, Fourth Edition, Vol. 1, Allen, A.O., Probability, Statistics, and Queueing Theory, Academic Press, 1978.

5 The Problem Half Track Read/Write: 90% Read-Hit Service Time SSCH Rate vs Service Time Service Time (msec) % Bus 44.5% Chan % Bus & Chan Busy 30.2% Chan 26.8% Bus % Bus & Chan Busy ,000 1,250 1,500 1,750 2,000 PAI/O Driver for OS/390 Data FMPLRESP.GMF SSCH Rate per Channel

6 The Problem (Cont.) For ESCON, channel utilization was a reliable steam gauge metric with proven rules of thumb for capacity planners and performance analysts With FICON channels, your service time can go to Pluto even though both of the available steam gauge utilization metrics are apparently within acceptable values Does either channel busy or bus busy provide an effective predictor of FICON channel saturation? The question this paper explores is not whether or not FICON is a good idea, it is! Rather, it is simply the question of how to manage a FICON environment

7 Do You Need a Utilization Metric? Captain, I am giving her all she s got and she is goingtoblow!

8 What Do Channel and Bus Busy Mean? Channel Busy is the measured utilization of the PowerPC microprocessor that manages the fibre channel protocol. We assert that microprocessor utilization is proportional to the total number of I/Os Bus Busy is the measured utilization of the PCI bus over which the data is transferred. We assert that bus utilization is proportional to the aggregate data transfer rate How can we test these and other assertions?

9 Correlation Analysis Correlation Coefficients Strong Weak Weak Strong Negative Correlation Positive Correlation -1.0 CORRCOEF.GMF Correlation analysis is a technique employed in exploratory data analysis to determine the relationship between variables Coefficients that approach 1.0 or 1.0 indicate that a strong positive or negative correlation exists between two variables The analysis presented in this session was performed with SAS PROC CANCORR

10 Channel Metrics Correlation Analysis Variable I/O Rate Channel MB/Sec % Channel Busy % Bus Busy Correlation Coefficients for Channel and Bus Busy SAS PROC CANCORR was employed to determine the correlation coefficients for channel and bus busy with the aggregate I/O and data rates for an entire PAI/O Driver for OS/390 test series for a supported by eight FICON Express channels The strong positive correlation coefficients verify the assertions we made about the two metrics So why aren t these reliable metrics for predicting service time problems? PROC CANCORR DATA=CHANDATA CORR VNAME='CHANNEL METRICS WNAME='I/O THRUPUT METRICS'; VAR PCTCHNBY PCTBUSBY; WITH IORATE CHANMBS;

11 Channel Metrics at Test Saturation Test % Chan % Bus Read Write Series Busy Busy MB/Sec MB/Sec 4K Read-Hit K R/W: 50% Hit K R/W: 90% Hit HT Read-Hit HT R/W: 50% Hit HT R/W: 90% Hit HT Read Sequential HT Write Sequential Channel Metrics at Saturation by PAI/O Driver Test Series Do any of these metrics correlate with measured service time or tell you when a channel is saturated?

12 Correlations to Subsystem Service Time Test % Chan % Bus Read Write Series Busy Busy MB/Sec MB/Sec 4K Read-Hit K R/W: 50% Hit K R/W: 90% Hit HT Read-Hit HT R/W: 50% Hit HT R/W: 90% Hit HT Read Sequential HT Write Sequential All Tests Channel Metrics and Subsystem Response Time Correlation Coefficients While some of the correlation coefficients are promising on a test by test basis, none of the metrics provide more than weak correlations to subsystem service time for all of the workload types! Are there other logical or physical resources for which we can develop metrics?

13 Other Channel Resources AnOpen Exchange (OE) is a logical resource that represents an I/O being processed by a channel. A FICON channel is limited to 32 concurrent OEs An Information Unit (IU) is a logical resource, comprised of 1 to 4 credits, which may contain a CCW, a CCW and associated data, or just data. A FICON channel can support a maximum of 16 concurrent IUs for each OE ACredit is a 2K physical data transmission buffer. According to the November 2003 revision of Cronin s paper, FICON and FICON Express channels can support a maximum of 60 and 120 credits, respectively, for each of the inbound and outbound fibres

14 IUs and Credits for a Chained Write IU 09 IU 08 IU ,998 bytes IU Write Data IU 05 IU 04 IU 03 IU Write Data 27,998 bytes IU Prefix FCNXFCCW.GMF Direction of Flow Consider an OE for a chained write of 2 half track blocks The OE requires 9 IUs and 29 credits

15 Estimating the Number of Active OEs Little s Law states that the multiprogramming level of a system is equal to the arrival rate times the average time in the system. Hence, we can estimate the number of active OEs: Active_OEs = Chan_I/O_Rate * OE_Duration For the OE_Duration, we will use the sum of the average DISC and CONN times for the I/Os performed by the channel Active_OEs = Channel_I/O_Rate * (DISC + CONN) Why do we exclude PEND time from OE_Duration?

16 Channel Metrics and IUs at Saturation Test % Chan % Bus Read Write Active Series Busy Busy MB/Sec MB/Sec OEs 4K Read-Hit K R/W: 50% Hit K R/W: 90% Hit HT Read-Hit HT R/W: 50% Hit HT R/W: 90% Hit HT Read Sequential HT Write Sequential Channel Metrics and OEs at Saturation by PAI/O Driver Test Series The number of active OEs is estimated using Little s Law How does the number of active OEs correlate to service time?

17 Correlating OEs to Service Time Test % Chan % Bus Read Write Active Series Busy Busy MB/Sec MB/Sec OEs 4K Read-Hit K R/W: 50% Hit K R/W: 90% Hit HT Read-Hit HT R/W: 50% Hit HT R/W: 90% Hit HT Read Sequential HT Write Sequential All Tests Channel Metrics and Subsystem Service Time Correlation Coefficients

18 Correlating OEs to Service Time (Cont.) With the exception of the 4K read-hit case, which saturated the channel microprocessor, very high correlations were observed for each of the workload types Even with the 4K read-hit case, the correlation coefficient to subsystem service time for all workloads was a strong 0.85 for direct channel to subsystem connections But can this explain why the service time went to Pluto?

19 Why Did The Service Time Go to Pluto? Half Track R/W: 90% Read Hit Service Time Open Exchanges vs Service Time Service Time (msec) SSCH Rate = SSCH Rate = PAI/O Driver for OS/390 Data FMPLJUMP.GMF Open Exchanges At a rate of 1,760 I/Os per second, the average number of OEs was about 8 and the service time was about 5 msec. When the service time jumped to 30 msec, then the number of OEs approached the limit imposed by the protocol The jump occurred since the half track blocks exhausted the credits and/or other underlying resources, elongating the average CONN and PEND times

20 Normal FICON Channel Operation SSCH SubChannels ORB IBM OE + 1 IBM OE < 32 FICON Channel IU: First CCW Information Unit (IU) FCHNSTP1.GMF Prior to OE saturation, when an ORB arrives at a subchannel it is immediately selected by a FICON channel and the concurrent OE count for the channel is increased by 1 The channel program and any associated data is then transmitted by the FICON channel as a sequence of IUs, each of which is comprised of 1 to 4 credits

21 Normal FICON Channel Operation (Cont.) Interrupt SubChannels ORB IBM FICON Channel OE - 1 OE<32 IU: Final Status IBM Information Unit (IU) FCHNSTP2.GMF When final status for the I/O arrives at the channel, the concurrent OE count for the channel is decreased by 1 and an interrupt is passed back to the address specified in the ORB The I/O request is now complete But what causes the explosive growth of PEND and CONN time at saturation?

22 Why Does Saturation Occur? There are a limitless number of scenarios that can result in OE saturation. Hence, let s discuss the pathological cases for small and large block transfers: For unchained operations with very small blocks, it is apparently possible to start 32 concurrent I/O operations. There would be one or two credits available to each of the I/O operations on a standard FICON channel. The value would be one to eight for a FICON Express channel For chained operations with large blocks, credit and/or other underlying resource starvation would significantly elongate PEND and CONN time for active I/Os. While the channel can accept 32 concurrent I/O operations, many of the I/Os would queue within the channel waiting for credits and/or other underlying resources

23 FICON Channel Operation at Saturation SSCH SubChannels ORB IBM SSCH ORB IBM SSCH SSCH SSCH ORB ORB ORB OE = 32 Credits FICON Channel SSCH ORB Information Unit (IU) FCHNSTP3.GMF When the OE count reaches 32, I/O requests queue in the subchannels waiting to be selected by a channel, just like they do when an ESCON channel is busy

24 FICON Channel Operation (Cont.) This queueing is reported as part of PEND time PEND time was excluded from the OE_Duration since an I/O can be PENDing outside the channel as well as PENDing within the channel for credits and/or other underlying resources as part of an OE The explosive growth in PEND time occurs as I/O requests queue in subchannels waiting on an available OE As the number of OEs increases, the channel exhausts its available credits and/or other underlying resources. This results in substantial CONN and PEND time elongation. After this point, the OEs just queue in the channel waiting on credits and/or other underlying resources This elongation exacerbates the saturation problem by decreasing the rate at which OEs are freed

25 What Happens to the Components? Half Track Read/Write: 90% Read-Hit Service Time Components Open Exchanges vs Service Time Service Time Components (msec) PEND DISC CONN SSCH Rate = 1,760 SSCH Rate = 1, PAI/O Driver for OS/390 Data FMPLCOMP.GMF Open Exchanges As can be seen in the figure, the average PEND and DISC times are almost constant through 1,760 I/Os per second

26 What Happens to the Components? (Cont.) Prior to 1,760 I/Os per second, there is approximately a 3:1 elongation in CONN time This initial CONN time elongation is a result of channel microprocessor utilization, subsystem interface microprocessor utilization, and interleaving delays on the fibre Then, as the number of OEs approaches 8, the credits and/or other underlying resources saturate and the PEND and CONN times explode!

27 Other Test Cases 4K Read/Write 90% Cache-Hit case that employed 10 open exchanges 4K Read/Write 50% Cache-Hit case that employed 31.5 open exchanges

28 4K Read/Write: 90% Read-Hit Case 4K Read/Write: 90% Read Hit Service Time Components Open Exchanges vs Service Time Service Time Components (msec) PEND DISC CONN Subsystem Saturation PAI/O Driver for OS/390 Data FMPL4KE1.GMF Open Exchanges For this case, the service time of the subsystem begins to increase after the OE count exceeded 8 However, the knee of the curve of the channel coincided with the internal saturation of the subsystem

29 4K Read/Write: 50% Read-Hit Case 4K Read/Write: 50% Read Hit Service Time Components Open Exchanges vs Service Time Service Time Components (msec) PEND DISC CONN Subsystem Saturation PAI/O Driver for OS/390 Data FMPL4KE0.GMF Open Exchanges For this case, OE saturation is induced by the low cache-hit ratio and the subsystem s back end FBA device configuration However, substantial CONN elongation still starts in the range of 6 to 8 OEs

30 Closing Thoughts FICON channel resource saturation FICON OE steam gauge Switched and cascaded configurations

31 FICON Channel Resource Saturation In the case of 100% cache-hit for small blocks, the channel microprocessor can saturate, resulting in very high PEND times At very high I/O rates for large and mixed blocksizes, OE saturation can result in explosive PEND and CONN time elongation As the utilization of the fibre increases, CONN time elongation can be expected. At the limit, credit and/or other resource saturation results in explosive PEND and CONN time elongation

32 FICON OE Steam Gauge 2 4 FICON Open Exchanges Danger FCNGAUGE.GMF We recommend operating directly connected channels in the 3 to5oerange.this will allow OE intensive online applications and OE/Credit intensive batch workloads to be intermixed on a channel. In most real world environments, FICON channels operate at OEs values less than 2

33 FICON OE Steam Gauge (Cont.) With the exception of the very high cache-hit ratio test cases that saturated the channel microprocessor, the jump in the number of active OEs occurred when the value exceeded 7. Then, the credits and/or other underlying resources become saturated and the number of OEs takes a step function jump to 30+ The number of OEs is very highly correlated with service time. While highly correlated with I/Os and data transfer, channel and bus busy have only weak correlations to service time In all but the pathological small block front end hit case, service time degrades as a result of the saturation of physical credits and/or other underlying resources; not as a result of physical channel or bus utilization Channel and subsystem OEs are not RMF reported metrics. They must be estimated using the average DISC and CONN times for the devices supported by the channel

34 Switched and Cascaded Configurations Switched Configuration IBM 2 4 FICON Open Exchanges 6 IBM Danger FICON Open Exchanges FICON Open Exchanges 8 Danger 6 IBM Danger 0 8 Cascaded Configuration IBM IBM 2 4 FICON Open Exchanges FICON Open Exchanges 6 IBM Danger FICON Open Exchanges FICON Open Exchanges 8 Danger Danger FICON Open Exchanges 6 IBM Danger Danger FCHCMPLX.GMF

35 Switched and Cascaded (Cont.) At the current time, we have not completed the analysis of the data for switched and cascaded FICON directors. The direct connect channel utilization analysis was a much bigger job than we anticipated. Those results will be included in an expanded version of the paper which will be posted on our web site during 2004 The complete experimental design is available at:

36 Questions