OB30 paper 07 KPI Report (August 2015) KPI and SLA regime: August 2015 performance summary Ref Jun 15 Jul 15 Aug 15 Target Description KPI A 100% 100% 99.87% 99% green 98% amber Service Restoration within 10 working days where household is a primary DTT user KPI A1 99.22% 99.20% 98.49% 97.5% green Engineer visits completed as scheduled with viewer 95.0% amber KPI A2 98.52% 97.16% 94.63% 90% green Engineer visits completed within 3 working days 50% amber KPI A4 99.10% 97.01% 94.25% 90% green Vulnerable visits completed within 3 working days 70% amber KPI A6 100% 100% Communal engineer visits completed as scheduled with landlord where access and liability is confirmed KPI A7 100% 98.39% 92.24% 98% green Reactive filters issued within 3 working days 94% amber [KPI A10] [SUSPENDED FOR REACTIVE-ONLY TRIAL] Addresses in very high pixels mailed filters SLA B1 40.7% 60.5% 86.7% 100% green 99% amber Identifiable addresses within forecast medium, high and very high risk pixels (high and very high risk only for London) to be mailed at least once no more than 12 weeks ahead of scheduled mast activation SLA C1 0.39% 0.14% 0.21% <1% green Complaints <2% amber SLA C4 Report on total number and categories of complaints SLA C5 Report on vulnerable complaints SLA D1 Measure of the numbers of mailings per report of / SLA D2 SLA D3 Qualitative and quantitative reports provided. No specific targets. case of interference Measure of the number of reported cases that occur outside of mailed area Measure of number of cases of interference per mast remaining between 0.17 and 1.66 (5k to 50k cases of interference for rollout) Page 1 of 11
KPIs August 2015: KPI and SLA performance summary, 9 September 2015 KPI A: Service Restoration within 10 working days where household is a primary DTT user (communal households and households where cable or satellite services are received are excluded). 99% (green); 98% (amber) Result: 99.87%; green Of the 787 confirmed 4G cases in August 2015, 786 were resolved within 10 working days. The case that breached is expected to take 24 working days to resolve, and is summarised below Result: 99.87% Green. In the case that breached, a cherry-picker is required to complete the job. The council needed to approve and arrange traffic management to suspend parking bays outside the viewer s home, for which they have a two- to four-week lead time. This has delayed the time to resolution. The viewer is on holiday for two weeks from 5 to 19 September and the job has been booked for 22 September. The viewer understands the reason for the delay and appreciates the regular updates. Excluding the viewer s holiday and requests for the viewer to delay appointments, the total journey length is expected to be 24 days, assuming the issue is resolved on 22 September. It is being reported in the August report because it became a breach before the end of August. KPI A1: Engineer visits completed as scheduled with viewer. 97.5% (green); 95% (amber) Result: 98.49%; green Of the 2,325 engineer visits originally scheduled for August 2015: o A total of 2,128 were undertaken and closed on schedule o Thirty-five visits were cancelled and rearranged by at800 to meet capacity constraints o A further 162 visits were cancelled by the viewer Total scheduled visits not missed by engineer / total visits scheduled (2,128 + 162) / 2,325 Result: 98.49% Green. KPI A2: Engineer visits completed within 3 working days. 90% (green); 50% (amber) Result: 94.63%; green Of the 2,325 scheduled engineer visits detailed above, there were 2,163 visits scheduled where the viewer did not cancel Of these 2,163 visits: o A total of 1,867 visits were completed within three working days o A further 180 were delayed beyond three working days at the viewer s request o Thirty-five visits were cancelled or rejected by the engineer o Eighty-one visits were completed outside of the three working day target (1,867 + 180) / 2,163 Result: 94.63% Green. KPIs Page 2 of 11
KPI A4: Vulnerable visits completed within 3 working days: 90% (green); 70% (amber) Result: 94.25%; green In August 2015, there were 1,078 scheduled engineer visits to people marked as vulnerable where the viewer did not cancel Of the 1,078 visits to be measured: o A total of 961 visits were completed within three working days o Fifty-five visits were delayed beyond the three working day window at the viewer s request o Fifteen visits were cancelled or rejected by the engineer o Forty-seven visits were delayed beyond the three working day target (961 + 55) / 1,078 Result: 94.25% Green. KPI A6: Communal engineer visits completed as scheduled with landlord where access and liability is confirmed. Reporting only Result: 100%; green All of the seven communal jobs that were scheduled to take place in August 2015 were undertaken on time and as scheduled Result: 100% Green. KPI A7: Reactive filters issued in 3 working days. 98% (green); 94% (amber) Result: 92.24%; red There were 3,916 filter requests in August 2015 Of these, 3,612 were sent within three working days Result: 92.24% Red. A total of 304 filters were issued outside of three working days during August. The management information that is used to track stock showed adequate volumes of each filter type prepacked and ready for labelling/posting. This reporting was incorrect and while there was sufficient stock of channel 60 indoor filters, none of the remaining stock had been prepacked. Instead, the filters were in raw stock pallets or engineers packs. The warehouse issued a request to prepack more filters which was missed by our supplier and this resulted in the channel 60 filters being put on hold by the warehouse and orders not being fulfilled. This issue was not initially picked up by at800 as the supplier sent confirmation files indicating that these filter requests had been fulfilled. The warehouse has an automated process for loading completion files in bulk and manually inputting/deleting dates for non-completed filter orders. The warehouse did not follow this process for channel 60 filters, and this meant at800 received daily confirmations of the filters being sent as normal. The issue was identified when at800 witnessed a spike in the number of calls into the contact centre from viewers saying they had not received a filter. As soon as the issue was identified, steps were taken to send out the filters that had not been sent, and to address the root cause of the problem. KPIs Page 3 of 11
The supplier has apologised for this issue and has changed internal processes as a result. Following the updated process, at800 now receives an email every day from the warehouse showing the number of filter orders dispatched from the warehouse in addition to the daily completion files. By the end of September, the warehouse will individually scan and process every filter rather than completing orders in bulk. This will remove the reliance on an individual to remember to manually intervene if an order cannot be fulfilled. A full stock count has been carried out by the supplier and the management information has been corrected. An escalation process has been agreed between all parties allowing any issues to be identified in a more timely manner. at800 is also in the process of carrying out a full stock count of all filters. This began in July and will continue with one stock count per month for the remainder of 2015. Channel 60 indoor filters will be counted on 17 September. KPI A10 [SUSPENDED]: 100% of identifiable addresses in very high pixels mailed proactive filter Not applicable. KPIs Page 4 of 11
SLA B Awareness August 2015: KPI and SLA performance summary, 9 September 2015 SLA B1: Identifiable addresses within forecast medium, high and very high-risk pixels (high and very high risk only for London) to be mailed at least once no more than 12 weeks ahead of scheduled mast activation. 100% (green); 99% (amber) Result: 86.7%; red A total of 282,528 households were identified as being eligible for mailing because of September 2015 rollout masts. These were all scheduled to be mailed in August, but a request was made to activate some of these masts before the beginning of September. As such, 17,035 of the households were mailed in July. The remaining 265,493 households were mailed in August and are reported in the table below A special request was made to activate a September rollout mast early. A special mailing file was created containing the 402 households that were impacted by the mast. Although the operator was informed that the mailing would be sent on 10 August (landing on 11 August), the operator activated the mast on 6 August As described in the July report, an operational issue was discovered in early July that identified 636,307 households that had not been mailed, despite their risk score making them eligible for mailing. These were mailed between 15 July and 17 August. The 304,345 households that were mailed in August are reported under SLA B2 below It should be noted that this SLA measures mailing against the originally scheduled monthly mast activation submission. Only around 7% of the masts generating the change in pixel risk, and therefore these mailing batches, had activated by 10 July. Therefore for the majority of recipients, the postcards will arrive closer to a mast activation date than it otherwise would have Of the above, 74,643 related to masts scheduled to be activated in August 2015 Of the 563,371 mailing issued in August, 75,045 were not sent in advance of the potential activation date of the masts with which they are associated Result for August: 86.7% Red. SLA B2: Report on households mailed and not mailed within forecast pixels (i.e. number of low risk households) The following table shows the breakdown of households by risk category and those confirmed as mailed. Very London Grand Mailing Mailing Date high High Medium medium Low Total Special mailing 10 August 251 151 402 September 2015* 17 August 1,256 12,634 12,233 26,123 18 August 1,511 16,140 19,114 36,765 19 August 482 15,412 20,871 36,765 20 August 368 12,432 23,963 36,763 21 August 393 14,085 22,287 36,765 24 August 394 12,008 24,363 36,765 25 August 890 16,216 19,656 36,762 26 August 74 7,104 11,607 18,785 Not mailed 60,987 336,096 397,083 SLA B: Awareness Page 5 of 11
Mailing Mailing Date Very high High Medium London medium Low Grand Total 3 August 46 14,151 15,800 29,997 4 August 9,271 20,729 30,000 5 August 1 8,330 21,669 30,000 6 August 32 14,237 14,170 28,439 Special missed 7 August 266 11,172 20,030 31,468 mailing** 10 August 30 12,132 17,838 30,000 11 August 9 10,710 19,281 30,000 12 August 20 12,155 16,820 28,995 13 August 26 12,539 17,435 30,000 14 August 18 10,194 18,365 28,577 Total 5,816 221,173 336,382 60,987 336,096 960,454 * This includes the unmailed volume for the entire September batch, including those that were mailed in July. ** Unmailed volumes have not been reported for the missed mailings. Note that from the August 2015 rollout mailing onwards, five seed addresses of people within the at800 organisation are being added to the mailing files. This is to introduce an additional operational check that mailings are being delivered. These five postcards will not be included in the numbers reported, unless the addresses are also identified as being eligible for mailings by the interference modelling. SLA B3: Report on households reporting interference within unmailed forecast pixels. More than 40% of total reports of interference at800 to address through operational adjustments (e.g. mailing low risk pixels in a certain area or adjusting the risk score for pixels), however, a de minimis threshold to apply In August 2015, at800 was contacted by a total of 4,964 households reporting interference that at800 believed could be caused by 4G. Engineers visited 2,175 properties, and filters were sent to a further 2,789. Of the 2,175 viewers to whom at800 sent engineers, 796 were confirmed as being impacted by 4G Of the 4,964 households, 366 (7.4%) were suppressed through the risk-based mailing. Mailing Status 4G confirmed Not 4G Job not complete Filter sent Grand Total Mailed 758 1,255 23 2,474 4,510 Suppressed 35 97 0 234 366 Not identified 3 4 0 81 88 Total 796 1,356 23 2,789 4,964 SLA B4: Awareness to be maintained (reminder mailings) with those viewers at continued risk of experiencing interference from masts that were not activated as scheduled. Report on reminder mailing numbers to be provided No reminder mailings were sent in August. The reminder mailing associated with masts originally expected to activate in February 2015 will be sent in September and will be reported in next month s report. SLA B: Awareness Page 6 of 11
SLA B7: Report on above the line communications and research outcomes, including awareness in rollout areas Seventeen pieces of media coverage appeared during August across print and online outlets. Fourteen online articles and three in print communicated at800 s messaging to a potential audience of around 225,000 people. Some examples of coverage are detailed below. Wilts and Gloucestershire Standard: New 4G mobile internet tower leaves Cirencester residents without TV for days Ballymena & Antrim Times: Solution for Freeview signal issues Henley Herald: Have your say 4G causing TV interference? SLA B: Awareness Page 7 of 11
SLA C Quality of Service August 2015: KPI and SLA performance summary, 9 September 2015 SLA C1: Percentage of households interacting with at800 that make a complaint. 1% (green); 2% (amber) Result: 0.21%; green at800 interacted with a total of 8,232 households in August 2015 Of these, 17 made a complaint that related to at800 s operational processes Result: 0.21% Green. SLA C4: Report on total number and categories of complaints There were a total of 47 complaints made in August 17 related to at800 s operational processes; 30 to its policies. Below is a breakdown of the topics of these complaints. Operations-related Dispute of non-4g diagnosis: 3 Engineer: damage/quality: 3 Engineer: late/missed appointment: 3 Appointment availability: 3 Length of journey: 2 Contact centre agent: 1 Engineer attitude/behaviour: 1 Filter-related: 1. Policy-related Support of satellite or cable viewers: 20 Support of communal properties: 5 Support of commercial properties: 3 4G rollout in general: 2. SLA C5: Vulnerable viewer complaints Out of 1,886 viewers flagged as being eligible for additional support with whom at800 interacted in August 2015, eight made a complaint, five of which related to at800 s operational processes and three to its policies. These are broken down as follows: Operations-related Dispute of non-4g diagnosis: 1 Engineer: damage/quality: 1 Engineer: late/missed appointment: 1 Filter-related: 1 Length of journey: 1. Policy-related 4G rollout in general: 2 Support of satellite or cable viewers: 1. SLA C: Quality of Service Page 8 of 11
SLA D Scale August 2015: KPI and SLA performance summary, 9 September 2015 SLA D1: Measure of the numbers of mailings per report of/case of interference The following table shows the number of mailings per engineer visit (4G and non-4g) and reactive filter sent, broken down by mast activated before the end of August 2015. Each case is tied to a single mast according to the rules specified in Appendix 1. Note that the cases are tied to the month in which the mast was scheduled to be activated (and therefore the masts that drove the mailing volumes); not the month in which the mast was activated. Engineer, Engineer, Reactive Initial Mailings Mailings Mailings per Mailings per visit or filter Rollout month 4G non-4g filter mailings per 4G per non-4g reactive filter sent July 2013 2,998 7,603 19,320 7,205,414 2,403 948 373 268 August 2013 295 320 777 153,875 522 481 198 140 September 2013 305 335 727 183,179 601 547 252 172 October 2013 123 151 386 117,425 955 778 304 219 November 2013 91 96 266 129,946 1,428 1,354 489 359 December 2013 36 77 200 103,043 2,862 1,338 515 372 January 2014 128 232 504 534,796 4,178 2,305 1,061 727 February 2014 102 132 302 128,943 1,264 977 427 297 March 2014 15 25 43 5,727 382 229 133 84 April 2014 6 17 79 4,133 689 243 52 43 May 2014 7 10 32 4,606 658 461 144 110 June 2014 142 159 404 94,100 663 592 233 167 July 2014 555 890 2,376 320,204 577 360 135 98 August 2014 186 241 647 64,130 345 266 99 72 September 2014 388 547 1,625 142,452 367 260 88 66 October 2014 477 521 1,105 343,052 719 658 310 211 November 2014 784 530 1,722 556,165 709 1,049 323 247 December 2014 76 100 270 296,536 3,902 2,965 1,098 801 January 2015 143 158 332 187,594 1,312 1,187 565 383 February 2015 138 167 287 95,345 691 571 332 210 March 2015 223 274 499 172,228 772 629 345 223 April 2015 215 256 478 244,807 1,139 956 512 334 May 2015 38 50 97 116,001 3,053 2,320 1,196 789 June 2015 114 100 207 296,835 2,604 2,968 1,434 967 July 2015 136 132 287 314,797 2,315 2,385 1,097 751 August 2015 147 89 227 205,731 1,400 2,312 906 651 September 2015 25 28 74 288,602 11,544 10,307 3,900 2,829 Nothing within 2km 828 1,849 6,476 Total 8,721 15,089 39,749 12,309,666 1,411 816 310 224 Note: Data is included for September 2015 masts as some of these masts were activated before the end of August 2015. SLA D2: Measure of the number of reported cases that occur outside of mailed area Of the 796 households confirmed as experiencing 4G interference in August 2015: Cases not mailed due to risk profile 35 (4.4%) Cases never identified in modelling 3 (0.4%) Total cases outside mailed area 38 (4.8%) Total cases caught by mailing 758 (95.2%) SLA D: Scale Page 9 of 11
SLA D3: Measure of number of cases of interference per mast remaining between 0.17 and 1.66 (5k to 50k cases of interference for rollout) exceeding or trending towards exceeding the range will be raised to the OB for review/action 1.80 Monthly 4G confirmed cases per mast activated in that month 1.66 1.60 1.40 1.20 1.00 0.80 0.69 0.73 0.79 0.81 0.82 0.85 0.88 0.93 0.95 1.00 1.02 1.03 1.02 1.03 1.06 1.07 1.10 1.15 0.60 0.55 0.61 0.40 0.33 0.41 0.20 0.17 0.11 0.11 0.19 0.25 - Jul 13 Aug 13 Sep 13 Oct 13 Nov 13Dec 13 Jan 14 Feb 14Mar 14 Apr 14 May 14 Jun 14 Jul 14 Aug 14 Sep 14 Oct 14 Nov 14Dec 14 Jan 15 Feb 15Mar 15 Apr 15 May 15 Monthly Cases Per Mast Range Lower Limit Range Upper Limit Poly. (Monthly Cases Per Mast) Jun 15 Jul 15 Aug 15 Note: The above thresholds are based on a total mast count of 50,000. SLA D: Scale Page 10 of 11
Appendix 1: Logic for SLA D1: Identifying the nearest mast Identifying an individual mast that is causing interference is an inexact science. Indeed two masts may individually not result in interference for a household, but when both are active at the same time, interference may result. A household that has interacted with at800 a number of times (either requesting multiple filters or receiving many engineer visits) further complicates the identification of a single affecting mast, as masts may have activated between in between their contacts with at800. For the purposes of SLA D1, the logic described below has been used to identify the dates against which households should be measured, and the logic for identifying the mast that is thought most likely to be causing the interference. In-scope households and their effective date The analysis looks separately at households that have received reactive filters and those that have had engineer visits. The few households that have received both count in both analyses. For households to which at800 has sent reactive filters, their analysis is based on the first date on which they made a request for a filter, a request that the triage process indicated warranted the dispatch of a filter. For engineer visits, each household is counted once, but all visits are analysed to inform the record. For any households that had a single visit, the date on which that visit was requested is used as the basis for analysis, and the household is categorised as 4G or not based on the outcome of that visit. For any households that had multiple visits, if one or more visits resulted in a 4G diagnosis, it is categorises as 4G and the date on which the earliest of these visits was requested is used as the basis for analysis. If all visits resulted in a non-4g diagnosis, it is categorised such and the request date of the earliest visit is used as the basis for analysis. Identifying mast most likely to be causing interference For each of the above cases, the following process is undertaken: Determine all masts that were activated on or before the household s effective date (see above), but only include masts that are within a 900 metre radius of the property. From these masts, determine the mast that was activated the soonest before the call came in. This is deemed to be the mast most likely to be the cause of the interference If the above yields no masts, undertake the same process using a 1,500 metre radius If the above yields no masts, undertake the same process using a 2,000 metre radius In the event that the above yield no masts (i.e. there was no active mast within 2,000 metres at the point at which the call came in), the case is not associated with any specific mast and is discounted from the analysis. This logic is used to tag each household with an affecting mast. The rollout month for which this mast was submitted is used as the basis for analysis. Page 11 of 11