This is a PDF file of an unedited manuscript that has been accepted for publication in Omega.

This is a PDF file of an unedited manusript that has been aepted for publiation in Omega. The manusript will undergo opyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the prodution proess errors may be disovered whih ould affet the ontent, and all legal dislaimers that apply to the journal pertain. The final version will be available at: http://dx.doi.org/10.1016/j.omega.2014.12.003 Benhmarking of maintenane and outage repair in an eletriity distribution ompany using the Value-Based DEA method MC Gouveia ISCAC, Quinta Agríola, Benanta, 3040-316 Coimbra INESC Coimbra, Rua Antero de Quental, 199; 3000-033 Coimbra mgouveia@isa.pt LC Dias Fauldade de Eonomia, Universidade de Coimbra, Av. Dias da Silva 165, 3004-512 Coimbra INESC Coimbra, Rua Antero de Quental, 199; 3000-033 Coimbra lmdias@fe.u.pt CH Antunes DEEC-FCT Universidade de Coimbra, Pólo II, 3030 Coimbra INESC Coimbra, Rua Antero de Quental, 199; 3000-033 Coimbra antunes@ines.pt J Bouinha EDP Serviço Universal, S.A., Rua Camilo Castelo Brano, nº 43; 1050-044 Lisboa julia.bouinha@edp.pt CF Ináio EDP Serviço Universal, S.A., Rua Camilo Castelo Brano, nº 43; 1050-044 Lisboa atarinafeteira.inaio@edp.pt ABSTRACT. Benhmarking of eletriity distribution utilities has been widely used as a means to ontribute for the adoption or reinforement of enhaned ompetitiveness and innovation praties to optimize osts, inrease ustomer satisfation, improve orporate image and maximize profits. The purpose of this paper is to present a benhmarking study for the maintenane and outage repair ativity arried out by a Portuguese eletriity distribution ompany, EDP Distribuição (EDP-D), using the Value-Based DEA method, whih builds on links between Data Envelopment Analysis (DEA) and Multiple Criteria Deision Analysis (MCDA). This study illustrates the impat of the inorporation of managerial preferenes in the lassifiation and ranking of 40 network areas served by EDP-D, onfronting the results with a previous study based on a BCC DEA model. In order to deal with the underlying unertainty, the Value-Based DEA method for performane evaluation is adapted to inlude 1

the onept of super-effiieny. Besides identifying best praties, soures of ineffiieny, gaps relatively to best praties and opportunities for improvement, this analysis supports the introdution of orretive measures and informs deisions about future goals. KEYWORDS. Data Envelopment Analysis; Multi-Criteria Analysis; Eletriity Distribution; Supereffiieny. 1. Introdution Eletrial energy is at the heart of modern soiety, as an essential omponent of lifestyle and a determining fator in the ompetitiveness of the eonomy. In the urrent ontext of inreasing ompetition and regulatory pressure on eletri utilities, ompanies must beome inreasingly effiient. In this framework, benhmarking is a very helpful instrument to identify the most effiient utilities in the setor, providing measures to evaluate the relative performane of the different utilities analyzed. Through benhmarking it is possible to quantify differenes in performane, identify the reasons for suh differenes and also the improvements needed to ahieve the targets set by the organization. Several approahes have been proposed to measure the relative effiieny of utilities with respet to an empirial effiient frontier defined by a set of units. More reently several benhmarking studies use Data Envelopment Analysis (DEA) for the identifiation of soures of ineffiieny in some of the most profitable ompanies (see, e.g. [1-8]). DEA [9] is a nonparametri approah based on linear programming for measuring the effiieny of a set of entities alled Deision Making Units (DMUs). A DMU is any entity under evaluation in terms of its abilities to onvert inputs into outputs, engaged in the same ativity. Sine the early 1980 s, after a first DEA study onduted by Färe et al. [10] to measure the relative effiieny of eletriity utilities in Illinois, several other studies to evaluate the effiieny of eletriity distribution ompanies have appeared in the literature. Jamasb and Pollit [11] reported and disussed the main benhmarking methods used by the eletriity regulators of the eletriity distribution ativity in the OECD and a few other ountries. Later, the same authors [12] presented an international benhmarking study of 63 regional eletriity distribution utilities in six European ountries, illustrating the methodologial and data diffiulties enountered in the use of international benhmarking for utility regulation. Haney and Pollitt [13] presented the results of an international survey of energy regulators in 40 ountries. There are many other studies in this area using DEA, some of whih evaluate the relative effiieny of distribution utilities in a single ountry, while others have an inter-ountry fous (for a omprehensive review see [14]). Although various studies on the use of DEA have been published, only a few inorporate managerial preferenes in the analysis (see e.g. [15-21]). Thanassoulis et al. [22] reported a number of reasons for the 2

inlusion of the Deision Maker s (DM s) preferenes in DEA. In a work relating possible areas of interation between DEA and Multiple Criteria Deision Making (MCDM), Bouyssou [23] onluded that both the hoie and the ranking of alternatives an be ahieved only by introduing a preferene struture. In this line of thought Köksalan and Tuner [24] proposed a DEA-based approah to ranking multi-riteria alternatives, inluding weight restritions to inorporate DM s preferenes into the analysis. Cook et al. [25] offered some larifiation and diretion on how DEA an be viewed as a tool for multiple-riteria evaluation problems. The present study ombines DEA with Multiple Criteria Deision Analysis (MCDA) inluding relevant preferential information eliited from the DM using Value-Based DEA [26-27]. This method is based on the additive DEA model with oriented projetions [28], making use of onepts developed in the field of MCDA under impreise information [29-30]. The present paper presents a benhmarking study for one of the ativities performed by EDP Distribuição (EDP-D) maintenane and outage repair. An internal benhmarking study had been previously undertaken using several DEA models to examine data on this ativity in the Portuguese eletriity distribution system during 2004 2005 [31]. More reently, for this partiular ativity, a study that uses DEA with the involvement of DMs was arried out by Amado et al. [21], whih ompares the ost effiieny of medium-voltage power lines belonging to the same regional distribution networks area operated by EDP-D. These authors analyze the impat of different design systems and different maintenane programs, ontributing to the redution of osts and improving servie delivery quality. In both aforementioned studies the experiene of the ompany has been used to draw some lessons about how performane measurement an be implemented within a ompany. In this ontext, the Value- Based DEA method an bring useful insights to the ompany managers, sine their preferenes and judgments are inorporated into the model. The present work intends to assess whether that is indeed the ase, by applying Value-Based DEA to a ontext the DMs already knew well. The study an be pereived as a learning ativity both for the method proponents (analysts) and for the ompany s DMs involved. This is in line with Dyson and Shale [32], when they state that a greater ollaboration between aademis and pratitioners or a greater involvement of aademis with pratie is neessary to obtain redible results and to foster the onfidene of the DMs in new developments of methods and tehniques. Although this study was developed for a partiular ativity, maintenane and outage repair, the same type of methodology an be applied to other ativities, as well as to the whole set of ativities developed within EDP-D. In order to be able to ompare results, this study uses the same data previously used to evaluate the effiieny of the 40 network areas then operating in the Portuguese mainland. This approah is useful not only for omparison with results obtained with the lassial DEA models, but also to understand the impat of the inorporation of managerial preferenes in the lassifiation of the units. With this analysis, besides identifying best praties, soures of ineffiieny, gaps relatively to best praties and opportunities for improvement, it is also possible to support the introdution of orretive measures and inform deisions about future goals, as well as to improve the knowledge of the ompany. 3

The remainder of this paper is organized as follows. Setion 2 introdues the Value-Based DEA method with the modifiations to inlude the super-effiieny onept [27]. Setion 3 gives a brief bakground on EDP-D. In setion 4 the input and output fators are presented as well as the protools used to eliit the DM s preferenes. The analysis of results is arried out in setion 5. Setion 6 highlights the many prospets for improvement in some of the ineffiient units, given their speifiity. Conluding remarks are presented in setion 7. 2. The Value-Based DEA method The main idea underlying DEA is that by omparing a set of similar DMUs, it is possible to identify best praties and find the effiient frontier formed by DMUs operating effiiently. Hene, the different models for DEA seek to determine the DMUs whih form the effiient frontier (or envelopment surfae) in the Pareto-Koopmans sense. Sine DEA is based on best praties, it is appealing for the manager who prefers to think in terms of benhmarks instead of omparisons with the mean, for example. DEA identifies benhmarks against whih the ineffiient units an be ompared, i.e., it provides measures for the relative effiieny of the non-frontier units. In other words, for the ineffiient units, whih do not represent the "best pratie" from the ombination of inputs and outputs, it is possible to identify the units belonging to the effiient frontier, with whih they should ompare and, onsequently, the redutions in inputs and/or inreases in outputs neessary for those units to beome effiient (this may be interpreted as a mehanism of projetion on the effiient frontier). Hene, this tehnique is widely used for benhmarking beause it is very effetive in determining the units with best praties. Classi DEA models may onsider both onstant returns to sale (CRS), as the CCR model [9], and variable returns to sale (VRS), as the BCC model [33]. In the first ase, a proportional hange in outputs is expeted from a given hange in inputs, at all levels of sale. In the seond ase, an inreasing or dereasing hange in outputs may our due to a given hange in inputs. Charnes et al. [34] proposed the additive DEA model as an alternative to the BCC model; the additive model also onsiders VRS but does not need a hoie between input-orientation and output-orientation. In oriented models, firstly all fators are redued or inreased at the same rate towards the envelopment surfae, and the seond stage yields an optimal set of slak values. The additive DEA model uses the seond stage only and measures the exess of inputs and the defiit of outputs for a DMU under evaluation, when onfronted with the DMUs operating on the effiient frontier. The method developed by Gouveia et al. [26] builds on Multi-Attribute Utility Theory (MAUT) [35] sine the input and output fators are onverted into utility funtions aording to the preferene information provided by DMs. In aordane with von Winterfeldt and Edwards [36], some protools were used in the proess of eliiting preferenes in order to build the marginal utility funtions, as well as onstraining the weights for the aggregation of marginal utilities into an additive overall utility funtion, instead of letting eah DMU hoose freely the weights assoiated with these funtions. 4

In the ontext of additive aggregation with impreise weights, the min-max regret rule [37] is a meaningful tool to ompare the alternatives [30, 38, 39]. In the Value-Based DEA method used in this study, one must find the sale oeffiients (weights) that, for eah alternative, minimize the utility differene to the best alternative, aording to the min-max regret rule, whih gives an intuitive meaning (the loss of utility) to the effiieny measure assigned to eah DMU. Let us onsider n DMUs to be evaluated, eah of them onsuming m different inputs to produe p different outputs. The DMU j onsumes the quantity x ij >0 of input i and produes the quantity y rj >0 of output r. Considering that the DMUs are evaluated aording to q (with q = m+p) riteria, q utility funtions u 1,, u q must be defined suh that the worst level has a 0 value and the best level has value 1. Hene, after being onverted into utility values, all fators are treated as outputs to be maximized. For eah alternative (DMU), aording to the additive MAUT model, the utility obtained is q =1 U(DMU j ) = å w u (DMU j ), where w 0, = 1,,q and å w =1 (by onvention). The sale oeffiients w 1,, w q are the weights of the utility funtions and reflet the DM s utility trade-offs, sine one unit in one marginal utility funtion is not neessarily valued as muh as one unit in the marginal utility orresponding to a different fator. The Value-Based DEA method [26] is extended to onsider the super-effiieny onept introdued in DEA by Andersen and Petersen [40], in the sense that a omplete ranking of all DMUs an be obtained [27]. For that purpose, the following linear program is solved (Phase 1): q =1 min d k d k,w s.t. q =1 q ( ) q ( ) å w u DMU j - å w u DMU k d k, j =1,...,n, j ¹ k å w =1, =1 w ³ 0," =1,..., q =1 (1) The optimal value d k denotes the distane defined by the utility differene to the best of all alternatives (exluding the one under evaluation). If d k < 0, then DMU k is effiient. This measure gives the extent to whih an effiient DMU may worsen its utility while remaining effiient. The purpose of the method is to alulate the vetor w of weights whih minimizes the distane (the utility differene) of DMU k to the best one (note that the best alternative will also depend on w), exluding itself from the referene set. Then, an effiient target is determined in ase the DMU is ineffiient (Phase 2). The details of this proess are as follows: Phase 1: Convert inputs and outputs into utility sales. Compute the effiieny measure, d k, for eah DMU, k = 1,,n, and the orresponding weighting vetor. 5

Phase 2: If d k 0 then solve the weighted additive model (2), using the optimal weighting vetor resulting from Phase 1, w, and determine the orresponding projeted point of the DMU under evaluation. min k = - q å w * s l,s =1 n s.t. å l j u DMU j j=1, j¹k n å l j =1, j=1 ( ) - s = u ( DMU k ), =1,...,q l j, s ³ 0, j =1,..., k -1, k +1,..., n, =1,..., q (2) Variables j, j =1,...,k-1,k+1,...,n, define a onvex ombination of the n-1 DMUs. The set of effiient DMUs (possibly only one) defining the onvex ombination (those DMU j suh that j >0) are the peers of DMU k under evaluation, i.e. the DMUs with whih it should ompare in terms of overall utility to ahieve effiieny. The onvex ombination orresponds to a point on the effiient frontier whih is better than DMU k by a differene given by s (slak) on eah riterion. This target point, onsidering these weights optimal for DMU k, is better than DMU k by a differene of d k in terms of global utility. 3. EDP Distribuição - Bakground for the Case Study Aording to Portuguese law, loal authorities, at Muniipal level, are entitled to perform all the ativities related with low voltage eletriity distribution. After the reation of EDP Eletriidade de Portugal, in 1976, as an integrated ompany in harge of eletriity generation and transmission aross the whole Portuguese mainland, a proess of integration of the distribution ativities into EDP has started to develop. Under 20 year ontrat agreements with eah of the muniipalities, EDP has progressively taken over the distribution ativity. By mid 1990s, the ompany was in harge of all the investment and maintenane ativities required in the distribution network. Aording to these ontrats, although the assets ownership remained within the loal authorities, EDP was in harge of all the operations, in exhange for the payment of a onession fee to eah muniipality. The distribution ativity was then organized into four ompanies within EDP, aording to the four main regions of the Portuguese mainland North, Center, Tagus Valley and South. In 2000, after deregulation and the reation of the Regulator (ERSE), these four ompanies were merged and onverted into a single ompany, named EDP Distribuição, unbundled from the other ativities, with a ompletely separate management. Given the very different network bakground, in terms of assets and organization, benhmarking 6

ativities are highly relevant for management purposes, in order to determine best performanes whih an be used as benhmarks for areas whose praties need to improve. 4. A Model for Maintenane and Outage Repair in Eletriity Distribution inluding the DM s Preferenes In a previous internal benhmarking study, a variety of DEA models (CCR, BCC and SBM) were used to evaluate the effiieny of maintenane and outage repair expenditures in the eletriity distribution networks operated by EDP-D [31]. The objetive of that study was to identify best praties, taking into aount all the relevant explanatory variables for this ativity. The Value-Based DEA method was suggested to EDP-D as an approah that might bring useful insights to the ompany, sine managerial preferenes would be inorporated into the model. This work relies on data for 2004-2005, allowing the omparison with results previously obtained, for the same period, in whih the ompany was organized into 40 different DMUs. After that period the ompany was re-organized into a onsiderably smaller number of units, whih are not omparable with the previous ones. Therefore, the value to the ompany is not only the omparison with results obtained through other DEA models, already used by Weyman-Jones et al. [31], but also to understand the impat of the inorporation of those preferenes on the lassifiation of the units and to assess the interest of using Value-Based DEA in benhmarking other ativities of the ompany. 4.1. Fators Weyman-Jones et al. [31], in the study undertaken for EDP-D, onstruted a model in whih the explanatory fators were seleted in an interative proess with the ompany engineers, with the purpose of evaluating the effiieny of 40 network areas in the years 2004 and 2005. The effiieny analysis was applied to the partiular ativity of maintenane and outage repairing. In that study, an input orientation was adopted. The analysis inluded the omparison of a variety of DEA models to examine the relationships between oriented and non-oriented models, and radial and non-radial analysis. Inputs and outputs, presented in Table 1, have been disussed with the engineering branhes of the ompany related to these ativities. The rationale was to inlude in the models the main variables that reflet the osts and performane of the different units analyzed. 7

Inputs Table 1. Fators. Outputs x OPEX : maintenane and outage repairing osts x MLL : supply interruptions (minutes of lost load) y CLI : lients (LV+MV) y NLL : network lines length (LV+MV) x CC : omplaints per ustomer x NI : number of inidents (LV and lients installations) x OPEX represents the resoures used, in terms of osts of that partiular ativity. Inputs x MLL, x CC and x NI are indiators for quality of supply and reflet undesirable outputs. Supply interruptions, measured in minutes of lost load, represent the number of minutes ustomers are without eletriity supply whih, ideally, should be zero. The number of omplaints per ustomer also reflets the performane of the area, as a higher number of omplaints indiates poorer ustomer servie. A higher number of inidents on the low voltage network or in ustomer installations also reflets poor servie and, hene, must be minimized. Outputs y CLI and y NLL reflet the ativity level of eah area and apply to both low voltage (LV) and medium voltage (MV) networks - more lients and a network with higher length will lead to higher osts. Output y CLI is a proxy for the number of ustomer servies provided. Network line length (y NLL ) is onsidered as an exogenous operating harateristi refleting the maintenane and repair load in the network, treated as an additional output in the input orientation. Regulators have used y NLL as a measure of the diffiulty in delivering eletriity [31]. These inputs and outputs are typial in the evaluation of this ativity. For instane, Amado et al. [21] took into aount most of these variables (namely x OPEX, x MLL, x NI and y NLL ) to assess this ativity, although in a different ontext, with the purpose of evaluating the impat of alternative poliies of design and maintenane on the effiieny of lines. The study uses the experiene of the ompany to draw some lessons about how performane measurement an be implemented within a ompany, in ontrast to the usual objetive of regulatory benhmarking proedures. In fat, there is a signifiant differene in purpose and implementation between publi regulatory benhmarking and internal ompany benhmarking, whih is related to the nature of inentives and rewards. The summary statistis of the inputs and outputs of all DMUs for both years is depited in Table 2. 8

Table 2. Summary statistis of fators for 2004-2005. Inputs Outputs Year 2004 x OPEX x MLL x CC x NI y CLI y NLL Average 2541523. 09 253.95 0.92 5333.53 145593.30 4832.00 Std. Dev. 1506629.27 136.82 0.39 6141.89 154080.24 2322.78 Max 7 802 302.10 606.45 1.75 33280.00 859831.00 14337.85 Min 1 129 899.13 86.00 0.39 1578.00 56730.00 2651.81 Year 2005 Average 2514733.13 221.00 1.17 4576.93 147685.40 4931.90 Std. Dev. 1501549.64 110.80 0.44 5141.97 155802.68 2354.68 Max 8 204 735.48 665.86 1.99 28880.00 868566.00 14717.25 Min 1 129 899.13 98.80 0.29 1479.00 57453.00 2725.99 4.2. Eliitation of fators utility funtions The use of the Value-Based DEA method allows tailoring the analysis aording to the DM s preferenes. von Winterfelt and Edwards [36] make a detailed presentation of various tehniques to question the DM in order to build utility funtions onsistent with the DM s answers, but these questions must be framed for eah partiular ontext. The eliitation of the DM s preferenes is a ruial step of a multiple riteria deision aiding proess. The purpose of fators onversion into a utility sale in the Value-Based DEA method developed by Gouveia et al. [26] is to reflet the DM s preferenes. The utility funtions have been onstruted using a preise protool (desribed by Almeida and Dias [41]) to eliit the differene in the DMU s relative merit orresponding to dereases in inputs or inreases in outputs, rather than the absolute utility of having these inputs available or outputs produed. The eliitation protool is based on omparing the merit of inreasing an output (or dereasing an input) from a to b versus inreasing the same output (or dereasing the same input) from a to b, all other performane levels being equal, and asking the DM to adjust one of these four values suh that the inrease of merit would be approximately equal. This onversion is done assuming the ontinuity of funtions and beause utility funtions are unique up to positive affine transformations it is usually assumed that both the global utility funtions and marginal utility funtions are saled between 0 and 1, as referred to in setion 2. For example, onsidering the variable x MLL a question raised to the DM was: Is it more meritorious to derease supply interruptions (minutes) from 800 to 300 or from 300 to 60, all the other performanes being equal? The answer was that it is more meritorious to derease from 300 to 60. Then an adjustment has been made and the question was reformulated as: Is it more meritorious to derease the number of supply interruptions from 800 to 200 or from 200 to 60, all the other performanes being equal? The answer was that the merit is the same. This means that u MLL (200)-u MLL (800) = u MLL (60)-u MLL (200), i.e., u MLL (200)=(u MLL (60)+u MLL (800))/2. The same proedure was used to dihotomize the intervals of merit [60,200] and [200,800]. 9

utility utility utility The DM answered questions about the differenes of merit between the performane levels on eah fator. A pieewise linear approximation was defined to represent the utility funtions for most fators, and known funtions (namely logarithmi funtions) were used when the DM s answer ould be adjusted to predefined urves. The eliited ranges were hosen to inlude the observed performane ranges plus or minus the highest tolerane value onsidered (in this ase = 20%). 01 1,00 01,900 01 01 01 01 00 00 00 00 00 y = -0,414ln(x) + 6,6703 R² = 1 0 3000 6000 9000 12000 Thousands maintenane and outage repairing osts (euros) (x OPEX ),800,700,600,500,400,300,200,100,00 0 200 400 600 800 1000 supply interruptions (minutes) (x MLL ) 1,00,900,800,700,600,500,400,300,200,100,00 0 5000 10000 15000 20000 25000 network lines lenght (y NLL ) Figure 1. Three of the utility funtions eliited for fators. Figure 1 displays the piee-wise linear utility funtions for the inputs x OPEX and x MLL and for the output y NLL. For example, for the supply interruptions: u MLL (60) u MLL (90) = u MLL (90) u MLL (200) = u MLL (200) u MLL (425) = u MLL (425) u MLL (800), all other performane levels being equal. The input fators x CC, x NI have utility funtions similar to the x MLL utility funtion and the output utility funtion y CLI is idential to the output utility funtion y NLL. The x OPEX utility funtion was obtained by making the orresponding 10

adjustment of a known funtion to the DM s preferenes. Note that the transformation of the original input/output data from original sales to a utility sale, on the basis of preferene information provided by the DM, allows dealing with undesirable outputs in a natural way by setting a dereasing utility funtion. Table 3 indiates utilities, for the 40 DMUs. Table 3. Performanes onverted into utility sales for 2004-2005. Fators in utility sales (2004) Fators in utility sales (2005) DMUs u OPEX u MLL u CC u NI u CLI u NLL u OPEX u MLL u CC u NI u CLI u NLL 1 0.518 0.426 0.221 0.668 0.063 0.326 0.523 0.466 0.166 0.696 0.064 0.334 2 0.668 0.582 0.464 0.709 0.080 0.094 0.687 0.695 0.347 0.730 0.082 0.103 3 0.672 0.644 0.205 0.687 0.068 0.081 0.619 0.730 0.280 0.715 0.070 0.091 4 0.477 0.462 0.287 0.596 0.139 0.393 0.511 0.693 0.233 0.645 0.142 0.406 5 0.100 0.697 0.438 0.169 0.669 0.724 0.069 0.702 0.291 0.243 0.674 0.725 6 0.841 0.217 0.282 0.777 0.017 0.104 0.882 0.453 0.109 0.880 0.018 0.114 7 0.590 0.478 0.169 0.686 0.073 0.336 0.632 0.591 0.097 0.717 0.075 0.343 8 0.593 0.595 0.322 0.607 0.138 0.178 0.581 0.658 0.243 0.659 0.142 0.189 9 0.677 0.783 0.330 0.632 0.091 0.156 0.683 0.655 0.174 0.668 0.094 0.164 10 0.811 0.700 0.516 0.792 0.029 0.249 0.801 0.560 0.345 0.857 0.030 0.253 11 0.887 0.432 0.317 0.741 0.019 0.117 0.831 0.401 0.315 0.786 0.020 0.123 12 0.900 0.490 0.470 0.749 0.021 0.166 0.901 0.481 0.384 0.856 0.022 0.171 13 0.588 0.417 0.226 0.689 0.088 0.278 0.628 0.610 0.213 0.711 0.089 0.282 14 0.836 0.669 0.571 0.728 0.061 0.326 0.827 0.338 0.489 0.747 0.062 0.332 15 0.801 0.601 0.533 0.856 0.031 0.162 0.804 0.469 0.330 0.865 0.032 0.175 16 0.858 0.668 0.481 0.792 0.036 0.153 0.888 0.659 0.409 0.812 0.037 0.175 17 0.878 0.402 0.527 0.791 0.023 0.171 0.897 0.682 0.632 0.836 0.024 0.181 18 0.397 0.528 0.144 0.563 0.183 0.419 0.462 0.485 0.118 0.602 0.186 0.428 19 0.438 0.363 0.236 0.612 0.150 0.338 0.428 0.358 0.118 0.636 0.154 0.343 20 0.606 0.547 0.419 0.682 0.084 0.144 0.634 0.525 0.228 0.700 0.085 0.155 21 0.659 0.725 0.255 0.662 0.127 0.236 0.618 0.738 0.318 0.689 0.129 0.251 22 0.710 0.505 0.472 0.777 0.032 0.175 0.688 0.428 0.390 0.805 0.033 0.187 23 0.754 0.376 0.360 0.714 0.030 0.226 0.770 0.384 0.138 0.716 0.031 0.239 24 0.529 0.486 0.301 0.632 0.124 0.303 0.497 0.489 0.281 0.680 0.127 0.310 25 0.536 0.177 0.183 0.606 0.109 0.337 0.550 0.089 0.122 0.647 0.112 0.350 26 0.607 0.355 0.300 0.655 0.120 0.200 0.594 0.464 0.215 0.682 0.123 0.208 27 0.646 0.331 0.198 0.671 0.073 0.242 0.614 0.283 0.190 0.697 0.074 0.252 28 0.102 0.593 0.426 0.084 0.828 0.798 0.125 0.649 0.363 0.139 0.835 0.811 29 0.221 0.498 0.306 0.384 0.475 0.459 0.195 0.556 0.205 0.416 0.485 0.464 30 0.564 0.215 0.230 0.680 0.093 0.307 0.530 0.382 0.335 0.695 0.095 0.310 31 0.776 0.131 0.390 0.643 0.031 0.265 0.758 0.316 0.270 0.656 0.031 0.274 32 0.492 0.129 0.180 0.726 0.105 0.379 0.516 0.322 0.118 0.733 0.107 0.381 33 0.439 0.611 0.436 0.544 0.247 0.289 0.470 0.780 0.370 0.579 0.253 0.288 34 0.593 0.651 0.536 0.530 0.246 0.191 0.592 0.732 0.433 0.570 0.249 0.200 35 0.597 0.479 0.406 0.662 0.075 0.365 0.621 0.472 0.438 0.688 0.077 0.375 36 0.635 0.259 0.432 0.714 0.055 0.381 0.675 0.418 0.461 0.730 0.057 0.395 37 0.750 0.266 0.462 0.743 0.025 0.287 0.773 0.442 0.342 0.794 0.026 0.293 38 0.538 0.468 0.482 0.625 0.128 0.304 0.583 0.461 0.414 0.614 0.135 0.307 39 0.554 0.575 0.447 0.592 0.145 0.311 0.591 0.540 0.339 0.589 0.150 0.315 40 0.699 0.450 0.508 0.704 0.055 0.203 0.702 0.436 0.394 0.701 0.058 0.211 11

4.3. Eliitation of weight restritions In DEA, DMUs hoose their best oneivable weights; the fat that those may be in ontradition with a priori knowledge leads to the introdution of managerial preferenes on the relative importane of the inputs and outputs used in the assessment [42]. The introdution of weight restritions in the model helps to reflet the organization s objetives, ensuring meaningful results whih are loser to what the DM onsiders as best praties. The weights employed in the additive aggregation MAUT model used in the Value-Based DEA method [26] are the sale oeffiients of the utility funtions, whih allow for utility trade-offs between different fators (see [35]). The weights of the additive aggregation model an be assessed, using again the DM s judgments. There are various tehniques available whih may help to obtain weights, suh as the diret rating, the swings method and the indifferene equations [36, 43]. The swings method was onsidered to be the most appropriate for this ase beause it is simpler and learer to the DM. The swings method begins by onstruting two extreme hypotheses, B and G, the first one displaying the worst performane on all riteria and the seond one the orresponding best performane. The preferene eliitation protool onsists in querying the DM to observe the potential gains from moving from B to G on eah riterion and then to deide whih of the riteria he/she prefers to shift to hypothesis G. Suppose that the transition from hypothesis B to hypothesis G on a speified riterion is worth 100 units in a hypothetial sale. Then, the DM is asked to give a value (<100) to the seond riterion moved to G, then to the third riterion and so on, until the last riterion is moved to G. The proedure used in this work was to start with a ranking of weights, via the swings method, and then to establish a limit to the ratio between the weights ranked first and last, by means of a trade-off question, to avoid null weights. Let W denote the set of weighting vetors ompatible with the eliited ranking and ratio limit. After the eliitation of weight restritions, formulation (1) (see setion 2) is modified to inlude the weight restritions (w 1,..., w q ) Î W. With this hange in Phase 1, it is neessary to hange formulation (2) allowing the slaks to have negative values; otherwise it might not be possible to keep the optimal utility differene d k derived from Phase 1 with the weight vetors inorporated (for details, see [41]). 5. Results 5.1. Comparison of results for standard DEA models and the Value-Based DEA method In this subsetion, we ompare the results of the BCC DEA model, obtained by Weyman-Jones et al. [31] and the Value-Based DEA method, without onsidering weight restritions. Although Weyman-Jones et al. s study presents results for the CCR, BCC and SBM models, for the sake of omparison with this approah only BCC results are referred to, as Value-Based DEA builds on the additive model with a VRS frontier and the results obtained with BCC and SBM models do not vary muh, maintaining the number 12

of effiient units. Figure 2 exhibits a omparison between the results obtained with the Value-Based DEA method and the standard BCC model (input oriented), for the year 2005. As stated in setion 2, if d is negative then the DMU under analysis is effiient and if d > 0 then the DMU is ineffiient. The number of effiient units dereases in Value-Based DEA method, sine the inorporation of preferenes during the onstrution of the utility funtions hanges the shape of the effiient frontier. In the BCC model 19 DMUs are effiient (the ones with effiieny 1 on the y-axis), but three of these DMUs lose effiieny in Value-Based DEA method: the ones with effiieny 1 (y-axis) but d > 0 (x-axis), that is in the first quadrant in Figure 2. DMUs that are lassified as effiient in the Value-Based DEA method are also effiient in the BCC model. There are no DMUs with d < 0 and with effiieny sore in the BCC model less than 1. For example, DMU 2 (in the 1 st quadrant) is effiient in the BCC model, but ineffiient onsidering Value-Based DEA method. This is mainly due to the DM s preferenes refleted in the utility funtions. DMUs 28 and 33 (2 nd quadrant) are effiient in both models (Phase 1 of Value-Based DEA method and BCC model); DMU 1 (1 st quadrant) is lassified as ineffiient in both. DMU 28 DMU 33 1 0,9 0,8 0,7 DMU 2 0,6 0,5 0,4 0,3 0,2 0,1 DMU 1 2005 0-0,2-0,15-0,1-0,05 0 0,05 0,1 Figure 2. Comparison of the BCC model and Value-Based DEA method (without weight restritions), for 2005 data (d in the x-axis and BCC sore in the y-axis). A study onsidering the ombined set of 2004 and 2005 observations, with 40 DMUs, was also arried out (Figure 3), in order to identify when variation of effiieny has ourred. In the BCC model, the number of effiient units inreases from 6 (2004) to 18 (2005), and in the Value-Based DEA method the number of effiient DMUs inreases from 7 to 14. Only one unit is lassified as ineffiient in the BCC and as effiient in the Value-Based DEA method (DMU 12, in 2004), all the other effiient units for the BCC model also having the same lassifiation in the Value-Based DEA method. On the other hand, four DMUs lose the effiieny status with the Value-Based DEA method in the year 2005 (DMUs 2, 3, 21, 37). 13

1 DMU 2 0,9 0,8 DMU 12 0,7 0,6 0,5 0,4 2004 2005 0,3 0,2 0,1 0-0,1-0,08-0,06-0,04-0,02 0 0,02 0,04 0,06 0,08 0,1 Figure 3. Comparison of the BCC model and Value-Based DEA method (without weight restritions), for 2004 and 2005 data (d in the x-axis and BCC sore in the y-axis). This analysis leads to the onlusion that the radial DEA model is more generous in the lassifiation of units in most ases, but there is a reasonable agreement between both approahes. The DM has learned what was the impat of his / her responses in terms of the overall strength of eah DMU, taking into aount the diverse fators aording to his / her managerial preferenes. Additionally, the DM was offered information providing further disrimination of the effiient DMUs, in omparison with standard DEA models. 5.2. Further results of Value-Based DEA method In the previous subsetion the number of effiient units was determined onsidering the year 2005 and ombining the observations of the years 2004 and 2005, when omparing the BCC model with the Value- Based DEA method. In this subsetion results for the same years will be presented, but fousing on the Value-Based DEA method to observe performane hanges from 2004 to 2005, sine this method yields more disriminating results. The inlusion of weight restritions in the Value-Based DEA method is also disussed. Table 4 shows the evaluation of DMUs effiieny aross the two years without weights restritions. The effiieny measure d* dereased for DMUs 5, 9, 11, 14, 15, 28, 31, 38, 39 and 40. DMU 9 is the only one that lost the effiieny status from 2004 to 2005. In fat, DMUs that hange from ineffiient to effiient, onsidering both years, namely DMUs 16 and 17, have better utility values in almost all fators (exept y CLI ) than DMU 9 (in 2005) and DMU 33. DMU 33, despite having a worse x OPEX utility value than DMU 9, improved from 2004 to 2005 in x OPEX, x MLL (the best unit in this fator), x NI (worse than DMU 9), and y CLI (muh better than DMU 9). The fators in whih DMU 33 worsened from 2004 to 2005 (x CC and y NLL ) have a better utility value when ompared with the utilities of the same fators in DMU 9 in 2005. 14

Table 4. d* for the 40 DMUs (2004-2005) and the differene between d onsidering both years (by inreasing order of d * (2005)). DMUs d (2004) d (2005) d (2005) d (2004) DMUs d (2004) d (2005) d (2005) d (2004) 17 0.0112-0.0715-0.0827 2 0.0504 0.0193-0.0311 28-0.0416-0.0356 0.0060 29 0.0425 0.0208-0.0217 10-0.0117-0.0291-0.0174 18 0.0424 0.0263-0.0161 33 0.0546-0.0246-0.0793 22 0.0391 0.0278-0.0113 36 0.0133-0.0213-0.0346 13 0.0600 0.0321-0.0279 6 0.0338-0.0208-0.0546 8 0.0716 0.0353-0.0363 5-0.0353-0.0191 0.0162 1 0.0549 0.0368-0.0181 4 0.0489-0.0094-0.0582 11 0.0091 0.0370 0.0279 12-0.0005-0.0077-0.0072 30 0.0529 0.0431-0.0098 14-0.0434-0.0072 0.0363 24 0.0695 0.0463-0.0232 32 0.0037-0.0033-0.0070 25 0.0726 0.0490-0.0236 15-0.0092-0.0031 0.0061 9-0.0413 0.0494 0.0907 34 0.0250-0.0024-0.0275 23 0.0572 0.0501-0.0071 16 0.0139-0.0014-0.0153 19 0.0643 0.0505-0.0138 21 0.0124 0.0020-0.0104 26 0.0676 0.0523-0.0153 3 0.0538 0.0061-0.0478 31 0.0488 0.0563 0.0075 37 0.0279 0.0090-0.0189 20 0.0663 0.0577-0.0086 7 0.0386 0.0112-0.0274 38 0.0531 0.0625 0.0094 35 0.0334 0.0191-0.0142 40 0.0580 0.0641 0.0061 27 0.0767 0.0643-0.0124 39 0.0641 0.0727 0.0086 Table 5 displays the results from Phase 1, only for effiient units, in the year 2005. When there are no restritions on the weights, some fators may be disregarded from the assessment beause DMUs an assign zero weights to some fators (namely those presenting low levels of outputs and high levels of inputs). For example, DMU 28 is effiient but all weights are null exept one, and there are several fators disregarded in other units. Moreover, DMUs an assign weights to their fators ignoring reognized opinions about the value of those fators [44]. The inorporation of weight restritions expressed by the DM on the effiieny assessment of DMUs is a way to overome the problem of having less redible effiieny sores. 15

Table 5. Results of Phase 1 for effiient units without weight restritions, for 2005 data. DMUs d w OPEX w MLL w CC w NI w CLI w NLL 3-0.001 0.000 0.724 0.000 0.276 0.000 0.000 4-0.020 0.000 0.278 0.000 0.350 0.000 0.372 5-0.025 0.000 0.800 0.000 0.000 0.000 0.200 6-0.021 0.091 0.000 0.000 0.909 0.000 0.000 10-0.030 0.000 0.069 0.000 0.570 0.000 0.361 12-0.008 0.286 0.000 0.000 0.468 0.000 0.246 14-0.042 0.490 0.000 0.000 0.000 0.000 0.510 15-0.006 0.000 0.000 0.055 0.690 0.254 0.000 16-0.001 0.510 0.003 0.000 0.000 0.486 0.000 17-0.169 0.099 0.165 0.736 0.000 0.000 0.000 21-0.002 0.000 0.504 0.000 0.281 0.000 0.214 28-0.161 0.000 0.000 0.000 0.000 1.000 0.000 32-0.003 0.000 0.000 0.000 0.466 0.146 0.389 33-0.047 0.000 0.937 0.000 0.000 0.052 0.011 34-0.006 0.251 0.491 0.000 0.000 0.259 0.000 36-0.026 0.000 0.017 0.052 0.409 0.000 0.521 Following the proedure explained in subsetion 4.3, weight restritions were eliited by asking the DM to ompare the swings of utility from 0 to 1 as depited in Table 6. The DM was asked to onsider one hypothetial unit with the performane level 0 for all fators and the question was: "if you ould improve one and only one fator to the maximum utility level (1), whih would you hoose?". The DM s answer was: x OPEX. This allows the inferene that w OPEX is the highest saling onstant. By repeating this question suessively for the remaining fators, the following ranking of the sale oeffiients was attained: w OPEX w MLL w NI w CC w NLL w CLI. Table 6. Extreme performanes assoiated with utility levels 0 and 1. Utility level x OPEX x MLL x CC x NI y CLI y NLL u(. ) = 0 10500000 800 2.5 40000 45000 2000 u(. ) = 1 900000 60 0 1000 1100000 20000 After the DM had established the ranking of the sale oeffiients and in order to avoid zero-value weights, an indifferene judgment question was asked to limit the ratio of the weights ranked in the first and last position. The answer to the question What would be the lowest amount h that would allow a unit with of 1.1 million lients and with maintenane and outage repairing osts of 10.5 million euros to be onsidered as having more merit than a unit with 45 000 lients and with maintenane and outage repairing osts of h? was h = 1 million euros. This answer is translated into: w CLI u CLI (1 100 000) + w OPEX u OPEX (10 500 000) w CLI u CLI (45 000) + w OPEX u OPEX (h). Substituting h in the previous expression yields: w OPEX 1.05 w CLI. Table 7 portrays the results from Phase 1 and Phase 2 under weight restritions and free slaks, only for effiient units, in the year 2005. 16

36 34 33 32 28 21 17 16 15 14 12 10 6 5 4 3 DMUs Table 7. Results of Phase 1 and Phase 2 under weight restritions and free slaks for 2005 data. Phase 1 Phase 2 d w OPEX w MLL w CC w NI w CLI w NLL s OPEX s MLL s CC s NI s CLI s NLL 0.125 0.171 0.171 0.163 0.171 0.163 0.163 0.28-0.05 0.35 0.12-0.05 0.09 0.104 0.167 0.167 0.167 0.167 0.167 0.167 0.39-0.01 0.40 0.19-0.12-0.22 0.091 0.167 0.167 0.167 0.167 0.167 0.167 0.83-0.02 0.34 0.59-0.65-0.54 0.131 0.171 0.171 0.163 0.171 0.163 0.163 0.02 0.23 0.52-0.05 0.01 0.07 0.068 0.171 0.171 0.163 0.171 0.163 0.163 0.10 0.12 0.29-0.02-0.01-0.07 0.072 0.174 0.165 0.165 0.165 0.165 0.165-0.00 0.20 0.25-0.02 0.00 0.01 0.076 0.174 0.165 0.165 0.165 0.165 0.165 0.07 0.34 0.14 0.09-0.04-0.15 0.096 0.171 0.171 0.163 0.171 0.163 0.163 0.10 0.21 0.30-0.03-0.01 0.01 0.045 0.171 0.171 0.163 0.171 0.163 0.163 0.01 0.02 0.22 0.02-0.01 0.01-0.046 0.169 0.169 0.169 0.169 0.161 0.161 0.085 0.167 0.167 0.167 0.167 0.167 0.167 0.28-0.06 0.31 0.15-0.11-0.07 0.055 0.167 0.167 0.167 0.167 0.167 0.167 0.77 0.03 0.27 0.70-0.81-0.63 0.175 0.167 0.167 0.167 0.167 0.167 0.167 0.38 0.36 0.51 0.10-0.08-0.20 0.086 0.167 0.167 0.167 0.167 0.167 0.167 0.43-0.10 0.26 0.26-0.23-0.11 0.080 0.167 0.167 0.167 0.167 0.167 0.167 0.31-0.05 0.20 0.27-0.23-0.02 0.086 0.167 0.167 0.167 0.167 0.167 0.167 0.22 0.26 0.17 0.11-0.03-0.21 When omparing results without weight restritions and with weight restritions, as expeted d is worse (i.e., higher) for all units when the weight restritions are inorporated into the model (Figure 4). Only DMU 17 (d = -0.046) is effiient when the weight restritions are onsidered in the Value-Based DEA method. The best unit (DMU 17) without onsidering weight restritions is still the best DMU onsidering the weight restritions previously stated. This means that all units should aim at ahieving DMU 17 s utility, but not neessarily trying to imitate the mix of inputs and outputs of that DMU (this is analyzed in setion 6). 17

Value-Based DEA method (2005) 0,3 0,25 0,2 0,15 0,1 0,05 0-0,05-0,1-0,15-0,2 17 12 16 6 10 28 14 11 15 21 34 36 5 37 33 2 35 4 9 13 3 39 22 38 40 8 23 7 24 31 20 30 29 26 18 1 32 27 19 25 2005 2005 WR Figure 4. Comparison of Value-Based DEA method results without and with weight restritions, ranked by 2005 effiieny measure with weight restritions. In onlusion, it an be stated that only one unit is lassified as effiient in the analysis when managerial preferenes are inorporated into the model. This is due to the fat that managerial preferenes did not give muh freedom to DMUs for hoosing the weights. In further experiments we used the 2005 data and tried a less stringent trade-off limit w OPEX 3.52 w CLI, (if the answer to the indifferene judgment question was h = 5 million euros, aommodating the response of DM) but the results were very similar. With this trade-off limit (w OPEX 3.52 w CLI ) and omparing the results in terms of units ranking with the one established by the DM (w OPEX 1.05 w CLI ), we onluded that there was another effiient DMU (DMU 16) besides DMU 17. The ranking hanges are small and have to do with a few exhanges of DMUs in onseutive positions. Another experiment was performed, in this ase for the ranking of weights. It onsisted in exhanging the positions of the fators plaed in seond and third plae (x MLL or x NI ), beause the first and last position in the ranking of the weights were undoubted. The ranking of sale oeffiients for all fators beame: w OPEX w MLL w CC w NLL w CLI w OPEX w NI w CC maintaining the less stringent trade-off limit w OPEX 3.52 w CLI. In fat, there were no hanges in the number of effiient units, some exhanges were deteted only in the ranking of ineffiient DMUs. The results are thus quite insensitive with regards to the respetive answers provided by the DM. 6. Prospets for Improvement in Ineffiient Units The main objetive of this study was to illustrate the results of a different possibility for making an internal benhmarking exerise with the introdution of managerial preferenes. Hene, the results are expeted to be loser to what the DM judges to be the best pratie for a speifi ativity of the ompany. The use of the 2004-2005 data allows a omparison of results from different approahes, the one performed by Weyman-Jones et al. [31] and Value-Based DEA. To illustrate the various possibilities for improvement in ineffiient units, only the ten units with 18

higher x OPEX in 2005 (the lower u OPEX value) were onsidered. The proposed solution of an effiieny target (projetion) for eah ineffiient DMU is displayed in Table 7 in the previous setion (Phase 2). For that year all units should math the overall utility of DMU 17, the only effiient unit. Note that to attain the effiieny status these ineffiient DMUs must hange their utility in eah fator by the amount indiated by s, whih does not neessarily orrespond to an improvement sine some of the ineffiient units may have negative slaks. In fat, an ineffiient DMU may be able to math its peers on the effiient frontier having a negative slak orresponding to an input (meaning it would inrease) or a negative slak orresponding to an output fator (meaning it would derease) if this is ompensated by enough improvement in other fators. For the DM this revealed to be diffiult to understand and thus there was a need to propose another solution. Let u k denote the utility of the best DMU using the optimal weighting vetor resulting from (1) with the weight restritions added, i.e., the utility value that DMU k ought to ahieve: q å u k * = w k * u (DMU k )+ d k * =1 (3) The problem solved in Phase 2 admits alternative optimal targets, orresponding to different ways of losing the gap d k. These targets orrespond to different projetions on the effiient frontier. The purpose is to onstrain the proposed effiieny targets to ahieve u k, not only to avoid those targets that imply an inrease of inputs or a derease of outputs, but also to hoose whih fators an be hanged, given the harateristis of eah unit. Hene, this requires that the utility value annot derease in any fator and targets are fored to maintain or improve the performane of all fators [41]. To aomplish this it is neessary to define two sets to develop new model formulations (4)-(5). Let S < = { Î { 1,...,q } : s * < 0 in Phase 2} free slaks; these slaks will now beome null onstants. Let denote the negative slaks in the optimal solution obtained for (2) with denote the remaining slaks, whih will be onsidered as non-negative variables. Therefore, a formulation whih yields an alternative target an be obtained by solving the linear problem (4) in whih the maximum slak (in terms of value) shall be minimized to ahieve the global utility target. No negative slaks are allowed, but the target will no longer be a onvex ombination of the observed DMUs. S ³ = { Î { 1,...,q } : s * ³ 0 in Phase 2} min, s k s. t. S u s w s DMU s k k * d s 0, S k * k 0, 1,..., q 1, 1,..., q (4) 19

A parameter σ is introdued to bound the value a slak may have. The purpose is to avoid setting unrealisti improvement targets. The restrition s - σ 0 in (4) implies that only the fators with negative slaks in Phase 2 have σ > 0 and an be hanged in order to overome the gap with the peer. Targets will never exeed the value 1 in any fator due to the restrition u (DMU k )+s 1. This ensures that the utility funtion does not spill over outside the ranges eliited for the performanes. A proposal to improve performane of all ineffiient units is to blok the hanges in the fators with negative slaks in Phase 2 (see proposal 1 in Table 8). However, for some of these ten ineffiient units, the x OPEX derease is attainable, but for others, given their speifi harateristis, the proposed value would be impossible to ahieve. For those units the DM does not onsider attainable a x OPEX redution greater than 40%. This requires a new formulation (5) aording to whih the gap to the peer is distributed by all the fators (inputs and outputs), in a balaned way (proposal 2 in Table 8): min, s k s. t. q 1 u s w s DMU s k k * d s 0, 1,..., q k * k 0, 1,..., q 1, 1,..., q (5) However, the result still did not satisfy the DM in some ases. With this proposal all units redued the x OPEX value by a perentage below 40%; however, for some units it was impossible to inrease the y CLI value reommended and the DM did not aept inreases in y NLL. Hene, the gap between the unit under evaluation and DMU 17 is losed by dereasing input fators x MLL, x CC and x NI and inreasing the output fator y CLI in some ases (DMU 29) (see proposal 3 in Table 8). Fators that have a limit to redution (whih is the ase of x OPEX ) or a limit to inrease (the ase of y CLI ) are displayed in bold typefae in Table 8 (proposal 3). The DM approved that the DMUs 4, 5, 28 and 33 maintain the dereases suggested in proposal 1. For the remaining ones there are proposals 2 and 3 to hoose from, aording to the speifi harateristis of eah DMU. It is relevant to point out that the DM was pleased with the possibility of having the entire range of targets available for making the hoie. 20