THE Internet of Things (IoT) is likely to be incorporated

This is the author's versio of a article that has bee published i this oural. Chages were made to this versio by the publisher prior to publicatio. IEEE INTERNET OF THINGS JOURNAL, VOL. 5, NO. 1, FEBRUARY 18 1 O Reducig IoT Service Delay via Fog Offloadig Ashka Yousefpour, Studet Member, IEEE, Geya Ishigaki, Studet Member, IEEE, Riti Gour, ad Jaso P. Jue, Seior Member, IEEE Abstract With the Iteret of Thigs (IoT) becomig a maor compoet of our daily life, uderstadig how to improve the quality of service (QoS) for IoT applicatios through fog computig is becomig a importat problem. I this paper, we itroduce a geeral framework for IoT-fog-cloud applicatios, ad propose a delay-miimizig collaboratio ad offloadig policy for fog-capable devices that aims to reduce the service delay for IoT applicatios. We the develop a aalytical model to evaluate our policy ad show how the proposed framework helps to reduce IoT service delay. Idex Terms Fog Computig, Iteret of Thigs, QoS, Cloud Computig, Markovia Queueig Networks, Task Offloadig I. INTRODUCTION THE Iteret of Thigs (IoT) is likely to be icorporated ito our daily life, i areas such as trasportatio, healthcare, idustrial automatio, smart home, smart city, ad emergecy respose. The IoT eables thigs to see ad sese the eviromet, to make coordiated decisios, ad to perform tasks based o these observatios [2]. I order to realize the full beefits of the IoT, it will be ecessary to provide sufficiet etworkig ad computig ifrastructure to support low latecy ad fast respose times for IoT applicatios. Cloud Computig has bee the mai eabler for IoT applicatios with its ample storage ad processig capacity. Noetheless, beig far from ed-users, cloud-supported IoT systems face several challeges icludig high respose time, heavy load o cloud servers ad lack of global mobility. I the era of Big Data, it may be iefficiet to sed the extraordiarily large amout of data that swarms of IoT devices geerate to the cloud, due to the high cost of commuicatio badwidth, ad due to the high redudacy of data (for istace, costat periodic sesor readig). Istead of movig data to the cloud, it may be more efficiet to move the applicatios, storage, ad processig closer to the data produced by the IoT. Fog computig is well suited to address this issue by movig the above services closer to where data is produced. Fog computig is a emergig cocept that puts the cloud services closer to the ed users (ad thigs) for better QoS [3], [4]. Fog is a itelliget layer sittig betwee cloud ad IoT, that brigs low latecy, locatio awareess, ad wide-spread geographical distributio for the IoT. Iheritig mai cocepts from cloud computig, fog provides computatio, storage, ad etworkig services to ed-users, aywhere alog the thig-tocloud cotiuum, accordig to OpeFog Cosortium. The idea The authors are with the Advaced Network Research Lab, Departmet of Computer Sciece, The Uiversity of Texas at Dallas, Richardso, TX, 7 USA. Email: {ashka, gishigaki, riti.gour, ue}@utdallas.edu A earlier versio of this paper appeared i IEEE EDGE 17, Hoolulu, Hawaii [1]. is to serve the requests that demad real-time ad low-latecy services at the fog, ad to sed the requests that demad permaet storage or require extesive aalysis to the cloud [5], [6]. Due to the coutless beefits of fog, the research i this area has bee gaiig attetio, ad researchers have recetly started to defie visios, basic otios, ad possible architectures of fog computig [3], [4], [7]. A. Related Work The problem of task offloadig i fog has recetly gaied attetio from researchers. The authors i [8] propose ad formulate a cooperative offloadig policy betwee two fog data ceters for load balacig. Similarly, the work i [9] aalyzes a offloadig policy betwee multiple fog data ceters i a rig topology. I this paper we formulate ad propose a geeral policy for fog odes to offload IoT requests to other fog odes or to forward them to the cloud, i order to miimize the IoT service delay. This policy is geeral, i the sese that the umber ad topology of fog odes are arbitrary. There are other similar efforts aimed at miimizig delay via fog computig, which are ot directly related to delay-miimizig fog offloadig. For istace, the authors i [5] itroduce a hybrid architecture that itegrates a cloud radio access etwork ad a fog radio access etwork to hadle etwork traffic. They propose the idea of servig delay-tolerat traffic i the cloud ad hadlig low-latecy traffic i the fog. A similar problem to task offloadig i the fog (fog offloadig), is the task or workload assigmet problem i the fog, which is assigig tasks or workloads to either fog odes or cloud servers, while miimizig delay, cost, or eergy. I these problems, the set of tasks are give ad the problem is modeled as a static optimizatio problem that determies the assigmet of the tasks or workloads. Namely, the authors i [10] study base statio associatio, task distributio, ad virtual machie placemet i fog-computigsupported medical cyber-physical systems, while miimizig the overall cost ad satisfyig QoS requiremets. Aother effort is the work i [11] that addresses power-delay trade-off i cloud-fog computig by workload allocatio. More recet work i [12] focuses o theoretical modelig of fog computig architectures, specifically, service delay, power cosumptio, ad cost. Nevertheless, task assigmet problems are static i ature ad have high complexity. I task assigmet problems, all the parameters (of odes ad etwork) are assumed to be kow to a etity that solves the overall problem for the optimal solutio, which may ot be practical. The fog offloadig problem does ot have such assumptios. Recet work i [13] addressed the desig of a policy for assigig tasks that are geerated by mobile subscribers to

This is the author's versio of a article that has bee published i this oural. Chages were made to this versio by the publisher prior to publicatio. The fial versio of record is available at http://dx.doi.org/10.1109/jiot.17.27882 IEEE INTERNET OF THINGS JOURNAL, VOL. 5, NO. 1, FEBRUARY 18 edge clouds, to achieve a power-delay trade-off. The problem is first formulated as a static optimizatio problem. The authors the propose a distributed policy for task assigmet that could be implemeted efficietly o the edge clouds. However, IoT-to-cloud or fog-to-cloud commuicatio ad computatio offloadig capability are ot cosidered. Moreover, i the distributed task assigmet policy, edge clouds broadcast their status cotiuously to mobile subscribers, which may limit scalability i IoT etworks with a large umber of edge devices ad ed users. I this paper we study the problem of fog offloadig for reducig IoT service delay, which is a fudametally differet problem from the problems discussed above (e.g. [5], [10] [13]). The proposed offloadig policy is geeral as opposed to studies [8], [9], as it does ot place ay restrictios o the umber or topology of the fog odes. I the rest of the paper, we discuss our proposed fog framework ad offloadig policy. B. Cotributio I this work, we itroduce a commo sese geeral framework to uderstad, evaluate ad model service delay i IoTfog-cloud applicatio scearios. We the propose a delaymiimizig offloadig policy for fog odes whose goal is to reduce service delay for the IoT odes. I cotrast to the existig works i the literature, the proposed policy cosiders IoT-to-cloud ad fog-to-cloud iteractio, ad also employs fog-to-fog commuicatio to reduce the service delay by sharig load. For load sharig, the policy cosiders ot oly the queue legth but also differet request types that have variat processig times. Additioally, our scheme is ot limited to ay particular architecture (such as a cellular etwork), ad the umber, type, or topology of IoT odes, fog odes, ad cloud servers are ot restricted. We also develop a aalytical model to evaluate service delay i the IoT-fog-cloud scearios, ad perform extesive simulatio studies to support the model ad the proposed policy. This paper exteds our previous work [1] as we itroduce a complete aalytical model for evaluatig service delay, ad we preset additioal results. C. Paper Orgaizatio The rest of this paper is orgaized as follows. We itroduce our IoT-fog-cloud framework i Sectio II ad propose the policy for reducig IoT service delay i Sectio III. We the formally itroduce the aalytical model for evaluatig IoT service delay ad its compoets i Sectio IV, ad explai the umerical results of our experimet i Sectio V. Fially, Sectio VI summarizes the paper ad provides future directios for this work. 2 Fig. 1. Geeral framework for IoT-fog-cloud architecture. Each layer is partitioed ito domais where a sigle applicatio is implemeted. processig uits, such as a rack of physical servers or a server with multiple processig cores. I each layer, odes are divided ito domais where a sigle IoT-fog-cloud applicatio is implemeted. For istace, a domai of IoT odes (i a factory, for istace) is show i dark gree, ad they commuicate with a domai of fog odes associated with the applicatio. A domai of IoT odes could comprise IoT devices i a smart home, temperature sesors i a factory, or soil humidity sesors i a farm where all the IoT devices i the viciity are cosidered to be i a sigle domai. Normally the fog odes i oe domai are placed i close proximity to each other, for example, i a eighborhood or i levels of a buildig. Each domai of fog odes is associated with a set of cloud servers for a sigle applicatio. The basic way i which IoT odes, fog odes, ad cloud servers operate ad iteract is as follows. IoT odes ca process requests locally, sed it to a fog ode, or sed it to the cloud; fog odes ca process requests, forward requests to other fog odes i the same domai, or forward the requests to the cloud; cloud servers process requests ad sed the respose back to the IoT odes. I this work, the aim is to miimize service delay for IoT devices i the proposed framework based o fog computig. The fog layer lies betwee IoT devices ad the cloud; thus, it ca hadle a maority of IoT service requests i order to reduce the overall service delay. Defiitio 1. Service delay for a IoT ode is the time required to serve a request, i.e. the time iterval betwee the momet whe a IoT ode seds a service request ad whe it receives the respose for that request. We first itroduce the delay-miimizig policy i the followig sectio, the formulate the IoT service delay to evaluate the policy aalytically. II. G ENERAL I OT-F OG -C LOUD F RAMEWORK III. F OG N ODE C OLLABORATION P OLICY Figure 1 illustrates a geeral framework for a IoT-fogcloud architecture that is cosidered i this work. There are three layers i this architecture: IoT layer, where the IoT devices ad ed-users are located, fog layer, where fog odes are placed, ad cloud layer, where distributed cloud servers are located. A cloud server ca be composed of several I this sectio, we itroduce the framework i which fog odes collaborate with each other to fulfill the requests set from IoT odes to the fog layer. If a fog ode ca accept a request based o its curret load, it processes the request; however, whe the fog ode is busy processig may tasks, it may offload the request to some other fog odes or to the

This is the author's versio of a article that has bee published i this oural. Chages were made to this versio by the publisher prior to publicatio. IEEE INTERNET OF THINGS JOURNAL, VOL. 5, NO. 1, FEBRUARY 18 3 TABLE I AN EXAMPLE OF REACHABILITY TABLE OF FOG NODE RTT (ms) Est. Waitig Time (ms) Node ID 3.85 30.4 129.110.83.81 * 5.3 72.3 129.110.5.75.. cloud (this is called load sharig). The cocept of load sharig is well studied i the literature [14], [15], ad we borrow similar cocepts for the desig of the policy by which fog odes collaborate. This collaboratio is discussed i detail the followig subsectios. A. Modes of Iteractio We propose two modes of iteractio for fog odes: oe mode is cetralized, i which a cetral authority cotrols the iteractio of fog odes, ad the other oe is distributed, where fog odes iteract with their eighborig fog odes usig a uiversal protocol. I the cetral mode of iteractio, i each domai of fog odes there is a Cetral Node that cotrols the iteractio amog the fog odes i that domai. I particular, based o the curret coditio of their queue, fog odes i domai M report their estimated time to process the requests i queue ad i service (i.e. estimated waitig time) to the Cetral Node of domai M. Defiitio 2. Estimated waitig time (W ) is the sum of the estimated processig delays of the requests i the queue (queueig delay), plus the estimated processig delay of the request uder service. The Cetral Node of domai M aouces the estimated waitig time of the fog odes i domai M to their eighborig fog odes i domai M. Upo receptio of estimated waitig times from the Cetral Node, fog odes record them i a Reachability table that they maitai. The Reachability table is utilized by fog odes whe makig a decisio as to which fog ode to offload a request for processig. The Reachability table has three colums as it is show i Table I. The iformatio i the Reachability table of a fog ode i domai M is updated by both the Cetral Node of domai M ad the fog ode itself; the Cetral Node aouces the estimated waitig times of eighborig fog odes, ad the fog odes measure the roud-trip delay amog themselves. A efficiet way to estimate the waitig time of fog odes is discussed i Sectio III-D. Usig estimated waitig time ad roud-trip delay i the Reachability table, each fog ode selects a fog ode from amog its eighbors as the best fog ode. It does so by selectig a eighborig fog ode i the same domai with the smallest estimated waitig time plus half of the roud-trip delay. The best fog ode is marked with star * i Table I. I the distributed mode of iteractio, there is o Cetral Node; istead, fog odes i domai M ru a protocol to distribute their state iformatio to the eighborig fog odes i the same domai. Each fog ode maitais the Reachability table usig the estimated waitig time it receives from. Request received Estimated Waitig Time < Threshold (W < θ ) Yes Place request i queue Update Waitig Time No N fwd = e M or All eighbors visited No Icremet N fwd of request Forward to best eighbor Fig. 2. Policy of fog ode for hadlig the received requests. Yes Forward to cloud eighborig fog odes, ad the roud-trip delay from itself to its eighbors. Similar to the cetral mode of iteractio, a fog ode always selects the best eighbor with the smallest estimated waitig time plus half of the roud-trip delay. Compariso: The cetral mode of iteractio ca be see as cetral resource orchestratio, where the Cetral Node is kowledgeable of the topology ad the state of the fog odes i a domai. Iheretly, the cetral mode of iteractio is easier to implemet, because there is o eed for a distributed commuicatio protocol amog fog odes; all the procedures of the estimated waitig time aoucemets will be implemeted o the Cetral Node. Moreover, the Cetral Node could be used to push fog applicatios ad software updates to the fog odes i a domai. O the other had, the distributed mode is more suitable for scearios i which fog odes are ot ecessarily static, or whe the fog etwork is formed i a ad hoc maer. Furthermore, i the distributed mode, there is o eed to have a dedicated ode to act as the Cetral Node, which is a reductio i the cost of deploymet ad less vulerable to a sigle poit of failure. For the purpose of this paper ad our simulatio studies, we have chose the distributed mode of iteractio, sice our policy oly eeds to aouce estimated waitig time updates, which is fairly easy to implemet o fog odes. B. Whe to Offload a Task I this subsectio, we discuss the decisio fog odes make for processig or offloadig a task to other fog odes. The cocept of computatio offloadig for mobile devices is studied i [16], [17], ad a umber of possible policies are discussed (based o eergy cosumptio, respose time, availability, etc.). I our scheme, the decisio to offload a task is based o the respose time of a fog ode, which depeds o several factors: the amout of computatio eeded to be performed o a task, ad the queueig status (i terms of curret load) ad processig capability of a fog ode. I particular, we propose a model that takes ito accout the differet processig times of differet idividual tasks. I other words, i our model, there is a distictio betwee heavy processig tasks ad light processig tasks. We assume requests have two types: type Light (light processig) with a average processig time of z at the fog ode ad Z k at the cloud server k, ad type Heavy (heavy processig) with a average processig time of z at the fog ode ad Z k at the cloud server k. For example, requests set by temperature sesors to fog odes for calculatig the

This is the author's versio of a article that has bee published i this oural. Chages were made to this versio by the publisher prior to publicatio. IEEE INTERNET OF THINGS JOURNAL, VOL. 5, NO. 1, FEBRUARY 18 4 average room temperature ca be see as light processig tasks. Similarly, a licese plate readig request i a recorded video of a vehicle, set by a traffic camera to fog odes is a example of a heavy processig task. Note that, i geeral, more tha two task types could be cosidered; however, i this paper, we oly cosider two task types for the simplicity of the presetatio. The procedure of processig or forwardig requests by fog odes is show i Fig. 2. Whe a fog ode receives a request, it first checks the estimated waitig time of tasks i the queue. A method for estimatig processig delay ad hece waitig time of the tasks i the queue is discussed i Sectio III-D. If the estimated waitig time is smaller tha a threshold θ at fog ode, the fog ode will accept the task. The task eters the queue ad the estimated waitig time is updated. If ot, the fog ode offloads this request, either to oe of its eighbors, or to the cloud. If the umber of times the request has bee offloaded (N fwd ) is less tha the offload limit e M for domai M (the domai where fog ode belogs), the request will be forwarded to a eighborig fog ode. If ot, it will be forwarded to the cloud. The value of θ depeds o the icomig traffic patter from IoT odes to fog odes i a domai. I geeral, if all fog odes have low load, offloadig is uecessary; ad if all fog odes have heavy load, offloadig will ot help to reduce the delay sigificatly. Offloadig helps whe there is a high degree of variace i the load amog fog odes. The value of θ could be adusted i implemetatio based o the give traffic demads, to reach a optimal value that miimizes the average service delay. C. Fidig Best Neighbor Roud-trip delays ca be measured at the time of the setup, so that fog odes have the values for the Reachability Table. Upo receivig a estimated waitig time sample, either from the Cetral Node or aother fog ode depedig o the mode, the fog ode updates the correspodig estimated waitig time i the Reachability table. As discussed, a fog ode selects as its best eighbor the fog ode for which the estimated delay for the curret request, if offloaded, is miimal. Whe a request is offloaded to the fog ode s best eighbor, it udergoes roughly half of the correspodig roud-trip delay to reach the best eighbor. If the request eters the best eighbor s queue, it speds time roughly equal to the estimated waitig time of the best eighbor, before it fiishes processig at that eighbor. If the request does ot eter the queue of the fog ode s best eighbor, the request should be offloaded agai (multiple offloads are possible whe eighborig fog odes are busy with requests). It is worth metioig that if the fog odes are mobile, they may eed to frequetly measure the roud-trip delays ad update them i the Reachability table. Note that for makig optimal offloadig decisio, other compoets, such as propagatio ad trasmissio delay betwee fog odes ad IoT odes should be cosidered. However, this may ot be practical, sice each fog ode would eed to kow the delay from the eighborig fog odes to the correspodig IoT odes associated with them. D. Checkig ad Updatig Queue Status Whe fog odes receive requests from IoT odes that participate i a applicatio, they eed to distiguish betwee type Light ad type Heavy requests, i order to update the queue parameters. To address this, we assume requests have i their header a field that idetifies the type of the request (for istace Traffic Class field i IPv6 header). The applicatio hece sets the field i the header of the packets beig geerated from IoT odes. Fog odes always eed to kow a estimate of the curret total processig delay of the tasks i their queue (i.e. estimated waitig time), both for whe they eed to make offloadig decisios, ad whe reportig their estimated waitig time to eighborig fog odes. A possible method for estimatig the waitig time is for the fog ode to store a estimate of the curret waitig time of the tasks i the queue. Upo arrival to or departure from the queue, the fog ode simply updates the estimated waitig time of the tasks i the queue. To calculate the estimated waitig time of the tasks i the queue W, fog ode periodically measures processig times of recetly processed tasks (ew z ) ad updates the estimated processig time of each task type (z ). For light processig tasks, we have z = (1 α) z + α ew z (a similar equatio holds for heavy processig tasks). This equatio is a weighted average that maitais a weight of α for ew measured processig times ad a weight of 1 α for the old measuremets (curret estimate). TABLE II TABLE OF NOTATIONS d i Service delay for a IoT ode i p I i Probability that IoT ode i processes its ow request p F i Probability that IoT ode i seds its request to fog layer p C i Probability that IoT ode i seds its request to the cloud Xst LL Propagatio delay from ode s i layer L to ode t i layer L, where s, t {i,, k} ad L, L {I, F, C} Sum of all trasmissio delays o liks betwee ode s Yst LL i layer L to ode t i layer L, where s, t {i,, k} ad L, L {I, F, C} A i Average processig delay of requests at IoT ode i a i Average processig delay of type Light requests at IoT ode i (a i is average proc. delay of type Heavy requests) Delay of processig ad hadlig requests of IoT ode i L i i the fog layer (ad possibly the cloud layer), where fog ode is the fog ode to which IoT ode i iitially seds its request (L i = L i (0)) Delay of processig ad hadlig requests of IoT ode i L i (x) i fog layer (ad possibly cloud layer), by fog ode durig the x th offload i the fog layer SD L Set of odes i domai D at layer L, where (L, D) {(I, P), (F, M), (C, N )} S L D SL D : set of odes (i all domais) at layer L H k Average waitig time at cloud server k k Average waitig time of a sigle processig uit at cloud server k ς i Average size of request data that IoT ode i geerates b i Probability that a geerated request at IoT ode i is Light W Waitig time of fog ode c Number of type Light requests i fog ode s queue P Probability that a icomig request is accepted by fog ode θ Offloadig threshold at fog ode e M Maximum offload limit at the fog layer i domai M q The fog fairess parameter

This is the author's versio of a article that has bee published i this oural. Chages were made to this versio by the publisher prior to publicatio. IEEE INTERNET OF THINGS JOURNAL, VOL. 5, NO. 1, FEBRUARY 18 5 To obtai the total processig time of the tasks i the queue o the fly, fog ode stores the curret umber of type Light ad Heavy requests i the queue c ad c, respectively, i additio to z ad z. Fog odes the multiply the estimated processig time of each task type by the umber of tasks of that type i the queue. I other words, the estimated waitig time W of the fog ode will be A. Network Model W = c z + c z. (1) IV. ANALYTICAL MODEL We model the etwork as a udirected graph G = [V ; E; w], where ode set V icludes set of IoT odes, fog odes, ad cloud servers (V = S I S F S C ). The Edge set E represets the commuicatio liks betwee the odes. For istace, there is a lik betwee IoT ode i ad fog ode if they commuicate. The edge weight set w represets the weight of the edges betwee odes. We use the tuple (X, R) for the edge weights, where X represets the propagatio delay, ad R the trasmissio rate betwee two odes. We will ot pose ay restrictios o the topology, except for the metioed logical three-layer architecture, as this makes the model easy to preset. B. Service Delay Recall that IoT odes process requests locally, sed it to a fog ode, or sed it to the cloud. Thus, service delay d i for a IoT ode i ca be writte as: d i = p I i (A i ) + p F i (Xi IF + Yi IF + L i ) +p C i (Xik IC + Yik IC + H k + Xki CI + Yki CI ); = f(i), k = g(i), (2) where p I i is the probability that the IoT ode i processes its ow request at the IoT layer, p F i is the probability of sedig the request to the fog layer, ad p C i is the probability that the IoT ode seds the request directly to the cloud; p I i + pf i + p C i = 1. A i is the average processig delay of the IoT ode i whe it processes its ow request. Xi IF is propagatio delay from IoT ode i to fog ode, Yi IF is sum of all trasmissio delays o liks from IoT ode i to fog ode. Similarly, Xik IC is propagatio delay from IoT ode i to cloud server k, Yik IC is sum of all trasmissio delays o liks from IoT ode i to cloud server k. Xki CI ad Yki CI are the propagatio ad trasmissio delays from cloud server k to IoT ode i. The trasmissio ad propagatio delay from the fog layer to IoT ode i will be icluded i L i, sice the request may be further offloaded to a differet ode i the fog layer (more details i Sectio IV-E). L i is the delay for processig ad hadlig requests of IoT ode i i the fog layer (ad possibly cloud layer, if fog odes offload the request to the cloud), where fog ode is the first fog ode to which IoT ode i iitially seds its request. Note that fog ode might offload the request to aother fog ode or to the cloud, ad that all the correspodig icurred delays are icluded i L i. H k is the average delay for hadlig the request at the cloud server k, which cosists of the queueig time at the cloud server k plus the processig time at the cloud server k. (L i ad H k will be discussed i further detail i Sectios IV-E ad IV-K respectively). f(i) ad g(i) are mappig fuctios that idicate the fog ode ad cloud server k to which IoT ode i seds its requests, respectively (refer to Table III). For istace, if i a applicatio, IoT ode i always seds its requests to fog ode i the fog layer, the f(i) =. I aother sceario if IoT odes always sed their requests to the closest fog ode, which traslates to the idex of the fog ode with smallest propagatio delay (distace) from IoT ode i. To formalize the problem further, let us defie a IoT-fogcloud applicatio Ψ. I the rest of this work all the equatios are defied o a sigle applicatio Ψ(N, M, P). i the fog layer, the f(i) = arg mi X IF i Defiitio 3. IoT-fog-cloud applicatio Ψ is a applicatio o domai N of cloud servers, domai M of fog odes ad domai P of IoT odes, ad is writte as Ψ(N, M, P). Examples of Ψ are: video processig, temperature sesor reportig, traffic road aalysis, ad oil rig pressure moitorig. We do ot assume ay distributio for p I i, pf i, ad pc i, sice their values will be defied by idividual applicatios ad based o QoS requiremets ad policies. I other words, they will be give to the scheme as iput. I Sectio V, we assume differet values for these probabilities for type Heavy ad Light requests, depedig o the type of IoT ode i. By usig the model to evaluate the service delay for the IoT odes i oe applicatio domai, our goal is to miimize delay through the defiitio of policies for exchagig requests betwee IoT odes, fog odes, ad cloud servers. We formulate the above statemet as the miimizatio of the average service delay of IoT odes i domai P, or mi 1 SP I d i, (3) i S I P where SP I deotes the set of IoT odes that are i domai P. Similar to above, SM F idicates the set of fog odes i domai M, ad SN C is the set of cloud servers i domai N. I the followig subsectio, we will discuss i more details the compoets of the service delay. C. Propagatio ad Trasmissio Delays I Equatio (2) we have the IoT-cloud delay terms Xik IC, Yik IC, XCI ki, Y ki CI whe the IoT ode seds its requests directly to the cloud layer. These terms are effective i the equatio i cases where the applicatio Ψ is ot implemeted i the fog layer (SM F = {}), or whe the request must be set to the cloud for archival purposes, or whe there is o fog layer ad the IoT ode commuicates directly to the cloud. Recall that Yi IF is the sum of all trasmissio delays o the liks from IoT ode i to fog ode. If IoT ode i ad fog ode are l-hop eighbors, we will have Y IF i = l ς i R l. (4)

This is the author's versio of a article that has bee published i this oural. Chages were made to this versio by the publisher prior to publicatio. IEEE INTERNET OF THINGS JOURNAL, VOL. 5, NO. 1, FEBRUARY 18 6 where ς i is the average amout of request data that IoT ode i geerates, ad R l is the trasmissio rate of the l th lik betwee IoT ode i ad fog ode. The expaded equatios for trasmissio delays betwee other layers (Y IC ik are derived similarly to Y IF i. D. Processig Delay of IoT ode, Y F C k, etc.) As explaied i Equatio (2), for IoT ode i, the average processig delay is A i. If b i deotes the probability that a geerated request at IoT ode i is type Light, ad b i = 1 b i is the probability that a geerated request at IoT ode i is type Heavy, A i could be writte as A i = b i a i + b i a i, (5) where a i is the average processig time of requests of type Light at IoT ode i, ad a i is the average processig time of requests of type Heavy at IoT ode i. If IoT ode i is of type Light (or type Heavy), i.e. it oly geerates type Light (or type Heavy) requests, b i = 1 (or b i = 0) ad A i = a i (or A i = a i ). Should more tha two types of tasks be cosidered, the equatio above (ad other equatios with two task types) could be geeralized to support more task types. Table III summarizes the importat parameters of differet layers ad shows the mappig fuctios that are used i the aalytical model. A few of the parameters ad mappig fuctios have already bee itroduced (such as a i ad f(i)); yet others will be itroduced i the followig subsectios. E. Delay i Fog Layer I this sectio, we defie a recursive equatio for L i. Let us defie L i (x) as the delay of processig ad hadlig requests of IoT ode i i the fog layer (ad possibly the cloud layer), by fog ode durig the x th offload i the fog layer (x 0). Also, let us label L i L i (0). If P deotes the probability that a request is accepted by fog ode (eters the queue of fog ode ) L i (x) ca be writte as: L i (x)= P (W + Xi F I + Yi F I ) [ +(1 P ) [1 φ(x)]. [ X F F + Y F F + L i (x + 1)] + φ(x) [X F C k + Y F C k + H k + X CI ki + Yki CI ] ] ; = best(), k = h(). (6) I the equatio above, W is the average waitig time i fog ode. φ(x) is the offloadig fuctio, which is defied as { 0 x < e M φ(x) =. (7) 1 x = e M If x < e M, the φ(x) = 0, which idicates that the request will be offloaded to aother fog ode. If x = e M, the φ(x) = 1, which meas that the forward limit is reached ad that the request will be offloaded to the cloud (recall Fig. 2). x takes o iteger values i [0, e M ]. best() ad h() are mappig fuctios that map a particular fog ode to its best fog eighbor ad the cloud server TABLE III PARAMETERS AND MAPPING FUNCTION OF DIFFERENT LAYERS Layer Mappig Node Arrival Service Avgerage Name Fuctio Idex Rate Rate Processig Time Cloud k l k, l k u k, u k Z k /m k, Z k /m k best h Fog λ, λ µ, µ z, z f g IoT i γ i p I i, γ i pi i ν i, ν i a i, a i associated with the fog ode, respectively (see Table III). Sice the choice of the best eighbor of a fog ode depeds o the curret state of the system, the system is dyamic ad best() will be a poiter to the curret best eighbor of fog ode. h() simply holds the idex of the associated cloud server for fog ode. Explaatio: Assume fog ode is the oe that is selected first by the IoT ode i. Whe a request from a IoT i ode reaches the fog layer, fog ode first tries to process the request. The request eters this ode s processig queue with probability P, ad does ot with probability (1 P ), which depeds o estimated waitig time. If the request eters the queue, it will experiece average waitig time W, ad propagatio ad trasmissio delays of Xi F I ad Yi F I to retur back to the IoT ode. Note that the processig delay of the curret task eterig the fog ode s queue is already icluded i the calculatio of W, because of the way waitig time is defied. If the request does ot eter fog ode, fog ode will offload the request to its best fog eighbor, which icurs a propagatio ad trasmissio delay of X F F 1 ad Y F F, respectively. The request also udergoes a delay of L i (x+1), which is the delay of processig ad hadlig the request i the fog layer (ad possibly the cloud layer), by fog ode durig the x + 1 st offload. Fially whe a request has bee offloaded e M times (φ(x) = 1), if the last fog ode eeds to offload the request, it will do so by offloadig it to the cloud, which icurs fog-cloud propagatio ad trasmissio delay of Xk F C ad Yk F C, respectively, cloud processig delay of H k, ad cloud-iot propagatio ad trasmissio delay of Xki CI ad Yki CI, respectively. The reaso Xi F I ad Yi F I are icluded i Equatio (6) ad ot Equatio (2) is because a request set from IoT ode i to fog ode could be received from fog ode (offloaded to ad processed at fog ode ). I this case the propagatio ad trasmissio delay from IoT layer to fog layer are Xi IF ad Yi IF, respectively, but the propagatio ad trasmissio delays from fog layer to IoT layer are X F I i ad Y F I i. Boudary case: Cosider a domai M of fog odes where e M = 0, which meas o forwardig is allowed. I this case, if a request does ot eter a fog ode s queue, it will be offloaded to the cloud. I this case, L i = P (W + Xi F I (1 P ) [Xk F C + Y k F C + H k + Xki CI + Yki CI ]. F. Average Waitig Time of Fog Node + Y F I i ) + To obtai a equatio for the average waitig time of fog ode, W, we eed to model fog odes. We assume a fog

This is the author's versio of a article that has bee published i this oural. Chages were made to this versio by the publisher prior to publicatio. IEEE INTERNET OF THINGS JOURNAL, VOL. 5, NO. 1, FEBRUARY 18 7 ode is a sigle server with a large eough queue to hold ifiite icomig requests. The icomig traffic to fog odes is cosidered to be Poisso (request type Light with rate λ ad request type Heavy with rate λ ) ad processig times to be expoetially distributed (request type Light with rate µ ad request type Heavy with rate µ ). Thus, fog odes ca be modeled as Markovia queuig systems with multi-class traffic. To derive the equatio for average waitig time of fog ode, we eed to defie the state of the system. We defie P, as the state i which there are N = type Light requests ad N = type Heavy requests i the fog ode (i.e. + 1 requests are i the queue ad 1 request is i service, if, 0). Thus W ca be calculated as: W = E(W ) = E(W N =, N = )P,, (8) where radom variable W deotes the waitig time of fog ode. We will derive a closed form equatio for P, i Sectio IV-G. E(W N =, N = ) is calculated as: E(W N =, N = ) = ad hece W = 1 µ + [( µ + µ )P, ]. G. Fog Node Steady State Probability P, 1, (9) To obtai the fog steady state probability P,, oe ca model the system as a two-dimesioal Markov chai ad solve for steady-state probabilities P,. The states are labeled as (, ), deotig type Light ad type Heavy requests i the queue. If r t:t sigifies the trasitio rate from state t to t, we ca obtai the trasitio rates of the state diagram as follows: r (, ):(+1, ) = λ, (10) r (, ):(, +1) = λ, (11) r (, ):( 1, ) = Q, µ, (12) r (, ):(, 1) = (1 Q, )µ, (13) where Q, is a state-depedet fuctio that is defied as q Q, = q + (1 q). (14) I the above defiitio, q (0, 1) is the fairess parameter ad its value depeds o how the fog ode selects obs from its queue. Fog odes segregate type Light ad Heavy requests i their queue, ad depedig o the value of q, select the ext ob to process from each of the two groups. The closer the value of q is to 0 (or 1), the higher priority is give to type Heavy (or type Light) requests i the queue for processig. Whe q = 0.5, Q, = +, ad this is equivalet to the case whe the fog ode simply selects the head of the queue for processig. More specifically, for q = 0.5, if the fog ode is i state (, ), o average the system processes a type Light request ad goes to state ( 1, ) with rate + µ, ad processes a type Heavy request ad goes to state (, 1) µ with rate + µ. This is the case because o average the head of the queue is a Light request with probability + ad is a Heavy request with probability +. H. Acceptace Probability: P P is the probability that a request is accepted by fog ode (eters the queue of fog ode ) ad is used i Equatio (6). P depeds o the queuig state of a fog ode; i particular, if fog ode s estimated waitig time is greater tha a threshold θ, it will offload the request to its best eighbor. Thus P is exteded by the followig probability: P = P [request eters the queue at fog ode ] (15) = P [waitig time of fog ode < θ ] = P [W < θ ]. To evaluate the equatio above, we eed to derive the probability desity fuctio (PDF) of waitig time W. Recall that waitig time W is the sum of the processig delays of all the requests i fog ode. Let the radom variables x l ad y l deote the processig delays of the l th request of type Light ad type Heavy i fog ode, respectively. If there are N type Light ad N type Heavy requests i the fog ode, the waitig time is W = N l=1 x l + N l=1 y l. Let X =, which is the sum of processig l=1 x l delays (expoetially distributed) of request type Light, ad Y = l=1 y l, which is the sum of processig delays (expoetially distributed) of request type Heavy. Thus W = X N +Y N. Note that the sum of m idepedet ad idetically distributed expoetial radom variables with parameter µ is a gamma radom variable with parameters (m, µ). Hece, the PDF of X ad Y will follow gamma distributios as follows: f X (t) = µ (µ t) 1.e µt u(t), (16) ( 1)! f Y (t) = µ (µ t) 1.e µ t ( u(t), (17) 1)! such that u(t) is the uit step fuctio. Note that the show PDFs are gamma distributios with parameters (, µ ) ad (, µ ), respectively. What follows is the derivatio of P : P = P [W < θ ] = P [X N + Y N < θ ] = P [X N + Y N < θ N =, N = ]P, =0 =0 = P 0,0 + P [X < θ ]P,0 + P [Y < θ ]P 0, + =1 =1 =1 = P 0,0 + [ + P [X + Y < θ ]P, θ =1 0 θ [ =1 =1 0 =1 f X (t)dt]p,0 + [ =1 θ 0 f Y (t)dt]p 0, f X +Y (t)dt]p, (18) I order to expad the summatio o the secod lie, we separated the sum ito four cases i the followig order:

This is the author's versio of a article that has bee published i this oural. Chages were made to this versio by the publisher prior to publicatio. IEEE INTERNET OF THINGS JOURNAL, VOL. 5, NO. 1, FEBRUARY 18 8 ( = 0, = 0), ( > 0, = 0), ( = 0, > 0), ad ( > 0, > 0). We have the equatios for f X (t) ad f Y (t), but we do ot have a equatio for f X +Y. Sice X ad Y are idepedet, the PDF of X + Y will be the covolutio of f X (t) ad f Y (t). We trasform f X (t) ad f Y (t) to their correspodig Laplace trasforms L{f X } ad L{f Y }, so that the Laplace trasform of X + Y is the product of L{f X } ad L{f Y }. We have L{f X }(s) = (µ ) (s + µ ), L{f Y }(s) = (µ ) (s + µ. (19) ) The PDF of X + Y is the give by: f X +Y (t) = (µ ) (µ ) L 1 1 { (s + µ ) (s + µ }. () ) We ow have all the compoets of Equatio (18). I the followig sectio, we discuss how to obtai the arrival rates to the fog odes. I. Arrival Rates to Fog Nodes The rates of arrival to fog odes (λ ad λ ) are eeded for the evaluatio of the fog steady-state probability P,. We ca obtai these rates by cosiderig the queueig model of the fog odes. Figure 3 shows the queueig model of fog ode for type Light requests. Note that i this subsectio we oly perform aalysis for obtaiig equatios for type Light requests, icludig λ. The equatio for λ is derived similarly to that of λ. The icomig type Light traffic from associated IoT odes to fog ode is deoted by I. The offloaded type Light requests from eighborig fog odes to fog ode are labeled δ 1, δ 2,..., δ em. These all accout for the icomig type Light traffic to fog ode. The icomig type Light traffic to fog ode is labeled as v. Thus, v = I + e M l=1 δ l. (21) Recall that P is the probability that a icomig request is accepted by fog ode. So λ, the arrival rate (type Light) of fog ode could be obtaied by the followig equatio: λ = P v = P (I + e M l=1 δ l ). (22) Whe the request is ot accepted by fog ode, the fog ode offloads it to its best eighbor, or to the cloud. The offloaded type Light requests from fog ode to its best eighbor are labeled β 1, β 2,..., β em. The offloaded type Light requests from fog ode to the cloud is labeled C. I order to evaluate the Equatio (22), we eed to have closed-form equatios for I ad δ l s. I is obtaied easily as follows. If f 1 () deotes the mappig fuctio that idicates the set of IoT odes that sed their requests to fog ode, the I is: I = [γ i p F i ], (23) i f 1 () C : requests offloaded to cloud from fog ode v λ.. β 1: requests offloaded 1 st time (to best eighbor) β 2: requests offloaded 2 d time (to best eighbor) β em : requests offloaded e M th time (to best eighbor) µ Respose back to IoT odes δ 1: requests offloaded 1 st time (from eighbor odes) δ 2: requests offloaded 2 d time (from eighbor odes) δ em : requests offloaded e M th time (from eighbor odes) I : requests from associated IoT odes to fog ode Fig. 3. The traffic model of fog ode (show oly for type Light requests) where γ i deotes the rate of geeratig Light requests from IoT ode i (i uits of Erlags). We assume IoT ode i geerates type Light (or Heavy) requests accordig to a Poisso process, with rate γ i (or γ i ), depedig o the type of IoT ode i. I order to obtai equatios for δ l s, we eed to solve the followig system of equatios: β 1 = (1 P ) I, (24) β 2 = (1 P ) δ 1, (25). β em = (1 P ) δ (em 1). (26) If fog ode caot accept a request set from IoT odes (I ), the request will be offloaded for the first time to fog ode s best eighbor (Equatio (24)). Similar to this explaatio, other equatios could be realized: if a l-times-offloaded request is ot accepted by the fog ode (δ l ), it will be offloaded for the (l + 1) st time to fog ode s best eighbor (β (l+1) ). I hece could be also be expressed as δ 0 ; type Light requests that are ot offloaded so far. Fially, if a request is already offloaded e M times, ad is ot accepted by fog ode, it will be offloaded to the cloud. This is realized by C = (1 P ) δ em. (27) β 1 could be calculated usig Equatio (24), because we have all the compoets of the equatio. Though, to attai β 2,..., β em, we eed to have the equatios for δ l s. Cosider fog ode, ad recall δ l represets the type Light requests that are offloaded for the l th time from eighborig fog odes to fog ode. Let ĵ be oe such eighbor. The chaces that fog ode ĵ s best eighbor is fog ode (ad 1 hece offloads the requests to fog ode ) is roughly, deg(ĵ) where deg(ĵ) is the umber of eighbors of fog ode ĵ. Therefore, δ l ca be obtaied by the cosiderig the chaces of receivig offloaded type Light requests from eighborig fog odes to fog ode as δ l = 1 [βĵl deg(ĵ) ] : 1 l e M, (28) ĵ ghbr()

This is the author's versio of a article that has bee published i this oural. Chages were made to this versio by the publisher prior to publicatio. IEEE INTERNET OF THINGS JOURNAL, VOL. 5, NO. 1, FEBRUARY 18 9 where ghbr() is the set of eighborig fog odes of fog ode. We ow have all the compoets of the system of equatios, so we ca get all the β l s, hece all the δ l s, ad hece the λ. As metioed before, λ could be derived similarly. J. Arrival Rates to Cloud Servers We eed the equatios for the arrival rate to cloud servers (l k ad l k ) to evaluate the average waitig time of cloud servers. The icomig traffic to the cloud servers are both from IoT odes ad fog odes (refer to Fig. 3). Similar to Equatio (23), we ca express the arrival rate of type Light requests to cloud server k as l k = [γ i p C i ] + C, (29) i g 1 (k) h 1 (k) where g 1 (k) is a mappig fuctio that idicates the set of IoT odes that sed their requests to cloud server k, ad h 1 (k) is a mappig fuctio that idicates the set of fog odes that offload requests to cloud server k. Equatio for l k, the arrival rate of type Heavy requests to cloud server k, is derived similarly to Eq. (29) by substitutig γ i ad C. K. Waitig Time of Cloud Server We model the cloud server k with m k iteral processig uits as m k M/G/1 queuig systems, with a load balacer that places the requests (uiformly) i the processig uits. Thus, the total arrival rate to each M/G/1 queue is give by Λ k = l k+l k m k. Sice we assume that the load balacer distributes the requests uiformly to the processig uits, the average waitig time at a cloud server is equal to the average waitig time of oe its processig uits (H k = k ). I order to obtai the average waitig time of a processig uit at the cloud server k, we use the Pollaczek-Khichie formula to determie the average queue legth, ad use Little s law to obtai the average waitig time. Thus, the average waitig time of a processig uit at the cloud server k will be: k = 2ρ k + Λ 2 k σ2 k ρ2 k (2 2ρ k )Λ k, ρ k = Λ k E(S k ). (30) where E(S k ) ad σk 2 are the overall average ad variace of service time of a request at a give processig uit of the cloud server k, respectively. At a processig uit of the cloud server k, the service time for a Light request, s k is expoetially distributed with a average service time of Z k, ad the service time for a Heavy request, s k is expoetially distributed with a average service time of Z k. We ca derive E(S k) ad σk 2 as: l k E(S k ) = ( l k + l k ) Z k + ( l k + l k ) Z k, (31) σk 2 = E(Sk) 2 E(S k ) 2, where l k E(Sk) 2 = ( l k + l k ) E(s 2 k) + ( l k + l k ) E(s 2 k ). (32) l k l k V. NUMERICAL EVALUATION I this sectio we evaluate the proposed mechaism through simulatio, ad also compare the simulatio results with our aalytical model. We first discuss the simulatio settigs i the followig subsectio, ad the explai the results i the ext subsectio. A. Simulatio Settigs 1) Network Topology: The etwork topology is a graph (hierarchical, similar to Fig. 1) with 0 IoT odes, 25 fog odes, ad 6 cloud servers. The IoT ode either processes its ow request, or seds it to its correspodig fog eighbor or to oe of the cloud servers. If the request is set to the fog layer, based o the proposed scheme, the request could be offloaded to other fog odes or to the cloud. The topology of the fog odes i the fog layer is geerated radomly i each experimet usig a radom graph geerator with average ode degree of 3. IoT odes are associated with the fog ode that has the smallest distace from them (i.e. has the smallest propagatio delay). 2) Lik Badwidth: If a IoT ode geerates type Light requests (e.g. sesor), the commuicatio betwee the IoT ode ad its correspodig fog ode is assumed to be through IEEE 2.15.4, or NB-IoT, or ZigBee, i which the trasmissio rates are 2 Kbps. If the IoT ode geerates Heavy requests (e.g. traffic camera), the commuicatio betwee the IoT ode ad its correspodig fog ode is assumed to be through IEEE 2.11 a/g, ad the trasmissio rate is 54 Mbps. The lik rates betwee fog odes i oe domai are 100 Mbps ad the lik rates o the path from fog odes to cloud servers are 10 Gbps. 3) Propagatio ad Trasmissio Delay: The propagatio delay ca be estimated by halvig the roud-trip time, RT T, which itself ca be expressed as RT T (ms) = 0.03 distace (km) + 5 [18]. Usig this, we assume the propagatio delay betwee the IoT odes ad the fog odes, amog fog odes, ad betwee fog odes ad the cloud servers are uiformly distributed betwee U[1,2], U[0.5,1.2], ad U[15,35] respectively (i ms). Request legths are expoetially distributed with a average legth of 100 bytes for light processig tasks, ad KB for heavy processig tasks. We assume that the legth of the respose is the same as the legth of its correspodig request, o average. 4) Processig Delay: To obtai realistic values for the processig ratio of IoT odes to fog odes, we looked at the processig capabilities of the Arduio Uo R3 microcotroller (a example of IoT ode geeratig Light requests) ad a Itel dual-core i7 CPU (a example of fog ode). I the worst case, a fog ode s processig capability is foud to be aroud 3000 times faster tha that of a IoT ode geeratig type Light requests ( Fog-to-IoT-Light ratio ), ad 0 times faster tha that of a IoT ode geeratig type Heavy requests ( Fogto-IoT-Heavy ratio ). We also assume that a cloud server is 100 times faster tha a fog ode ( Cloud-to-Fog ratio ), o average, ad that the average processig time of IoT ode for Light ad Heavy requests is 30 ms ad 0 ms, respectively. Other simulatio parameters are summarized i Table IV for 5 differet settigs for parameters. To accout for the

This is the author's versio of a article that has bee published i this oural. Chages were made to this versio by the publisher prior to publicatio. IEEE INTERNET OF THINGS JOURNAL, VOL. 5, NO. 1, FEBRUARY 18 10 TABLE IV SIMULATION PARAMETERS (p F i VALUES ARE SHOWN FOR AFP) Settig p I i p F i b i θ e M γ i γ i q 1 0 1 0.8 0.2 1 0.1 0.25-2 0 0.85 0.5 0.0002-0.5 0.6 0.5 3 0.1 0.75-0.2 1 0.05 0.005 0.5 4 - - 0.9 0.0002 1 0.01 0.001 0.5 5 0 0.75 0.02 0.0002 1 0.1 0.05 0.5 variatio of values of the above parameters i real IoT-fogcloud applicatios, we altered the parameters uiformly as: Fog-to-IoT-Light ratio, U[0,00]; Fog-to-IoT-Heavy ratio, U[100,0]; ad Cloud-to-Fog ratio, U[,0]; we foud that the result (average delay) fluctuates oly by -0.77% to +5.51%. 5) Operatio Modes: To see the beefits of the proposed offloadig scheme, we compare the average service delay i three modes of the IoT-fog-cloud operatio. The proposed scheme is labeled as All Fog Processig (AFP), while other possible modes are labeled Light Fog Processig (LFP) ad No Fog Processig (NFP). I No Fog Processig (NFP) mode, the IoT ode either processes its ow requests, or seds them directly to the cloud (that is, o request is set to the fog). I this case, p F i = 0, ad p C i = 1 p I i for both Light ad Heavy request types. Coversely, the other two modes beefit from processig requests i the fog layer. I All Fog Processig (AFP) mode, the IoT ode either processes its ow requests, or seds them to the fog or to the cloud; i AFP, both request types Light ad Heavy ca be set to ad processed i the fog layer. The values of p I i ad pf i are as show i Table IV, ad are the same for both type Light ad type Heavy requests. Light Fog Processig (LFP) is similar to AFP i all the cases, except that oly type Light requests could be set to the fog layer, ad type Heavy request must be set to the cloud if they are ot processed at the IoT odes. Said differetly, type Heavy requests could be either processed at the IoT ode or i the cloud, but type Light requests could be processed at IoT, fog, or cloud. I this case, the values of p I i ad pf i for type Light requests are as show i Table IV; however, for Heavy requests, p I i is as show i Table IV, but p F i is set to 0. I all the cases, the value of p C i is determied by p C i = 1 p I i pf i. 6) Figure Settigs: 5 differet parameters for the scheme settigs are cosidered i the simulatio results, ad these settigs are summarized i Table IV. Each sample poit i the graphs for simulatio is obtaied usig 1 millio requests usig a evet-drive simulatio. For eve more detailed aalysis, the delay for type Light ad Heavy requests is plotted separately for the three modes i the color figures (Namely, ad ). All time uits are i ms. I Fig. 4b ad 4d, i additio to simulatio values (black), the results of our aalytical model are plotted (gold), to show the accuracy of the aalytical model. B. Numerical Results Figure 4a shows the average delay as a fuctio of fairess parameter q. For this figure, the simulatio Settig 1 i Table IV is used. Recall that q is the fairess parameter, ad whe q is closer to 1, more priority is give to light processig 70 30 10 NFP H NFP L LFP H LFP L 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 100 70 90 70 30 q: fairess parameter for fog odes NFP H NFP L (a) LFP H LFP L 0 0.2 0.4 0.6 0.8 1 b: Probability of Type 1 (LIGHT) request (c) 30 NFP H NFP L 10 LFP H 0 LFP L 0 0.2 0.4 0.6 0.8 1 p F : prob. of sedig request to fog layer (p I =0) (e) 1 110 100 90 70 NFP NFP-al AFP AFP-al LFP LFP-al 0 5 10 15 25 e M : Fog Layer Offload Limit 100 90 70 (b) NFP NFP-al 30 AFP AFP-al LFP LFP-al 10 0 0.2 0.4 0.6 0.8 1 1 1 1 100 b: Probability of Type 1 (LIGHT) request NFP H NFP L LFP H LFP L (d) 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 p F : prob. of sedig request to fog layer (p I =0.2) Fig. 4. Simulatio results. I figures (b) ad (d) simulatio values are compared with aalytical model values. tasks. Thus whe q is closer to 1, the delay of light processig tasks is decreased ad the delay of heavy processig tasks is icreased. Note that this chage is oly see i AFP, as the fairess parameter q is ot defied i NFP (there is o fog) ad LFP (all Light requests). Figure 4b shows the average service delay as a fuctio of e M (obtaied usig simulatio Settig 2). For AFP, the optimal value of e M where the service delay is miimum is achieved for e M = 1 usig the parameters metioed i simulatio Settig 2, ad whe e M > 5, AFP performs worse tha NFP. It is iterestig to see that chages i e M do ot chage the average service delay i LFP, sice the icurred trasmissio ad propagatio delay to offload a request amog fog odes is egligible for Light requests with small legth. The suffix -al i the leged of the figures deotes the aalytical model values for the correspodig mode of the policy. It ca be iferred that the aalytical model results (f)