Relationship-based Intercom Platform for Smart Space

Transcription

1 Int'l Conf. Wireless Networks ICWN' Relationship-based Intercom Platform for Smart Space Daecheon Kim, Duc-Tai Le, and Hyunseung Choo School of Information and Communication Engineering, Sungkyunkwan University, Suwon , South Korea Abstract - Internet of Things (IoT) is rising recently for providing environment-based services to human in smart spaces. The concept of smart space has emerged from ubiquitous technology and becomes one of the promising fields of IoT. Services in a smart space require an interaction of shared information from devices in the space. This paper presents a relationship-based intercom platform for smart spaces. Although there are papers offering smart intercom in literature, there has not yet been any intercom developed for a smart space. This is due to the absence of dominant protocols and the limitations of processing resources. The proposed platform extends the area of a smart space from a door, which is the entrance of the space, to each area in the space by using an IoT gateway and cameras. The platform facilities an efficient use of a smart space by allowing users to get familiarize themselves with the smart space easily. Keywords: Internet of Things, Smart Space, Platform, Smart Intercom 1 Introduction Internet of Things (IoT) is a network of interconnecting things that connects all the things in the world to the Internet. For example, everything that has existed as a physical object, such as cups, books, walls, etc., as well as devices capable of logical operations such as wearable devices, smart phones, and PCs, is now connected to the Internet through IoT technology [1]. The basic IoT environment is achieved by attaching sensors, actuators and small embedded devices to physical objects to create an IoT echo system around things. Various sensors play an important role in the IoT environment, and the data collected by sensors improves efficiency, accuracy, and economic benefits in the user's environment while reducing human intervention. A large number of sensors and actuators have been introduced into the IoT environment with IoT, and expanded to include systems such as smart space, smart grid, intelligent transportation and smart city. Among them, the concept of smart space emerges from ubiquitous technology and is now considered as one of the promising fields of IoT. Smart space can be thought of as an advanced computing environment in which knowledge is learned and applied from the surrounding environment to improve the user experience and the service is then applied to the user environment again [2], [3]. A smart space user can be provided with an autonomous information processing system that can run on a peripheral device. Smart space services are the result of interaction for shared information. In terms of smart space research, four element technologies such as semantic-based spatial context understanding technology, smart object creation and control technology, human-based user interface technology, and networking technology are typical examples of each elementary technology. In this approach, we argue that intercoms are the best fit for smart space construction. Intercoms are signaling devices that are usually placed near the door of the building entrance. Wireless intercom systems that do not require wall wiring in recent decades have become widespread. When the button is pressed, the transmitter transmits a radio signal to a receiving device connected to a wall outlet inside the building. In the metropolitan area, there has been a tendency to use wireless technology over the last decade to wirelessly signal the intercom, open the door, and release the electric shock remotely. In many cities around the world, this is the main form of buzzer signaling. Although many papers currently propose smart intercoms, intercoms for smart space platforms have not yet been developed. This is because there is no dominant standard for IoT communication, control and data management, and there is a limit to how smart intercoms can be handled by gateways provided by each company. In addition, to use only the platform of the smart intercom itself, users must use multiple control interfaces such as mobile apps to use smart devices at home, or must use the devices sold by a single vendor to ensure the best user experience It is difficult to use a single platform due to problems. Another problem is the lack of processing resources or the lack of storage for data management to perform heavy processing due to the restriction of the intercom's own resources [4], [5]. In this paper, we propose an intercom bell - based platform that can expand the role of smart space and utilize more abundant resources through solving the resource constraint problem in a IoT gateway. The proposed intercom based platform identifies the person near the intercom and displays the corresponding intercom screen on the screen near the person inside the space. When an intercom screen is displayed to a person, check whether the screen is correctly delivered to the user or deliver it to another person. When the connection with the intercom is ended, the relationship data is generated and the intercom

2 114 Int'l Conf. Wireless Networks ICWN'17 screen is transmitted to the person closest to the relationship data when the intercom is triggered by the same person. The person who entered the space also accumulates relationship data when entering each room. In addition, if the identified person is a criminal relating to the police database, he will not open the house. This can prevent people in the house from danger. The composition of this paper is as follows. Chapter 2 explains the current state of smart intercom's research and the applicable technologies for the proposed platform. Next, we propose an intercom-based IoT platform that utilizes intercoms, IoT gateways, and smart devices, and Chapter 4 presents usages and detailed development techniques. We conclude the last chapter. 2 Related work In the existing paper, we can notice the status of smart intercom. DoorStep, proposed in 2016, is an intelligent intercom solution that can detect when a user is at the front door and send an alert to the elderly at home [6]. The elderly can review the captured image of the outsider and can get help from a guardian. The guardian may present recommendations to the elderly to prevent them from coming in or out. The doorstep system differs from other intercoms in that it focuses on providing security by reviewing who has approached the elderly population and the disabled community and by asking for help from the people. Also, the elderly can inform and ask for help if they need it, and they can freely maintain and control their house. However, this system is not proposed for the IoT environment and has a totally immature context. Especially when the intercom device is interlocked with the client terminal, the intercom search function itself is not likely to be useful because of the characteristics of each house near to each house. It is also difficult to provide the elderly with a proper way of recommendation which is better behavior when suspicious people come in front of their door without third person's comments. The person cannot watch the screen everyday. The ADBS, proposed in 2016, is also an automatic intercom system [7]. At the heart of ADBS is an ultrasonic sensor that detects people in front of the door and sends an SMS to the registered mobile number if there is no response in the house a few seconds later. However, such a system only sends an SMS to a user using only an ultrasonic sensor, so that many SMSs can be sent to a user when a detected person performs repetitive movements in front of the door. This is a very unnecessary action, and it is possible to construct an efficient system by using the camera image of the intercom. The most famous model on the market today is Ring [8]. Ring has mobile access, HD camera, motion detection, and night vision. This system lacks face recognition capability and can only confirm motion. The Ring intercom is intended for general homeowners and simply receives an intercom alarm when the homeowner is not at home. I-BELL is mentioned again for general homeowners but specifically mentions that they can be used for the elderly or disabled [9]. This allows family members or caregivers' smartphones to display detailed information about the app, making it easier for someone to monitor when they are on their way to work. The current intelligent intercom systems investigated do not consider the detailed processing capabilities due to their respective resource limitations. We use the OpenCV [10] and SIFT algorithms [11] to measure distances from humans and devices with screens preregistered on the IoT gateway. OpenCV (Open Source Computer Vision) is an open source computer vision C library. Originally developed by Intel. It can be used on multiple platforms, such as Windows and Linux. It is a library focused on real-time image processing. Supports Intel Performance Primitives (IPP), which can be used to speed up when used on Intel CPUs. The SIFT algorithm is a scale-space theory that can be used as a scale-independent property. The feature of an object can be extracted from an image regardless of the size of the object. Photo contrasts between photographs of people, extracted from the intercom and those stored in the database, can be contrasted using the project Oxford. Project Oxford is one of the machine learning projects created by Microsoft [12]. If user registers with Microsoft, everyone can use the machine learning technology API. Project Oxford has four machine learning APIs. The Face API provides features such as finding a person's face in an image and capturing its functionality to find the same person in another image. Vision API image recognition and thumbnail generation for voice and image processing that can be used with speech recognition technologies such as Cortana or Siri API, and Language Understanding Intelligence Service (LUIS), which provides functions for natural language recognition. Among them, we want to implement the proposed platform by using Face API. Our paper creates and stores a relational tree of user data through a high-performance IoT gateway for the environments. The data is generated when the visitor returns, the user can view the intercom screen directly from the nearest screen without further push notification. Communication between an IoT gateway and devices can be made using Constrained Application Protocol (CoAP). It is an Internet application protocol for limited devices. Such a constraint device can communicate with the Internet using similar protocols. The CoAP is designed for connections between devices on the same limited network, between devices on the Internet and common nodes, and between devices on a limited number of networks. In addition, CoAP is being used through other mechanisms such as SMS on mobile communication networks. Recently, IoTivity, which stands out in the field of IoT middleware, supports CoAP based communication [13]. Open source IoTivity is an ideal framework for building IoT infrastructure. IoTivity is an open source framework that enables interoperability between high-end Internet devices

3 Int'l Conf. Wireless Networks ICWN' and rapid product development. IoTivity helps you manage each object as a resource and implement resource discovery. Virtual Network Computing (VNC) is a graphical desktop sharing system that uses a Remote Frame Buffer (RFB) protocol to remotely control other computers [14]. RFPs support image compression, allowing display screens to be shared within lower bandwidths. Since the VNC system is implemented as open source, it can be used freely on any platform and is useful for implementing intercom- server. Finally, the Google Cloud Platform (GCP) is used for relational data storage [15]. GCP is Google's cloud computing service that provides hosting on the same support infrastructure that Google uses internally for end-user products such as Google Search and YouTube. The GCP provides a developer environment for developing a variety of programs, from simple web sites to complex applications. Here, GCP is used as the basic processing and storage use of relational data. 3 Design of Proposed Platform The Relationship-based Intercom Platform for Smart Space addresses the problems of the current intercom system mentioned above and is designed for the smart space. The proposed IoT platform is configured for each role group to perform the function of smart space. Figure 1 shows the structure of the intercom-based IoT platform. 3.1 Architecture Fig. 1 Doorbell-based IoT Platform Architecture The proposed platform consists of three parts: Endpoint, Middleware, and Cloud Computing. The endpoint refers to all devices available in the smart space. Typically, it refers to intercom, smart TV, smart phone, and CCTV. Middleware is an essential appliance for achieving smart space, consisting of South interface, XML processing, database, CoAP-HTTP proxy, image processing and North interface part. Finally, Cloud Computing has a database that can manage and store all the data collected from IoT gateways and relationship data to satisfy smart spatial elements. Details of each part are as follows Endpoint A common endpoint is the detection of an actual entity at various sensor endpoints that provide raw measurements. In the case of a device, it locally connects the name, model number, hardware type, device, version, type and timestamp to the sensor value, which generates measurement metadata. Through the interface of the smart device, the sensor metadata can be read as a GET request or pushed to the south interface of the gateway. In the data retrieval phase, records of these endpoints must be kept. Therefore, adding or deleting devices and endpoints informs the IoT gateway to update the internal database and configuration accordingly. Also, when the first device is added, the configuration list is sent to the gateway and stored in the local database. Intercoms and CCTV also perform this basic data processing and support streaming servers. The streaming server is opened one by one for each device, and the IoT gateway manages a list of all the streaming servers. This way of managing the list allows you to clearly distinguish between the Server-Client parts by separating the necessary processing from the administrative part and the necessary processing from the dependent part Middleware Middleware is computer software that provides additional services that application software provides in addition to the services provided by smart devices. Application software should be flexible, easy to expand and shrink, and to meet these needs, there is no need to rebuild the platform between different platforms of the developer. Such as database systems, electronic communication software, message and query processing software. Currently, Middleware is installed in the IoT gateway. IoT gateways are centralized devices that support high performance processing. The IoT gateway is directly connected to the data of each device. IoT gateways must be able to handle resources. It means recognizing each smart device as a resource and managing information such as protocol, type and resource name at once. It handles all the tasks that all IoT Smart Gateways need to take, such as device operation, device connection, device discovery, and more. The resource management process also has data on who each device is. This data is primarily used to create relationship data based on the human location and location data of each device obtained through image processing. The south interface is the interface responsible for communication between the endpoint and the IoT gateway mentioned above. This interface utilizes CoAP to create a network between the smart device and the IoT gateway, and to open the necessary sockets when the gateway searches for devices. To enable the CCTV identifying a device close to the user, the device must be registered in the IoT gateway, and XML registration of the device for easy device registration is

4 116 Int'l Conf. Wireless Networks ICWN'17 supported. Southbound is connected to the smart device and works with the gateway's database and manages the sensor value or device metadata of each device. It also connects the client to the north-bound. The device adds an XML format header to the metadata portion of each device and has information on how each device operates. The database is the data storage space required by the device for all the processing performed on the IoT gateway, where data temporarily goes through for fast processing. Here, measurement data, a device list, an intercom image, and relationship data generated by each device are temporarily stored. The database of the proposed IoT gateway is a database that takes data from the cloud server and writes it as a cache. Northbound and Southbound refer directly to clients and all smart devices. The CoAP-HTTP Proxy is a proxy required to connect the network and the Internet for connection between the GCP and the device. The image processing part extracts a person or device from a photo and compares and analyzes the relationship data stored in the GCP or the registered device. The following describes the role of each part of the platform. Image processing is a process of comparing pictures received from the intercom to those stored in the cloud to make sure the pictures are correct. Also, the position of each device extracted from the camera installed in the space and the position of the device are mapped to confirm if the device is located near the registered device. The above processing allows the user to intuitively register the relationship data. Relational data behave differently depending on whether the first statement bell has detected a person and whether the person is in the database. The presence of the recognized person in the database can be accomplished by sending the door type screen to the person whose relationship data is already associated with the recognized person in the space. 3.2 Role The proposed platform can be roughly divided into three roles. Each function implements a technology that understands the space situation necessary for the smart space and observes the location of the object, the behavior of the person, etc., and then understands the situation of the space occupied by the object and the person. Face recognition, distance measurement, and data accumulation are described to play a role as such a smart space platform Face recognition Face recognition is a process that occurs when a person is detected in front of a intercom. The intercom has a streaming server and a picture transfer function that can send video to the IoT gateway. The image transfer function considers the face recognition processing overload. Since the face recognition process must be performed quickly and accurately, the intercom, which is relatively weaker than other smart devices, takes a photo of the other party and sends it to the IoT gateway. CCTV, outside the smart space, also captures a multifaceted picture of a person when they are detected and sends them to the IoT gateway. This is effective in reducing the error rate of image processing. Figure 2 is a flow chart showing how the proposed IoT platform provides intercom-based services in some form. Image processing is divided into two parts. It is the part that compares and analyzes the images of the prestored people in the intercom image and the relational data, and the part that grasps the The North interface requires that the IoT gateway and the GCP be connected for working together. The GCP should handle the Internet connection. This part is closely related to the CoAP-HTTP Proxy. The north interface, which is already connected to the GCP via the Internet, also requires processing for connection to the inside of the IoT gateway. The following describes the platform's role Cloud Computing In the proposed platform, cloud computing can receive push notifications of intercoms from the outside or collect the entire data generated by IoT gateways. This is the idea that multiple smart spaces can be created in the same building, and a total data management system is required to manage the IoT gateways in all smart spaces. In addition, bringing external data such as criminal records provided by the police directly to the IoT gateway can cause a heavy load on each IoT gateway. That's why all public data need to come from the cloud computing side. Fig. 2 Flow of proposed IoT platform location of the person and the device. The intercom image transfers the person's picture to the IoT gateway when the person stands in front of the intercom for 3 seconds or more. By using not only intercoms but also external CCTV photos, we collect as much data as we can compare and analyze. The relationship data contains pictures of the criminals who have previously visited the smart space or received from the police, their profile data and relationship

5 Int'l Conf. Wireless Networks ICWN' points. The first image processing sends the intercom to the person with the highest relationship point within the space based on the data obtained by comparing and analyzing these two photographs, or if the person is a criminal, automatically notifies the police immediately without user's judgment to prevent any future crime Distance measurement Locating people and devices is basically determined by using CCTV or internal camera located in smart space. As a discrimination criterion, it is judged that the current person is near to the device when the recognition part of the face and the part recognizing the device having the screen overlap or come close to each other. Also, the screen of the intercom is prevented from appearing on the personal screen of another person. In order to accomplish this technology, the devices registered in the IoT gateway are given priority according to the person, so that devices irrelevant to the relationship data are excluded from the image processing step. For example, when the outsider is a person related to A, it means that the intercom screen does not go to the monitor of A and the nearest monitor of B, but it means that the screen is transmitted to A's smartphone. This priority can be assigned by registering the device to the IoT gateway by storing the location of the device, image data of the multi-angle, and a record of the user's object. 4 Prototypes We implement a prototype of a intercom-based platform to verify the proposed architecture. This prototype manages real-time metadata from remotely deployed intercoms, IoT gateways, and smart TVs and performs basic structure and functions for the above-mentioned technologies. Figure 3 shows the prototype implementation scope of the proposed platform, and shows the open source and protocol used for each part. Fig. 3 Scope of the proposed platform prototype implementation Accumulation of relationship data A smart space platform based on intercoms stores spatial contextual data in a smart spatial sense. Representative spatial contextual data in the proposed platform include devicespecific data, location data, and person-to-person relationship data. Among these, the focus is on how the proposed platform collects relational data. Relational data accumulation is first divided into fixed relationship data registration and dynamic relationship data registration. Fixed relationship data registration occurs when an external person first presses the intercom. The door type screen is transmitted to the person who is initially set as the default. If the primary user sends the intercom screen to another person and the connection ends normally, the streaming connection is broken and the relationship data is provided to the external user and the final screen recipient. The composition of the relationship data is a room number, a person name, a guest name, and a relationship creation score. In particular, the score is awarded when a normal connection is made through the intercom. Registration of dynamic negotiation data uses camera data depending on each space. When a stranger enters a building, the camera keeps track of the person and gives a score of 0.01 when the person enters a person's room related to someone. These scores are cumulative and continue to have meaningful results in scores accumulated later in the relationship data. The prototype of the proposed platform includes intercom, smart TV, IoT gateway, client, and GCP. In terms of prototype characteristics, it shows what kind of prototypespecific components are in the main part of the current system. Intercoms and smart TVs are developed using the highly versatile raspberry pie. Raspberry Pie is a low-end device, but it is optimized for basic processing. The Raspberry Pi can be connected to a intercom switch via a general purpose input / output (GPIO), so it can operate like a traditional intercom. Raspberry Pi also has the flexibility to have Ethernet, Wi-Fi or 3G or 4G data connectivity. The intercom can be placed outside the door at the door. It can also be stored as a standalone device, including a camera. The intercom will have a streaming server for VNC when the operation starts. Smart TV uses Kodi's open source software and shows through the Picture In Picture screen in case the intercom screen turns on the smart TV. The PIP screen can display the image processing result value processed by the IoT gateway and confirm whether the correct recognition value is delivered to the user. If the user does not need the screen, press the button to send the intercom screen to the other person's device. Due to the limitation of using the minimum space due to the characteristics of smart TV, the overlay is displayed on the PIP screen to indicate who the person is. IoT gateway can manage all of the resources and are implemented so that the platform can operate for smart space through image processing. For basic data communication, we

6 118 Int'l Conf. Wireless Networks ICWN'17 use the IoTivity framework, which utilizes the resource discovery and CoAP-HTTP proxy functions in this framework. Additional sources are added to manage streaming server data on the intercom and enable smooth communication with external cloud and client. The gateway uses Ubuntu as a base for access points and laptops. Figure 4 shows the results after image processing to detect faces and devices. For image processing, faces of users and outsiders are detected through Project's Oxford Face API. The detected face is necessary to generate pre-stored relational data, to determine the location of the user, or to check whether the person is an offender. Fig. 5 Push Notification When user is outside, user also take advantage of push notifications. Figure 5 shows the registration method for all devices. When the token registered with MySQL is considered considering the usage method of PHP token, the FCM server sends the notification message to the client. Outside push notifications can be used to show that an event has occurred when an event occurs in the smart space from the outside. Fig. 4 Face Detection (above) Device Detection (below) OpenCV and SIFT are used to detect devices. If the image and feature points of the device are stored in the IoT gateway for the first time, the feature points of the device are detected in the camera image using the camera in the space. This allows the user to know where the registered device is in the smart space, and when the detected device and the detected user are close to each other, outputs the intercom screen to the adjacent screen of the user. The relationship data generated through this is transmitted to the cloud database. The transmitted data is forwarded to the IoT gateway periodically every time the intercom is triggered. Clients are basically based on smart TVs and smartphones. Smartphones utilize IoTivity and VNC protocols to deliver data or streaming data that an IoT gateway needs. If the external camera determines to whom to send the intercom screen based on the relationship data, the screen is transmitted to the device of the most relevant person. In the case of smart TV, since the screen is displayed without infringing the basic function of the smart TV of the space, the overlay shows the person who is detected in the intercom. On the other hand, Based on the person's perception and relationship data. If it is not the person's device, press the button to turn the intercom over to another person, or if the person is related to the user, close the intercom. Then, one point is added to the relationship data, and the relationship data between the person and the user detected in the intercom is thickened. Finally, cloud computing installs Linux on the GCP and provides the computing power needed for intercom-based platforms. The GCP provides the external IP, and the relationship data generated by the IoT gateway is transferred to the database inside the GCP via HTTP. In addition, when the IoT gateway establishes a Server-Client relationship with the IoT gateway, when the IoT gateway requests the relational data, it transmits the relevant data to the corresponding IoT gateway through third party authentication. 5 Conclusions We are living in a smart space environment with large amounts of sensors and actuators entering the IoT environment with IoT. However, due to the absence of dominant protocols or the resource constraints of devices, the role of smart space has been greatly reduced. In this paper, we propose an intercom based platform for smart space using intercom which is thought to be suitable for expanding the area of smart space to solve these problems. The proposed platform extends the area of smart space from the door which is the beginning of all space to the location area of each space by using IoT gateway and camera in space. The platform will allow users to familiarize themselves with smart spaces by solving the difficulties users have had to make effective use of smart space. Future research will strengthen the roles of relationship data generation and cloud computing, and introduce personal authentication methods in smart spaces.

7 Int'l Conf. Wireless Networks ICWN' ACKNOWLEDGMENT This research was supported by Basic Science Research Program through NRF of Korea, funded by MOE(NRF ), and G-ITRC support program (IITP ). References [1] G. Fortino and P. Trunfio, "Internet of things based on smart objects," Fortino & P. Trunfio, eds., Cham: Springer International Publishing, [13] IoTivity, " [Online]. Available: [Accessed: 29-May-2017]. [14] T. Richardson, Q. Stafford-Fraser, K. R. Wood, A. Hopper, "Virtual network computing," IEEE Internet Computing. vol. 2, no. 33, [15] GCP, "cloud.google.com", [Online]. Available: [Accessed: 29-May-2017]. [2] R. Harper, "Inside the smart home," Springer Science & Business Media, [3] S. Chae, Y. Yang, J. Byun, and T. D. Han, "Personal Smart Space: IoT based User recognition and Device control. In Semantic Computing," IEEE Tenth International Conference on,pp , Feb, [4] D. Miorandi, S. Sicari, F. D. Pellegrini, and I. Chlamtac, "Internet of things: Vision, applications and research challenges," Ad Hoc Networks, vol. 10, no. 7, [5] A. Sinclair, "Vision of smart home the role of mobile in the home of the future," GSMA, London, [6] A. Ennis, I. Cleland, T. Patterson, C. D. Nugent, F. Cruciani, C. Paggetti, and R. Taylor, "Doorstep: A doorbell security system for the prevention of doorstep crime," In Engineering in Medicine and Biology Society, IEEE 38th Annual International Conference, pp , Aug, [7] T. V. N. Rao, and K. R. Yellu, "Automatic Safety Home Bell System with Message Enabled Features," IJCSET, vol. 6, no. 12, pp , [8] Ring, Ring.com, [Online]. Available: [Accessed: 29-May-2017]. [9] I-Bell, i-bell.co.uk, [Online]. Available: [Accessed: 29-May-2017]. [10] OpenCV, "opencv.org," [Online]. Available: [Accessed: 29-May-2017]. [11] SIFT algorithm, "docs.opencv.org/trunk/da/df5/tutorial_py_sift_intro.html," [Online]. Available: l/. [Accessed: 29-May-2017]. [12] Project oxford, "azure.microsoft.com/enus/services/cognitive-services," [Online]. Available: [Accessed: 29-May-2017].