SEC-2010.2.3-3 Architecture for the recognition of threats to mobile assets using networks of multiple affordable sensors D7.2: Human Machine Interface Filename: ARENA_D7_2_v3.1 Revision of v3 Deliverable D7.2 Report Version 3.1 Classification: PU Grant Agreement number: 261658 Contract Start Date: May 16, 2011 Duration: 36 Months Project co-ordinator: FOI Partners: FOI, BMT, ITTI, Sagem, Morpho, TNO, UoR Responsible for report: BMT Project website address: www.arena-fp7.eu - 1(38) -
Executive summary The ARENA Human Machine Interface (HMI) is what an operator using the system actually sees. It provides several screens and graphical interfaces that allow users to visualise the video streams (playback of live streams) coming from the cameras and the output of the algorithms. The aim is to highlight any detected potential criminal behaviour against the driver of the truck or the truck itself. In particular, users can watch the playback video from all the cameras on a multi-camera screen containing three small video panels and a large one. The large one displays the currently selected camera, whereas the smaller ones allow the user to switch the large panel to a different camera. In addition, annotations such as object detections and the tracks of the object detections are displayed on the multi-camera view on top of the video. Any detected threats are displayed as well, including threat information and confidence levels. Moreover, the ARENA HMI also provides a Truck Status Monitor and a Threat History Viewer. The former shows the latest threat detected by flashing in red (high possibility of criminal behaviour) or yellow (suspicion of criminal behaviour) and by sounding an alarm. The user then has the choice of dismissing or just silencing the alarm. The Threat History Viewer shows all the alarms that have been detected by the ARENA platform in chronological order until the current time. Moreover, the previous two views display the time/date of the threat event as well as further information about it. Finally, the HMI provides a Map View screen which displays the area around the truck from a bird s eye view. Object detection (red colour is criminal behaviour, yellow is potential criminal behaviour and green is normal behaviour) and tracks are also shown on this screen.
Version Management Version Date Author Modification V0.01 16/07/13 P. Lohrmann/ T. Cane V2 15/09/13 P.Lohrmann/T. Cane/S. Katsoulacos V3 25/04/14 P.Lohrmann/T. Cane/S. Katsoulacos V3.1 16/05/14 P.Lohrmann/T. Cane/S. Katsoulacos Initial version based on report template First Draft Final document Revision of final document
Table of contents Executive summary 2 Version Management 3 1 Introduction 5 2 HMI Requirements 6 2.1 Use Cases... 6 2.1.1 Cargo theft from a parked truck... 6 2.1.2 Piracy attack on a ship at sea... 7 2.2 Requirements... 8 3 Specifications 9 3.1 Desktop HMI... 10 3.2 Mobile HMI... 11 3.3 Context Module HMI... 12 4 Implementation 13 4.1 Desktop HMI... 13 4.1.2 Truck Status Monitor... 14 4.1.3 Drill Down Capabilities (Threat, Situation and Object Viewer)... 15 4.1.4 Database Management Module... 18 4.1.5 Video Player... 18 4.1.6 Multi-camera screen... 20 4.1.7 Single-camera screen... 22 4.1.8 Map View... 22 4.1.9 Threat Thresholds Manager... 24 4.2 Maritime HMI... 26 4.2.1 Maritime Threat Detection... 26 4.2.2 Boat Tracking... 28 4.3 Mobile HMI... 29 4.3.1 Server Side... 30 4.3.2 Client Side... 30 4.4 Context Module HMI... 33 5 Conclusion and Future Work 38
1 Introduction The ARENA Human Machine Interface (HMI) is one of the main development components of the ARENA project. There are two main reasons that HMI is such an important and integral component of this project. These are a) it provides the developers of the algorithms used in the project to test and optimize their algorithms and b) it allows the end users to see in a user friendly format the output of the system while it is working. Finally, using the HMI we can show proof of concept by demonstrating playbacks of real life scenarios and the outputs of the algorithms regarding these scenarios in a smart and user friendly way for a non-expert.
2 HMI Requirements There are three important reasons why a HMI must be included in the development of the ARENA system: 1. The HMI is what the users of the system will see - it is the main means of interacting with the system. For the demonstrator cases the HMI plays an important role because the end-users will not be data fusion experts. Their feedback on the project outputs will be largely based on what they experience through the HMI. 2. During the project, the HMI should play the role of a development and testing environment for the different algorithms to be developed in WP5, 6 and 7. Individual algorithms can be developed in lab conditions but this will become more difficult when they are linked together as part of an integrated system. The HMI should provide facilities for combining algorithms, tuning parameters, querying and visualising inputs/outputs for debugging, running tests, and so on. 3. Finally, the HMI could provide a very useful tool for configuring the ARENA system for use on different platforms, including tools for adding and removing different sensor and algorithm capabilities. 2.1 Use Cases In Annex H of Deliverable D2.1 1, two use case scenarios were defined for the ARENA system. We reproduce these use cases for the reader s convenience: 2.1.1 Cargo theft from a parked truck Truck is parked at non-secured parking lot. Driver is at truck stop restaurant. ARENA-truck system detects people loitering near vehicle, followed by the actual opening of the cover to look inside the vehicle. The ARENA-truck system alerts the ARENA-headquarters system, where the operator checks the video live feed 2. The operator immediately raises the alarm to the local authorities who dispatches a police car, and contacts the truck driver by mobile phone. The truck driver rushes to his vehicle, which scares the thieves and they get away in their vehicle. 1 D2.1 ARENA WP2: Threat analysis and user requirements Overview Report 2 A prototype of live video feed has been implemented and then tested in the Reading data campaign. However, it has been discontinued since then.
The ARENA-truck notes the license plates 3 of the get-away vehicle, while the driver assures that his cover is partly opened but that his cargo is still all there. The police car arrives at the scene, and apprehends the thieves later based on the ARENA-truck system evidence, or sent to the local police for intelligence purposes. Variant 1: The initial alarm and live video feed is sent to the smartphone of the truck driver, after which the same sequence of events occurs. Variant 2: The live video feed is sent to police control room. 2.1.2 Piracy attack on a ship at sea An oil tanker is sailing through the Gulf of Aden (known as Pirate Alley ). Variant 1: The tanker is sailing alone and is fitted with the ARENA system. Suspicion is first aroused by detection of a large vessel (i.e. mothership) which does not have an AIS signal. ARENA compares its location to the most recent information received from authorities relating to positions/sightings of suspected motherships in the area. If close enough, it also fuses visual information to assist with the identification and confirmation of the mothership. Skiffs cannot be detected by radar because they are too small. Long range detection (500m 10,000m) through other means is very difficult, also due to their small size. However, the ARENA system is able to detect their presence up to 500m and warns the crew. The crew gathers near the secure citadel. When the skiffs are within 10m of the vessel, ARENA system detects a proximity breach and crew therefore have enough time to ensure that all crew members are safely inside the secure citadel before the pirates are able to board the ship. Following the boarding of the pirates, the ARENA system can be used from inside secure citadel to monitor the situation as it evolves and communicate with shipping company and law enforcement authorities as appropriate. Variant 2: The tanker decides to join a convoy for security. Several of the ships in the convoy (including the oil tanker) are fitted with the ARENA system. The journey takes approximately 12 days to pass through the area of highest risk. During this time, the ARENA system(s) are placed on most sensitive setting. A group of pirates has captured another large vessel and is using it as a mothership to increase the range of their attacks. The pirates launch 2 skiffs (small, fast vessels) from the mothership, each carrying 3 pirates armed with AK- 3 The licence plate recognition was originally planned but it was not implemented.
47 rifles. Their aim is to board the tanker, take the crew hostage, and demand a ransom for their release. The ARENA system behaves as in Variant 1, but it also facilitates the sharing of information between the ships in the convoy. This enables further functionality: Sensing/monitoring of mothership from different positions in convoy to confirm or reject threat hypothesis Send warnings to other ships (e.g. to allow them to change course, speed up) Fusion of AIS and other sensor data to monitor larger area (investigate optimal convoy formation) 2.2 Requirements In the stakeholder workshop at the beginning of the project, a list of requirements for the ARENA system was identified in Deliverable 2.1 1. The following requirements are relevant to the HMI (ranked in order of priority given by the end users): User friendliness; Allow the identification of potential threats; Provide simple information from complex situation and multiple sensors to allow people under pressure to make the best decisions; Playback of real-time stream video of an attack to assist response planning; Provide information on a need to know -principle. As the above list is very high-level, we have added a number of additional requirements to guide the specification and development of the system: The HMI should provide an intuitive, graphical overview of the sensor arrangements to allow the user to switch between different fields of view. On receipt of an alarm, users should be able to dismiss threats. Users should be able to drill down into a scenario, for example by displaying the tracks associated with an object. The system should provide playback capabilities for real-time video feed.
3 Specifications The following system specifications are meant to guide the implementation of the Human Machine Interface. We provide a detailed description of the system components and its functionality. The ARENA consortium has decided to develop two different types of HMI: a desktop version (to be used in truck operators headquarters or on ships) and a mobile version (to be used by truck drivers on handheld devices). The second type of HMI is only relevant for land-based scenarios; both interfaces are included in the general architecture description (Deliverable D3.2 4 ). Truck's dispathing center Truck / Lorry Arena Analyst ARENA HQ Node Truck driver HMI ARENA Service IR camera Company ERP System Integration Platform ARENA Node Fish-eye camera ARENA Repository GPS on-board computer Radar CCTV camera AN Truck with ARENA Node Parking site with CCTV cameras Figure 1: The place of the HMIs in the Arena architecture 4 D3.2 ARENA WP3: Architecture Specification Final Version
3.1 Desktop HMI The desktop HMI is built according to a modular design. This allows the switching on and off of components to cater for different users needs. In particular, a developer using the system will need interfaces and functionality that differ from those required by an end user (e.g. a headquarter operator). Ideally, a "plug-and-play approach will allow the quick and easy configuration of the system. The HMI should consist of the following modules: An Object Browser for displaying properties of objects. A Situation Browser for displaying objects as part of a situation and relationships between them. A Threat Browser for displaying information about detected threats A Playback Engine which will simulate the real-time behaviour of the system for testing and demonstration purposes. A Database Manager which allows users to connect/disconnect to the databases, create/delete, and perform a variety of manipulations on them such as querying them. A Truck Status Monitor which notifies the ARENA system operator about any newly detected threats, providing information about the threats such as confidence levels and the date and time that occurred; allows dismissal of threats. A Multi-camera window which displays streaming of all the cameras attached to the truck, accompanied with related annotations. In additions it shows the driver latest detected position as well as a history of the latest alarms. A Single Camera window which displays the streaming of only one camera with the ability to switch between them, accompanied with annotations. A Map View screen which displays the area around the truck from a bird s eye view. It displays object detections (colour coded depending on the status normal/suspicious/criminal) and their tracks. A Threat Thresholds Manager which allows the configuration the thresholds for the incoming threats on the system. This includes picking ranges for warning (suspicious behaviour), alarms (high probability of criminal behaviour) and also a cutting limit to ignore threat with low confidence levels.
3.2 Mobile HMI The mobile HMI will be installed on a smartphone or tablet that is used by the truck driver. Its main purpose is to alert the driver of potential threats while he/she is away from the vehicle. Therefore, as a minimum, the on-board system will be able to wirelessly send notifications to the handheld device, which will then display details of the detected threat. Ideally, a screenshot or (given enough bandwidth) short video sequence of the scenario should be included. We intend to use two levels of severity: Warnings are broadcast if a suspicious behaviour (e.g. loitering near the truck) has been detected; Alarms are sent if ongoing criminal behaviour (e.g. breaking into truck) is detected. Both notifications will be displayed in the HMI in a similar manner, with alarms being more visible. For a more advanced version of the HMI, the driver should be able to respond to notifications. In particular, on receipt of a warning, the driver should have two options: Dismiss the warning after establishing that there is no threat. In this case, the warning is logged at the headquarters, but not flagged up as a warning or alarm. Escalate to the headquarters if assistance is required. This triggers an alarm at the headquarters. In case of an alarm (i.e. criminal behaviour is detected), both the driver and the headquarters are notified. The driver cannot dismiss the alarm with the press of a button; such functionality could be abused by the attackers, who might either get hold of the mobile device or force the driver to dismiss the alarm. Lastly, the driver can escalate at any point, even if no warning or alarm has been received. Thus the HMI provides a kind of panic button that allows the driver to request assistance from the headquarters.
3.3 Context Module HMI The Context Module HMI (CM HMI) is a separate HMI from Arena HMI and is designed to support the needs of the ARENA system operator (e.g. logistic dispatcher). The approach is the following: The operator first needs to supply the information about the configuration of the truck fleet it operates (e.g. truck ids, dimensions, location of the GPS module in cabin, locations of cameras mounted, camera characteristics, etc.) then based on the knowledge about the planned routes of trucks (to their destinations) the operator needs to provide to the Context Module the coordinates of the parking places that will be visited by the trucks on their way to the destination In turn the operator will task Context Module to retrieve the metainformation about the facilities within a particular parking lot. This operation is automated but the quality of data retrieved is service-specific (e.g. the data provided in responses may be more accurate/detailed when called against the GIS service in country X than similar, or the same, GIS service in country Y) As a result of the step above the meta-information about the parking will be populated into the local repository of the Context Module (i.e. the ontology). The information in the ontology may then be accessed directly (operating the ontology file) this mode however is not advised, indirectly via dedicated front-end application (e.g. Protégé) or in a target way i.e. using the HMI part of Context Module.
4 Implementation 4.1 Desktop HMI Figure 2: ARENA HMI Architecture The desktop HMI system serves as the head quarter operator s main tool for the surveillance of parked trucks. The surveillance relies on cameras, which constantly monitor the area. Furthermore, special algorithms will analyse the data retrieved from the cameras in order to detect potential threats. The results from the analyses are distributed by the integration platform developed in WP4 and will be displayed by the HMI. The HMI consists of multiple graphical user interfaces (GUIs), which allow the representation of the incoming information about potential threats in a user-friendly format. Features that are included in these GUIs include a player for playback of previously recorded videos selected via the playback player, a multi-camera window or a single camera window that shows the playback accompanied by annotations (object detections, tracks and threats), a map view screen which displays the area around the truck from a bird s eye view, a database manager, drill down capabilities for threats to inspect the details of a potential threat and an alarm system to draw the user s immediate attention to potential threats.
4.1.2 Truck Status Monitor The Truck Status Monitor window (see Figure 3: Truck Status Monitor with an alarm active) is the main window that informs the user about the latest alarm or warning detected by the threat detection algorithms of the system. The table in the window will show with flashing red colour the alarms (high level of confidence for criminal behaviour) and with flashing yellow colour the warnings (potentially criminal behaviour). The alarms and the warnings are accompanied by a siren sound, which can be silenced (only temporarily until another alarm is triggered) without stopping the rows from flashing. There is also the option to completely turn off the sound notifications. In addition, if the user wants to see only the threats for a specific track and drill down to see more info about it (such as related Situations and then related Objects to the Situations), then they can select the row in the table and press the Filter Threats button, in which case the window with the drill down capabilities (Threat Viewer - History) will display only the threats for this particular truck. After finishing with the investigation of the threats, the operator can press the Show all Threats button on the Truck Status Monitor and the Threat Viewer will be showing again all the threats for all the trucks. Finally, the window provides functionality to completely dismiss an alarm or warning which will also silence the siren and the selected row will stop flashing. Figure 3: Truck Status Monitor with an alarm active
4.1.3 Drill Down Capabilities (Threat, Situation and Object Viewer) The drill down capabilities of the HMI provides users with an intuitive way to see threats and drill down to the level of individual objects in a video. As shown in the figure below (Figure 4: Threat Viewer - History), the user can see all the current threats on all trucks that they are currently invigilated by the system. By selecting a threat from the table and then by pressing the Show Situations button the system will provide the user with the Situations related to the selected threat (Figure 6: Drilling down from a Threat Viewer via Situations Viewer to Objects Viewer Second table from left). Each situation contains a number of objects (i.e. people and vehicles), which have been detected and analysed by the threat recognition algorithms and can be shown by selecting situation and pressing the Show objects button (Figure 6: Drilling down from a Threat Viewer via Situations Viewer to Objects Viewer First table from right). In addition, the three tables mentioned above can also been seen individually (a separate Situations Viewer without seeing the Threats has the capability of expanding and showing the related Objects of the selected Situation) Figure 5: Situations Viewer to Objects Viewer (Drill down) in Dynamic mode. We should also notice that the Situations Viewer and Objects Viewer screen have three modes. These three modes are Static, Dynamic, and History mode. The Static mode shows everything that is in our database (at any time) depending on which table you are using (either situations or objects), the dynamic mode shows only related information for the current timestep in the video and the History mode shows everything up to the current timestep (note that the Dynamic and History mode share the same button and these two modes can only be used while a playback video is playing or is paused not when is stopped). These three modes can be selected using the buttons provided under the tables. Finally, the Truck Status Monitor window (see Section 4.1.2) has filters activated to restrict the view of the threats.
Figure 4: Threat Viewer - History Figure 5: Situations Viewer to Objects Viewer (Drill down) in Dynamic mode
Figure 6: Drilling down from a Threat Viewer via Situations Viewer to Objects Viewer
4.1.4 Database Management Module The database has been implemented using a relational database management system (HSQLDB) and object-relation mapping (i.e. classes to database tables) has been implemented using Hibernate. The database management module (Figure 7: Database Manager Screen) allows the user to perform a variety of database manipulations. To begin with, it allows the user to create a database. In addition, the user can perform DDL (Data Definition Language) and DML (Data Manipulation Language) statements to create or drop and query the database respectively (Figure 7: Database Manager Screen green square). Finally, from the database management module you can select a database and connect or disconnect from it (Figure 7: Database Manager Screen red square). 4.1.5 Video Player One of the most important GUIs of the HMI is its ability of showing playback of live stream of the cameras in the trucks. The Video player (Figure 8: Video player) allows the manipulation of the video providing capabilities such as choosing which video to play, pausing and stopping the video. In addition, it shows whether there exists data (threats, annotations, etc.) for the selected video, using either green (exists) or red (does not exist) colour next to the labels Data available and Video available. The video can been seen on two different screens: a) a Multi-camera screen and b) a Single Camera screen. These two screens are described below.
Figure 7: Database Manager Screen
Figure 8: Video player 4.1.6 Multi-camera screen The Multi-camera screen consists of two main components. The top component shows four small windows (see Figure 9: Multi-camera Screen Red square) named Driver, History(1), History(2) and Alarm. These 4 displays show pictures (with annotation - squares) of the drivers last position, previously detected threats (History 1 & 2) and the latest threat detected respectively. The second component of the Multi-camera screen is the four camera displays (see Figure 9: Multicamera Screen Blue square). In particular, users can watch the video from all the cameras on a multi-camera screen containing three small video panels and a large one. The large one displays the currently selected camera, whereas the smaller ones allow the user to switch the large panel to a different camera (by clicking on a small camera panel). In addition, annotations such as object detections and their tracks each track has a unique colour and is a line from the starting point of the object detection in the picture to the current point/location of the object detection that relates to - (and their id blue text next to circle) of the object detections are displayed on the large camera video panel. Any detected threats are displayed as well, including threat information and confidence levels (text next to the circle annotating the object detection with yellow or red colour depending on the confidence levels if threat). The threat can be seen in this case by observing the colour of the annotations (circles) around an object detection. Red circle means that there is high probability of criminal behaviour, yellow means that there is potentially a criminal behaviour (lower confidence levels than the red one) and green means normal behaviour.
Figure 9: Multi-camera Screen 5 5 In this picture we can see three camera panels (a large one and two small ones) instead of four (the fourth one has a test image) and this is because in Paris we have used only three visual cameras so the fourth ones remains empty.
Figure 10: Single-camera Screen 4.1.7 Single-camera screen The Single camera screen (Figure 10: Single-camera Screen) does not have the top component that the Multi-camera screen has and only can display one camera at the time. In addition, the user can switch between the cameras using a list provided at the bottom of the single camera screen. Finally, annotations are also displayed the same way it is done on the large video panel of the Multi-camera screen. 4.1.8 Map View The Map View screen (Figure 11 & Figure 12) which displays the area around the truck from a bird s eye view. Object detections (red colour is criminal behaviour, yellow is potential criminal behaviour and green is normal behaviour) and tracks (uniquely coloured lines from the starting point of object detection in the map to the current point/location of the object detection that relates to) are also shown on this screen. In addition, the map view displays as a black rectangle the truck, as blue-green wedge areas the camera field of view of the cameras (read from the camera calibration files), as crosses the points of the buildings in the surrounding area (connecting the crosses with a line will show the shape of the buildings) and
with white lines some characteristic lines of the surrounding area such as a roundabout or a junction of roads. The information about the buildings is found via the Context Module from an ontology. Finally, it displays text next to an object detection annotation if an abnormality has been detected, for example a person staying for too long in an area near the truck. Figure 11: Map View (Paris area) with object detections, their tracks and text indicating abnormality
Figure 12: Map View showing clearly the area around (buildings, camera angles and road characteristics) 4.1.9 Threat Thresholds Manager The threats threshold manager (Figure 13: Threats Thresholds Manager) allows the user to configure the thresholds for displaying criminal behaviour, potentially criminal behaviour and none displaying of the threats at all. These thresholds configure the system so it knows which threat to show with yellow colour, red colour as mentioned in the Multi-camera, Single camera and Map view screens sections) or not displayed at all. The thresholds are following: Alarm range: This range represent the maximum and minimum values of confidence levels that the system will use to show an alarm (high probability of criminal behaviour red colour row in TruckStatusMonitor and ThreatViewer, and red colour circle on Multi-camera, Single-camera and Map View screens). Warning range: This range represent the maximum and minimum values of confidence levels that the system will use to show a warning (potential criminal behaviour yellow colour row in TruckStatusMonitor and ThreatViewer, and red colour circle on Multi-camera, Single-camera and Map View screens). Cutting limit: The system will ignore any threats with confidence level from the value given by the user to value 0.
Figure 13: Threats Thresholds Manager
4.2 Maritime HMI The generic ARENA architecture covers both land-based scenarios (e.g. theft from trucks) as well as maritime scenarios (e.g. pirate attacks). However, the main focus of the work in the project was the land-based use case (Section 2.2.2), i.e. the protection of a truck in a parking area. The maritime use case (Section 2.2.2) is covered in two ways: 4.2.1 Maritime Threat Detection We have developed several rule-based maritime threat detection algorithms which take radar and AIS tracks as input. Thus the focus is on providing early warning for the crew through the utilisation of long range sensors. Full details on these algorithms are given in Deliverable 7.3 6. The threat detection algorithms are implemented in Matlab 4. However, to demonstrate the use of the ARENA platform for maritime scenarios we have integrated a simple graphical user interface (GUI) that displays the output of the algorithms in the desktop HMI. This interface takes as input the XML files produced by the algorithms; the same data model 7 as in the land-based scenario is used. Below we give a screenshot of the maritime threat detection GUI which is part of the ARENA HMI (Figure 14: Incoming radar track displayed in the maritime GUI). Tracks are drawn on a (simulated) sea chart; the black square in the centre denotes the ship that is equipped with the ARENA system. This example shows the detection of an incoming radar track; the radar range is indicated by the blue circle. In this case, no threat detection algorithms are switched on. 6 ARENA WP7 Threat Recognition D7.3 Threat reasoning engine 7 ARENA WP3: Architecture Specification Final Version
Figure 14: Incoming radar track displayed in the maritime GUI In Figure 15, the same scenario is replayed, but this time the radarorphan algorithm is activated (the corresponding box on the GUI is ticked). This algorithm detects radar tracks which cannot be associated with a matching AIS track within a certain time window and displays them as a threat (by painting the track red). This scenario corresponds to the use case quoted in Section 2.1.2. Figure 15: The same radar track with threat detection algorithm activated
4.2.2 Boat Tracking In addition to the threat detection algorithms, we have implemented a boat tracking algorithm for visual and infrared cameras. For full details on this algorithm, please refer to Deliverable D5.2 8. The boat tracking algorithm has been tested on the PETS dataset 9. In the HMI, we display the outputs of this algorithm in the form of an annotated video stream (similar to the land-based truck scenario). This stream is shown on a video player GUI which is integrated in the desktop HMI (Figure 16: Boat Tracking Screen). This approach has not been developed further within the ARENA project; we display the output of the boat tracker as a proof of concept. Figure 16: Boat Tracking Screen 8 D5.2 ARENA WP5: Tracking 9 http://www.cvg.rdg.ac.uk/pets2009/a.html
4.3 Mobile HMI The mobile HMI has been implemented using Google Cloud Messaging (GCM) for Android 10. GCM is a lightweight messaging service allowing developers to send data from a server to one or several registered Android device(s). We chose this service because it is free of charge and automatically handles all aspects of message queuing and delivery. GCM messages can accommodate a payload of up to 4kb, which is easily sufficient to send a description of the detected threat. It is not, however, enough to send images or video sequences. For this reason we transfer images through synchronised folders on the desktop and the device. The threat notification that is sent to the mobile device simply contains the name of the frame on which the threat was first detected. On receipt of the notification, the image is then loaded and displayed. Alternatively, it is possible to store the frame in an online repository and send its URL to the device, which subsequently downloads the image. In this implementation of the system we do not send video sequences to the mobile device because of bandwidth limitations. The GCM messages contain the following data: Threat type: This is a brief description of the type of threat that has been detected; examples are Person Loitering, Person Checking out Truck, Theft in Progress etc. Severity: This is a flag indicating the severity of the threat; values are either Warning or Threat. Picture URL: This is the filename of the image frame in which the threat was first detected (or the URL of the online repository from which the image is to be downloaded). The message format used by GCM is JSON. An example of the payload of a typical message is given below: data: {threattype=person loitering,url=https://s3-eu-west- 1.amazonaws.com/arenascreenshots/samplethreat.png,severity=Warning} 10 http://developer.android.com/google/gcm/index.html
4.3.1 Server Side GCM messages are sent from an application server via the GCM server; the message takes the form of a simple HTTPS request. The application server has been integrated into the ARENA HMI to allow for an automatic notification of the driver if a threat is detected. This creates a fully-automated processing sequence: monitoring threat detection notification, implementing the use case in Section2.1.2. 4.3.2 Client Side The client side is realised by an Android application deployed for testing purposes on a smartphone (HTC Desire) and a tablet (Samsung Nexus 10). The user interface is shown in Figure 17: Screenshot of the ARENA HMI on the tablet (before receipt of threat notification). On installation, the application registers with the GCM server and receives a unique device ID. This data is passed on to the application server to identify devices that will receive notifications. When the ARENA system detects a threat, the application server sends a message via HTTPS POST request to the GCM server. There, the message is queued and delivered when the device is online. On receipt of the message, the mobile HMI extracts the payload data, loads and displays the image (if a filename or URL has been sent) and triggers an acoustic signal to notify the user (truck driver). Figure 18 shows the display the user will see. On receipt of a notification, the user is expected to perform any necessary checks and either dismiss or escalate the alert. Ideally, this should be done through the ARENA mobile application using the buttons provided. However, the currently used version of GCM does not allow the sending of upstream message (device to server); it is possible, though, to use GCM Cloud Connection Server (CSS) which provides this functionality via a XMPP connection 11. 11 https://developer.android.com/google/gcm/ccs.html
Figure 17: Screenshot of the ARENA HMI on the tablet (before receipt of threat notification)
Figure 18: On receipt of a warning
4.4 Context Module HMI The Context Module HMI includes four main tabbed panels, which enable the user to introduce general information about the Context Module, information about the (truck) platform parking lot elements, infrastructure and zones. Users can add, remove or change such information. Main dialog window of Context Module is presented in Figure 19: Context Module HMI (CM HMI). Particular tabs of Context Module HMI are presented in Figure 19 - Figure 24 with exemplary definition of the platform, elements and parameters at the parking lot and their visualization. At each step of Context Module HMI usage user can Refresh, Save or Cancel introduced information. Figure 19: Context Module HMI (CM HMI)
Tab Platform comprises of two sub-tabs related to platform and camera details. The Platform sub-tab enables the user to define parameters of platform (vehicle) such as width, height and length (in meters). The user can also introduce the last platform location, i.e. geographical coordinates, x and y (see Figure 20: "Platform" tab of CM HMI). Figure 20: "Platform" tab of CM HMI The tab Camera provides capabilities to manage the cameras installed on the platform. The user can add new cameras and remove previously defined ones. Moreover, it enables one to define the camera s angle of view, i.e. vertical and horizontal angles, camera s position on platform parameters x and y (distances as shown in Figure 18) and angle according to the platform s axis (see Figure 21: "Camera" tab of CM HMI). The management of the information about parking lots can be introduced using the Parking lots tab and the General and Buildings sub-tabs. The first one enables the user to provide general information about parking lots. It includes a description of the parking lot, and its general location country, town, street. Moreover, the user should define parking lot coordinates, can select or remove the existing details as well as add new coordinates (see Figure 22: "General" tab of CM HMI).
Figure 21: "Camera" tab of CM HMI Figure 22: "General" tab of CM HMI
The Context Module HMI also provides the capability of defining buildings in the neighbourhood of the parking lot. The user can add new buildings or remove buildings from pre-defined list. For the new one, the user should define the type of building and provide a description. Each building needs to have assigned geographical coordinates, the user can select or remove existing coordinates of given building or introduce new coordinates (see Figure 23: "Buildings" tab of CM HMI). Figure 23: "Buildings" tab of CM HMI Important from the Context Module capabilities point of view is the definition and management of zones at the parking lot. First, in the tab Zones, the user selects the parking lot at which he/she would like to define zones. Next, the user can add new zones, select existing, pre-defined zones, or remove them. As the last step, the zones need to have assigned geographical coordinates, which the user can select from existing ones or add new coordinates (see Figure 24: "Zones" tab of CM HMI). Zones can be also inferred automatically based on the Zone learning algorithm available for the ARENA system as a result, a set of rules will be populated into the ontology of Context Module.
Figure 24: "Zones" tab of CM HMI
5 Conclusion and Future Work In conclusion, the ARENA HMIs have been successful in various areas. These areas are a) developing a user friendly Human Machine Interface for the end users which will allow them to see the output of the algorithms in an understandable way, b) allowing the algorithm developers to test their algorithms and optimize them quicker and finally c) showing proof of concept at the end of the project through demonstrations to journalists and to the public in the TRA 12 conference (22 potentially interested stakeholders). In particular, we have validated the use of a generic architecture and data model for visualising sensor fusion and threat information in a configurable implementation for both land and maritime cases. The desktop HMI was also invaluable in demonstrating the whole data processing chain from raw sensor data, through object assessment and situational awareness, to threat detection and classification. Most importantly, both the desktop and mobile HMIs provide the end user with automatic notifications of potential and emerging threats. In future projects, the camera-based detection and tracking in the maritime case should be further developed and fused with the radar and AIS data to provide a more complete situational picture. Close range threat detection algorithms are needed to monitor the development and escalation of a threat scenario (e.g. boat coming alongside, attackers trying to board the ship etc.). Some of these problems will be tackled as part of the IPATCH project 13. 12 Transport Research Arena. Paris, 14-17 April 2014. 13 http://www.ipatchproject.eu/