30.9.2016 DOAG 2016 Big Data Days Guido Schmutz BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH
Guido Schmutz Working for Trivadis for more than 19 years Oracle ACE Director for Fusion Middleware and SOA Co-Author of different books Consultant, Trainer, Software Architect for Java, SOA & Big Data / Fast Data Member of Trivadis Architecture Board Technology Manager @ Trivadis More than 25 years of software development experience Contact: guido.schmutz@trivadis.com Blog: http://guidoschmutz.wordpress.com Slideshare: http://www.slideshare.net/gschmutz Twitter: gschmutz 2
Agenda 1. Introduction 2. Towards an IoT Architecture 3. IoT Refererence Architecture 4. Summary 3
Introduction 4
Internet of Things (IoT) Wave Internet of Things (IoT): Enabling communication between devices, people & processes to exchange useful information & knowledge that create value for humans Term was first proposed by Kevin Ashton in 1999 Source: The Economist 5 Source: Ericsson, June 2016
Reasons why IoT opportunity is occurring now? Affordable hardware Costs of actuators & sensors have been cut in half over last 10 years Smaller, more powerful hardware Form factors of hardware have shrunk to millimeter or even nanometer levels Ubiquitous & cheap mobility Cost for mobile devices, bandwidth and data processing has declined over last 10 years Availability of supporting tools Big data tools & cloud based infrastructure have become widely available Mass market awareness IoT has surpassed a critical tipping point Vision of a connected world has reached such a followership that companies have initiated IoT developments Commitment is irreversible 6
The Sensing-as-a- Model 7
Towards an IoT Architecture 15
Key Challenges for building an IoT application 1. Connect: How to collect data from intelligent devices? Abstract complexity associated with device connectivity Standardize integration of devices with enterprise 2. Analyze: How to analyze IoT data? Reduce noise and detect business event at real-time Enable historical big-data analysis 3. Integrate: How to integrate IoT data & events with enterprise infrastructure? Make enterprise processes IoT friendly Allow enterprise & mobile applications to control devices 18
Today) Existing -/API Architecture as a base Mobile Apps D B Rich (Web) 19Client Apps D B / SOAP / SOAP API Gateway API Gateway / SOAP Bus Oracle Data Integrator Enterprise Bus (ESB) / Data Integration SOAP SOAP Various SQL WS WS External Cloud Providers BPM and SOA Platform Processes SOA Suite s Enterprise Apps Logic/Rules Intelligence Visualization BPM Suite Activity Monitoring 19 = one way = request/response
IoT 1a) Reuse exiting -/API-based Architecture Mobile Apps D B Rich (Web) 20Client Apps D B IoT Smart Devices / SOAP / SOAP API Gateway API Gateway / / SOAP Bus Oracle Data Integrator Weblogic Enterprise Bus (ESB) / Data Integration SOAP SOAP Various SQL WebSocket WS WS External Cloud Providers BPM and SOA Platform Processes SOA Suite s Enterprise Apps Logic/Rules Intelligence Visualization BPM Suite Activity Monitoring 20 = one way = request/response
IoT 1a) Challenges Do IoT devices contain enough resources to communicate directly over the internet (HTTP or )? Should the device only collect data (sense) or is there also the way back necessary (actuator)? Can be used from external devices (firewalls allow traffic over )? How many IoT devices are planned short and long term? How often do they send to the backend? Are the server as well as the ESB capable to deal with the resulting message volume? What are the operations on IoT messages / events, only simple transformations, filter and routing operations? 21
IoT 1b) Reuse existing -/API-based Architecture Mobile Apps D B Rich (Web) 22Client Apps D B IoT Smart Devices / SOAP / SOAP HTTP API Gateway API Gateway HTTP / SOAP Bus Oracle Data Integrator Weblogic Enterprise Bus (ESB) / Data Integration SOAP SOAP Various SQL WebSocket WS WS External Cloud Providers BPM and SOA Platform Processes SOA Suite s Enterprise Apps Logic/Rules Intelligence Visualization BPM Suite Activity Monitoring 22 = one way = request/response
IoT 2) Adding Hub and optional IoT Gateway Mobile Apps D B Rich (Web) 23Client Apps D B IoT Smart Devices IoT Devices / SOAP / SOAP IoT Gateways Kura API Gateway API Gateway Bus / SOAP / / / / Oracle Data Integrator Hub ESB / Data Integration SOAP SOAP Various SQL WebSocket WS WS External Cloud Providers BPM and SOA Platform Processes SOA Suite s Enterprise Apps Logic/Rules Intelligence Visualization BPM Suite Activity Monitoring 23 = one way = request/response
How to implement an Hub? Apache to the rescue Distributed publish-subscribe messaging system Designed for processing of high-volume, real time activity stream data (logs, metrics collections, social media streams, ) Producer Producer Producer Cluster Consumer Consumer Consumer Topic Semantic does not implement standard Initially developed at LinkedIn, now part of Apache 24
Oracle s Bus as a consumer of Web Apps Bus 12c Mobile Apps Proxy Pipeline Routing Cloud Cloud API Cloud Apps Sensor / IoT Backend Apps Database CDC Stream Processing Proxy Pipeline Routing WSDL WSDL Backend Apps 25
IoT 2) Solutions & Challenges Solutions Hub solves the potential scalability issue of IoT Gateway makes sure that lightweight sensors can connect to the internet / send their data Challenges Where to do complex analytics on the events? Is it scalable? Can we really send all data down to backend? Network bandwidth? 26
IoT 3) Adding Stream Processing / in Backend 27 Mobile Apps D B Rich (Web) Client Apps D B / SOAP / SOAP API Gateway API Gateway / SOAP Bus Oracle Data Integrator (ESB) / Data Integration SOAP SOAP Various WS WS External Cloud Providers BPM and SOA Platform Processes SOA Suite s Enterprise Apps Logic/Rules BPM Suite IoT Smart Devices IoT Devices / / IoT Gateways Kura / / Hub Stream Processing ESP / CEP Stream SQL Hub WebSocket Intelligence Visualization Activity Monitoring 27 = one way = request/response
IoT 3) Solutions & Challenges Solutions Stream Processing handles complex analytics on events in a scalable manner before sending events to ESB / backend systems Challenges Can we really send all data down to backend? Network bandwidth? 28
Oracle s Stream as consumer of / Web Apps Mobile Apps Sensor / IoT Stream Machine Data Oracle Stream CDC 29
IoT 4) Adding Industry 4.0 Data Sources (machine data) 30 Mobile Apps D B Rich (Web) Client Apps D B / SOAP / SOAP API Gateway API Gateway / SOAP Bus Oracle Data Integrator (ESB) / Data Integration SOAP SOAP Various WS WS External Cloud Providers BPM and SOA Platform Processes SOA Suite s Enterprise Apps Logic/Rules BPM Suite IoT Smart Devices IoT Devices I 4.0 Machine CDC GoldenGate / / IoT Gateways Kura / / / / Hub Stream Processing ESP / CEP Stream SQL Hub WebSocket Intelligence Visualization Activity Monitoring 30 = one way = request/response
Oracle s GoldenGate for Change Data Capture of existing database Machine Data Capture Pump Delivery HDFS Oracle GoldenGate Machine Data GoldenGate Gateway Capture Pump Delivery HBase Oracle GoldenGate 31
IoT 5) Adding Stream Processing / at Edge 32 Mobile Apps D B Rich (Web) Client Apps D B / SOAP / SOAP API Gateway API Gateway / SOAP Bus Oracle Data Integrator (ESB) / Data Integration SOAP SOAP Various WS WS External Cloud Providers BPM and SOA Platform Processes SOA Suite s Enterprise Apps Logic/Rules BPM Suite IoT Smart Devices IoT Devices I 4.0 Machine CDC GoldenGate / / IoT Gateways ESP/CEP Edge / / / / Hub Stream Processing ESP / CEP Stream Hub SQL WebSocket Intelligence Visualization Activity Monitoring 32 = one way = request/response
IoT 5) Solutions & Challenges Solutions Stream Processing at the edge / gateway allows to reduce the amount of messages send to the backend (cloud / on premises) if necessary Challenges What if we need the raw events persisted? Where do to that? 33
IoT 6) Adding Raw Data and Batch 34 Mobile Apps D B Rich (Web) Client Apps D B IoT Smart Devices IoT Devices I 4.0 Machine CDC GoldenGate / SOAP / SOAP / / IoT Gateways ESP/CEP Edge / / / / API Gateway API Gateway Hub / SOAP Oracle Big Data Appliance HDFS Big Data Processing Batch Processing Hadoop / Spark Stream Processing ESP/CEP Stream HDFS Bus Oracle Data Integrator SQL Hub (ESB) / Data Integration SOAP SOAP Various SQL WebSocket WS WS External Cloud Providers BPM and SOA Platform Processes SOA Suite s Enterprise Apps Logic/Rules Intelligence Visualization BPM Suite Activity Monitoring 34 = one way = request/response
IoT 6) Solutions & Challenges Solutions Adding Big Data platform allows to store all raw data in the distributed file system in a scalable and reliable manner Challenges How can we leverage the Big Data platform for more than just storing raw data? How does it combine with the stream processing? 35
Continuous Ingestion / Fan-In from the Edge File Source 37 Log 37 Log Log Source CDC Source Log Social IoT Sensor Native Dataflow GW Log Topic CDC GW CDC Dataflow GW Dataflow GW Topic Native Connect Hub Topic Topic Topic Topic Topic Topic Topic Big Data Stream Processing IoT Sensor IoT Sensor IoT GW Queue 37
Challenges for Ingesting Sensor Data Multitude of sensors Multiple Firmware versions Bad Data from damaged sensors Data Quality 38
40 IoT 6a) Adding Data Mining / Machine Learning and Model execution 40 Mobile Apps D B Rich (Web) Client Apps D B IoT Smart Devices IoT Devices I 4.0 Machine CDC GoldenGate / SOAP / SOAP / / IoT Gateways ESP/CEP Edge / / / / API Gateway API Gateway Hub / SOAP Oracle Big Data Appliance HDFS Big Data Processing Batch Processing Hadoop / Spark Stream Processing ESP/CEP Stream HDFS Bus SQL Oracle Data Integrator = one way = request/response (ESB) / Data Integration Hub SOAP SOAP Various SQL WebSocket WS WS External Cloud Providers BPM and SOA Platform Processes SOA Suite s Enterprise Apps Logic/Rules Intelligence Visualization BPM Suite Activity Monitoring
IoT Reference Architecture 42
IoT Logical Reference Architecture Big Data / BI IoT Device Sensor Actuator BLE ZigBee WIFI Wired IoT Gateway Streaming App UI CoAP XMPP DDS AMQP KAFKA WIFI Hub Hub Bus Endpoint Management IoT s Streaming Bulk UI API SOAP HTTP KAFKA Hub Bus Bus Streaming Bulk s App BPM and SOA Platform Processes Enterprise Applications UI UI 43
IoT Logical Reference Architecture Oracle on premises Oracle Data Integrator Big Data Big Data / BI Appliance IoT Device Sensor Oracle IoT CS Client Library Actuator BLE ZigBee WIFI Wired IoT Gateway Edge Streaming Oracle IoT CS Gateway App UI CoAP XMPP DDS AMQP KAFKA WIFI Hub Hub Bus Endpoint Management IoT s Streaming Stream Bulk UI API Gateway API SOAP HTTP KAFKA Hub Bus Bus Bus Bus Streaming Bulk UI BPM and SOA SOA Suite Platform s Processes BPM Suite Enterprise Applications Activity Monitoring App UI 44
IoT Logical Reference Architecture Oracle Cloud s IoT Device Sensor Oracle IoT CS Client Library Actuator BLE ZigBee WIFI Wired IoT Gateway Edge Streaming Oracle IoT CS Gateway App UI CoAP XMPP DDS AMQP KAFKA WIFI Hub Hub Bus Endpoint Management Oracle IoT CS IoT s Streaming Bulk UI API Hub Bus Bus Oracle Messaging CS SOAP HTTP KAFKA Oracle Integration CS Oracle Big Data CS Oracle Big Big Data Data / BI Discovery CS Oracle DataFlow ML CS Streaming Big Data Preparation CS Oracle Streaming Bulk CS UI Oracle BI CS Oracle SOA BPM CS and SOA Platform Oracle Process CS s Processes Oracle Mobile Enterprise CS Application Applications Container CS Container App CS UI 45
IoT Logical Reference Architecture Microsoft Azure Big Data / BI HD Insight IoT Device Sensor Actuator BLE ZigBee WIFI Wired IoT Gateway Streaming Azure AppIoT Gateway UI SDK CoAP XMPP DDS AMQP KAFKA WIFI Hub Hub Bus Azure IoT Hub Endpoint Management IoT s Azure Azure Streaming Streaming Bulk UI API SOAP HTTP KAFKA Azure Hub Hub Bus Azure Bus Bus Streaming Azure Azure Machine Learning Bulk UI Azure Power BI s App BPM and SOA Platform Processes Enterprise Applications UI 46
IoT Logical Architecture Open Source Big Data / BI IoT Device Sensor Actuator BLE ZigBee WIFI Wired IoT Gateway Streaming App UI CoAP XMPP DDS AMQP KAFKA WIFI Hub Hub Bus Endpoint Management IoT s Streaming Bulk UI API SOAP HTTP KAFKA Hub Bus Bus Streaming Bulk s App BPM and SOA Platform Processes Enterprise Applications UI UI 47
Summary 48
Summary Treat events as events! Infrastructures for handling lots of events are available! IoT tends to make Big Data / Fast Data infrastructures necessary Know your use case/requirements to choose the right architecture! Can my existing backend landscape handle the new IoT load? Do I have to handle huge amount of events in real-time? Do I need to filter/aggregate data before invoking existing backend systems? Do I want to do Advanced (predictive analytics) where historical information is necessary? What is the network bandwidth between device/gateway and cloud/backend? Centralized or Decentralized IoT solution? 49
J 50
Guido Schmutz Technology Manager guido.schmutz@trivadis.com 51