• Complex systems often include many different streaming data formats produced by different system outputs, sensors and sub-systems.
• Often there is a need to integrate these homogeneous data streams in real-time in an efficient, scalable and fault tolerant way.
We have implemented a heterogeneous streaming data integration demonstrator on Apache Spark Streaming which is:
• Tolerant to out-of-order event arrival
• Merges and integrates events by key
• Scalable and efficient for high velocity, high volume data
• The technology can be used across many different scenarios where heterogeneous streaming data is present such as:
• Integration of back end streaming machine generated data such as system logs, sensor data, telemetry data
• Customer data integration such as social media streams, customer touch points, website interactions etc.
•Dr. Guangyu Wu, UCD School of Computer Science and Informatics
•Dr. Oisín Boydell, UCD School of Computer Science and Informatics
•Dr. Brian MacNamee, UCD School of Computer Science and Informatics
For more information on buying a report evaluating the general task of computing statistical metrics over streaming data using three popular open-source stream processing platforms: 1) Apache Spark Streaming, 2) Apache Storm, and 3) Apache Storm Trident, please visit: https://licence.ucd.ie/tech/537