In computing, an event is an occurrence that triggers a response within a system or application. User actions such as clicks or keystrokes, or system-generated occurrences, like incoming network traffic, are commonly known events. These events are central in an event-driven architecture to communication and coordination among decoupled components. This approach allows systems to respond to events as they happen, making it ideal for real-time processing and scalability.
However, an event-driven architecture presents several challenges, especially those related to the huge amount of generated data and its management. For example, ensuring data consistency can be tricky, especially with asynchronous processing. Managing event order and schema compatibility, handling scalability, and ensuring fault tolerance for reliable event delivery are essential but complex.
In this context, observability has also become a fundamental aspect of daily operations, as it allows tracking all activity within your applications. With accurate data, decision-making is facilitated to ensure system reliability, performance, and security. And this task must be carried out smoothly, leveraging the best resources and tools available.
The basics of an event-based monitoring tool
Event-driven architectures are closely linked to real-time event-monitoring tools. These tools enable continuous monitoring and alerting based on the events occurring within a system. This integration ensures that anomalies and critical updates are detected and handled promptly, enhancing system reliability and performance.
An ideal observability and event-based product needs to meet certain requirements to be truly efficient:
- Be capable of processing large amounts of data in real time.
- Generate logs with detailed information about the events.
- Offer high reliability and resilience.
- Ensure the availability and persistence of data.
- Be compatible with distributed architectures.
Azion Real-Time Events was developed to meet these critical requirements, aiming to monitor application behavior comprehensively. It achieves this by:
- Collecting data from diverse sources across the platform.
- Generating detailed logs for each processed request, including user, application, and API interactions.
- Capturing metadata to identify and register the performance of individual requests.
- Enabling real-time observation of system behavior, including traffic handling, anomaly detection, and issue identification.
This approach allows for a granular view of your data, which can be accessed through precise and flexible queries. Users can leverage ready-built tools for data source selection, filtering, and sorting via the Azion Console or CLI, or utilize the GraphiQL Playground for more advanced analysis.
How does Real-Time Events work
The data volume that feeds Real-Time Events comes from all the events occurring at the more than 100 edge locations in our distributed network. All these events are collected and transformed to subsequently be classified and categorized according to each product so that our customers can consult them at any time they require. The workflow of Real-Time Events can be illustrated as follows:
Data Generation Layer
- A request reaches Azion’s platform.
- The request is processed by the Azion services in a given edge location.
- The system generates logs, containing all the relevant information related to that request.
Data Transformation Layer
- The Distributed Event Broker handles the distribution, management, and routing of event logs across different components in a distributed environment. In this case, it’s based on Kafka, receiving the events and processing them in a row.
- It enables the decoupling of event sources from event consumers, facilitating asynchronous communication between them.
- The ETL Processor transforms the data received from the broker into a suitable format and loads it into the target data storage system.
- This process consolidates data from multiple sources into a single, cohesive view.
- Data is stored in Columnar Database Events Databases, allowing for more efficient data retrieval and compression, and overall performance.
- This step allows storage as well as persistence and retention of data.
API and Presentation Layer
Once the data is processed, transformed, and stored:
- Azion customers can access the data via Real-Time Events, either through Azion Console, CLI, or the Real-Time Events GraphQL API.
Real-Time Events main features
Considering that real-time systems must process events instantly, without delays or interruptions, concerns around reliability and resilience appeared during the Real-Time Events development. In this matter, being an edge platform was a huge advantage to accelerate our plan. This is because:
- A robust distributed architecture, with more than 100 edge locations globally, facilitates handling high traffic, unexpected spikes, and all the generated data.
- This distributed architecture also helps to process events accurately and consistently, even during network failures or server downtime, capable of recovering from failures and functioning without losing data.
- Real-Time Events stores event data efficiently for historical analysis, auditing, or debugging purposes, ensuring data persistence and accessibility.
- Processing events closer to the source, at the edge, reduces latency and improves performance overall.
To summarize, by distributing the workload across multiple edge locations, together with redundant systems and automatic failover strategies, the platform can handle higher volumes of data with greater resilience and fault tolerance.
With Real-Time Events, you can perform a deep dive analysis, checking raw logs in real time and querying data from different data sources, related to each Azion product or service that generated the events, from the last 168 hours, equivalent to 7 days.
Besides the detailed log view, storage, and filtering features, the product also offers integration with other analytics tools, either Azion’s or third-party tools, and automation with edge functions to execute dynamic queries.
Once your applications start receiving traffic, it’s possible to consult and diagnose these events. Each data source exhibits detailed information from events related to topics such as access, behavior, and performance of products connected to your edge applications.
Get to know more about Real-Time Events Data Sources or check the documentation to know How to add filters on Real-Time Events.
Real-Time Events GraphQL API
Real-Time Events already provides you with a detailed view of your data. However, thinking of giving more control and granularity to our customers, we decided to add GraphQL-based API endpoints to further enhance the possibilities of this product. With the Real-Time Events GraphQL API, you can explore different variables from your raw data logs, all in a single query, simplifying the data retrieval process and improving performance.
Why use GraphQL? GraphQL allows developers to request specific data, different from REST APIs, where each endpoint is tied to a specific resource and multiple requests may be needed to gather all the required data. This makes GraphQL more efficient, avoiding over-fetching or under-fetching, consequently reducing the server load and operational costs. Additionally, GraphQL API popularity was key, as well as being an easy-to-learn query language, enabling experienced developers and non-technical roles to query data intuitively.
The Real-Time Events GraphQL API features include:
- Datasets: a dataset is a structured collection of data that can be queried, including specific fields related to the context of each product.
- Filtering: using a more specific and granular query is a way of getting exact and direct data from your requests. For example, querying information from a certain field instead of all the data from a dataset, as well as using parameters to define further your analysis.
- Sorting: this feature allows enables to organize and sort the received data of a dataset according to the event’s order, either ascending or descending.
- Pagination: Azion GraphQL API uses offset and limit pagination to receive data within the specific range you’ve set.
- Resampling: this feature allows you to resample your data to display a smaller amount of data points being exhibited on charts. It works according to the rule you pass in the function field.
Visit the documentation to explore query examples for GraphQL API and read more on How to investigate requests with Real-Time Events GraphQL API.
GraphiQL Playground
Looking to test some queries and become familiar with the Real-Time Events GraphQL API? The GraphiQL Playground is a tool to run queries or explore how to build them.
Events vs. Metrics
Azion Observe product suite includes Real-Time Metrics and Real-Time Events. Even when they can seem similar for the more beginner users, each one has a specific focus. By using them together, you have a powerful tool that makes your analysis more comprehensive and in-depth.
Metrics are specific, quantifiable indicators that represent data, collected at regular intervals, and presented in an aggregated format (averages, sums, counts), being predictable and consistent, providing a continuous stream of data. In this case, data can be exhibited in graphs, facilitating analysis of trends, understanding long-term behavior, and identifying performance bottlenecks.
On the other hand, events represent an action that has taken place. Events aren’t continuous and occur irregularly, based on specific triggers or conditions. Events are exhibited in logs, providing qualitative data, often with contextual information such as timestamps, metadata, and descriptions. Ideal for incident detection, auditing, logging, and real-time alerting.
In this matter, Real-Time Metrics show long-term performance trends, while Real-Time Events zoom in to understand better the context for anomalies and changes in the system related to performance and behavior. For example: through Real-Time Metrics, you can observe the trend in your successfully attended request. In certain periods, you detect a decline in metrics related to requests from a specific geolocation. Consulting the logs related to the events around these requests can help you identify the problem and fix it.
Conclusion
Azion Real-Time Events was developed as a solution for your needs involving:
- Collecting data from different sources.
- Monitoring application behavior.
- Creating complex queries and obtaining real-time data related to applications’ functionality and performance.
- Troubleshooting, debugging, and identifying errors, and taking preventative measures.
It can also be useful for more comprehensive use cases such as:
- Security observability: monitoring real-time security events across the network, detecting and responding to threats as they emerge, ensuring prompt mitigation of potential risks at the edge, before reaching your origin.
- Compliance: tracking and enforcing compliance with regulations by monitoring event data across systems, certifying that all processes meet legal and industry.
- Auditing: giving businesses a comprehensive view of actions, changes, and accesses across systems.
- Data-driven development: enabling feedback by analyzing real-time events and user interactions, allowing developers to optimize and fine-tune features based on real-world usage data.
This way, you’re able to interact with and utilize your event data more strategically and increase your observability, getting comprehensive real-time insights. All of this also empowers you to make more informed decisions quickly and efficiently to guarantee the best performance and experience in your digital environment.
Additionally, by incorporating advanced features like GraphQL-based endpoints, it offers enhanced granularity and flexibility, enabling users to tailor their data queries precisely to their needs.
Get to know more about Azion Real-Time Events and how to start using it in the documentation.
