Azion Data Stream is our data delivery tool, and a critical player in our observability full-stack of tools. Good analytics is critical to the success of your enterprise security, web maintenance, and business insights, and data delivery methods like Azion Data Stream provide the raw information that analytics tools need to produce valuable metrics. Check out our introductory article for a refresh on the basics of What is Data Stream. In this post, we’re going to take a closer look at our Data Stream tool, how it works, and how you can use it to better serve your data analytics needs on the Azion Edge platform.
The sheer volume of data being processed in the world is truly staggering. A 2019 Forbes report pegged it at over 4 million Gigabytes per minute in the United States alone, and the prevalence of data generation and consumption has continued to exponentially skyrocket each year since, particularly after the recent global pandemic pushed more people and businesses into the digital space than ever before. In order to make sense of the data that your business and your customers generate you need modern data analytics solutions, and a data delivery service like Azion Data Stream to feed all that raw data to your analytics tools.
How It Works
Putting it simply, Azion Data Stream is our way of rendering the data your applications generate on our edge platform transparent and observable to you. This allows you to then plug that data into your preferred analytics tools (Either Azion or third party), to convert it into metrics and visualizations that can provide key insight for marketing strategy and infrastructural maintenance.
This data can be used to power your:
- Big Data
- Stream Processing
- SIEM (Security Information and Event Management)
To optimize transparency, security, and network efficiency.
Azion Data Stream plugs into a Source, which defines the type of data received. You can connect Azion Data Stream to
- Edge Applications
- Edge Functions
- WAF Events
This gives Data Stream access to all relevant data pertaining to application health, user/customer trends, and attack patterns targeting your system. Azion Data Stream then connects to the Stream Ingestion, which will be the receiving end for all this data. The Stream Ingestion can be any of a number of third party tools focused on this step in the Stream journey.
From there you can store, process, and analyze your data in whatever manner you choose, including having the option of using Azion analytics tools like Real-Time Metrics and Real-Time Events. With Azion Data Stream, this whole journey takes place in real-time, so your Stream Ingestion will receive your data in the moment that data is being generated. With Azion’s modern edge network, real-time streaming is able to scale to the largest of enterprises, using distributed edge nodes to ensure consistent ultra-low latency.
Parsing Best Practices
When working with some of our largest enterprise clients, we have to account for a truly massive amount of data being generated through our platform, which subsequently must all be funneled through Azion Data Stream. As a result, we’ve made sure to build a tool that is not only able to operate at competitive real-time speeds, but also consistently preserves the clarity of data being passed along, implementing powerful safeguards against data corruption.
Slight communication deviances can generate parser/syntax errors, rendering portions of transmitted information illegible to the receiving Stream Ingestion. Even if .01% of data developed syntax errors in transit, that’s still a massive loss at the scale we stream at. Illegible data is useless data, and even slightly illegible data introduces sub-optimal lag in what is supposed to be a real-time process. With Azion Data Stream, information clarity is protected even for the largest of streaming enterprises. Our best practices include using ASCII encoding on all our streams to optimally minimize the occurrence of any such issues.
Putting Azion Data Stream to Work
Azion Data Stream is built to follow our core design philosophies of maximizing customizability and user experience. When you’re ready to implement Data Stream into your analytics strategy, simply start by logging into your Azion account on Real-Time Manager.
Once there, you can navigate to Data Stream. Either by clicking directly under the Edge Analytics column on the home page, or by opening the menu in the upper left-hand corner and finding it in the OBSERVE section.
Then, click Add Streaming in the upper right to set up your first stream.
This brings you to the stream creation portal, where you can name your stream, define, and customize the requisite parameters that will determine what data you process and how it will be received.
As mentioned above, Azion Data Stream has three sources to select from. Edge Applications, Edge Functions, and WAF Events. You can only select one source per stream, but for optimal observability best practices, you’ll want to eventually have at least one stream running from each source. You can always provision additional streams by returning to the initial RTM Data Stream menu and selecting Create New Stream.
Once you’ve chosen a source, Real-Time Manager does the rest for you. You can select the appropriate data set from a series of templates designed to fit most standard streaming configurations. Each template is open and customizable in the plaintext window below the templates, giving you the flexibility to merge, aggregate, and refine your data to best fit your needs. This flexibility is powerful, easy, but most importantly optional. That’s because the templates are already complete and fully functional from the start. Simply selecting one is enough to get your stream up and running without needing to write any code yourself.
There’s a couple of further customization options that you can play with to enhance your stream efficiency. Data sampling is an optional feature that can be turned on to stream a randomized sampling of a percentage of your data instead of all the data your source generates. This is not usually optimal for security-focused streaming, where you want to be able to build insights from all the data you have. But it can lighten your data load when streaming content focused on user experience, as a randomized sampling in that use-case can pretty effectively represent the larger data set.
Finally, all that’s left to do is complete the streaming journey by selecting your desired endpoint. Azion does not currently offer a Stream Ingestion service of our own, but Azion Data Stream is built to be compatible with a wide variety of the most popular Stream Ingestion solutions. This open-form collaborative approach to working with third-party tools is one of the major strengths of Azion Data Stream. We believe vendor-locked products are counterintuitive to enterprise growth, so whenever possible we build tools with full compatibility, allowing you to pick and choose what works best for your enterprise at each step in your Data Stream flow.
Current Stream Ingestion offerings that we’ve achieved compatibility with include:
- Apache Kafka
- Google BigQuery
- S3
- Splunk
- Elasticsearch
- IBM QRadar
- AWS Kinesis Data Firehose
- Data Dog
And we’re continuously working to offer even-more third party support. If your offering of choice isn’t provided, selecting Standard HTTP/HTTPS Post will provide you a simple and effective option that is compatible with most third party tools.
And that’s it! From there, just save it and you’ll have your first Data Stream ruleset, available to access and demo at any time.
Summary
Data Stream is the lifeblood of any data analytics strategy. If you’re not making use of Azion Data Analytics, you’re losing out on the chance to apply a holistic observability philosophy to your use of our edge platform. Hopefully this article has demonstrated the importance of using Azion Data Stream to gain insights into your web infrastructure, security, and customer base, allowing you to develop data-based strategies to accelerate the growth of your business.
It’s also a far from daunting process to get set up. Just click through our intuitive user interface on Real Time Manager and you’ll have secure low-latency streams up in no time, delivering Real-Time data directly from your Azion edge applications to feed your analytics tools. Go ahead and try it out for yourself.
Curious about how other companies have achieved observability optimization using Azion Data Stream with Azion’s distributed edge platform? Check out our success stories from client company GetNinjas and see how Azion Data Stream improved their security and performance optimization through better observability practices.
Got further questions about Azion Data Stream beyond the scope of this article? Feel free to reach out to our team of experts to learn more and find out why Azion Data Stream might be a good fit for your company.
