FlinkSQL上的完整流处理旅程

In the fast-paced world of data analytics, real-time processing has become a necessity. Modern businesses require insights not just quickly, but in real-time to make informed decisions and stay ahead of the competition. Apache Flink has emerged as a powerful tool in this domain, offering state-of-the-art stream processing capabilities. In this blog, we introduce our FlinkSQL interactive solution in accompanying productionising automation, and enhancing our users’ stream processing development journey.

在快速发展的数据分析世界中，实时处理已成为一种必要性。现代企业不仅需要快速获取洞察，还需要实时获取，以便做出明智的决策并保持竞争优势。Apache Flink 已成为该领域的强大工具，提供最先进的流处理能力。在这篇博客中，我们介绍了我们的 FlinkSQL 交互式解决方案，以配合生产自动化，并增强我们用户的流处理开发旅程。

Preface

前言

Last year, we introduced Zeppelin notebooks for Flink, as detailed in our previous post Rethinking Stream Processing: Data Exploration in an effort to enhance data exploration for downstream data users. However, as our use cases evolved over time, we quickly hit a few technical roadblocks.

去年，我们为 Flink 引入了 Zeppelin notebooks，详细信息请参见我们之前的文章 重新思考流处理：数据探索，以增强下游数据用户的数据探索能力。然而，随着我们的用例随着时间的推移而演变，我们很快遇到了一些技术障碍。

Flink version maintenance

Flink 版本维护

Zeppelin notebook source code is maintained by a community separate from Flink’s community. As of writing, the latest Flink version supported is 1.17, whilst the latest Flink is already on version 1.20. This discrepancy in version support hinders our Flink upgrading efforts.

Zeppelin notebook 源代码由一个与 Flink 社区分开的社区维护。截至撰写时，支持的最新 Flink 版本是 1.17，而最新的 Flink 已经是 1.20 版本。这种版本支持的差异阻碍了我们对 Flink 的升级工作。

Cluster start up time

集群启动时间

Our design to spin up a Zeppelin cluster per user on demand invokes a cold start delay, taking roughly around 5 minutes for the notebook to be ready. This delay is not suitable for use cases that require quick insights from production data. We quickly noticed that the user uptake of this solution was not as high as we expected.

我们设计的按需为每个用户启动 Zeppelin 集群会引发冷启动延迟，笔记本准备好大约需要 5 分钟。这种延迟不适合需要快速从生产数据中获取洞察的用例。我们很快注意到，这种解决方案的用户接受度并没有我们预期的那么高。

Integration challenges

集成挑战

Whilst Zeppelin notebooks were use...