使用 StarRocks 构建 Spark 可观察性产品：实时和历史性能分析

At Grab, we’ve been working to perfect our Spark observability tools. Our initial solution, Iris, was developed to provide a custom, in-depth observability tool for Spark jobs. As described in our previous blog post, Iris collects and analyses metrics and metadata at the job level, providing insights into resource usage, performance, and query patterns across our Spark clusters.

在 Grab，我们一直在努力完善我们的 Spark 可观察性工具。我们的初始解决方案 Iris 是为了提供一个自定义的、深入的 Spark 作业可观察性工具。正如我们之前的博客文章中所描述的，Iris 在作业级别收集和分析指标和元数据，提供有关资源使用、性能和查询模式的洞察，涵盖我们的 Spark 集群。

Iris addresses a critical gap in Spark observability by providing real-time performance metrics at the Spark application level. Unlike traditional monitoring tools that typically provide metrics only at the EC2 instance level, Iris dives deeper into the Spark ecosystem. It bridges the observability gap by making Spark metrics accessible through a tabular dataset, enabling real-time monitoring and historical analysis. This approach eliminates the need to parse complex Spark event log JSON files, which users are often unable to access when they need immediate insights. Iris empowers users with on-demand access to comprehensive Spark performance data, facilitating quicker decision-making and more efficient resource management.

Iris 解决了 Spark 可观察性中的一个关键缺口，通过提供 Spark 应用级别的实时性能指标。与传统监控工具通常仅在 EC2 实例级别提供指标不同，Iris 更深入地探讨了 Spark 生态系统。它通过使 Spark 指标可通过表格数据集访问，弥补了可观察性缺口，从而实现实时监控和历史分析。这种方法消除了解析复杂的 Spark 事件日志 JSON 文件的需要，而用户在需要即时洞察时往往无法访问这些文件。Iris 使用户能够按需访问全面的 Spark 性能数据，从而促进更快的决策和更高效的资源管理。

Iris served us well, offering basic dashboards and charts that helped our teams understand trends, discover issues, and debug their Spark jobs. However, as our needs evolved and usage grew, we began to encounter limitations:

Iris 为我们提供了良好的服务，提供了基本的仪表板和图表，帮助我们的团队理解趋势、发现问题并调试他们的 Spark 作业。然而，随着我们的需求发展和使用量增加，我们开始遇到一些限制：

Fragmented user experience and access control: Observability data is split between Grafana (real-time) and Superset (historical), forcing users to switch platforms for a complete view. The complex Gra...