Uber在日志记录方面的现代化改进（第二部分）

This is the second installment in our series detailing the modernization of Uber’s logging infrastructure using CLP. In our last blog, we described how we split CLP’s compression algorithm into two phases: (1) distributed, row-based streaming compression that produces a compressed intermediate representation on each container or host, and (2) full columnar compression into more efficient archives. We developed a Log4j appender that uses the streaming compression and integrated it into our Spark platform. This resulted in a substantial compression ratio (169:1), which contributed to the resolution of the SSD burnouts caused by writing large amounts of log data while allowing us to extend our log retention period by 10x. 

这是我们系列文章的第二部分，详细介绍了Uber使用CLP对其日志基础设施进行现代化的过程。在我们的上一篇博客中，我们描述了如何将CLP的压缩算法分为两个阶段：（1）分布式、基于行的流式压缩，在每个容器或主机上生成压缩的中间表示，以及（2）完全列式压缩为更高效的存档。我们开发了一个使用流式压缩的Log4j appender，并将其集成到我们的Spark平台中。这导致了一个巨大的压缩比（169:1），从而解决了由于写入大量日志数据而导致的SSD烧毁问题，同时允许我们将日志保留期延长10倍。

In collaboration with CLP’s developers, we have further developed an innovative end-to-end system to manage unstructured logs across Uber’s various data and ML platforms. Many log management systems, including the one described in a previous blog, tend to focus solely on search and aggregation, which are useful for developers to bootstrap a debugging session (e.g. locating an error event or detecting anomalies occurring at a certain point in time). However, to understand how and why the error occurred, developers need to further view the sequence of logs leading up to the event. Thus, search is only useful if integrated with convenient log-file viewing, a feature often overlooked by existing tools. Our new system includes advanced log viewing and programmable analytics directly on the compressed logs, unlocking the full potential of text logs using CLP.  Figure 1 below shows our current end-to-end deployment. Most of the core tools and basic features described in this blog are now open source.

与CLP的开发人员合作，我们进一步开发了一种创新的端到端系统，用于管理Uber各种数据和ML平台上的非结构化日志。许多日志管...