知鸦日报2023-10-06

At Mixpanel, our proprietary multi-tenant columnar store database, known as ARB, serves as the backbone of our analytics platform. Following the ingestion of customer events, these data sets undergo a compaction process to create columnar files, which are then uploaded to Google Cloud Storage (GCS). In this blog post, we will delve into the recent updates we’ve implemented in this columnar file format. These updates enhance our query servers’ ability to accommodate more ARB files in their local SSDs, all without actually reducing file sizes. In some instances, these changes may even result in larger files.

High-quality data is necessary for the success of every data-driven company. It enables everything from reliable business logic to insightful decision-making and robust machine learning modeling. It is now the norm for tech companies to have a well-developed data platform. This makes it easy for engineers to generate, transform, store, and analyze data at the petabyte scale. As such, we have reached a point where the quantity of data is no longer a boundary. Yet this has come at the cost of quality.

In this post we will define data quality at a high-level and explore our motivation to achieve better data quality. We will then introduce our in-house product, Verity, and showcase how it serves as a central platform for ensuring data quality in our Hive Data Warehouse. In future posts we will discuss how Verity addresses data quality elsewhere in our data platform.

尽管软件开发一直致力于追求高效、可读性强、易于维护的特性，但这些特性却像是一个不可能三角，相互交织，此消彼长。就像底层语言（如汇编和C语言）能够保持高效的运行性能，但在可读性和维护性方面却存在短板和劣势；而高级语言（如Java和Python）在可读性和可维护性方面表现出色，但在执行效率方面却存在不足。