Ensuring Data Consistency for Continuously-modified Data (ft. Topological Sorting)

Ensuring Data Consistency for Continuously-modified Data (ft. Topological Sorting)

摘要

At Mixpanel, data trust and reliability are of utmost importance. In order to safely roll out code and infrastructure changes, we maintain two independent zones of ingestion infrastructure per region. However, having two zones is not sufficient to ensure safety and we also need a way to detect changes that cause data corruption. Previously, we have written about ensuring data consistency across replicas for real-time data. This post is a follow-up that covers the work the storage infrastructure team at Mixpanel has done to guarantee that all write and modification operations on the data we store are checked for cross-replica consistency.