调试Ubuntu18升级中的PininfoService死锁。第1部分，共2部分

Author: Kangnan Li | Software Engineer, Key Value Systems

作者：李康南李康南 | 软件工程师, 关键价值系统

on the left, an octopus all tangled up in its own tentacles. On the right, the octopus is untangled and happy. There is an arrow between the two pointing to the untangled octopus. The arrow is labeled “unlock”.

Unblock deadlock for PininfoService

为PininfoService解开死锁

This is part 1 of a two-part blog series.

这是由两部分组成的博客系列的第一部分。

Reading both parts of this series will give you insight into some debugging techniques we are using in the Pinterest Engineering Key Value Systems team (a team derived from the previous Serving System). Related projects owned by this team can be seen in blogs and presentations on Terrapin, Rocksplicator (1 and 2), Aperture and Realpin.

阅读这个系列的两部分内容，可以让你了解我们在Pinterest工程关键价值系统团队（由之前的服务系统衍生出来的团队）中使用的一些调试技术。这个团队拥有的相关项目可以在Terrapin、Rocksplicator（1和2）、Aperture和Realpin的博客和演讲中看到。

In this series, we divide the contents as below:

在这个系列中，我们把内容划分如下。

Part 1

第一部分

How to set up a test/canary environment
如何建立一个测试/罐头环境
How to design tests with control variables
如何设计带有控制变量的测试
How to design tests for root/leaf (ie. routing/persistence layer) separation
如何设计根/叶（即路由/持久层）分离的测试

Part 2

第二部分

How to generate a heap dump
如何生成一个堆转储
How to use GDB for debugging a running process
如何使用GDB对运行中的进程进行调试

Pinterest is a platform with more than 400M monthly active users, which means engineers have a chance to work on big data problems at scale. PininfoService (PIS) is one of Pinterest’s many backend services.

Pinterest是一个拥有超过4亿月活跃用户的平台，这意味着工程师有机会大规模地处理大数据问题。PininfoService（PIS）是Pinterest的众多后台服务之一。

PininfoService is a rootless read-only cache service serving >10M batch requests per second per cluster to support Pinterest Homefeed, Related Pins, and others. It fetches customer-defined signals from various source data services (i.e. upstreams) and holds the data with customized time-to-live (TTL).

PininfoService是一个无根的只读缓存服务，为每个集群每秒超过1000万的批量请求提供服务，以支持Pinterest Homefeed、Related Pins及其他。它从各种源数据服务（即上游）获取客户定义的信号，并以自定义的生存时间（TTL）保存数据。

Note: In this article, we use root, leaf, rootless to classify systems architecture. Root, also called routing layer, is responsible for routing requests and aggregating responses. Leaf (i.e. persistence layer) is responsible for seri...