In the first two posts about the Duplo initiative, we described why we decided to revamp our mobile codebases, the initial phase to clean up tech debt, and our efforts to modularize our iOS and Android codebases (post 1, post 2). In this final post, we will discuss the last theme of the Duplo initiative, Modernization, and look at the overall results and impact on developers.

Slack experienced a major incident on February 22 this year, during which time many users were unable to connect to Slack, including the author — which certainly made my role as Incident Commander more challenging!

This post describes the path we have taken to minimize the number of flaky tests through an approach of automated test failure detection and suppression. This is not a new problem that we are trying to solve; many companies have published articles on systems created for handling flaky tests. This article outlines how test flakiness is an increasing problem at scale and how we got it under control at Slack.

A story of evolving socio-technical workflows that increased developer velocity and redefined confident testing and deploy workflows at Slack.

On September 30th 2021, Slack had an outage that impacted less than 1% of our online user base, and lasted for 24 hours. This outage was the result of our attempt to enable DNSSEC — an extension intended to secure the DNS protocol, required for FedRAMP Moderate — but which ultimately led to a series of unfortunate events.

The internet relies very heavily on the Domain Name System (DNS) protocol. DNS is like a phone book for the entire internet. Web sites are accessed through domain names, but web browsers interact using IP addresses. DNS translates domain names to IP addresses, so that browsers can load the sites you need. Refer to ‘What is DNS?’ by Cloudflare to read more about how DNS works and all the necessary steps to do a domain name lookup.

At Slack, proactively securing our systems is a top priority. One way we achieve this is by automating the detection of security issues with static code analysis, which are tools that inspect programs without executing them. They’re often used with security-based rules to automate identification of vulnerabilities and insecure programming practices, which frees up more bandwidth for security engineers. For us, expanding our static code analysis program became critical as we looked to grow into the public sector, where there are rising demands to show our feature work is secure and to meet security certification requirements. We view static code analysis as guardrails; it prevents the worst kinds of security vulnerabilities from joining our codebases. As a result, static code analysis has been top of mind for the security team at Slack for the past three quarters and remains one of the major focuses for next quarter.

Our codebase is largely written in Hack. While Hack comes from work that Facebook performed to develop a typed version of PHP, it is a separate language and there are no static analysis tools broadly available for it. Given that over 5 million lines of code at Slack are written in Hack, how can we ensure it remains secure at scale?

A deep dive on how we crafted an order of magnitude change in our spend (10x reduction compared to baseline growth) over the last two years with iterative understanding and changes in Slack’s Continuous Integration (CI) infrastructure.

What are you really selling?

We recently rolled out support for Conversation Bubbles for DMs and Group DMs on Android 11.

In case you’re not familiar with Conversation Bubbles, take a look at the video below. Basically, they are a way to pop out a conversation from a notification into a Bubble that will draw over other apps, making multitasking while chatting easy and comfortable.

This post will describe how we built support for these bubbles, the challenges we faced along the way, and how we overcame them. Some details were simplified where it made sense to, but stayed relatively close to our actual implementation overall.

Reinventing how the world does work inevitably creates a lot of data. Each year, Slack’s scale has increased and the volume of data ingested and stored has kept pace. To make it possible to understand relationships within our data, we’ve invested heavily in an automated data lineage framework. This facilitates producer/consumer coordination, improves risk mitigation, impact analysis, and better execution of data programs here at Slack.

More than five years ago, we launched the Slack Platform, giving developers an easy way to build apps in Slack and publish them in our App Directory. Today, millions of users bring their work into…

A deep-dive in how we built an eventual consistency data model to predict Slack Connect invites at Slack by smart classification system for email domains.

We use Dagger heavily in the Slack Android app for compile-time dependency injection. It’s powerful, flexible, supports basic Kotlin idioms, and allows for advanced dependency injection patterns with less boilerplate. It’s not without its sharp edges though. It slows down our builds with kapt, has a steep learning curve, and can often be tedious to write out its module+component plumbing.

Anvil augments Dagger’s own boilerplate. Factory generation enables us to remove kapt from much of our project, @ContributesTo and @ContributesBinding allow us to automatically wire dependencies without modules, and @MergeComponent allows us to automatically wire together component interfaces. Factory generation alone recently helped us reduce our incremental build times by as much as 25%!

Anvil’s core features are fantastic, but they are not (nor pretend to be!) turnkey solutions for every use case, though they cover the common ones. Every codebase has its own patterns and nuances, so Anvil 2.3.0 introduced a new compiler-api artifact that allows us to extend Anvil’s own code generation to suit those remaining needs. In this post we’ll detail a little of how it works with our own Activity injection pattern as an example.

Troubleshooting plugin upgrades by going down the rabbit hole of debugging Jenkins.

控制哪些用户能够采取哪些行动并不是一件简单的事情。在Slack中建立这一点，一直是一个有趣的挑战。在大型企业组织中，我们提供给客户的标准角色类型过于宽泛，委托一个通用的管理员角色可能会授予某人过多的权力--如果你只想让一个特定的用户能够管理特定的频道，怎么办？当你让他们成为管理员时，他们能够执行超出预期目的范围的各种行动，并且可以查看仪表盘，看到与管理渠道无关的信息。我们需要建立一个更加灵活并允许细化权限的系统。我们想分享一下我们在角色方面面临的问题，我们实施的解决方案，以及我们对未来的计划。

复杂的系统很难进行大规模的推理；我们往往不能准确地推断系统的行为和性能，所以我们需要根据经验得出这些数据。我们使用负载测试来做到这一点：找到我们系统的极限，并在一个受控环境中大规模地剔除错误。Slack是一个相当复杂的系统--无论你是为成千上万的成员触发一个工作流，还是将一个文件上传到一个线程中，所有的东西都是相互关联的 为我们的用户提供发送消息并使其立即出现在可能是数以百万计的客户端的体验所需的技术，对于大规模构建和测试是非常具有挑战性的。为了充分地测试我们的系统，我们需要建立一个既现实又具有成本效益的工具，以反映实际的用户流量和行为。