APEC- How to ensure your application is operated properly?

1. APEC: How to ensure your application is operated properly? DevOps Finland meetup 10.10.2023 Uri Savelchev

2. DevOps Finland meetup Agenda APEC - How to ensure your application is operated properly? 1. The problem and a possible solution 2. Goals we want to achieve 3. What’s inside the checklist? 4. APEC History in Zalando 5. Learnings 6. Q & A

3. DevOps Finland meetup APEC - How to ensure your application is operated properly?

4. DevOps Finland meetup APEC - How to ensure your application is operated properly? Zalando Tech ● ● ● ● 2500+ software engineers 300+ engineering teams 200+ K8s clusters 3500 active applications How to maintain operation excellence?

5. DevOps Finland meetup APEC - How to ensure your application is operated properly?

6. DevOps Finland meetup APEC - How to ensure your application is operated properly? Solution: a checklist? Application Production Engineering Checklist (APEC) is a set of checks that teams should have in place for all applications. The items on APEC have been identiﬁed from system reviews, post-mortem documents, existing production systems, and Zalando requirements. Together the items on the checklist form a shared understanding of what production readiness is for teams shipping to production. Outcome is a status (GREEN, AMBER, RED) and a list of what’s missing

7. DevOps Finland meetup APEC - How to ensure your application is operated properly?

8. DevOps Finland meetup APEC - How to ensure your application is operated properly? Operational excellence metrics ● per team ● per department ● per organization

9. DevOps Finland meetup APEC - How to ensure your application is operated properly? A tool to ﬁnd gaps in operations 1. The APEC status 2. A summary of what’s missing 3. The review process itself: answering questions helps you to look at your application from a diﬀerent angle.

10. DevOps Finland meetup APEC - How to ensure your application is operated properly?

11. DevOps Finland meetup APEC - How to ensure your application is operated properly? APEC is NOT about compliance ● ● ● ● Security, Copyright, GDPR, … need separate tools and processes.

12. DevOps Finland meetup What’s inside? APEC - How to ensure your application is operated properly?

13. DevOps Finland meetup APEC - How to ensure your application is operated properly? APEC Sections (non-complete) ● ● ● ● ● ● ● ● ● ● Application registry information Technical stack (and if it is included in the Tech Radar) Tier evaluation Naming and Code Of Conduct Data Grade API speciﬁcation Service Level Objectives Monitoring and tracing Playbooks Production Readiness Review

14. DevOps Finland meetup APEC - How to ensure your application is operated properly? https://opensource.zalando.com/tech-radar/

15. DevOps Finland meetup APEC - How to ensure your application is operated properly? Tier evaluation A way to classify the apps by their business criticality

16. DevOps Finland meetup APEC - How to ensure your application is operated properly?

17. DevOps Finland meetup APEC - How to ensure your application is operated properly? Service Level Objectives API Speciﬁcation is a contract, SLOs are an important appendix to the contract.

18. DevOps Finland meetup APEC - How to ensure your application is operated properly? Monitoring, tracing, playbooks ● ● ● Every application running in production must have: ○ A dashboard with metrics ○ A reasonable set of alerts Every application running in production should be instrumented with tracing (OpenTracing / OpenTelemetry) Every application being supported 24x7 must have reliable playbooks

19. DevOps Finland meetup APEC - How to ensure your application is operated properly?

20. DevOps Finland meetup APEC History APEC - How to ensure your application is operated properly?

21. DevOps Finland meetup APEC - How to ensure your application is operated properly? APEC History 1. 2. 3. A plain document ● Proof of Concept ● Small set of teams and application to try the approach A very long form ● Google-form style ● Painful to ﬁll out ● A typical review took 20-30 minutes for every app More compact form with expandable sections ● Autodiscovery (API specs, tracing, monitoring, playbooks,...) ● Memoization — no need to repeat the same answer twice ● A repeated review takes 1-5 minutes usually

22. DevOps Finland meetup APEC - How to ensure your application is operated properly?

23. DevOps Finland meetup APEC - How to ensure your application is operated properly? Backstage integration

24. DevOps Finland meetup APEC - How to ensure your application is operated properly?

25. DevOps Finland meetup APEC - How to ensure your application is operated properly? Learnings ● ➔ Engineers: “ﬁlling forms is a bureaucracy” The process should be as fast/painless as possible ❏ Memoization (DRY principle) ❏ Autodetect and autoﬁll everything you can ● ➔ There are no rules without exceptions Let people to use them ● Comments are important! (even though they aren’t processed automatically) ● The “color” of the APEC should be deﬁned by most critical things

26. DevOps Finland meetup APEC - How to ensure your application is operated properly? Learn more ● A nice presentation of Heinrich Hartmann at DevOpsCon Berlin in 2022 ● Henning Jacobs’ blog post ● Zalando engineering blog

27. DevOps Finland meetup APEC - How to ensure your application is operated properly?