Why we don't use the term DevOps2

1. Why we don't use the term DevOps 2019-06-13 HENNING JACOBS @try_except_ MOSTAFA NAGEEB @MostafaNageeb

2. ZALANDO AT A GLANCE ~ 5.4 billion EUR > 250 million revenue 2018 2 > 15.000 > 79% employees in Europe of visits via mobile devices visits per month > 300.000 > 26 product choices million ~ 2.000 17 brands countries active customers

3. THE BIRTH OF DEVOPS - 2009 “10+ Deploys per Day: Dev and Ops Cooperation at Flickr.” DevOps Days Belgium 3 https://devops.com/the-origins-of-devops-whats-in-a-name/

4. slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr

5. slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr

6. slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr

7. THE PHOENIX PROJECT - 2013 "The Three Ways" 1. Systems Thinking 2. Amplify Feedback Loops 3. Culture of Continual Experimentation And Learning 7

8. WHAT HAPPENED NEXT 8

9. CARGO CULTING 9

10. DEVOPS IS … if every person uses the same tool for the same job … codified knowledge - everybody contributing their part to common automation … if all people have the same privileges in their tooling … if human error is equally possible for Dev and Ops … replacing people interfaces by automated decisions and processes … a result 10 bit.ly/5pdops

11. DevOps is normal 11

12. 12

13. WHAT HAPPENED NEXT 13

14. WHAT HAPPENED NEXT 14

15. DEV + OPS ⇒ DEVOPS Devs "DevOps" Team Ops 15

16. DEV + OPS ⇒ ENGINEERING Devs "DevOps" Team Ops 16 Engineering Team

17. PRODUCT DEVELOPMENT TEAMS "Great products emerge from empowered, fully cross-functional teams." - John Cutler, 2019 17 https://amplitude.com/blog/on-being-product-led

18. DEVOPS IS It is a culture, in which people work together to improve the product delivery cycle. 18

19. DEVOPS ANTI-PATTERNS Dev and Ops Silos DevOps Team Silo Dev Don't Need Ops DevOps as Tools Team Rebranded SysAdmin Ops Embedded in Dev Team 19 web.devopstopologies.com

20. 20

21. TEAM INTERACTION MODES • Collaboration: 2 teams working together • X-as-a-Service: 1 provides, 1 consumes • Facilitating: 1 team helps another 21 Product teams need a family too - DevOpsDays Portugal

22. A BRIEF HISTORY OF ZALANDO TECH 22

23. 2010 "Sysop-Test" "QA-Test" 23

24. DEVOPS IS ABOUT SKIN IN THE GAME Absence of skin in the game: People who are isolated from the impacts of their decisions do not learn. They remain captive to their erroneous ideas about how the world works. 24

25. 2013: SELF SERVICE 25

26. 2015: RADICAL AGILITY DOCKER DEPLOY SSH ACCESS AUDIT REPORTS STUPS AWS 26 FULL AWS ACCESS Teams have admin access & full responsibility

27. 2015: ISOLATED AWS ACCOUNTS Internet *.abc.example.org ELB Team ABC EC2 27 *.xyz.example.org ELB Team XYZ EC2

28. 2019 > 200 development teams > 1100 developers 28 Platform

29. YOU BUILD IT, YOU RUN IT The traditional model is that you take your software to the wall that separates development and operations, and throw it over and then forget about it. Not at Amazon. You build it, you run it. This brings developers into contact with the day-to-day operation of their software. It also brings them into day-to-day contact with the customer. - A Conversation with Werner Vogels, ACM Queue, 2006 29

30. ON-CALL: YOU OWN IT, YOU RUN IT When things are broken, we want people with the best context trying to fix things. - Blake Scrivener, Netflix SRE Manager 30

31. "SKIN IN THE GAME" 31

32. DEVELOPER JOURNEY Consistent story that models all aspects of SW dev 32

33. Developer Journey 33

34. Correctness Compliance GDPR Security Cost Efficiency 24x7 On Call Governance Resilience Capacity ... 34 Developer Journey

35. DEVELOPER PRODUCTIVITY Setup Code Build Test Deploy Cloud Native Application Runtime 35 Operate

36. CLOUD NATIVE .. uses an open source software stack to deploy applications as microservices, packaging each part into its own container, and dynamically orchestrating those containers to optimize resource utilization. Cloud native technologies enable software developers to build great products faster. - https://www.cncf.io/ 36

37. CONTAINERS END-TO-END Setup Code Build Test Deploy Cloud Native Application Runtime 37 Operate

38. CONTAINERS 38

39. CONTAINERS 39

40.

41. PLAN & SETUP 41

42. Plan Stories Rules of Play Tech Radar 42

43.

44. Setup Application Bootstrapping 44

45.

46.

47. BUILD & TEST 47

48. CONTINUOUS DELIVERY PLATFORM: BUILD push Git code 48 CDP

49.

50. DEPLOY 50

51. Kubernetes Deploy 51

52. DEPLOYMENT CONFIGURATION ├── deploy/apply │ ├── deployment.yaml │ ├── credentials.yaml # Zalando IAM │ ├── ingress.yaml │ └── service.yaml └── delivery.yaml # Zalando CI/CD 52

53. INGRESS.YAML kind: Ingress metadata: name: "..." spec: rules: # DNS name your application should be exposed on - host: "myapp.foo.example.org" http: paths: - backend: serviceName: "myapp" servicePort: 80 53

54. TEMPLATING: MUSTACHE kind: Ingress metadata: name: "..." spec: rules: # DNS name your application should be exposed on - host: "{{{APPLICATION}}}.example.org" http: paths: - backend: serviceName: "{{{APPLICATION}}}" servicePort: 80 54

55. CONTINUOUS DELIVERY PLATFORM 55

56. CDP: DEPLOY "glorified kubectl apply" 56

57. CDP: OPTIONAL APPROVAL 57

58. STACKSET: TRAFFIC SWITCHING 58 github.com/zalando-incubator/stackset-controller

59. STACKSET CRD kind: StackSet ... spec: ingress: hosts: ["foo.example.org"] backendPort: 8080 stackLifecycle: scaledownTTLSeconds: 1800 limit: 5 stackTemplate: spec: podTemplate: ... 59 github.com/zalando-incubator/stackset-controller

60. TRAFFIC SWITCHING STEPS IN CDP 60 github.com/zalando-incubator/stackset-controller

61. EMERGENCY ACCESS SERVICE Emergency access by referencing Incident zkubectl cluster-access request \ --emergency -i INC REASON Privileged production access via 4-eyes zkubectl cluster-access request REASON zkubectl cluster-access approve USERNAME 61

62. INTEGRATIONS 62

63. CLOUD FORMATION VIA CI/CD "Infrastructure as Code" ├── deploy/apply │ ├── deployment.yaml │ ├── cf-iam-role.yaml │ ├── cf-rds.yaml │ ├── kube-ingress.yaml │ ├── kube-secret.yaml │ └── kube-service.yaml └── delivery.yaml 63 # Kubernetes # AWS IAM Role # AWS RDS Database # CI/CD config

64. ZALANDO IAM/OAUTH VIA CRD kind: PlatformCredentialsSet .. Extensible Kubernetes spec: application: my-app tokens: read-only: privileges: - com.zalando::foobar.read clients: employee: grant: authorization-code realm: users redirectUri: https://example.org/auth/callback 64 API

65. POSTGRES OPERATOR Application to manage PostgreSQL clusters on Kubernetes >700 clusters running on Kubernetes 65 github.com/zalando/postgres-operator

66. Elasticsearch 2.500 vCPUs 1 TB RAM Elasticsearch in Kubernetes github.com/zalando-incubator/es-operator/

67. SUMMARY • Application Bootstrapping • Git as source of truth and UI • 4-eyes principle for master/production • Extensible Kubernetes API as primary interface • OAuth/IAM credentials • PostgreSQL • CloudFormation for proprietary AWS services 67

68. Platform as a Product 68

69. “.. means establishing empathy with internal consumers (read: developers) and collaborating with them on the design. Platform product managers establish roadmaps and ensure the platform delivers value to the business and enhances the developer experience.” - ThoughtWorks Technology Radar

70. PRODUCT MINDSET ... means we are developing all our products for the best customer experience and biggest business impact while all actions are aligned with our strategy. 70

71. KNOW YOUR AUDIENCE 71

72. TALK TO USERS ● Be Prepared. ● Talk to both users and NON users. ● Ask about problems, not solutions. ● Open ended questions. 72

73. IDENTIFY COMMON PROBLEMS "Documentation is hard to find" "Documentation is not comprehensive enough" "Remove unnecessary complexity and obstacles." "Get the documentation up to date and prepare use cases" "More and more clear documentation" "More detailed docs, example repos with more complicated deployments." 73

74. TRANSPARENTLY COMMUNICATE WHAT YOU ARE SOLVING 74

75. LET THEM INFLUENCE YOUR PLANS 75

76. ENGAGE CUSTOMERS EARLY ON We are working on speeding up builds by providing basic overlays for the most used languages/frameworks so you don't have to download the internet every time you are building your app. This is the approach and the documentation (not live yet), please add your comments to guide us in the right direction. 76

77. SUPPORT IS A FEATURE 77

78. DOCUMENTATION • Restructure following www.divio.com/en/blog/documentation/ • Concepts • How Tos • Tutorials • Reference • Global Search • Weekly Health Check: Support → Documentation 78

79.

80. COMMUNICATION Guidelines • Target announcements • Use the right tags • Timing 80

81. NEWSLETTER "You can now.." • You can now upload your Scala code coverage report to SonarQube "This enables you to.." • This enables you to browse your code quality results and track it over time 81

82. “Useful information, good level of details and pleasant to read. It's one of the few newsletters that I took time to read entirely :)” - a reader, July 2018 82

83. EMPOWER YOUR TEAM 83

84. WITH GREATER POWER COMES GREATER RESPONSIBILITY 84

85. DELIVERY PERFORMANCE METRICS • Lead Time • Release Frequency • Time to Restore Service • Change Fail Rate 85 srcco.de/posts/accelerate-software-delivery-performance.html

86. DELIVERY PERFORMANCE METRICS • Lead Time • Commit to Prod • Release Frequency • Deploys/week/dev • MTRS from incidents • Time to Restore Service • Change Fail Rate 86 • N/A

87.

88. DEVELOPER SATISFACTION 88

89. INSIGHTS THAT GUIDE THE PRODUCT “Kubernetes/CDP deploys are too slow - ours takes over 10 minutes, mainly due to having to download SBT dependencies every time.” 89

90. INSIGHTS THAT GUIDE THE PRODUCT “CDP should somehow cache builds or something like that. The current approach forces everyone to re-invent the cache wheel for their own projects - or face a crazy slow build.” 90

91. 91

92. TESTIMONIALS “So, thank you, Team Automata, for listening to our community, taking our upvotes in consideration when developing new solutions and building every day 'the first CI that doesn't suck'.” - a user, October 2018 92

93. TESTIMONIALS “Thank you very much for improving the color scheme of the UI. It really helps me with my color blindness. It used to be hard to distinguish between statuses previously.” - a user, February 2019 93

94. 94

95. Wrap Up 95

96. DEVOPS IS … if every person uses the same tool for the same job … codified knowledge - everybody contributing their part to common automation … if all people have the same privileges in their tooling … if human error is equally possible for Dev and Ops … replacing people interfaces by automated decisions and processes … a result 96 bit.ly/5pdops

97. DEVOPS IS … if every person uses the same tool for the same job … codified knowledge - everybody contributing their part to common automation … if all people have the same privileges in their tooling … if human error is equally possible for Dev and Ops … replacing people interfaces by automated decisions and processes … a result 97 bit.ly/5pdops

98. DEVOPS IS … if every person uses the same tool for the same job … codified knowledge - everybody contributing their part to common automation … if all people have the same privileges in their tooling … if human error is equally possible for Dev and Ops … replacing people interfaces by automated decisions and processes … a result 98 bit.ly/5pdops

99. DEVOPS IS … if every person uses the same tool for the same job … codified knowledge - everybody contributing their part to common automation … if all people have the same privileges in their tooling … if human error is equally possible for Dev and Ops … replacing people interfaces by automated decisions and processes … a result 99 bit.ly/5pdops

100. DEVOPS IS … if every person uses the same tool for the same job … codified knowledge - everybody contributing their part to common automation … if all people have the same privileges in their tooling … if human error is equally possible for Dev and Ops … replacing people interfaces by automated decisions and processes … a result 100 bit.ly/5pdops

101. DEVOPS IS … if every person uses the same tool for the same job … codified knowledge - everybody contributing their part to common automation … if all people have the same privileges in their tooling … if human error is equally possible for Dev and Ops … replacing people interfaces by automated decisions and processes … a result 101 bit.ly/5pdops

102. DevOps is normal 102

103. DevOps is normal 103

104. You build it, you run it is normal 104

105. WAIT A SECOND.. What about my job title? 105

106. "DEVOPS ENGINEER" AS A JOB TITLE 106

107. T-SHAPED ENGINEER "Generalizing specialist" Technical Breadth of Knowledge 107

108. DevOps Engineer 108 Software Engineer / Software Developer (T-Shaped)

109. ON THE DEVOPS JOB TITLE "Everyone’s a devops engineer these days. I’m a devops engineer, my wife’s a devops engineer, even my dog’s a devops engineer" - James Betteley, 2014 109 devopsnet.com/2014/08/13/the-devops-team-myth/

110. 110

111. OPEN SOURCE & MORE Kubernetes on AWS github.com/zalando-incubator/kubernetes-on-aws Skipper HTTP Router & Ingress controller github.com/zalando/skipper External DNS github.com/kubernetes-incubator/external-dns Postgres Operator github.com/zalando-incubator/postgres-operator More Zalando Tech Talks github.com/zalando/public-presentations 111

112. QUESTIONS? HENNING JACOBS @try_except_ MOSTAFA NAGEEB @MostafaNageeb Illustrations by @01k