Why we don't use the term DevOps1

1. Why we don't use the term DevOps DESTINATION AARHUS 2019-06-04 HENNING JACOBS @try_except_

2. EUROPE’S LEADING ONLINE FASHION PLATFORM 2

3. ZALANDO AT A GLANCE ~ 5.4 billion EUR > 250 million revenue 2018 3 > 15.000 > 79% employees in Europe of visits via mobile devices visits per month > 300.000 > 26 product choices million ~ 2.000 17 brands countries active customers

4. THE BIRTH OF DEVOPS - 2009 “10+ Deploys per Day: Dev and Ops Cooperation at Flickr.” DevOps Days Belgium 4 https://devops.com/the-origins-of-devops-whats-in-a-name/

5. slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr

6. slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr

7. slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr

8. THE PHOENIX PROJECT - 2013 "The Three Ways" 1. Systems Thinking 2. Amplify Feedback Loops 3. Culture of Continual Experimentation And Learning 8

9. WHAT HAPPENED NEXT 9

10. CARGO CULTING 10

11. DEVOPS IS … if every person uses the same tool for the same job … codified knowledge - everybody contributing their part to common automation … if all people have the same privileges in their tooling … if human error is equally possible for Dev and Ops … replacing people interfaces by automated decisions and processes … a result 11 bit.ly/5pdops

12. DevOps is normal 12

13. 13

14. WHAT HAPPENED NEXT 14

15. WHAT HAPPENED NEXT 15

16. DEV + OPS ⇒ DEVOPS Devs "DevOps" Team Ops 16

17. DEV + OPS ⇒ ENGINEERING Devs "DevOps" Team Ops 17 Engineering Team

18. PRODUCT DEVELOPMENT TEAMS "Great products emerge from empowered, fully cross-functional teams." - John Cutler, 2019 18 https://amplitude.com/blog/on-being-product-led

19. DEVOPS IS It is a culture, in which people work together to improve the product delivery cycle. 19

20. DEVOPS ANTI-PATTERNS Dev and Ops Silos DevOps Team Silo Dev Don't Need Ops DevOps as Tools Team Rebranded SysAdmin Ops Embedded in Dev Team 20 web.devopstopologies.com

21. 21

22. TEAM INTERACTION MODES • Collaboration: 2 teams working together • X-as-a-Service: 1 provides, 1 consumes • Facilitating: 1 team helps another 22 Product teams need a family too - DevOpsDays Portugal

23. A BRIEF HISTORY OF ZALANDO TECH 23

24. 2010 "Sysop-Test" "QA-Test" 24

25. DEVOPS IS ABOUT SKIN IN THE GAME Absence of skin in the game: People who are isolated from the impacts of their decisions do not learn. They remain captive to their erroneous ideas about how the world works. 25

26. 2013: SELF SERVICE 26

27. 2015: RADICAL AGILITY DOCKER DEPLOY SSH ACCESS AUDIT REPORTS STUPS AWS 27 FULL AWS ACCESS Teams have admin access & full responsibility

28. 2015: ISOLATED AWS ACCOUNTS Internet *.abc.example.org ELB Team ABC EC2 28 *.xyz.example.org ELB Team XYZ EC2

29. 2019 > 200 development teams > 1100 developers 29 Platform

30. YOU BUILD IT, YOU RUN IT The traditional model is that you take your software to the wall that separates development and operations, and throw it over and then forget about it. Not at Amazon. You build it, you run it. This brings developers into contact with the day-to-day operation of their software. It also brings them into day-to-day contact with the customer. - A Conversation with Werner Vogels, ACM Queue, 2006 30

31. ON-CALL: YOU OWN IT, YOU RUN IT When things are broken, we want people with the best context trying to fix things. - Blake Scrivener, Netflix SRE Manager 31

32. "SKIN IN THE GAME" 32

33. DEVELOPER JOURNEY Consistent story that models all aspects of SW dev 33

34. Developer Journey 34

35. Correctness Compliance GDPR Security Cost Efficiency 24x7 On Call Governance Resilience Capacity ... 35 Developer Journey

36. DEVELOPER PRODUCTIVITY Setup Code Build Test Deploy Cloud Native Application Runtime 36 Operate

37. CLOUD NATIVE .. uses an open source software stack to deploy applications as microservices, packaging each part into its own container, and dynamically orchestrating those containers to optimize resource utilization. Cloud native technologies enable software developers to build great products faster. - https://www.cncf.io/ 37

38. CONTAINERS END-TO-END Setup Code Build Test Deploy Cloud Native Application Runtime 38 Operate

39. CONTAINERS 39

40. CONTAINERS 40

41. PLATFORM AS A PRODUCT "A good platform is treated as a product (reliable, usable, fit for purpose)." - Manuel Pais, 2019 41 Product teams need a family too - DevOpsDays Portugal

42.

43. PLAN & SETUP 43

44. Plan Stories Rules of Play Tech Radar 44

45.

46. Setup Application Bootstrapping 46

47.

48.

49. BUILD & TEST 49

50. CONTINUOUS DELIVERY PLATFORM: BUILD push Git code 50 CDP

51.

52. DEPLOY 52

53. Kubernetes Deploy 53

54. DEPLOYMENT CONFIGURATION ├── deploy/apply │ ├── deployment.yaml │ ├── credentials.yaml # Zalando IAM │ ├── ingress.yaml │ └── service.yaml └── delivery.yaml # Zalando CI/CD 54

55. INGRESS.YAML kind: Ingress metadata: name: "..." spec: rules: # DNS name your application should be exposed on - host: "myapp.foo.example.org" http: paths: - backend: serviceName: "myapp" servicePort: 80 55

56. TEMPLATING: MUSTACHE kind: Ingress metadata: name: "..." spec: rules: # DNS name your application should be exposed on - host: "{{{APPLICATION}}}.example.org" http: paths: - backend: serviceName: "{{{APPLICATION}}}" servicePort: 80 56

57. CONTINUOUS DELIVERY PLATFORM 57

58. CDP: DEPLOY "glorified kubectl apply" 58

59. CDP: OPTIONAL APPROVAL 59

60. STACKSET: TRAFFIC SWITCHING 60 github.com/zalando-incubator/stackset-controller

61. STACKSET CRD kind: StackSet ... spec: ingress: hosts: ["foo.example.org"] backendPort: 8080 stackLifecycle: scaledownTTLSeconds: 1800 limit: 5 stackTemplate: spec: podTemplate: ... 61 github.com/zalando-incubator/stackset-controller

62. TRAFFIC SWITCHING STEPS IN CDP 62 github.com/zalando-incubator/stackset-controller

63. EMERGENCY ACCESS SERVICE Emergency access by referencing Incident zkubectl cluster-access request \ --emergency -i INC REASON Privileged production access via 4-eyes zkubectl cluster-access request REASON zkubectl cluster-access approve USERNAME 63

64. INTEGRATIONS 64

65. CLOUD FORMATION VIA CI/CD "Infrastructure as Code" ├── deploy/apply │ ├── deployment.yaml │ ├── cf-iam-role.yaml │ ├── cf-rds.yaml │ ├── kube-ingress.yaml │ ├── kube-secret.yaml │ └── kube-service.yaml └── delivery.yaml 65 # Kubernetes # AWS IAM Role # AWS RDS Database # CI/CD config

66. ZALANDO IAM/OAUTH VIA CRD kind: PlatformCredentialsSet .. Extensible Kubernetes spec: application: my-app tokens: read-only: privileges: - com.zalando::foobar.read clients: employee: grant: authorization-code realm: users redirectUri: https://example.org/auth/callback 66 API

67. POSTGRES OPERATOR Application to manage PostgreSQL clusters on Kubernetes >700 clusters running on Kubernetes 67 github.com/zalando/postgres-operator

68. Elasticsearch 2.500 vCPUs 1 TB RAM Elasticsearch in Kubernetes github.com/zalando-incubator/es-operator/

69. SUMMARY • Application Bootstrapping • Git as source of truth and UI • 4-eyes principle for master/production • Extensible Kubernetes API as primary interface • OAuth/IAM credentials • PostgreSQL • CloudFormation for proprietary AWS services 69

70. DELIVERY PERFORMANCE METRICS • Lead Time • Release Frequency • Time to Restore Service • Change Fail Rate 70 https://srcco.de/posts/accelerate-software-delivery-performance.html

71. CONTAINERS 71 From "Accelerate: The Science of Lean Software and DevOps"

72. DELIVERY PERFORMANCE METRICS 72 • Lead Time ≙ Commit to Prod • Release Frequency ≙ Deploys/week/dev • Time to Restore Service ≙ MTRS from incidents • Change Fail Rate ≙ n/a

73. “.. means establishing empathy with internal consumers (read: developers) and collaborating with them on the design. Platform product managers establish roadmaps and ensure the platform delivers value to the business and enhances the developer experience.” - ThoughtWorks Technology Radar

74. PRODUCT MINDSET ... means we are developing all our products for the best customer experience and biggest business impact while all actions are aligned with our strategy. 74

75.

76. DEVELOPER SATISFACTION 76

77. DOCUMENTATION "Documentation is hard to find" "Documentation is not comprehensive enough" "Remove unnecessary complexity and obstacles." "Get the documentation up to date and prepare use cases" "More and more clear documentation" "More detailed docs, example repos with more complicated deployments." 77

78. DOCUMENTATION • Restructure following www.divio.com/en/blog/documentation/ • Concepts • How Tos • Tutorials • Reference • Global Search • Weekly Health Check: Support → Documentation 78

79.

80. NEWSLETTER "You can now.." • You can now benefit from the most recent Kubernetes 1.12 features, e.g. .. • You can now analyse your Kotlin project with SonarQube and upload your Scala code coverage report to SonarQube 80

81. SIGNAL: ISSUE UPVOTES 81

82. TESTIMONIALS “Useful information, good level of details and pleasant to read. It's one of the few newsletters that I took time to read entirely :)” - a reader, July 2018 82

83. TESTIMONIALS “So, thank you, Team Automata, for listening to our community, taking our upvotes in consideration when developing new solutions and building every day 'the first CI that doesn't suck'.” - a user, October 2018 83

84. DEVOPS IS … if every person uses the same tool for the same job … codified knowledge - everybody contributing their part to common automation … if all people have the same privileges in their tooling … if human error is equally possible for Dev and Ops … replacing people interfaces by automated decisions and processes … a result 84 bit.ly/5pdops

85. DEVOPS IS … if every person uses the same tool for the same job … codified knowledge - everybody contributing their part to common automation … if all people have the same privileges in their tooling … if human error is equally possible for Dev and Ops … replacing people interfaces by automated decisions and processes … a result 85 bit.ly/5pdops

86. DEVOPS IS … if every person uses the same tool for the same job … codified knowledge - everybody contributing their part to common automation … if all people have the same privileges in their tooling … if human error is equally possible for Dev and Ops … replacing people interfaces by automated decisions and processes … a result 86 bit.ly/5pdops

87. DEVOPS IS … if every person uses the same tool for the same job … codified knowledge - everybody contributing their part to common automation … if all people have the same privileges in their tooling … if human error is equally possible for Dev and Ops … replacing people interfaces by automated decisions and processes … a result 87 bit.ly/5pdops

88. DEVOPS IS … if every person uses the same tool for the same job … codified knowledge - everybody contributing their part to common automation … if all people have the same privileges in their tooling … if human error is equally possible for Dev and Ops … replacing people interfaces by automated decisions and processes … a result 88 bit.ly/5pdops

89. DEVOPS IS … if every person uses the same tool for the same job … codified knowledge - everybody contributing their part to common automation … if all people have the same privileges in their tooling … if human error is equally possible for Dev and Ops … replacing people interfaces by automated decisions and processes … a result 89 bit.ly/5pdops

90. DevOps is normal 90

91. DevOps is normal 91

92. You build it, you run it is normal 92

93. WAIT A SECOND.. What about my job title? 93

94. "DEVOPS ENGINEER" AS A JOB TITLE 94

95. T-SHAPED ENGINEER "Generalizing specialist" Technical Breadth of Knowledge 95

96. DevOps Engineer 96 Software Engineer / Software Developer (T-Shaped)

97. 97

98. OPEN SOURCE & MORE Kubernetes on AWS github.com/zalando-incubator/kubernetes-on-aws Skipper HTTP Router & Ingress controller github.com/zalando/skipper External DNS github.com/kubernetes-incubator/external-dns Postgres Operator github.com/zalando-incubator/postgres-operator More Zalando Tech Talks github.com/zalando/public-presentations 98

99. QUESTIONS? HENNING JACOBS HEAD OF DEVELOPER PRODUCTIVITY henning@zalando.de @try_except_ Illustrations by @01k