Kubernetes + Python
如果无法正常显示，请先停止浏览器的去广告插件。
相关话题：
#zalando
1. Kubernetes
+ Python
=
CLOUD NATIVE PRAGUE
2019-09-05
HENNING JACOBS
@try_except_
2. ZALANDO AT A GLANCE
~ 5.4
billion EUR
> 250
million
revenue 2018
2
> 15.000 > 79%
employees in
Europe of visits via
mobile devices
visits
per
month
> 300.000
> 26 product choices
million ~ 2.000 17
brands countries
active customers
3. SCALE
396
Accounts
140
3
Clusters
4. PYTHON
4
• Very popular
• Easy to learn
• Huge community
5. PYTHON
5
6. PYTHON IN ZALANDO
61 (Go) vs 257 (Python) profiles
14k (Go) vs 52k (Python) internal GitHub hits
Machine Learning / Data Science
6
7. RUNNING PYTHON ON KUBERNETES
7
8. RUNNING PYTHON ON KUBERNETES
⇒ ~114 MiB Docker image
8
9. DEPLOYMENT MANIFEST
Never without a
readinessProbe
Security "stuff"
9
10. HELLO, WORLD
kubectl apply -f deployment.yaml
kubectl port-forward service/aiohttp-helloworld 8080:80
http localhost:8080
HTTPie
10
11. HOW TO USE KUBERNETES
11
1. run
"Hello World"
2. configure
the rest of your app
12. IS IT FAST ENOUGH?
echo 'GET http://localhost:8080' | vegeta attack -rate 200 | vegeta report
Requests
Duration
Latencies
[total, rate]
[total, attack, wait]
[mean, 50, 95, 99, max]
Bytes In
[total, mean]
Bytes Out
[total, mean]
Success
[ratio]
Status Codes [code:count]
3300, 200.06
16.498s, 16.494, 3.261ms
3.185ms, 3.229ms,
4.182ms, 4.791ms,
21.153ms
39600, 12.00
0, 0.00
100.00%
200:3300
local run on my laptop, single Pod in kind cluster
12
13. RUNNING PYTHON ON KUBERNETES
13
14. "GRACEFUL" TERMINATION IN KUBERNETES
1. kubectl delete pod
2. SIGTERM + remove endpoints
3. grace period (default: 30s)
kube-proxy/ iptables,
Load Balancer update
4. SIGKILL
⇒ Going down on SIGTERM leads to request errors
14
15. TESTING THE ROLLING UPDATE
tests/e2e/test_update.py::test_rolling_update_no_signal_handling
Elapsed:
41.60 seconds
Successes: 2056
Errors:
0
Requests/s: 49.42 rps
tests/e2e/test_update.py::test_rolling_update_with_signal_handling
Elapsed:
15.16 seconds
Successes: 816
Errors:
2
Requests/s: 53.95 rps
15
16. ALTERNATIVE: THE PRESTOP TRICK
...
16
17. TESTING WITH PYTEST AND KIND
17
18. RUNNING PYTHON ON KUBERNETES
18
19. TESTING WITH PYTEST AND KIND
https://pypi.org/project/pytest-kind/
19
20. USING KIND CLUSTER DIRECTLY
https://pypi.org/project/pytest-kind/
20
21. ACCESSING KUBERNETES FROM PYTHON
21
22. ACCESSING KUBERNETES FROM PYTHON
•
•
Official Kubernetes client - github.com/kubernetes-client/python
Pykube-ng - github.com/hjacobs/pykube
du -csh /../../lib/python3.7/site-packages/kubernetes/
23M
total
du -csh /../../lib/python3.7/site-packages/pykube/
160K
22
total
23. LISTING CONTAINER IMAGES
23
24. OFFICIAL CLIENT VS PYKUBE-NG
•
•
•
•
•
•
•
24
"Official"
Generated (Swagger Codegen)
Less "Pythonic"
urllib3
Supports many auth methods
Heavy (23 MiB)
Python 2.7 and 3
•
•
•
•
•
•
•
Kel project 2015
Dynamic
More "Pythonic"
requests
Limited auth methods
Light (160 KiB)
Only Python 3
25. INTERACTIVE CONSOLE
$ pip3 install pykube-ng
$ python3 -m pykube
Pykube v19.9.1, loaded "/home/hjacobs/.kube/config" with context "mycluster".
Example commands:
[d.name for d in Deployment.objects(api)]
# get names of deployments in default namespace
list(DaemonSet.objects(api, namespace='kube-system')) # list daemonsets in "kube-system"
Pod.objects(api).get_by_name('mypod').labels
# labels of pod "mypod"
Use Ctrl-D to exit
>>>
25
26. WRITING A SIMPLE CONTROLLER
26
27. DOWNSCALING DURING OFF-HOURS
Weekend
27
github.com/hjacobs/kube-downscaler
28. DOWNSCALING DURING OFF-HOURS
DEFAULT_UPTIME="Mon-Fri 07:30-20:30 CET"
annotations:
downscaler/exclude: "true"
28
github.com/hjacobs/kube-downscaler
29. HOUSEKEEPING
● Delete prototypes
after X days
● Clean up temporary
deployments
● Remove resources
without owner
29
30. KUBERNETES JANITOR
● TTL and expiry date annotations, e.g.
○ set time-to-live for your test deployment
● Custom rules, e.g.
○ delete everything without "app" label after 7 days
30
github.com/hjacobs/kube-janitor
31. JANITOR TTL ANNOTATION
# let's try out nginx, but only for 1 hour
kubectl run nginx --image=nginx
kubectl annotate deploy nginx janitor/ttl=1h
31
github.com/hjacobs/kube-janitor
32. EXTENDING KUBERNETES WITH OPERATORS
32
33. ELASTICSEARCH OPERATOR
33
github.com/zalando-incubator/es-operator
34. WRITING AN OPERATOR
34
35. MAKE A FRAMEWORK!
github.com/zalando-incubator/kopf
35
36. KOPF: SIMPLE SPY-HANDLERS
React on events from K8s API.
Raw payload, no interpretation. Fire-and-forget, ignore errors.
36
github.com/zalando-incubator/kopf
37. KOPF: CONVENIENT CAUSE AND DIFF HANDLERS
37
github.com/zalando-incubator/kopf
38. RUNNING FROM DEVELOPMENT ENVIRONMENT
$ kopf run scripts.py [--verbose]
And here we are! Creating: {'duration': '1m', 'field': 'value',
'items': ['item1', 'item2']}
[2019-02-25 14:06:54,742] kopf.reactor.handlin [INFO
]
[asf-preprocessing/kopf-example-1] Handler create_fn succeeded.
[2019-02-25 14:06:54,856] kopf.reactor.handlin [INFO
]
[asf-preprocessing/kopf-example-1] All handlers succeeded for
38
creation.
$ kubectl apply -f ../obj.yaml
$ kubectl describe -f ../obj.yaml
Name:
kopf-example-1
...
Status:
create_fn:
Message: hello world
Events:
Type
Reason
Age From
----
------
---- ----
Normal Success 81s kopf
github.com/zalando-incubator/kopf
Message
-------
Handler create_fn succeeded.
39. "HELLO WORLD" EXAMPLE: CRD
39
github.com/zalando-incubator/kopf
40. 8 LINES OF PYTHON
40
github.com/zalando-incubator/kopf
41. 4 LINES OF DOCKER
41
github.com/zalando-incubator/kopf
42. CRD OBJECT
42
github.com/zalando-incubator/kopf
43. STATUS: "HELLO PRAGUE!"
43
github.com/zalando-incubator/kopf
44. VMWORLD 2019 SESSION WITH KOPF
44
github.com/embano1/kopf-operator-vmworld
45. KOPF: FEATURES
•
•
•
•
•
•
45
Custom & built-in resources supported (CRDs, Pods, Services, etc)
Agnostic to API clients: kubernetes-client, pykube-ng, raw HTTP, etc
Immediate reaction to changes and events
Predefined behavioural patterns:
• Simple spy-handlers for event watching
• Advanced cause & diff detection for actual change tracking
• Retry-until-success approach to handlers
Operator resilience:
• Restores its state on restarts
Operator testing toolkit (minimally sufficient)
github.com/zalando-incubator/kopf
46. TESTING WITH KIND
46
pypi.org/project/pytest-kind/
47. Kubernetes Web View
"kubectl for the web"
48. MOTIVATION: CHAT PING-PONG
48
49. MOTIVATION: PENDING PODS INCIDENT
⇒ I want to see all "Pending" Pods across all clusters!
49
50. MOTIVATION
50
51. SOME USE CASES
Support interactions with colleagues
•
•
•
share links
avoid human mistakes (wrong cluster login on CLI, "-n" param, ..)
deep link to resource YAML
Incident response and investigation
•
•
•
51
find a resource by name across clusters to help troubleshoot
find pending pods across all clusters
help identify if a problem is only in one cluster or multiple ones
52. MOTIVATION
52
srcco.de/posts/kubernetes-web-uis-in-2019.html
53. SOME USE CASES
53
54. SEARCHING FOR DOCKER IMAGE
54
55. SOME USE CASES
55
56. CRDS
additionalPrinterColumns
56
57. UPGRADE TO KUBERNETES 1.14
"Found 1223 rows for 1 resource type in 148 clusters in 3.301 seconds."
57
58. RESOURCE YAML VIEW
58
59. MULTIPLE CONDITIONS
kubectl -l application=coredns
59
60. MULTIPLE RESOURCE TYPES
kubectl get
pods,stacks,deploys,..
60
61. LABEL & CUSTOM COLUMNS
labelcols=application
customcols=CPU+Requests=join(', ', spec.containers[*].resources.requests.cpu)
uses JMESPath expression
61
62. URL STRUCTURE
/clusters/{cluster}/namespaces/{namespace}/{resource-type}
cluster: cluster aliases separated by "," or "_all"
namespace: namespace name or "_all"
resource-type: plural names separated by "," or "all" (like kubectl get all)
62
https://kube-web-view.readthedocs.io/
63. QUERY PARAMETERS FOR LIST VIEW
sort: column name to sort by, optionally with ":desc"
labelcols: additional label columns (comma separated) or "*"
selector: label selector (like kubectl -l), will be passed to Kubernetes API
filter: post-processing filter, either a simple string, Column=Value or
Column!=Value
hidecols: hide certain columns
customcols: custom columns using JMESPath
limit: number of rows to show
63
https://kube-web-view.readthedocs.io/en/latest/features.html#listing-resources
64. CUSTOMIZATION
Nearly all aspects can be customized for your org via CLI options:
64
• External Links (monitoring dashboards, ..)
• Default Label and Custom Columns
• Sidebar, HTML templates
• OAuth2 integration
kube-web-view.readthedocs.io/en/latest/customization.html
65. LINKING FROM DEPLOYMENT STATUS
65
66. KUBE-WEB-VIEW.DEMO.J-SERV.DE
66
67. TESTING WITH PYTEST-KIND AND PURE HTML
•
•
•
•
67
Pykube
aiohttp
pytest-kind
Jinja2 templates
68. KUBERNETES + PYTHON?
• Pykube
• asyncio / aiohttp
• pytest-kind
• Jinja2 templates
codeberg.org/hjacobs/kube-web-view/
68
69. KUBERNETES RESOURCE REPORT
69
github.com/hjacobs/kube-resource-report
70. https://github.com/hjacobs/kube-ops-view
71. https://github.com/hjacobs/kube-ops-view
72. KUBERNETES + PYTHON =
github.com/hjacobs
72
73. KUBERNETES + PYTHON =
code examples from this talk can be found in
codeberg.org/hjacobs/kubernetes-and-python
73
74. ZALANDO OPEN SOURCE
Kubernetes on AWS
github.com/zalando-incubator/kubernetes-on-aws
AWS ALB Ingress controller
github.com/zalando-incubator/kube-ingress-aws-controller
External DNS
github.com/kubernetes-incubator/external-dns
Postgres Operator
github.com/zalando/postgres-operator
Elasticsearch Operator
github.com/zalando-incubator/es-operator
Kubernetes Metrics Adapter
github.com/zalando-incubator/kube-metrics-adapter
74
75. QUESTIONS?
HENNING JACOBS
henning@zalando.de
@try_except_
Illustrations by @01k