Going Infinite, handling 1M websockets connections in Go
如果无法正常显示，请先停止浏览器的去广告插件。
1. Going Infinite, handling 1M
websockets connections in Go
Eran Yanay, Twistlock
2. The goal
Developing high-load Go server that is able to manage millions of concurrent
connections
●
●
●
●
How to write a webserver in Go?
How to handle persistent connections?
What limitations arise in scale?
How to handle persistent connections efficiently?
○ OS limitations
○ Hardware limitations
3. How a Go web server works?
package main
import (
"io"
"net/http"
)
func main() {
http.HandleFunc("/", hello)
http.ListenAndServe
(":8000", nil)
}
func hello(w http.ResponseWriter, r *http.Request) {
io.WriteString(w, "Hello Gophercon!"
)
}
4. How a Go web server works?
package main
import (
"io"
"net/http"
)
func main() {
http.HandleFunc("/", hello)
http.ListenAndServe
(":8000", nil)
}
func hello(w http.ResponseWriter, r *http.Request) {
io.WriteString(w, "Hello Gophercon!"
)
}
5. How a Go web server works?
// Serve accepts incoming connections on the Listener l, creating a
// new service goroutine for each. The service goroutines read requests and
// then call srv.Handler to reply to them.
func (srv *Server) Serve(l net.Listener) error {
// ...
for {
rw, e := l.Accept()
// ...
c := srv.newConn(rw)
c.setState(c.rwc, StateNew) // before Serve can return
go c.serve(ctx)
}
}
6. How a Go web server works?
// Serve accepts incoming connections on the Listener l, creating a
// new service goroutine for each. The service goroutines read requests and
// then call srv.Handler to reply to them.
func (srv *Server) Serve(l net.Listener) error {
// ...
for {
rw, e := l.Accept()
// ...
c := srv.newConn(rw)
func hello(w http.ResponseWriter, r *http.Request) {
)
c.setState(c.rwc, StateNew) io.WriteString(w, "Hello Gophercon!"
// before Serve can return
}
go c.serve(ctx)
}
}
7. The need for persistent connections
●
●
●
●
●
●
Message queues
Chat applications
Notifications
Social feeds
Collaborative editing
Location updates
8. What is a websocket?
WebSockets provide a way to maintain a full-duplex persistent connection
between a client and server that both parties can start sending data at any time,
with low overhead and latency
GET ws://websocket.example.com/ HTTP/1.1
Connection: Upgrade
Host: websocket.example.com
Upgrade: websocket
Client
Server
HTTP/1.1 101 WebSocket Protocol Handshake
Connection: Upgrade
Upgrade: WebSocket
9. Websockets in Go
10. Websockets in Go
func ws(w http.ResponseWriter, r *http.Request) {
func main() {
// Upgrade connection http.HandleFunc("/", ws)
upgrader := websocket.Upgrader{} http.ListenAndServe(":8000", nil)
conn, err := upgrader.Upgrade(w, r, nil)
if err != nil {
return
}
for {
_, msg, err := conn.ReadMessage()
if err != nil {
log.Printf("Failed to read message %v", err)
conn.Close()
return
}
log.Println(string(msg))
}
}
}
11. Demo!
12. Demo! - Cont’d
13. Too many open files
● Every socket is represented by a file descriptor
● The OS needs memory to manage each open file
● Memory is a limited resource
● Maximum number of open files can be changed via ulimits
14. Resources limit
Ulimit provides control over the resources available to processes
15. Resources limit
Ulimit provides control over the resources available to processes
●
●
●
●
●
The kernel enforces the soft limit for the corresponding resource
The hard limit acts as a ceiling for the soft limit
Unprivileged process can only raise up to the hard limit
Privileged process can make any arbitrary change
RLIMIT_NOFILE is the resource enforcing max number of open files
16. Resources limit in Go
func SetUlimit() error {
var rLimit syscall.Rlimit
if err := syscall.Getrlimit(syscall.RLIMIT_NOFILE, &rLimit); err !=nil {
return err
}
rLimit.Cur = rLimit.Max
return syscall.Setrlimit(syscall.RLIMIT_NOFILE, &rLimit)
}
17. Demo! (#2)
18. Memory consumption
19. pprof
Package pprof serves via its HTTP server runtime profiling data in the format
expected by the pprof visualization tool.
import _ "net/http/pprof"
go func() {
if err := http.ListenAndServe
("localhost:6060" nil); err != nil {
,
log.Fatalf("Pprof failed: %v" err)
,
}
}()
● Analyze heap memory: go
● Analyze goroutines: go
tool pprof http://localhost:6060/debug/pprof/heap
tool pprof http://localhost:6060/debug/pprof/goroutine
20. pprof - Demo!
21. Memory consumption
Each connection in the naive solution consumes ~20KB:
22. Memory consumption
Each connection in the naive solution consumes ~20KB:
func ws(w http.ResponseWriter, r *http.Request) {
// ...
}
23. Memory consumption
Each connection in the naive solution consumes ~20KB:
func ws(w http.ResponseWriter, r *http.Request) {
// ...
}
upgrader := websocket.Upgrader{}
conn, err := upgrader.Upgrade(w, r, nil)
if err != nil {
return
}
24. Memory consumption
Each connection in the naive solution consumes ~20KB:
func ws(w http.ResponseWriter, r *http.Request) {
// ...
}
upgrader := websocket.Upgrader{}
conn, err := upgrader.Upgrade(w, r, nil)
if err != nil {
return
}
Serving a million concurrent connections would consume over 20GB of RAM!
25. Optimizations
If we could…
●
●
●
Optimize goroutines
Optimize net/http objects allocations
Reuse allocated buffers across websockets read/write
26. Optimization #1: goroutines
Knowing when data exists on the wire would allow us to reuse goroutines and
reduce memory footprint
●
●
●
goroutines
select / poll
epoll
27. Optimization #1: goroutines
Knowing when data exists on the wire would allow us to reuse goroutines and
reduce memory footprint
func ws(w http.ResponseWriter, r *http.Request) {
● goroutines
●
●
// Upgrade connection …
select / poll
epoll
for {
_, msg, err := conn.ReadMessage()
if err != nil {
log.
Printf("Failed to read message %v" err)
,
conn.Close()
return
}
log.Println(string(msg))
}
}
28. Optimization #1: goroutines
Knowing when data exists on the wire would allow us to reuse goroutines and
reduce memory footprint
t := &syscall.Timeval{ /* timeout for the call */ }
●
goroutines
if _, err := syscall.Select(maxFD+1, fds, nil, nil, t); err != nil {
return nil, err
● select / poll
●
epoll
}
for _, fd := range fds {
if fdIsSet(fdset, fd) {
// Consume data
}
}
29. Optimization #1: goroutines
Knowing when data exists on the wire would allow us to reuse goroutines and
reduce memory footprint
epfd, _ := unix.EpollCreate1(0)
●
●
goroutines
select / poll
● epoll
_ := unix.EpollCtl(epfd, syscall.EPOLL_CTL_ADD, fd,
&unix.EpollEvent{Events: unix.POLLIN | unix.POLLHUP, Fd: fd})
events := make([]unix.EpollEvent, 100)
n, _ := unix.EpollWait(e.fd, events, 100)
for i := 0; i < n; i++ {
// Consume data from connection who's fd is events[i].Fd
}
30. Epoll - Demo!
fd, err := unix.EpollCreate1(0)
if err != nil {
return nil, err
}
fd := websocketFD(conn)
err := unix.EpollCtl(e.fd, syscall.EPOLL_CTL_ADD, fd, &unix.EpollEvent{Events: unix.POLLIN | unix.POLLHUP, Fd:
int32(fd)})
if err != nil {
return err
}
31. Epoll - Results
We managed to reduce the memory consumption by ~30%
But..is it enough?
32. Optimization #2: buffers allocations
gorilla/websocket keeps a reference to the underlying buffers given by Hijack()
var br *bufio.Reader
if u.ReadBufferSize == 0 && bufioReaderSize(netConn, brw.Reader) > 256 {
// Reuse hijacked buffered reader as connection reader.
br = brw.Reader
}
buf := bufioWriterBuffer(netConn, brw.Writer)
var writeBuf []byte
if u.WriteBufferPool == nil && u.WriteBufferSize == 0 && len(buf) >= maxFrameHeaderSize+256 {
// Reuse hijacked write buffer as connection buffer.
writeBuf = buf
}
c := newConn(netConn, true, u.ReadBufferSize, u.WriteBufferSize, u.WriteBufferPool, br, writeBuf)
33. Optimization #2: buffers allocations
github.com/gobwas/ws - alternative websockets library for Go
●
●
●
No intermediate allocations during I/O
Low-level API which allows building logic of packet handling and buffers
Zero-copy upgrades
import "github.com/gobwas/ws"
for {
// Fetch ready connections with epoll logic
func wsHandler(w http.ResponseWriter, r *http.Request) {
msg, _, err := wsutil.ReadClientData(conn)
conn, _, _, err := ws.UpgradeHTTP(r, w)
if err == nil {
if err != nil {
log.Printf("msg: %s", string(msg))
return
} else {
}
// Close connection
// Add to epoll
}
}
}
34. gobwas/ws - Demo!
35. Buffer allocations - Results
We managed to reduce the memory usage by 97%
Serving over a million connections is now reduced from ~20GB to ~600MB
36. Recap..
Premature optimization is the root of all evil, but if we must:
● Ulimit: Increase the cap of NOFILE resource
● Epoll (Async I/O): Reduce the high load of goroutines
● Gobwas - More performant ws library to reduce buffer allocations
● Conntrack table - Increase the cap of total concurrent connections in the OS
37. Thank you!
Code examples are available at https://github.com/eranyanay/1m-go-websockets
Questions?