Using Envoy as an Explicit CONNECT and Transparent Proxy to disrupt malicious traffic

Dean Liu

·

Follow

Published in

Lyft Engineering

·

12 min read

·

Nov 2, 2022

--

Listen

Share

Unrestricted egress traffic from services poses a significant security risk as it allows external threats to exfiltrate data and download arbitrary payloads from untrusted, dangerous hosts. While ingress filtering from the Internet is ubiquitous using firewalls, it is far less common that companies are controlling and observing traffic leaving their network.

As part of implementing traffic filtering, we achieved observability of our egress traffic, which has also enabled opportunities for our Security team to write detection rules based on anomalous traffic, perform network forensics, and conduct proactive threat hunting exercises.

In this post, we aim to cover how our Security team achieved egress filtering on behalf of all service owners at Lyft. We will go over design decisions, different proxy types, and how we leverage Envoy to act as our Internet Gateway (IGW).

Note that this post addresses service filtering and not employee traffic filtering which is a different threat model in itself.

Goals

Our primary goal is to disrupt malicious traffic by ensuring all Internet bound network traffic originating from Lyft services are filtered through Envoy.

There are numerous ways to control egress traffic that Lyft weighed where the main challenges stemmed from obtaining observability of encrypted TLS traffic and preventing exfiltration via DNS.

We focused on two main goals to drive our design decisions:

1. Observability: we want to know the upstream Internet hosts our services communicate with and be able to attribute this traffic to individual downstream services. We effectively want to create a mapping of services to a list of upstream Internet hosts: {service_a: [lyft.com, example.com], service_b:[eng.lyft.com],…}
2. Restriction: ability to restrict upstream hosts on a per downstream service basis. Each service will have a unique allowlist of Internet domains for a service.

Terminology:
Downstream: a host that connects to Envoy, sends requests, and receives responses.
Upstream: a host that receives connections and requests from Envoy.

A driving principle of the Security team at Lyft is to make security easy for engineers. Security at Lyft focuses on building and providing secure infrastructure and services for our engineers so that engineers can focus on shipping features for our customers.

Lyft uses a micro-service architecture comprising of close to a thousand services, and one of the core challenges of this endeavor was to re-route our services’ egress traffic through the Envoy Internet Gateway (IGW) with zero downtime.

Proxy Types

Before diving into our final implementation, we briefly cover how explicit and transparent proxies work, which Lyft ultimately uses as the foundation to filter traffic.

HTTP CONNECT

The HTTP CONNECT proxy is a type of explicit proxy. The client knows about the proxy and is “explicitly” configured to use it.

The HTTP CONNECT method is an instruction sent by the client to the proxy to open a TCP connection to the target server. After a TCP connection is established, the proxy just forwards packets.

TLS provides end-to-end encryption and ensures there’s no MITM occurring between the client and server. CONNECT proxies are especially useful for proxying TLS traffic because HTTP libraries know they must first establish a TCP connection through the proxy to the destination host prior to establishing the TLS connection. The initial TCP connection establishment enables us to obtain Layer 7 HTTP observability into both downstream services and upstream hosts.

Tcpdump snippet of an HTTP CONNECT call:

CONNECT proxy.lyft.net:80 HTTP/1.1
Host: server.example.com:443
Proxy-Authorization: basic bHlmdDpiZXJyaWVzNGxpZmUK
Proxy-Connection: Keep-Alive

A client can be configured to use the proxy in many ways such as PAC files, code (python-requests example), or through HTTP_PROXY environment variables where the values are set to the hostname or IP of the proxy. We used the environment variable approach because it did not require any service code changes. When an HTTP client sees these environment variables, it knows it must use the proxy.

Let’s walk through the steps that occur when a client makes a web request to the Internet:

1. The client’s HTTP library looks for HTTP_PROXY environment variables, and if present, initiates a CONNECT call to the proxy.
2. The proxy receives the instruction from the client, and then resolves the requested hostname via DNS.
3. The proxy establishes a connection to the destination IP and returns a HTTP 200 response.
4. The client then makes its original web request. Packets are forwarded back and forth between the client and destination through the established TCP connection.

The below diagram demonstrates a typical HTTPS web request through a CONNECT proxy.

Transparent

Transparent proxies, on the other hand, work by re-routing client traffic without the client knowing. This can be done by a Layer 4 switch or with iptables. Contrasted with the CONNECT proxy, obtaining observability into the downstream client and upstream host is much more challenging. There are also multiple options on how to forward traffic that the proxy must decide on.

Let’s walk through the steps that occur when a client makes a web request to the Internet using a transparent proxy:

1. Client resolves the IP via DNS of the host it wants to connect to.
2. Client initiates a connection to the IP.
3. Iptables re-routes the traffic by replacing the destination IP to that of the proxy.
4. Proxy receives the re-directed traffic. This is where things get complicated as the proxy needs to decide how to forward the traffic depending on the traffic type:

• If traffic is TLS, route via SNI; however, there’s no guarantee that all TLS traffic will have SNI, and TLS 1.3 introduces encrypted SNI.
• If traffic is plaintext HTTP, route based on hostname.
• Forward based on IP: this would also mean visibility and restriction would be limited to IP instead of hostnames.

Our original goal is to be able to map downstream services to upstream Internet hosts. Filtering traffic with a transparent proxy alone comes with many challenges to this, including the fact that the proxy is only able to reliably see the downstream IP with TLS traffic (which comprises the majority of web requests). In the Final Architecture section, we will discuss in detail how we create this mapping of downstream services to upstream hosts using both CONNECT and transparent proxying.

Man-in-the-Middle

We wanted to mention this option briefly as we explored this, but ultimately we did not pursue it.

In a man-in-the-middle (MITM), the proxy decrypts and re-encrypts all traffic. The proxy intercepts TLS handshakes between the client and server and generates certificates on behalf of the upstream server. This requires managing an additional certificate authority (CA) where we would need to generate a private certificate and ensure all services at Lyft trust the CA.

While visibility into payloads does enable more options for forensics and filtering, the complexity of managing certificate lifecycles; additional latency, memory, and CPU overhead; and introducing a single point where all egress traffic is decrypted added too much risk for us to adopt this option.

Final Architecture

We deployed a new Envoy edge proxy and decided to use an explicit proxy as the primary egress method with a transparent proxy to capture traffic not obeying the explicit proxy. The explicit proxy is a far simpler configuration compared to transparent proxies for the reasons stated earlier above.

The transparent proxy acts as a catch-all where we capture “leaky” traffic in order to surface problematic HTTP libraries that don’t obey the explicit proxy. We then corrected these services to onboard them fully to the explicit proxy. As a temporary stop gap, we used the transparent proxy to block on SNI.

We used iptables to force all non-RFC1918 traffic through the transparent proxy. If we observe no traffic from the client reaching the transparent proxy then it means all traffic is going through the explicit proxy, giving us confidence that it is safe to shut-off passthrough traffic at the transparent proxy.

Explicit Proxy

1. Services egress directly through the IGW via the explicit proxy. The service will authenticate to IGW by passing in a username as a proxy-header allowing the IGW to attribute the downstream service with the upstream destination

HTTP_PROXY=http://pyexample2:password@igw.lyft.net
HTTPS_PROXY=http://pyexample2:password@igw.lyft.net
NO_PROXY=127.0.0.1,localhost

2. Client HTTPS traffic will create a CONNECT call to IGW. Plaintext HTTP traffic will forgo the CONNECT call and make a request with a proxy-authorization header only.

3. Envoy’s HTTP RBAC filter will either allow or deny the traffic based on the HTTP headers tuple: (downstream service, upstream destination). Engineers can update these RBAC rules which get dynamically updated by another service, acting as our control-plane.

4. If the traffic is allowed, Envoy’s dynamic forward proxy cluster will resolve the host and then forward the traffic to the external upstream host.

Transparent Proxy

The goal for services is to not need transparent proxying, but we need observability to determine if services have HTTP libraries that are not obeying the explicit proxy configurations. The transparent proxy is allowed temporarily for services, and its path is eventually shut off once we are certain all traffic is using the client-aware path. A positive of the transparent proxy is that it works for all ports and protocols because we have the option to forward by IP.

Our transparent proxy setup is really a combination of both a transparent and CONNECT proxy. We use an Envoy sidecar to add additional transport properties that allow our IGW to allow or deny and eventually route upon.

TLS traffic is forwarded based on IP where restoring the original IP that was overwritten by iptables presents an additional challenge. The original IP is stored in the SO_ORIGINAL_DST socket when a connection has been redirected by iptables. This saved IP is only available on the local filesystem; therefore, we need a proxy that also exists locally to the service to read it. We use an Envoy sidecar to read the stored IP so it can append it as an additional transport property for IGW to restore and route on.

Walking through an HTTPS request first:

1. Traffic not obeying the explicit traffic is captured via iptables and redirected to an Envoy sidecar. The proxy-protocol filter preserves the client’s original destination IP which is needed for TLS traffic because the host header is encrypted.
2. The transparent proxied traffic is CONNECT tunneled to IGW with the downstream service name and upstream SNI (if it exists) added as a header. This allows IGW observability to both downstream and upstream as hostnames instead of just IP.
3. CONNECT terminates on the IGW.
4. RBAC is performed on the HTTP headers which allow / deny on the tuple (downstream service name, upstream SNI).
5. Terminate proxy-protocol so we can restore the original destination IP and route using Layer 4 if the traffic is TLS. Otherwise, if the traffic is HTTP plaintext, we route at Layer 7 using the hostname.

The traffic flow for plaintext HTTP is roughly similar to that of explicit proxy. We don’t need to use proxy protocols as we can forward based on the host header. Again, for downstream attribution, the service name is added as an additional header by the Envoy sidecar.

1. Iptables redirects plaintext HTTP traffic to a separate listener on the Envoy sidecar
2. The sidecar adds the downstream service name by appending it to the header and forwarding it to IGW.
3. CONNECT terminates on the IGW.
4. RBAC is performed on the HTTP headers which allow / deny on the tuple (downstream service name, upstream hostname).
5. No need for proxy-protocol because we’re routing on hostname, so this filter does not exist in this path.
6. Traffic is forwarded to the original destination using a dynamic forward proxy.

Edge Internet Gateway vs Sidecar

Today, each service at Lyft is deployed alongside a dedicated Envoy sidecar to handle service-to-service communication. We weighed leveraging the existing sidecar to filter egress Internet traffic versus deploying a new Envoy edge proxy as a centralized gateway that all services would share. We decided to adopt a centralized gateway for the following reasons:

1. This positions us well to migrate services to private subnets. More on this later.
2. We wanted to minimize changes to sidecars because deploying changes to them requires service instances to turn-over before those changes take effect, making iteration slow and rollbacks exceptionally painful.

Rollout Strategy

After standing up the IGW, we began onboarding services to egress through it by automatically injecting the HTTP_PROXY variables during deployment. Lyft uses Kubernetes, and all containers that are deployed through our CI pipeline get HTTP_PROXY variables injected. Additionally, an init container gets added that applies iptables rules to re-route traffic through the transparent proxy.

Lyft has many environments and levers to safely roll-out widely impacting changes like this. We were able to carefully onboard services in batches, incrementally shifting traffic through the IGW with no involvement from service owners. Security would monitor metrics and logs to not only validate success metrics of newly shifted traffic, but also to detect any increases in HTTP 5XX and 4XX to indicate if an immediate rollback is needed.

We first operated the IGW as a passthrough and gathered observability for the upstream domains that services were reaching. Using this data, we bootstrapped a mapping of downstream services to external upstream domains. To help condense configurations and limit changes, a small, global allowlist of Internet hosts was created, which grants all services access to certain Internet hosts that are needed by default such as AWS STS.

This mapping of services to Internet hosts {service_a: [lyft.com, example.com], service_b:[eng.lyft.com],…} was used to hydrate the IGW’s HTTP RBAC filter. We first enabled RBAC in shadow mode, logging stats on the number of denies. Once shadow deny stats reached 0, this gave us the confidence that our RBAC definitions were complete, and we flipped the switch from shadow denies to real active enforcement.

We then shut off the transparent proxy path for services not using it, which eliminated the ability to circumvent the explicit proxy.

We’ve squeeze tested our solution using synthetic traffic reflective of typical request and response sizes that were sampled from IGW. Our Envoy nodes run in Kubernetes which allows us to quickly horizontally scale where we have seen well over 200k+ concurrent outbound Internet connections across multiple Envoy nodes.

What was surprising in our squeeze testing was that although the proxy adds a p99 latency of ~5 ms, subsequent requests were faster compared to requests without a proxy because Envoy re-uses existing established connections and caches DNS resolutions.

Developer Workflow

Developers are responsible for adding any new upstream Internet domains that their service requires. In practice, we’ve found these updates to be relatively infrequent; however, when service owners need to allow a new upstream domain for their service, then we want to make it as seamless as possible.

We alarm service owners directly if the IGW blocks an outbound request, linking the alarm with a log of the downstream service and upstream domain blocked. This tells the developer the exact Envoy configuration change that is required and includes a runbook on how to perform it themselves.

HTTP Client Library Behaviors

While onboarding services to egress through our IGW, we encountered libraries that did not play nicely with our CONNECT proxy which the Security team needed to troubleshoot and address:

Boto

We discovered services that were still on boto that needed to migrate to boto3. https://github.com/boto/boto/issues/3561

Boto was deprecated alongside the deprecation of Python 2.7. This was a nice forcing function to get a small number of services migrated to boto3.

Golang

When making a gRPC request and leaving the hostname blank, Go will implicitly route to the local interface. Golang’s NO_PROXY environment variable does not match on empty hosts: there’s no way to exclude localhost traffic from routing to the IGW. https://github.com/golang/go/issues/48325

To get around this issue, we ensured all services explicitly specified localhost or 127.0.0.1.

Axios

A popular javascript HTTP client does not properly support CONNECT proxies when dealing with HTTPS requests. https://github.com/axios/axios/issues/4531

As a stop gap, we use the transparent proxy and block on SNI. Alternatively, hpagent can be used with axios explicitly configured with the library.

Next Steps: Private Subnets

Private subnets are defined by two characteristics:

1. Internet egress traffic cannot go directly to the Internet: it must traverse a proxy gateway living on a public subnet.
2. No public IP addresses are assigned to services living in private subnets, so the Internet cannot route directly to the service.

While this would have been easier to build upon at the very start, Lyft uses many of AWS’ offerings that traverse the public Internet by default. The end goal is to delineate services into private and public subnets where only a small set of services that need direct access to the Internet will live on public subnets. This would make it impossible for services to egress through to the Internet without first traversing through our Envoy Internet Gateway.

Gratitudes

Special thanks to Ruwaifa Anwar who was with me at the start until now, implementing the project.

Yueren Wang, Giuseppe Petracca and the Security team who are actively involved in contributing to the project.

Gaston Kleiman, Tom Wanielista, Jacky Tian, Junchao Lyu, Matt Klein, Snow Pettersen who’ve helped steer and advise on architectural and technical decisions.

Matthew Webber, Michael Anderson, Nico Waisman for leadership support.

Ashley Cutalo and Jake Kaufman for leading Public Readiness Review prior to shipping. 🚢

If you’re interested in working on Security problems like these then take a look at our careers page.