A bug in how Akamai – a Content Distribution Network (CDN) and cybersecurity firm – handles HTTP headers could have allowed an attacker to launch a range of potentially devastating attacks including placing malicious code on millions of the world’s websites which rely on the service provider, according to security researchers.
Yet when they reported the Akamai vulnerability – promptly checked, replicated and acknowledged by the firm, which has since quietly mitigated it – they were told that the company doesn’t have a bug bounty programme.
So in a “race against time” Jacopo Tediosi and Francesco Mariani set out to report the vulnerability to exposed Akamai customers who might have been at risk before it was patched, including Airbnb, Goldman Sachs, Hyatt Hotels, PayPal, Playstation, and Starbucks in the hope of securing some reward for their White Hat work.
Their caustic write-up on the process makes for eye-opening and sometimes amusing reading, including for the vast divergence in responses to their discloses: PayPal parting with $25,200, Airbnb paying $14,875, Goldman Sachs a meagre $100, Starbucks saying it wasn’t a major security issue, and Playstation being unable to replicate the attack. Others closed tickets with no response or said the report replicated previous disclosures.
Cuttingly, they noted: “On Bugcrowd [another bug bounty platform], they were not competent enough to understand the vulnerability and closed both our reports for Tesla.com as “duplicated” (of a ticket clearly not related to ours) and for LastPass.com as “not applicable” because they were unable to reproduce”.
Akamai vulnerability: HTTP headers again...
The two were working on a private Bug Bounty program organised by White Hat platform Whitejar to search for bugs on a website that was using the widely used Akamai CDN. In the process they identified a HTTP Smuggling Vulnerability with significant potential impact. The bug was in how Akamai proxies handled so-called “hop-by-hop” headers – HTTP headers that are meaningful only for a single transport-level connection, and which typically must not be retransmitted by proxies or cached, or risk being abused by enterprising hackers.
As Jacopo Tediosi put it in a technical report, published on Medium on September 29, 2022: “‘Smuggled’ responses were being server-side cached from Akamai Edge Nodes for the entire geographic area close to the IP sending the malicious request. This allowed us to semi-permanently (depending on cache times) create new arbitrary contents within almost any domain served by Akamai, resulting in a HUGE impact!”
(Sergey Belov, Application Security Lead at Acronis, notes to The Stack, this kind of server-side cache poisoning would let hackers “place malicious content by [on] any URL. Which means they can post fake news or place a malicious JavaScript code that will attack users of this website (stealing accounts, money etc.)”
Akamai fixed the issue by applying new rules to header values, but as the two noted “we are not sure that there are no bypasses or some other unexpected similar ways to split the requests” – sharing a response from Akamai that notes frankly that a deeper fix wil require “major changes to our… HTTP processing logic.”
Follow The Stack on LinkedIn to connect with the team
Despite the absence of a bounty Akamai responded swiftly to the security researchers’ initial disclosure on March 23, confirming the vulnerability within 25 hours and pushing a silent fix out on April 2.
Whilst public bug bounties are increasingly common, not everybody buys into the need for them and many large software providers also do not have them; VMware being one case in point. (As The Stack noted earlier this year “Find a critical pre-authentication exploit that lets you attack VMware’s vCentre Server and you could earn $100,000 selling it to a zero day broker like Zerodium. Do the right thing as a security researcher and report it to the software giant and you will get a big fat nothing other than the warm fuzzy glow of being a good person — because the $11 billion (by 2021 revenues) company does not run a paid bug bounty programme.”)
Key Akamai rival Cloudflare meanwhile, only launched a paying public bug bounty programme via HackerOne at the start of 2022, noting that its earlier internal disclosure programme had resulted in a “tremendous amount of time to triaging false positive reports and helping the researchers understand their errors”.
That was due to challenges for many in “understanding where Cloudflare fits into the HTTP request/response pipeline”, Cloudflare wrote, in part because the company did not “provide much supporting documentation about how our products worked…” resulting in a situation in which of 1,197 reports disclosures, only 158 were valid. (The company has an interesting blog about how it turned around the signal to noise ration here.)
Acronis’ Belov meanwhile notes to us that “this type of HTTP smuggling request is still common if the company is big and has a lot of users, that should be balanced, or a system that requires specific logic on the backend side. Also, companies are starting to use not only a single web servers and application server behind, but several web servers. And since the HTTP protocol that we use in WebApps isn't strict enough, such issues are happening and will continue to happen… Vendors should try to use the same webservers for balancing, and not implement some custom HTTP. They should try to simplify their infrastructure as much as they can..."
Akamai had not responded to a request for comment as we published.
