Several weeks after a major security incident at Heroku, the company said this week it "will not be reconnecting to GitHub until we are certain that we can do so safely, which may take some time" -- as news of the early April breach continues to percolate slowly down to confused users dealing with the consequences of the incident.
UPDATED May 5, 2022: Heroku says the attacker "obtained access to a Heroku database [on April 7] and downloaded stored customer GitHub integration OAuth tokens. Access to the environment was gained by leveraging a compromised token for a Heroku machine account... Our investigation also revealed that the same compromised token was leveraged to gain access to a database and exfiltrate the hashed and salted passwords for customers’ user accounts. For this reason, Salesforce is ensuring all Heroku user passwords are reset and potentially affected credentials are refreshed. We have rotated internal Heroku credentials and put additional detections in place." It continues its investigation
Heroku, a powerful platform that lets developers build, run, manage and scale applications, was bought by Salesforce for $212 million in 2010 and received investment for several years, but is widely regarded as having been somewhat neglected in recent years, even if it mains a core part of many developers' workflows.
https://twitter.com/Deffex_/status/1519466834021564424 Plenty of customers still haven't noted the incident...
(Customers include Citrix, PensionBee and many others. The former built a video chat tool "GoToMeeting Free" on Heroku, using it to host both the API and the front-end web experience including the production and staging environments, with AWS used to handle audio and video traffic and the Firebase for synchronisation).
Heroku outage, GitHub disconnection follow breach
Earlier this month Heroku's GitHub account was illicitly accessed by an unknown hacker.
GitHub told Heroku that the "threat actor was enumerating GitHub customer accounts using OAuth tokens issued to Heroku’s OAuth integration dashboard hosted on GitHub" in an incident that still remains somewhat opaque -- Heroku owner Salesforce and a third-party incident response team continue to investigate how it occurred.
See also: GitHub breached after 0Auth tokens stolen, abused in alarming supply chain incident
Alarmingly for many, compromised OAuth user tokens from Heroku and Travis-CI applications were stolen and abused to download private GitHub repositories belonging to numerous victim organizations.
GitHub said earlier this month that: "Our analysis of other behavior by the threat actor suggests that the actors may be mining the downloaded private repository contents, to which the stolen OAuth token had access, for secrets that could be used to pivot into other infrastructure" and in an update on April 27 added:
"GitHub’s analysis of the attacker’s behavior reveals the following activities carried out on GitHub.com using stolen OAuth app tokens" (note that it has not yet put a hard number on how many victims there were).
- "The attacker authenticated to the GitHub API using the stolen OAuth tokens issued to Heroku and Travis CI.
- "For most people who had the affected Heroku or Travis CI OAuth apps authorized in their GitHub accounts, the attacker listed all the user’s organizations.
- "The attacker then selectively chose targets based on the listed organizations.
- "The attacker listed the private repositories for user accounts of interest.
- "The attacker then proceeded to clone some of those private repositories."
GitHub CSO Mike Hanley added: "This pattern of behavior suggests the attacker was only listing organizations in order to identify accounts to selectively target for listing and downloading private repositories.
"GitHub believes these attacks were highly targeted based on the available information and our analysis of the attacker behavior using the compromised OAuth tokens issued to Travis CI and Heroku."
Sympathy from some Heroku customers was limited, with one voicing frustration at what they characterised as -- "after sitting into their debrief meeting on Monday" --- an hour of the company "saying 'hey guys someone got access to our 0Auth credentials, sorry we still don’t know how, if you need help that’s a GitHub problem'".
https://twitter.com/madbyk/status/1518659306249465856
Heroku users cut off from the GitHub integration can still use the CLI and others have found workarounds. Software engineer Burak Yigit Kaya, for example, said: "After the Heroku-GitHub integration was severed last week due to the security incident, we quickly came up with an efficient GitHub Action to restore the functionality... Now you might be asking "what's so special about this, aren't there like a dozen actions that do the same thing?" and you'd be right.
" The issue is, as far as I could tell, all of them were using the `git push` mechanism that required pulling and pushing the *entire* repo history. With an active monorepo, we quickly realized this wouldn't be a sustainable nor efficient solution. We use GitHub API to generate a GitHub repo tarball link on the fly, pass that to Heroku API, and then follow the build and release to the end. Hope you find it useful!"
Although Heroku's GitHub cut-off has triggered some predictable snark, it has also triggered some real soul searching and discussion about alternatives for what, despite its apparent neglect at Salesforce, has remained a powerful tool for building and deploying applications -- a lively discussion featuring engineers from many of the proposed alternatives as well as Heroku itself can be tracked on HackerNews here.
As one commentator put it: "Heroku has been constantly praised since it took off because it does massive amounts of things behind the scenes to abstract away all the operations you don't want to worry about. You can launch a tiny prototype or small to medium startup in under an hour.
"You can add a DB and Redis w/ snapshots and automated backup and monitoring all w/i the Heroku API or UI. Does it expose the full power of the datastore and let you do everything you can do with RDS or a VPS? Of course not, but thats a totally valid trade off when you are just trying to get something shipped to see if it has traction."
End-users more concerned about the security than the developer implications meanwhile will no doubt be eagerly awaiting the outcome of Salesforce and Travis-CI (decidedly quiet on the incident)'s investigations, as well as potentially further follow-up from GitHub on the scale of the alarming supply chain incident.
Heroku continues to ask customers to share any logs obtained from GitHub that reveal potential indicators of compromise (IOCs) via email to security@salesforce.com...
Affected by the incident? Strong views on it? Get in touch with our team.
