Uber has moved thousands of services to an Arm-based architecture in a landmark shift that follows its migration to Oracle’s and Google’s clouds.
Uber said it migrated to Arm to “improve price-performance, and secure hardware flexibility in an unpredictable supply chain environment.”
This, explained senior engineers Andreas Lykke and Jesper Borlum in an intriguing new Uber blog, was not entirely straightforward…
Yet despite it now having more than 400,000 container image builds per week that need to be multi-architecture, the costs this has incurred are offset by the savings it’s made by shifting from x86 to Arm, they said.
Uber uproots from on-prem… and x86
The move follows a cloud migration that started in 2023, when Uber signed seven-year deals with Google Cloud and Oracle – and has seen Uber shift workloads to Arm-based “Ampere” Oracle cloud instances.
Previously, it had run 95% of its IT on-premises, with its own servers helping to power some meaty workloads including over 10 million predictions per second across the popular rideshare’s marketplace.
(Oracle was an early adopter of Arm-based Ampere semiconductors for Oracle Cloud Infrastructure. Uber appears to have heavily shifted to Ampere on OCI, which its engineers name-check regularly in the blog. Notably, in 2023 Oracle, which has also invested heavily in Ampere, said that its ubiquitous Oracle Database was also now Arm-compatible.)
5,000 services tightly tied to x86
Uber’s engineers explained in the February 13 blog that shifting to a new multi-architecture environment was not entirely straightforward.
It entailed, to start with, ensuring host-level software was compatible with Arm, making code repositories and container images compatible with Arm and executing a tricky migration “workload by workload.”
Uber’s container image stack relied on a centralized Buildkite (a continuous integration and continuous delivery or “CI/CD” platform) pipeline that it dubs Makisu, which couldn’t cross-compile for Arm.
Moreover, Uber had 5,000 services which had “several custom build steps tightly coupled to Makisu’s build flow. So, migrating away from Makisu was a substantial undertaking,” its engineers explained this week.
They subsequently decided to “bootstrap” an Arm-compatible version of Makisu using Google’s open-source Bazel – a decision that they said was “...driven by Bazel’s ability to build container images for architectures different from the host it runs on [using] OCI container image rules.)
Join peers following The Stack on LinkedIn
A cascade of dependencies resulted in a real engineering challenge that they detail here. And shifting to multi-architecture builds involved some “significant downside” they admitted – including “the doubling of build costs, as container images needed to be built for both architectures.”
"With more than 400,000 container image builds per week (at the time of writing), this additional cost quickly became substantial. However, despite the increase in build costs, the unit economics of transitioning to Arm still made it worthwhile" Uber's Andreas Lykke and Jesper Borlum
(They had also teamed up with Ampere to share details later last year, detailing how Uber split the migration into four phases - SKU qualification, infrastructure readiness, technology readiness, and production rollout (shown below), after evaluating four different classes of workloads: stateless, stateful, batch, and AI/ML, saying that "while the overall migration process was serial, we could operationalize some work across all workloads at relatively the same time depending on the phase.")
The blog is the latest in a refreshingly detailed and candid series of updates from Uber’s team on its architectural shifts over the years – and further emphasises Arm’s growing popularity to power CPU workloads.
In 2023, for example, Uber migrated 4,500 stateless services to a new platform as part of its ongoing transformation that involved setting up ways to maintain and manage stateless microservices at scale. (Each week, those microservices are deployed more than 100,000 times by 4,000 engineers and many autonomous systems, it explained at the time.)
As part of that programme, Uber switched from its in-house development “µDeploy” platform to a new multi-cloud setup “Up”, moving “about 2,000,000 compute cores to the new platform” in 2022. The shift teed it up for the cloud migration and meant that “service teams are largely distanced from the infrastructure detail of this transition” it it put it.
It has also described its mammoth Hadoop migration.
The blog comes a week after Uber’s Q4 earnings call, in which it reported record results. Quarterly revenues were $12.0 billion, up 20% year-on-year. Net profit for the quarter was $6.9 billion (which admittedly “includes a $6.4 billion benefit from a tax valuation release”).
Annual revenues were $43.9 billion. CEO Dara Khosrowshahi said: “Our performance has been powered by rapid innovation and execution across multiple priorities, including the massive opportunity presented by autonomous vehicles. We enter 2025 with clear momentum and will continue to be relentless against our long-term strategy.”
