Buildbot end of life date is March 1, 2019. As of March 1, all Chrome/Chromium builds hosted on chromium.org will be running on LUCI.
LUCI which stands for Layered Universal Continuous Integration is a replacement for Buildbot (our existing single-threaded monolithic continuous integration system). Over time, we have been swapping out Buildbot responsibilities with LUCI services. As such, most builds today on Buildbot are already running using LUCI services. The next step is to migrate Buildbot builders to LUCI builders, with the end result to turndown Buildbot.
As a Chromium developer, read this to understand the CQ and CI user expectations and how to contact us.
As of March 1, 2019, all Buildbot masters run by Chrome Operations will be permanently shut down. All existing builders will have been migrated to LUCI before this date.
A migration guide is available to help you with migrations. Please reach out to the Chrome Operations team at firstname.lastname@example.org for references and scheduling or timeframe info.
The existing Buildbot builders are not running on the LUCI stack, as such, they are still prone to Buildbot limitations such as single process scheduling and deployment issues (restarts needed) to name a few. We have been in preparation to migrate builders to LUCI with as minimal impact as possible to Chromium devs.
We have started migration of existing Buildbot builders to LUCI builders in Q1 2018 and will first focus on the Chromium CQ builder sets. A Chromium CQ builder set consists of try-builders that are part of the commit queue and their corresponding waterfall (continuous integration) ci-builders. A builder set will be migrated together whenever possible. Target migration time per builder set is approx. 1 week per set with a few days of overlap with existing builders to ensure correctness and performance. Note that times can vary per builder set.
We have created a LUCI migration task force (email@example.com) comprised of Chrome Operations developers. Owners of Chromium builders are responsible for approvals for final go ahead to switch a LUCI builder to production.
No, there should be minimal to no disruption. Our migration plan is to make it as seamless as possible for Chromium devs to continue their work and use our CQ and CI systems effectively.
While we're working to make the transition as seamless as possible, there can be a few hiccups that may occur when a builder is migrated. Below are the known limitations:
All Chromium developers should already be using the new LUCI UI (ci.chromium.org). The views will continue to look the same once LUCI builders are switched to production. Sheriff-O-Matic, Gatekeeper, and Findit build integrations will all work as intended. The main differences to expect are:
Builds on LUCI use a different URL path that starts with
/p/chromium/builders/… If the build is still running on Buildbot, “buildbot” will still be part of the URL path. Also, in the Code Review UI, try-builders on Buildbot is shown with a “Buildbot” tag. The default with no tag represents a LUCI build.
In some cases, builds on LUCI can also have the following URL
ci.chromium.org/p/chromium/builds/b<buildbucket_build_id>. This occurs when the build does not have a build number which should not occur for any Chromium build
Buildbot builders will be turned down progressively and won't be turned down all at once. Once a CQ builder set has completed migration, we will schedule a date within the following month to turn down the corresponding Buildbot builders. This date will be coordinated between the development teams and Chrome Operations.
Contact the Chrome Operations Foundation team (firstname.lastname@example.org) and include email@example.com, firstname.lastname@example.org, email@example.com and firstname.lastname@example.org directly for an immediate response. Also, refer to the Contact us section below for other options to reach us.
Builder migration involves 3 major steps.
There are many ways you can help. Either through contributing to LUCI, helping with builder migrations or providing feedback. Reach out to the Chrome Operations Foundation team (email@example.com) and CC firstname.lastname@example.org and email@example.com to let us know.
We are continually making improvements to the migration process and to LUCI services. Our initial goal is to migrate all Chromium CQ builders sets first as seamless as possible for Chromium devs to continue to work unimpacted and use our CQ and CI systems effectively.
If you have issues or feedback you would like to share with us, we would love to hear it and incorporate it into our ongoing LUCI improvements.