chromium/src has millions of lines of code. Sometimes, we want to change code that has been used for years and is referenced from thousands of places throughout the codebase. These sorts of changes are referred to as LSCs (Large Scale Changes) and generally require complex planning.
This page and the referenced docs describe the policies and best practices that go into the planning, creating, and executing changes that impact many distinct packages.
Fixing old mistakes, crufty interfaces, and bad usage requires making many small changes across the codebase. Individually, these changes are tiny, but taken all together, they allow us to deprecate old interfaces and pay down our technical debt. The simpler we can make the code base, the more that developers can focus on their actual problems, instead of coping with artificial complexity because of accumulated cruft.
Although there are many ways that these changes can be handled, most will be handled by an Eng Review reviewer and a domain reviewer.
Given the cost of large scale changes (review cost often being a significant factor), we want to prioritize changes that have high value.
For example, while cosmetic changes have some positive value, we want reviewers to spend their time on higher value changes. This means that changes which are primarily cosmetic are rarely worth doing unless they interact with other automation, such as fixing a spelling mistake that then makes other refactorings work more consistently. If in doubt, send us your proposal before spending too much time on it.
Anyone making “large” changes. (Currently this is roughly defined as changes covering more than 10 distinct directories with OWNERS files.) We are aiming to enable a process where the processes and practices (using git cl split) for making sweeping changes are more broadly accessible enabling teams to more readily change the APIs they export while keeping a bound on how much time is spent coping with codebase churn.
At the highest level, we want to make sure anything going out in tens/hundreds/thousands of CLs is moving the codebase in the right direction and that it‘s worth the work it takes to get such changes tested, reviewed, and submitted. Eng Review will rarely say "This isn’t impactful enough," although it may happen (especially in cases where there is little or no distinct benefit). The more common role of the review is to ensure that the necessary documentation is in place: you will be sending CLs that are seen by dozens or hundreds of engineers who may not have any knowledge about your change. Some CL description updates and documentation can go a long way to making the process smooth.
In Chrome, we will grant the LSC Requestor the Gerrit Global Owners Approver power for the duration of their change, i.e. a member of Eng Review will approve the LSC Proposal which when then gives the LSC Requester the ability to bypass OWNERS in the interest of efficiency. (For external contributors, a Googler sponsor will act as the Global Owners Approver power holder. If you’re an external contributor, everything else is the same.) Each CL will still need a second human to review it to satisfy “two sets of eyes” code review policy requirements. This reduces the code-review burden on the rest of Chrome reviewers and increases the rate at which your change gets submitted. However, this also means that these kinds of LSCs have a high bar of being low-risk and non-controversial.
As the LSC Requestor, the process of getting the LSC fully submitted looks like:
After the LSC is fully submitted, the LSC Requestor will lose their Global Owners Approver power.
If global owners approval does not seem appropriate, we will ask you to send your changes for local approval by owners of the individual directories you're changing.
To build Chrome, we have many other repos which are included via DEPS. Most of these are under the ownership of the Chrome org and this process (and have identical policies). But, some are not. For those that are under Chrome, this policy applies to all repos. V8 and Skia are not yet included in this process.
As a LSC Requestor, you can split your CLs across these boundaries.
Do I have to use this process? This depends on how large your change is—specifically, how many child CLs your large refactoring will split it into—and whether all sub-CLs affect the same OWNERS.
| Number of child CLs | Action |
|---|---|
| >= 30 | Follow the Chrome LSC Workflow (i.e. create an LSC doc, get domain approval, email lsc-review@, wait for approval). |
| < 29 | No requirements |
What do I have to do? Follow the steps at Chrome LSC Workflow.
How long will this process take? You should expect an initial response from a cleanup approver within two days.
I'm in a hurry; Is there a fast track? Eng Review may fast-track low-risk changes, with a response estimated between a few hours to one business day. If you‘d like to request fast tracking, email lsc-review with a link to the doc that you created end of Chrome LSC Workflow and we’ll get to it as quickly as we can.
Why this process? Changes that operate broadly across the codebase affect many engineers and teams and these changes may generate discussion on the change or the best way to roll it out. This process is in place to make sure that these changes are useful, well-communicated, and to minimize the risk of having to attempt difficult rollbacks.
I have an idea for a change, but I’m not sure who to ask about it? Definitely email the lsc-review@chromium.org list. We should be able to give you at least a vague idea of whether your idea has merit and who to approach as a domain expert to move ahead.
My LSC touches third_party, do I have to do anything special? These changes are likely to be rare. If it does occur, please obey the normal rules for third_party changes. Be aware that some parts of third_party have an external codebase as their source-of-truth, and so are mechanically generated (e.g. by Copybara [external, internal, Googlers only]). This means that your changes will be overwritten by the next import.
My proposal was accepted for “local approval”, now what? Get your CLs submitted. For changes at this scale, it's usually best to use Find Owners and Auto-Submit. If anything surprising comes up during this process, or you need to ask questions to someone familiar with the process, try your assigned committee reviewer (see your LSC document). As with any wide-scale change, you should also consider announcing it on a mailing list that covers the target audience.
My proposal was accepted for “global approval”, now what? Contact your Eng Review reviewer: they should have granted you (or your Googler sponsor) Global Approver power. The techniques for getting these submitted via global owner approval vary based on what is being changed, your approver will know what's what for your particular LSC. As with any wide-scale change, you should also consider announcing it on a mailing list. Any coworker or contributor can be your second-set of eyes to mass Code-Review +1 all of your CLs.
My proposal requires multiple large changes, do I need multiple LSC docs & approvals? No. You can use a single LSC document to describe your entire series of changes even if you are asking for a mix of local and global approvals. Be sure to break each change out into an easily identifiable step in the document.
How do I deal with backsliding? Backsliding (others submitting changes which reverse the intended change) is something which cannot be completely avoided in all cases. A useful techniques to reduce them is to use Tricium (see go/luci/tricium [internal, Googlers only] for details) or Presubmit.
Chrome’s LSC process was introduced to reduce the use of self-code-review to bypass OWNERS as we moved to make code review mandatory. It (and this doc) were heavily copied from a Google-internal process that has been in place for more than a decade. We needed an improved workflow for submitting large scale changes (i.e. refactorings or architectural changes) across a big codebase (e.g. chromium/src). Because it had been so slow to get OWNERS approval for dozens of directories across dozens of CLs, folks in chromium/src had the convention of code-review from a main owner and then TBR= every other owner.