| # Merge Request Process |
| |
| [TOC] |
| |
| ## tl;dr |
| |
| * Release managers (and delegates like the security team) must review all |
| merges made to release branches |
| * Merge criteria become more strict as the stable release date approaches; use |
| Chromium Dash's [Branches page](https://chromiumdash.appspot.com/branches) |
| to understand which branches are active and what merges are acceptable for |
| each branch |
| * Ensure your change is [safe to merge](#verifying-eligibility-and-safety) |
| before initiating the merge review process unless it's time-sensitive |
| * Use Chromium Issue Tracker's [project queries](#monitoring-merge-requests) to |
| track your approved merges as well as your pending requests |
| * Use Gerrit or git to land your merge only after it's been approved |
| |
| ## Introduction |
| |
| Chromium is a main-first development team; generally, all code should land on |
| main then roll out to stable users only after the milestone containing the code |
| is branched, stabilized and shipped to the stable channel (to learn more about |
| the release cycle, click |
| [here](https://chromium.googlesource.com/chromium/src.git/+/main/docs/process/release_cycle.md)). |
| This is because merging (also known as cherry-picking) code to an older release |
| branch introduces risk and costs time across the team. However, there are times |
| when the benefits outweigh the costs and a merge might be appropriate, e.g. to |
| fix a web platform regression, address a crash or patch a security |
| vulnerability. |
| |
| To ensure we make the right decisions, release managers leverage a merge review |
| process to evaluate each request. They'll ask questions about the reason you |
| would like to merge a change and the risk of the merge itself, and you'll work |
| together to make a judgement call on whether or not the merge should be approved |
| or rejected. |
| |
| Generally, merges follow these high-level steps: |
| |
| * Developers update bug with relevant details and request a merge by updating |
| the *Merge-Request* field with the requested milestones, then wait for |
| review. |
| * Release managers and automation review and approve, reject, or ask questions |
| about the merge within two business days. |
| * Developers wait for review and, if approved, land the merge ASAP. |
| |
| For details on each step, see below. |
| |
| **NOTE:** Because security issues (identified with *Type=Vulnerability*) follow |
| a more complex flow, you may simply mark security issues as *Fixed* in the Issue |
| Tracker and [automation](#security-merge-triage) will handle the remainder of |
| the merge request process flow for you; simply process the merge if it is |
| requested and approved. |
| |
| ## Requesting a merge |
| |
| ### Verifying eligibility and safety |
| |
| Before requesting a merge, first ensure your change is a good merge candidate: |
| |
| * Ensure it meets the merge criteria (via |
| [Chromium Dash](https://chromiumdash.appspot.com/branches)) of the |
| branch(es) you'd like to merge to; merge criteria become more strict the |
| older the branch is, more details on criteria |
| [below](#merge-criteria-phases) |
| * Verify merging the change to an older branch would be safe, e.g. unlikely to |
| introduce new regressions, no major merge conflicts, automated test coverage |
| present, etc; chat with your TL for input if you're not sure |
| * Confirm your change fixes the issue at hand, preferably by testing on and |
| monitoring the canary channel for 24 hours post-release (see |
| [Chromium Dash](https://chromiumdash.appspot.com/commits) to determine if |
| your change has shipped) |
| |
| * You may skip this step if a release manager or security team member has |
| told you that the merge is urgent, e.g. is actively blocking a release |
| |
| ### Updating Chromium Issue Tracker |
| |
| Next, update the bug (generally the bug being fixed by the merge) with the |
| following information present and accurate: |
| |
| * Title and description clearly describing the bug being fixed |
| * Priority (*Priority*), OS (*OS*) and target milestone(s) (*Milestone*) |
| fields are set |
| * Consider all available data when setting the priority, such as existing |
| metrics for usage of a broken feature, to ensure important merges are |
| not missed. Consider collecting new data, such as by landing new metrics |
| and estimating severity with pre-stable data. For Web Platform changes, |
| [compat |
| tools](https://www.chromium.org/blink/platform-predictability/compat-tools/) |
| such as |
| [UseCounters](https://www.chromium.org/blink/platform-predictability/compat-tools/#usecounter) |
| , [Cluster |
| Telemetry](https://www.chromium.org/blink/platform-predictability/compat-tools/#on-demand-crawl) |
| , and |
| [HTTPArchive](https://www.chromium.org/blink/platform-predictability/compat-tools/#the-http-archive) |
| may be useful. |
| |
| * Owner, generally the person requesting / performing the merge |
| * [Release block label](./release_blockers.md) if applicable (*ReleaseBlock* |
| field*) |
| * Issue status: |
| |
| * Fixed: You're confident the issue is fixed on main, e.g. you've locally |
| built and tested the issue, no additional crash reports are generated |
| after the fix was released, etc (most issues) |
| * In Progress (Accepted): Diagnostic merges only, e.g. to merge code to |
| track down the root cause of an issue that only exists on branch |
| |
| ### Setting the Merge-Request field |
| |
| Once you've verified all the above, you're ready to request a merge! Simply |
| update the issue's *Merge-Request* field with the milestone(s) you'd like to |
| merge to. Within the next ~15 minutes, automation will create a new Merge |
| Request issue, link it to the original (parent) issue, copy relevant metadata, |
| and assign the issue to you. |
| |
| If you don't have an existing Chromium Tracker issue, you can request a merge by |
| filling out the bug template at |
| [crbug.com/merge-request](https://crbug.com/merge-request). |
| |
| ### Submitting the Merge Request |
| |
| Follow the instruction in the new Merge Request issue. You will be asked to |
| update a number of custom fields (found in the issue's sidebar) and provide a |
| rationale for the merge request. After you have updated and verified the |
| request, you can submit the request for review by assigning the issue to |
| *merges@chromium.org*. |
| |
| ## Monitoring merge requests |
| |
| After assigning the issue to *merges@chromium.org*, automation will evaluate |
| your request and either approve it or pass it along to a release manager for |
| manual evaluation; see [here](#merge-request-triage) to learn more about this |
| automation. If manual review is required, release managers strive to answer all |
| merge requests within two business days, but extenuating circumstances may cause |
| delays. |
| |
| At this point, following along via bug comments sent by email will always keep |
| you in the loop, but you can also use the following queries in the Issue Tracker |
| to track your merges: |
| |
| * [Approved and TBD merges](https://issues.chromium.org/issues?q=assignee:me%20customfield1223087:(Approved%20%7C%20TBD)): |
| Merges that require your follow-up, either by landing the relevant merge (if |
| approved) or determining whether or not a merge is actually required and if |
| so, requesting it (if TBD) |
| * [Pending merges](https://g-issues.chromium.org/issues?q=assignee:me%20customfield1223087:Pending): |
| Merges that are pending review by automation and need additional action from |
| you. Frequently, issues remain in the state because developers have failed |
| to update the merge request issue with all required information, or the |
| request was not assigned to **merges@chromium.org**. Please follow the |
| instructions in the issue to complete the merge request, and reach out to |
| a release manager if you have any questions. |
| * [Requested |
| merges](https://issues.chromium.org/issues?q=assignee:me%20(-customfield1223134:none%20%7C%20customfield1223087:Review)): |
| Merges that are waiting for input from release managers or automation; feel |
| free to ping bugs that sit in this queue for two business days (assuming you |
| verified that the change was already deployed to canary ahead of requesting |
| a merge) |
| * [Rejected and NA merges](https://issues.chromium.org/issues?q=assignee:me%20customfield1223087:(Rejected%20%7C%20NA)): |
| Merges that were either rejected by release managers, or not applicable to |
| be merged; generally, no action is needed for these items unless you |
| disagree with a merge's rejection and wish to escalate |
| * [All merges](https://issues.chromium.org/issues?q=assignee:me%20(-customfield1223087:none%20%7C%20-customfield1223134:none)): |
| Includes every possible merge state, useful when wanting to find an item you |
| considered for merging but can't recall the state it was last in. |
| |
| For a description of each label used to track the merge process, see the |
| appendix [below](#merge-states-and-labels). |
| |
| ## Landing an approved merge |
| |
| Once your merge has been approved for a given milestone (via the release manager |
| or automation updating the *Merge* field with *Approved-###*), you have two |
| options to land the merge: |
| |
| * Gerrit UI, easiest for clean cherry-picks or those requiring only minor |
| changes |
| * git, for more complex cherry-picks and / or when local verification may be |
| beneficial |
| |
| Regardless of which method you choose, please ensure you land your cherry-pick |
| ASAP so that it can be included in the next release built from the branch; if |
| you don't merge your cherry-pick soon after approval, it will eventually be |
| rejected for merge. |
| |
| **NOTE:** Ensure you link to the bug that has merge approval for the relevant |
| milestone. Not linking to a bug that has approval can cause delay to your CL |
| landing. If the merge request is for a single change, add `Fixed: <bug number>` |
| to the change description. If the merge request is for multiple changes, add |
| `Fixed: <bug number>` to the final change description, `Bug: <bug number>` |
| to all other change descriptions. |
| |
| Once the cherry-pick has landed, a bot will update the *Merge* field with |
| *Merged-###* label. If `Fixed:` was used, the bug will be closed. |
| |
| ### Using Gerrit UI |
| |
| Select the "..." button in the Gerrit UI, then choose "Cherry Pick". When |
| prompted for a branch, enter *refs/branch-heads/####*, where #### corresponds to |
| the release branch you are merging to (available on |
| [Chromium Dash](https://chromiumdash.appspot.com/branches) in the "Chromium" |
| column). |
| |
| Once the cherry-pick CL is prepared, you can bypass code review (but not OWNERS |
| approval) within 14 days of the original change by adding the Rubber Stamper bot |
| (rubber-stamper@appspot.gserviceaccount.com) as a reviewer. If the CL meets the |
| [Rubber Stamper criteria](https://chromium.googlesource.com/infra/infra/+/refs/heads/main/go/src/infra/appengine/rubber-stamper/README.md), |
| the bot will vote *Bot-Commit+1* to bypass code review. If the CL is marked |
| *Auto-Submit+1*, the bot will also submit the CL to the CQ on your behalf. |
| |
| ### Using git |
| |
| The commands below should set up your environment to be able to successfully |
| upload a cherry-pick to a release branch, where *####* corresponds to the |
| release branch you are merging to (available on |
| [Chromium Dash](https://chromiumdash.appspot.com/branches) in the "Chromium" |
| column): |
| |
| ``` |
| $ gclient sync --with_branch_heads |
| $ git fetch |
| $ git checkout -b BRANCH_NAME refs/remotes/branch-heads/#### |
| $ git cl upstream branch-heads/#### |
| $ git cherry-pick -x COMMIT_HASH_MAIN |
| $ gclient sync |
| ``` |
| |
| From here, your environment should be ready to adjust the change as required; |
| use ninja to build and test your changes, and when ready upload for review: |
| |
| ``` |
| $ git cl upload |
| ``` |
| |
| **Adjust the change description** to omit the "Change-Id: ..." line from |
| original patch, otherwise you may experience issues when uploading the change to |
| Gerrit. Once complete, use Gerrit to initiate review and approval of the merge |
| as TBR has been discontinued. |
| |
| Other tips & tricks when merging with git via release branches: * Consider using |
| multiple working directories when creating the release branch * Editing the |
| change description to denote this is a merge (e.g. "Merge to release branch" at |
| the top) will help reviewers distinguish between the cherry-pick and the |
| original change |
| |
| ## Merge automation |
| |
| The Chrome team has built automation via |
| [Blintz](https://www.chromium.org/issue-tracking/autotriage), formerly known as |
| Sheriffbot, to assist in several merge flows: security merge triage, general |
| merge request triage, and preventing missed merges. |
| |
| ### Security merge triage |
| |
| Given the additional complexity inherent in security merges, the security team |
| has built custom automation to handle this flow end to end; simply mark any |
| security issue as *Fixed* and Blintz will evaluate applicable milestones, |
| determine if merges are required and automatically request them if need be. |
| |
| ### Merge request triage |
| |
| To reduce release manager toil, Blintz performs the first pass review of all |
| merge requests; it may auto-approve the issue if it can detect the issue meets |
| the right criteria for the current merge phase (e.g. a ReleaseBlock-Dev issue |
| requesting a merge before beta promotion). If it cannot decide, it will pass the |
| issue to a release manager for manual review. |
| |
| Blintz only takes action on merge requests when the merge request issue (not the |
| original issue) is assigned to *merges@chromium.org*. |
| |
| ### Preventing missed merges |
| |
| To avoid the situation where a critical issue is present on a release branch |
| but the fix isn't merged, Blintz evaluates all release-blocking issues |
| targeting a milestone that has already branched and updates the *Merge* field |
| with *TBD-##* if the issue was marked as fixed after branch day but hasn't been |
| merged. When this occurs, developers should evaluate the issue and either |
| request a merge if required (e.g. the fix did miss the release branch point) by |
| updating the *Merge-Request* field, or update the *Merge* field with *NA-###* |
| (e.g. the fix is present in the release branch already or the merge is |
| unnecessary for other reasons). |
| |
| ## Appendix |
| |
| ### Merge criteria phases |
| |
| The table below describes the different phases that each milestone progresses |
| through during its release cycle; this data is available via the Chromium Dash |
| [front-end](https://chromiumdash.appspot.com/branches) and |
| [API](https://chromiumdash.appspot.com/fetch_milestones). |
| |
| | Branch Phase | Period Begins | Period Ends | Acceptable Merges Include Fixes For: | |
| | --- | --- | --- | --- | |
| | beta | M(X) Branch | M(X) Stable Cut | Non-functional issues for Finch-gated features (e.g. add metrics, fix crash), noticeable new regressions, any release blockers, any security issues, emergency string issues (.GRD changes) | |
| | stable | M(X) Stable Cut | M(X+1) Stable | Urgent new regressions (especially user reports), urgent release blockers, important security issues (medium severity or higher) requested by the security team | |
| | extended (if applicable) | M(X+1) Stable | M(X+2) Stable | Important security issues (medium severity or higher) applicable to any platform supported by Chrome Browser requested by the security team | |
| |
| ### Merge states and labels |
| |
| The table below describes the different merge states applied via a bug's |
| metadata fields. All merge states (except *Pending*) follow the form |
| *[State]-###*, where ### corresponds to the applicable milestone. If multiple |
| merges are required, these labels may appear multiple times on the same bug in |
| different states. |
| |
| | Field | Value | Step Owner | Next Steps | |
| | --- | --- | --- | --- | |
| | Merge-Request | ### | Release manager | Automation will review and either approve / reject directly, or pass the review to a release manager for manual evaluation | |
| | Merge | Pending | Issue owner | Issue owner should follow the instructions in the merge request and assign it to *merges@chromium.org* to begin review | |
| | Merge | Review-### | Release manager | Release manager will evaluate and either approve, reject, or request additional information within two business days | |
| | Merge | Approved-### | Issue owner | Issue owner should cherry-pick the fix to the appropriate release branch ASAP | |
| | Merge | Merged-### | None | N/A; merge has already been landed, no further work required for given milestone | |
| | Merge | Rejected-### | Issue owner | Issue owner should re-request a merge to escalate if they feel the merge was erroneously rejected and should be re-evaluated | |
| | Merge | TBD-### | Issue owner | Issue owner should evaluate if a merge is required, then remove *TBD-##* and replace it with *NA-##* (if no merge needed) or re-request a merge (if needed) | |
| | Merge | NA-### | None | N/A; merge is not required to the relevant milestone, no further work required for given milestone | |