Chromium OS uses the repo tool to manage its source checkout.
Some random concepts as names might be overloaded leading to confusion.
repo program that people run directly.~/chromiumos/.The repo launcher is the program that developers run directly. It isn‘t the full repo source, it is just a wrapper to set up an initial repo client checkout by checking out the manifest & the repo sources. Once that’s been initialized, the launcher serves as a trampoline to reexec itself using the local copy of the repo source checkout under .repo/repo/.
NB: You should never mess around with .repo/repo/ as repo will reset any changes you make in there, or it will wedge/break it requiring manual fixing. See the updating repo section for more details.
For CrOS, developers get repo by putting depot_tools into their $PATH. The copy in depot_tools has a single change to point it to our fork of the repo source, and it saves developers from having to install it themselves.
Many distros include packages to install it at /usr/bin/repo, and the official Android documentation suggests downloading & installing it under e.g. ~/bin. Those launchers will point to the upstream repo tool. While not strictly required, CrOS users should stick to the depot_tools copy.
There are two source repositories to be aware of:
repo from depot_tools.repo from standard locations.The CrOS fork used to contain a lot of customizations, but nowadays is almost exactly the same as upstream. We are actively working to reduce the differences (we‘re down to 1 patch), and we do not want to add anymore custom code. That means any changes/requests should be sent to the upstream repo tool project for implementation, and once they’re merged+released there, we can update our fork to the new release.
The repo tool will keep itself up-to-date automatically. There is usually no need for users to ever worry about this. Whenever you run repo, it will check once/24hrs to see if there‘s an update. If there isn’t, then it won't try again for another 24hrs.
If you know a new release was made and you're itching to try it out immediately, you can run repo selfupdate to force it.
This guidance applies equally to rollbacks -- repo will automatically roll itself back as neeeded; see that section for more details.
If the repo launcher version is increased, we need to update our copy of the launcher to avoid confusing users. This does not happen with every repo release as the repo launcher is updated much less frequently.
Ignoring unintended bugs, it is completely safe to have a new repo launcher with an old release of repo sources. It is also safe to have an old repo launcher with a new release of repo, but this combination will trigger user-visible warnings.
So if you‘re aware that a new repo launcher version has been released, you should try to update it in the various places well in advance of pushing the new release. Developers often don’t update their checkouts every day, but the repo release is self-updating, so when a new release is pushed, they'll get it automatically.
If a repo release needs to be rolled back for some reason, you usually do not have to worry about rolling back the repo launcher version. This scenario has yet to happen in the history of the CrOS project :).
You can easily check to see if this is needed by running & comparing the version output of the launchers.
repo$ ./repo --version <repo not installed> repo launcher version 2.4 <-- compare this line to ... (from ./repo) ... depot_tools$ ./repo --version <repo not installed> repo launcher version 2.4 <-- ... this line! (from ./repo) ...
Here are all the places you'll want to update.
REPO_URL customization to point users to the CrOS repo fork!~/chromiumos/manifest/.~/chromiumos/manifest-internal/.You will need to get a checkout of the two repo source repositories. The snippet below shows how to initialize things, but all other examples will simply assume your working directory is a repo source checkout.
# Our copy of repo. ~$ git clone https://chromium.googlesource.com/external/repo ~$ cd repo # Upstream repo. ~/repo$ git remote add upstream https://gerrit.googlesource.com/git-repo ~/repo$ git fetch upstream
Since all changes happen in the upstream repo tool project, we don‘t utilize Gerrit for updates to our fork anymore. The only exception is if we actually need to make a CrOS-specific change to our code base, but that hasn’t happened in years. All work should go through upstream first.
While good git practices say to always fast-forward branches, non-fast-forward pushes to any of these branches will not break users or bots. The repo tool will safely checkout whatever commit a branch is at.
This is not a recommendation to be cavalier in branch management, but a protip for extraordinary cases like rollbacks.
These are the branches in our repository that we care about.
stable: This is production. Push here carefully. The branch must always have a signed tag in order for repo to use it.next: This is staging. Used to test new repo release before being sent to production. Does not require signed tags, but won't be used unless explicitly opted-in to.master: Only used for review of CrOS-specific changes. Normally kept in sync with stable. If desired, changes can be pushed here first for sanity/review by others before being pushed to stable or next.upstream/*: These refs are kept exactly in sync with upstream. They are not used by any tooling. They exist as a convenience to compare the state of our fork to upstream at any time. Thus they may safely point at the exact same refs as upstream even if our own branches are behind.To mirror upstream branches to our fork:
# This will fetch the latest branches from upstream. $ git fetch upstream # Sanity check what branches will be pushed to our fork. $ git push origin '+refs/remotes/origin/*:refs/heads/upstream/*' -n # Do the actual push! $ git push origin '+refs/remotes/origin/*:refs/heads/upstream/*' # NB: If upstream does a rollback, some branches might have to be force pushed. # Make sure you've carefully reviewed the -n output to avoid mistakes! $ git push origin remotes/origin/stable:upstream/stable remotes/origin/maint:upstream/maint -n $ git push origin remotes/origin/stable:upstream/stable remotes/origin/maint:upstream/maint -f
Upstream tags use standard names like v2.4. CrOS specific tags append a -cr# suffix to keep them distinguished. If we make local changes, we increment the suffix.
So if our branch is based on/has merged upstream v2.4, we'll use v2.4-cr1. Any changes before merging the next upstream release would be e.g. v2.4-cr2.
Tags may be safely pushed/mirrored at any time. They will never have a direct effect -- repo watches branches, not tags. So if upstream releases v2.5, but our branches are still on v2.4-cr1, the new v2.5 tag may be safely pushed to our fork.
To mirror upstream tags to our fork:
# This will fetch the tags from upstream. $ git fetch --tags upstream # Sanity check which tags will be pushed to our fork. $ git push origin --tags -n # Do the actual push! $ git push origin --tags
Once upstream has made a new release, we need to merge it into our fork. See the previous sections for syncing the upstream branches & tags first.
The example below will assume the state:
master branch is sitting at v2.4-cr1.~/repo has a local master branch checked out at the same state as the remote origin/master branch (and is tracking it).stable branch is sitting at v2.5.v2.5 into our master branch and create v2.5-cr1.# Merge in the new release to our local master branch. $ git merge --log=1000 v2.5 # If there are conflicts, git merge will tell you that they need to be resolved. # You should resolve them at this point :). If there are no conflicts, skip to # the next step. Once you resolve the conflicts, commit them. $ git commit # Your editor should have opened at this point to generate a commit message for # the merge commit. You can look at previous releases (git log -1 v2.3-cr1) for # example formats. If you resolved conflicts, you should note them here. <save the commit message & exit your editor> # Sign the new release. The script defaults to signing HEAD. # NB: This requires access to official repo keys. See go/repo-release. $ ./release/sign-tag.py v2.5-cr1 # Push the new tag. This is safe to do as you're only pushing the tag, not # updating any branches. See the next sections for those steps. $ git push --tags -n # If the -n output looked sane, push for real. $ git push --tags
In the sections below, we use a ^0 syntax that you might not be familiar with. This is to distinguish between “lightweight” (a.k.a. “normal” or “plain”) and “annotated” (a.k.a. signed) tags. The Git manual has a section explaining things, but it is vitally important that you always use this syntax when pushing to branches.
If an annotated tag is pushed to a branch, both git & repo get extremely upset, and this will break users & bots. The ^0 syntax refers to the commit that the signed tag is signing and that is what branches must always track.
repo selfupdate explicitly, so rollout to them will be gradual. There are no plans to fix this since we're working to migrate all legacy builders to recipes.CrOS builders will explicitly run repo selfupdate at the start of all of their builds. This makes sure they're always running the latest published version, otherwise the random 24hr rollout can be quite confusing for everyone. For example, if a new release causes problems, only the bots that pick it up will exhibit that problem, and CQ/release builders will fail inconsistently. This will in turn cause developers/oncall to think specific bots are flaking which will in turn cause delays in attributing problems to repo itself.
After the builder runs repo selfupdate, it will not automatically update itself again during the build (since it resets the 24hr check). So if bots are in the middle of a build and a new repo release is made, they won‘t see it until the next build starts. Thus you don’t have to worry about waiting for all bots to finish their builds just so a new release can be pushed out.
Before pushing any upgrades to production, they need to be run through our staging instances first (ignoring “normal” emergency pushes of course).
CrOS bots are configured to watch the next branch in our repo fork, and do so without enforcing signed tags (see https://crbug.com/1055913). This allows us more flexibility in testing out new repo changes before they go into a release.
So before pushing a new release to production, push it to the next branch, and watch the staging bots to see how they handle things. Pay particular attention to the staging CQ as they will cycle the fastest.
^0 syntax is required; see the tagging sidenotes section.# Push the new release to the next branch. The staging bots will pick this up # immediately, so start monitoring the bots. $ git push v2.5-cr1^0:next -n # If that looked good, push it for real. $ git push v2.5-cr1^0:next # If you don't have a tag but just want to test some new changes on staging # bots, you can push a commit to the next branch. $ git push <commit>:next -n # If that looked good, push it for real. $ git push <commit>:next # If things go wrong, it's perfectly safe to do non-fast-forward pushes. # Bots will rollback to the right version immediately. $ git push <old commit>:next -f
repo that their launcher is out of date. That will just confuse them and cause them to raise support issues with the build team.Before pushing any upgrades to production, they need to be run through our staging instances first (ignoring “normal” emergency pushes of course).
Once a release has been vetted in staging, it's time to roll it out to prod. Pushing to the stable branch will make the release available to users and bots immediately. So make sure you want to do this :).
Signed tags are required in order to push to production. Any commits pushed to stable that are not signed will be ignored and repo will fallback to whatever closest signed tag it can find in the branch.
NB: Releases to production must be treated like any other production service. That means:
Before you push the release, make sure you let people know! This helps users find out about new features, but it also helps devs/oncall triage failures faster that might be related to the new release.
^0 syntax is required; see the tagging sidenotes section.# Push the new release to the master branch. This is to make sure things are # in sync with the stable branch, and allow for any last minute sanity checks # via e.g. gitiles. NB: This will not affect any bots or users. $ git push v2.5-cr1^0:master -n # If that looked good, push it for real. $ git push v2.5-cr1^0:master # Push the new release to the stable branch. This makes it live in production! $ git push v2.5-cr1^0:stable -n # If that looked good, push it for real. $ git push v2.5-cr1^0:stable
If a new release is causing a problem, the best course of action is most likely to rollback to the previous working version.
Non-fast-forward pushes to the stable branch is perfectly safe. Repo will always sync to the latest branch state exactly. As noted in the production section, only signed commits will be respected!
Make sure all rollbacks are coordinated with go/cros-oncall.
# Push the an old release to the stable branch. This makes it live in production! $ git push v2.4-cr1^0:stable -f -n # If that looked good, push it for real. $ git push v2.4-cr1^0:stable -f
Yes! If the stable branch is not pointing to a signed commit, then repo will search backwards through the branch's history for the first signed commit. It will warn about this state before falling back.
Nope! Our staging bots are configured to use whatever commits are in the next branch even if they aren't yet signed. This allows more CrOS infra developers to help in testing new changes.
Contact the CrOS infra discuss group for assistance.
See go/repo-release for details.
We don't need to rebase our branches. Merging commits is fine.
Of course, rebasing our local branches isn‘t problematic for users/bots as they’ll gladly pull down any rewritten refs. If it helps to clean up history or something, it's permissible.
We love feature requests! Please send requests to the upstream issue tracker so they can be triaged & implemented.
Awesome, we love when people take initiative. Please send your CL upstream and have it merged there.
Once upstream makes a new release, we'll roll it back into our fork.
Sorry, but we don't do that anymore. It is expensive to deviate from upstream which is why we want everyone to merge there first. Plus, they can do reviews for general appropriateness of the request.
Zero. When you run repo sync, it takes care to not delete modified checkouts.
Check out the upstream repo tool project. It has user & developer & internal documentation.
Check out our source checkout documentation.