This document is for CrOS developers & users only.
Some random concepts as names might be overloaded leading to confusion.
repoprogram that people run directly.
The repo launcher is the program that developers run directly. It isn‘t the full repo source, it is just a wrapper to set up an initial repo client checkout by checking out the manifest & the repo sources. Once that’s been initialized, the launcher serves as a trampoline to reexec itself using the local copy of the repo source checkout under
NB: You should never mess around with
.repo/repo/ as repo will reset any changes you make in there, or it will wedge/break it requiring manual fixing. See the updating repo section for more details.
For CrOS, developers get
repo by putting depot_tools into their
$PATH. The copy in depot_tools has a single change to point it to our fork of the repo source, and it saves developers from having to install it themselves.
Many distros include packages to install it at
/usr/bin/repo, and the official Android documentation suggests downloading & installing it under e.g.
~/bin. Those launchers will point to the upstream repo tool. While not strictly required, CrOS users should stick to the depot_tools copy.
There are two source repositories to be aware of:
repofrom standard locations.
The CrOS fork used to contain a lot of customizations, but nowadays is almost exactly the same as upstream. We are actively working to reduce the differences (we‘re down to 1 patch), and we do not want to add anymore custom code. That means any changes/requests should be sent to the upstream repo tool project for implementation, and once they’re merged+released there, we can update our fork to the new release.
NB: For CrOS bots/builders, see the updating bots section.
The repo tool will keep itself up-to-date automatically. There is usually no need for users to ever worry about this. Whenever you run
repo, it will check once/24hrs to see if there‘s an update. If there isn’t, then it won't try again for another 24hrs.
If you know a new release was made and you're itching to try it out immediately, you can run
repo selfupdate to force it.
This guidance applies equally to rollbacks -- repo will automatically roll itself back as neeeded; see that section for more details.
If the repo launcher version is increased, we need to update our copy of the launcher to avoid confusing users. This does not happen with every repo release as the repo launcher is updated much less frequently.
Ignoring unintended bugs, it is completely safe to have a new repo launcher with an old release of repo sources. It is also safe to have an old repo launcher with a new release of repo, but this combination will trigger user-visible warnings.
So if you‘re aware that a new repo launcher version has been released, you should try to update it in the various places well in advance of pushing the new release. Developers often don’t update their checkouts every day, but the repo release is self-updating, so when a new release is pushed, they'll get it automatically.
If a repo release needs to be rolled back for some reason, you usually do not have to worry about rolling back the repo launcher version. This scenario has yet to happen in the history of the CrOS project :).
You can easily check to see if this is needed by running & comparing the version output of the launchers.
repo$ ./repo --version <repo not installed> repo launcher version 2.4 <-- compare this line to ... (from ./repo) ... depot_tools$ ./repo --version <repo not installed> repo launcher version 2.4 <-- ... this line! (from ./repo) ...
Here are all the places you'll want to update.
REPO_URLcustomization to point users to the CrOS repo fork! NB: Update the
repo_launcherfile instead of
NB: Only CrOS build/infra developers have access to push to this repo. Do not attempt to do so if you aren‘t on the team. Well, you can try, but it’ll fail, so maybe don't be surprised when it does?
You will need to get a checkout of the two repo source repositories. The snippet below shows how to initialize things, but all other examples will simply assume your working directory is a repo source checkout.
# Our copy of repo. ~$ git clone https://chromium.googlesource.com/external/repo ~$ cd repo # Upstream repo. ~/repo$ git remote add upstream https://gerrit.googlesource.com/git-repo ~/repo$ git fetch upstream
Since all changes happen in the upstream repo tool project, we don‘t utilize Gerrit for updates to our fork anymore. The only exception is if we actually need to make a CrOS-specific change to our code base, but that hasn’t happened in years. All work should go through upstream first.
While good git practices say to always fast-forward branches, non-fast-forward pushes to any of these branches will not break users or bots. The repo tool will safely checkout whatever commit a branch is at.
This is not a recommendation to be cavalier in branch management, but a protip for extraordinary cases like rollbacks.
These are the branches in our repository that we care about.
stable: This is production. Push here carefully. The branch must always have a signed tag in order for repo to use it.
main: This is staging.
upstream/*: These refs are kept exactly in sync with upstream. They are not used by any tooling. They exist as a convenience to compare the state of our fork to upstream at any time. Thus they may safely point at the exact same refs as upstream even if our own branches are behind.
Upstream branches are automatically mirrored to our fork via GoB copy configuration rules, so the
upstream/* namespace is always up-to-date.
Upstream tags use standard names like
v2.4. CrOS specific tags append a
-cr# suffix to keep them distinguished. If we make local changes, we increment the suffix.
So if our branch is based on/has merged upstream
v2.4, we'll use
v2.4-cr1. Any changes before merging the next upstream release would be e.g.
Tags may be safely pushed/mirrored at any time. They will never have a direct effect -- repo watches branches, not tags. So if upstream releases
v2.5, but our branches are still on
v2.4-cr1, the new
v2.5 tag may be safely pushed to our fork.
To mirror upstream tags to our fork:
# This will fetch the tags from upstream. $ git fetch --tags upstream # Check which tags will be pushed to our fork. $ git push origin --tags -n # Do the actual push! $ git push origin --tags
Once upstream has made a new release, we need to merge it into our fork. See the previous sections for syncing the upstream branches & tags first.
The example below will assume the state:
mainbranch is sitting at
~/repohas a local
mainbranch checked out at the same state as the remote
origin/mainbranch (and is tracking it).
stablebranch is sitting at
mainbranch and create
# Merge in the new release to our local main branch. $ git merge --log=1000 v2.5 # If there are conflicts, git merge will tell you that they need to be resolved. # You should resolve them at this point :). If there are no conflicts, skip to # the next step. Once you resolve the conflicts, commit them. $ git commit # Your editor should have opened at this point to generate a commit message for # the merge commit. You can look at previous releases (git log -1 v2.3-cr1) for # example formats. If you resolved conflicts, you should note them here. <save the commit message & exit your editor> # Sign the new release. The script defaults to signing HEAD. # NB: This requires access to official repo keys. See go/repo-release. # NB: --force is required since ChromeOS tag naming convetion is different than # upstream. $ ./release/sign-tag.py --force v2.5-cr1 # Push the new tag. This is safe to do as you're only pushing the tag, not # updating any branches. See the next sections for those steps. $ git push origin --tags -n # If the -n output looked correct, push for real. $ git push origin --tags
In the sections below, we use a
^0 syntax that you might not be familiar with. This is to distinguish between “lightweight” (a.k.a. “normal” or “plain”) and “annotated” (a.k.a. signed) tags. The Git manual has a section explaining things, but it is vitally important that you always use this syntax when pushing to branches.
If an annotated tag is pushed to a branch, both git & repo get extremely upset, and this will break users & bots. The
^0 syntax refers to the commit that the signed tag is signing and that is what branches must always track.
NB: For end users/developers, see the updating users section.
NB: This documentation applies to builders using LUCI recipes. Legacy builders, most notably release & branch builders, do not run
repo selfupdateexplicitly, so rollout to them will be gradual. There are no plans to fix this since we're working to migrate all legacy builders to recipes.
CrOS builders will explicitly run
repo selfupdate at the start of all of their builds. This makes sure they're always running the latest published version, otherwise the random 24hr rollout can be quite confusing for everyone. For example, if a new release causes problems, only the bots that pick it up will exhibit that problem, and CQ/release builders will fail inconsistently. This will in turn cause developers/oncall to think specific bots are flaking which will in turn cause delays in attributing problems to repo itself.
After the builder runs
repo selfupdate, it will not automatically update itself again during the build (since it resets the 24hr check). So if bots are in the middle of a build and a new repo release is made, they won‘t see it until the next build starts. Thus you don’t have to worry about waiting for all bots to finish their builds just so a new release can be pushed out.
CrOS bots are configured to watch the
main branch in our repo fork, and do so without enforcing signed tags (see https://crbug.com/1055913). This allows us more flexibility in testing out new repo changes before they go into a release.
So before pushing a new release to production, push it to the
main branch, and watch the staging bots to see how they handle things. Pay particular attention to the staging CQ as they will cycle the fastest.
^0syntax is required; see the tagging sidenotes section.
# Push the new release to the main branch. The staging bots will pick this up # immediately, so start monitoring the bots. $ git push origin v2.5-cr1^0:main -n # If that looked good, push it for real. $ git push origin v2.5-cr1^0:main # If you don't have a tag but just want to test some new changes on staging # bots, you can push a commit to the main branch. $ git push origin <commit>:main -n # If that looked good, push it for real. $ git push origin <commit>:main # If things go wrong, it's perfectly safe to do non-fast-forward pushes. # Bots will rollback to the right version immediately. $ git push origin <old commit>:main -f
NB: If the new repo release contains an updated launcher, make sure to update the launcher before pushing the release to the CrOS fork, otherwise all CrOS developers will see warnings whenever they run
repothat their launcher is out of date. That will just confuse them and cause them to raise support issues with the build team.
Once a release has been vetted in staging, it's time to roll it out to prod. Pushing to the
stable branch will make the release available to users and bots immediately. So make sure you want to do this :).
Signed tags are required in order to push to production. Any commits pushed to
stable that are not signed will be ignored and repo will fallback to whatever closest signed tag it can find in the branch.
NB: Releases to production must be treated like any other production service. That means:
- Avoid new releases during off-hours. If something goes wrong, staff/oncall need to be active in order to respond quickly & effectively.
- Stick to normal working hours: Mon - Thu, 9:00 - 14:00 PT (i.e. MTV time).
- Avoid US holidays (and large international ones if possible).
- Follow the normal Chops/Google production freeze schedule.
Before you push the release, make sure you let people know! This helps users find out about new features, but it also helps devs/oncall triage failures faster that might be related to the new release.
^0syntax is required; see the tagging sidenotes section.
# Push the new release to the main branch. This is to make sure things are # in sync with the stable branch, and allow for any last minute checks via # e.g. gitiles. NB: This will not affect any bots or users. $ git push origin v2.5-cr1^0:main -n # If that looked good, push it for real. $ git push origin v2.5-cr1^0:main # Push the new release to the stable branch. This makes it live in production! $ git push origin v2.5-cr1^0:stable -n # If that looked good, push it for real. $ git push origin v2.5-cr1^0:stable
If a new release is causing a problem, the best course of action is most likely to rollback to the previous working version.
Non-fast-forward pushes to the
stable branch is perfectly safe. Repo will always sync to the latest branch state exactly. As noted in the production section, only signed commits will be respected!
Make sure all rollbacks are coordinated with go/cros-oncall.
# Push the an old release to the stable branch. This makes it live in production! $ git push origin v2.4-cr1^0:stable -f -n # If that looked good, push it for real. $ git push origin v2.4-cr1^0:stable -f
Yes! If the
stable branch is not pointing to a signed commit, then repo will search backwards through the branch's history for the first signed commit. It will warn about this state before falling back.
Nope! Our staging bots are configured to use whatever commits are in the
main branch even if they aren't yet signed. This allows more CrOS infra developers to help in testing new changes.
Contact the CrOS infra group for assistance.
See go/repo-release for details.
We don't need to rebase our branches. Merging commits is fine.
Of course, rebasing our local branches isn‘t problematic for users/bots as they’ll gladly pull down any rewritten refs. If it helps to clean up history or something, it's permissible.
Please send requests to the upstream issue tracker so they can be triaged & implemented.
Nice work! Please report issues to the upstream issue tracker so they can be triaged & addressed.
THe upstream repo project is a cross-collaboration between CrOS, Android, and other teams & customers who utilize repo in their project. Each project tends to have a subteam who is responsible for their own users.
In CrOS, issues that affect CrOS developers & systems are supported by folks across the CrOS Infra team. It's a group effort.
Awesome, we love when people take initiative. Please send your CL upstream and have it merged there.
Once upstream makes a new release, we'll roll it back into our fork.
Sorry, but we don't do that anymore. It is expensive to deviate from upstream which is why we want everyone to merge there first. Plus, they can do reviews for general appropriateness of the request.
Zero. When you run
repo sync, it takes care to not delete modified checkouts.
Check out the upstream repo tool project. It has user & developer & internal documentation.
Check out our source checkout documentation.