| # DockerBuild | |
| `infra.tools.dockerbuild` is a management and execution environment for Chrome | |
| Infrastructure's Docker toolchain build environment. Fundamentally, it handles: | |
| - Generation and internment of Chrome Infrastructure Docker build images. | |
| - Specific build instructions for Python wheels. | |
| - Entry point for build operations using Docker toolchains. | |
| DockerBuild is complex. It justifies this complexity by solving some annoying | |
| problems: | |
| - Cross-compiling is really useful. It is difficult to have continuous builders | |
| for each supported platform, especially the more esoteric ones. Adding support | |
| for a new platform is decoupled from actually having that platform running. | |
| - Cross-compiling Python wheels is nuanced and full of errors. Python build | |
| scripts, in general, do not really accommodate cross-compiling without hacks. | |
| These hacks, and the overall logic, should be enshrined somewhere. | |
| - Building the Python `cryptography` wheel for Linux really sucks. This logic | |
| and all of its prerequisites should be enshrined in a script somewhere. | |
| - In general, it is really good to zip up build toolchains and environments | |
| and use them hermetically instead of relying on the configuration of a | |
| builder somewhere. | |
| ## DockerBuild Images | |
| DockerBuild images are based on | |
| [dockcross](https://github.com/dockcross/dockcross) project images. Each | |
| `dockcross` image bundles a platform's toolchain with an entry point script | |
| which can be used to execute commands within that toolchain. The script, when | |
| run, mounts a directory to `/work` within the Docker image and invokes the | |
| provided command on that directory. This allows a user to almost-seamlessly | |
| use the `dockcross` image's toolchain to operate on local files. This, in turn | |
| makes it really easy to cross-compile. | |
| DockerBuild extends `dockcross` images to add: | |
| - A host and cross-compiled Python environment. | |
| - A host Perl installation. | |
| - Several packages in support of generation of complex Python wheels. | |
| See [dockcross.py](dockcross.py) for more details. | |
| DockerBuild images, like `dockcross` images, can be run using a generated entry | |
| point script. This can be created by running the image: | |
| ```bash | |
| docker run <image-name> > ./entrypoint.sh | |
| chmod +x ./entrypoint.sh | |
| ./entrypoint.sh | |
| ``` | |
| Alternatively, the script can be run through `dockerbuild` using the `run` | |
| subcommand. | |
| ## Note on Windows and Mac | |
| It is important to note that Docker image environments only exist for Linux | |
| systems at the moment. Chrome Infrastructure's build strategy for Windows and | |
| Mac is to actually perform the builds on real Windows and Mac systems. Support | |
| for Windows and Mac Python wheels is built into DockerBuild, but serves mostly | |
| as a wrapper around using `pip` to download pre-generated packages and upload | |
| them to CIPD. | |
| Windows and Mac support are gated on the ability to have a fully-functioning | |
| Windows and Mac build toolchain running in a Linux (i.e., Docker) environment. | |
| This may be possible with a combination of `wine` and/or a Mac cross-compiler, | |
| but this is not currently explored or implemented. | |
| ## Lifecycle | |
| Walking backwards through the lifecycle of this script: | |
| - Building Python wheels requires | |
| - Python wheel source files, which requires | |
| - Downloading and storing Python sources in CIPD. | |
| - It also requires a DockerBuild image, which requires | |
| - A `dockcross` base image and a set of sources. | |
| - The sources need to be interned in CIPD for consistency. | |
| - The `dockcross` base image needs to be built independently. | |
| Minimal operation will use cached pre-built DockerBuild images and cached | |
| CIPD sources, allowing a user to avoid building everything from scratch. | |
| However, the logic to build everything from scratch is included in DockerBuild | |
| for updates and reproducibility. | |
| ### Subcommand: sources | |
| The various sources used to build Python wheels and DockerBuild images must be | |
| downloaded from the Internet and interned in CIPD packages. This ensures that | |
| any given image will be reproducible and resilient to external packaging and | |
| infrastructure changes. | |
| The `sources` subcommand scans through all sources used by DockerBuild, | |
| acquires them, and creates CIPD packages for them. Run `sources` with the | |
| `--help` flag for more information. | |
| If new sources are added, or sources without CIPD packages are identified, | |
| a warning message will be printed at the end of DockerBuild operation | |
| encouraging the user to upload source CIPD packages for future reproducible | |
| builds. | |
| ### Subcommand: docker-mirror | |
| The `dockcross` project's images are used as base images for DockerBuild's | |
| specialized images. Rather than rely on upstream Docker hosts, DockerBuild | |
| offers the `docker-mirror` subcommand to load upstream `dockcross` images and | |
| store them in our Google Cloud Docker image registry. | |
| `docker-mirror` can be run to synchronize and/or update the local images. All | |
| DockerBuild images are generated from the local images, not the upstream images, | |
| so an explicit synchronization step is required to update their `dockcross` | |
| bases. | |
| ### Subcommand: docker-generate | |
| DockerBuild's specialized Docker images can be built locally using the | |
| `docker-generate` subcommand. This will pull the base `dockcross` images | |
| and required sources and construct local DockerBuild images. | |
| Optionally, the local images can be pushed to the Google Cloud Docker image | |
| registry using the `--upload` flag. This ensures that other users of the script | |
| can retrieve and use the images without requiring them to build them locally. | |
| ### Subcommand: wheel-build | |
| DockerBuild has specific support for building Python wheels and uploading them | |
| to CIPD. This synergizes with | |
| [vpython](/go/src/infra/tools/vpython)'s CIPD wheel packaging expectations. | |
| Wheels can be built and uploaded by using the `wheel-build` subcommand, and | |
| uploaded using the `--upload` flag. | |
| Support for additional Python wheels and/or wheel versions can be added by | |
| editing [wheels.py](wheels.py) and adding entries for the new wheels. | |
| Adding support for universal wheels is easy. Adding support for **most** | |
| platform-specific binary wheels can range from easy to hacky-difficult depending | |
| on the wheel's binary requirements and suitability for cross-compiling. This has | |
| to be evaluated on a case-by-base basis, sadly. | |
| ### Subcommand: run | |
| DockerBuild offers an entry point into a DockerBuild image's toolchain | |
| environment through its `run` subcommand. `run` will mount the specified | |
| directory inside of the DockerBuild environment and execute the specified | |
| command against that directory. | |
| `run` can be used to cross-compile software (e.g., `git`) for other Infra | |
| platforms. | |
| ## Problems with Known Fixes | |
| ### Got permission denied while trying to connect to the Docker daemon socket... | |
| Fix: Add your local user to the docker POSIX group: | |
| ``` | |
| sudo usermod -a -G docker $USER | |
| ``` | |
| You need to restart your shell for this to take effect. | |
| ### Got permission denied while trying to upload CIPD package... | |
| Upload rights are restricted to troopers and members of | |
| [`vpython-wheels-uploaders`]( | |
| https://chrome-infra-auth.appspot.com/auth/groups/vpython-wheels-uploaders) | |
| First, make sure you are logged into cipd with your @google.com account. | |
| ```bash | |
| cipd auth-info | |
| cipd auth-login | |
| ``` | |
| Second, [file a bug with the Infra\>LUCI\>Admin component](https://bugs.chromium.org/p/chromium/issues/entry?components=Infra%3ELUCI%3EAdmin) to add you to this group. | |
| ### Got permission denied while downloading docker image | |
| When running commands that require docker images, such as the | |
| `docker-generate` subcommand, you need to be authenticated. | |
| It is recommended to use gcloud to handle this; try running | |
| `gcloud auth configure-docker`. Authenticating with a google.com | |
| account using `gcloud auth login` may be necessary. | |
| ### (Googlers only) Got permission denied or "Not logged in" while downloading Python packages manually | |
| DockerBuild accesses internal package registries via Artifact Registry, which requires an OAuth token. | |
| If you get a 401 Unauthorized or a "Not logged in" error during operations that download wheels or | |
| sources from `us-python.pkg.dev` (e.g. `sources` or `wheel-build`), ensure you are authenticated: | |
| ```bash | |
| luci-auth login -scopes "https://www.googleapis.com/auth/cloud-platform" | |
| ``` | |
| ### (Googlers only) Got report that the dependency is missing from repository | |
| That might happen if the dependency is not present in the root repository at https://airlock.corp.goog. | |
| Search for the package at https://airlock.corp.goog/search?type=Python&upstreamType=pyPi. | |
| If it is not there, you will need to add it to the root repository by following the | |
| screen prompts at package import or by following go/airlock-ui-guide. | |
| There also might be a case when the package was explicitly banned. In that case, try | |
| to work around by using a different package instead. | |
| ### Error response from daemon: squash is only supported with experimental mode | |
| Fix: Run docker in experimental mode (yes, this is a hack): | |
| ``` | |
| vim /etc/default/docker | |
| # Edit file such that DOCKER_OPTS includes "--experimental=true" | |
| sudo service docker restart | |
| ``` | |
| ## Examples | |
| ### Upload all sources to CIPD | |
| To upload all sources to CIPD, run: | |
| ```bash | |
| vpython3 -m infra.tools.dockerbuild --upload-sources sources | |
| ``` | |
| ### Update dockcross base image mirrors | |
| To update the Google Cloud repository's mirrored `dockcross` images, run: | |
| ```bash | |
| vpython3 -m infra.tools.dockerbuild docker-mirror --upload | |
| ``` | |
| ### Build and upload DockerBuild images | |
| To build new DockerBuild images and upload them to the Google Cloud repository, | |
| run: | |
| ```bash | |
| vpython3 -m infra.tools.dockerbuild docker-generate --upload | |
| ``` | |
| ### Regenerate all configured wheels | |
| To ensure that all configured Python wheels are uploaded to CIPD for all known | |
| platforms, run the following command: | |
| ```bash | |
| vpython3 -m infra.tools.dockerbuild --upload-sources wheel-build --upload | |
| ``` | |
| - `--upload-sources` instructs DockerBuild to upload CIPD packages for any | |
| new wheel sources that are not already locally mirrored. | |
| - `--upload` instructs the `wheel-build` subcommand to upload any wheels that | |
| are not currently present in CIPD. | |
| This can be run after adding a new wheel configuration, or after adding a new | |
| platform to support. |