Lots of infrastructure, building, and testing code is built using Python. Chrome Operations supports the deployment, management, and augmentation of basic Python installation on infrastructure systems.
Chrome Operations currently supports Python 3.8 on Windows, Linux, and Mac OSX.
Chrome Operations provides a bundled Python distribution. This Python distribution is hermetic, ignoring system packages and software. The set of packages available by default to a bundled Python version is fixed and deployed within that bundle.
Using Python bundles enables Chrome Operations and its clients to reliably create Python package sets, canary new Python distributions, and deploy bug- and security fixes independently from underlying Operating System updates. This results in reliability and long-term reproducibility of Python code and programs.
With a hermetic Python bundle, traditional methods of deploying Python packages (e.g., pip) are not available. Python hermetic bundle environments are read-only, as they must be to satisfy hermeticity. Users oftentimes need to augment the default Python package set, typically with public packages from pip or PyPi. For this reason, Chrome Operations has built a tool called vpython which allows Python programs to explicitly request their dependencies.
vpython is designed to make the specification, instantiation, and execution of VirtualEnv simple and reliable. Code that requires additional Python packages will use vpython to start with a read-only hermetic python bundle and augment it with specific packages in a disposable VirtualEnv, producing a final Python environment that is fully specified and completely under their control.
POSIX systems typically include their own version of system Python and a set of associated packages composed of:
apt-get, dpkg).pip.Because these sets of packages widely vary with the version and lifecycle of a given system, they are not suitable for hard dependencies for infrastructure, build, or test code.
Further, upgrading system Python involves upgrading all of the system software and packages that depend on that Python. This makes system Python upgrades necessarily heavy operations instead of the quick low-risk agility enabled by using non-system bundles.
Python versioning and dependencies have traditionally been managed differently on different platforms. This is both a consequence of the fundamental differences between those platforms and the native offerings of those platforms:
depot_tools bootstrap.For security- and bug-related reasons, systems and users require a higher standard of Python package management than that natively provided by the myriad systems that Chrome Operations supports. Chrome Operations is solving this by using hermetic deployable Python bundles built on top of CIPD, using 3pp packages.
Python bundles are installed on and used in various parts of the system:
C:\infra-python3\bin\python3.exe/opt/infra-python3/bin/python3luci-config Swarming Task Template.