TL;DR: go/vpython-and-you
Chrome Operations builds and maintains a tool called vpython (the current version is invoked as vpython3), which offers a simple, easy, reliable, and dependable mechanism for the instantiation of arbitrary user-specified Python virtual environments for Operations and client teams.
A vpython invocation looks largely like a standard python invocation, with the notable difference of the tool name. vpython accepts Python interpreter arguments and forwards them to an underying python interpreter, surrounding the invocation with on-demand VirtualEnv setup and maintenance. See Invocation for more information.
Alongside vpython, Chrome Operations also maintains several Python wheel bundles for popular or necessary packages. See the Available Wheels section for a non-exhaustive list.
vpython takes a hermetic Python bundle and augments it with bundled Python wheels using VirtualEnv to create an effective Python environment that is tailored to each script's needs.
A Python wheel is a pre-packaged formally-specified Python distribution. For more information on wheels, see Wheel vs Egg. A wheel may be either universal, written in pure Python, or binary, including binary content specialized to a specific operating system and/or architecture.
Users can expect that vpython3 will be available in PATH in a standard Chromium development or bot environment.
vpython3 is deployed:
depot_tools, through a bootstrap wrapper ( Windows, Linux and Mac).PATH:PATH in the luci-config Swarming Task Template.To reliably invoke vpython3 in both user and bot environments, Windows users and scripts should use vpython3.bat and Linux and Mac users should use vpython3.
For documentation on the vpython tool itself, see:
For information about using vpython in a Chrome Operations environment, continue reading.
A user leverages vpython by:
vpython3.In the expected case, Chrome Operations will have already created wheel packages for all of the dependencies, and the user just has to pick and choose from that list. See Available Wheels for the most recent list of available Python wheels.
Some packages or wheel combinations are non-trivial and frequently used. To facilitate their inclusion, vpython specification snippets for these packages are included in the Templates section.
Prior to critically depending on vpython, please review its Caveats.
Below are some simple walkthroughs that a user can use to get a feel for the vpython installation process.
The goal of this walkthrough is to take a script and transform it so that it works in vpython. For example purposes, the script will be very simple: it wants to use psutil to get a count of running processes on the current system.
The Python script, test.py, looks like:
#!/usr/bin/env python3 import psutil print('Number of running processes is:', len(psutil.pids()))
As-is, this script may or may not work, depending on whether or not the target system has the psutil package installed. Let's try running it on our developer system:
python3 ./test.py Number of running processes is: 1337
Great, ship it! Well, first let's try running it on a bot:
python3 ./test.py Traceback (most recent call last): File "test.py", line 3, in <module> import psutil ImportError: No module named psutil
Yikes! File an infra ticket! But wait, another bot yielded:
python3 ./test.py Number of running processes: Traceback (most recent call last): File "test.py", line 4, in <module> print 'Number of running processes:', len(psutil.pids()) AttributeError: 'module' object has no attribute 'pids'
On this system, there is a psutil, but it‘s really old and doesn’t have the pids member. Let's file an infra ticket to upgrade it on all bots ... but now some other builder is red because it depended on that older version...
Enter VirtualEnv (featuring vpython3)!
We can use vpython to download psutil and create a VirtualEnv just for this script! Because we have a separate, hermetic, and isolated VirtualEnv for this script, it will not interfere with other system installations or other scripts. Other scripts may continue to work as they always have, may use vpython themselves with the same package set (in which case this VirtualEnv will be re-used) or may choose their own package set with their own wheel versions.
vpython3 ./test.py Traceback (most recent call last): File "test.py", line 3, in <module> import psutil ImportError: No module named psutil
This makes sense - we didn‘t create a specification that includes psutil, so sans specification, vpython created a standard, empty VirtualEnv. This behavior would be observed on any of the three prior systems, though, so already we have (albeit broken) consistency down. Now, let’s add psutil to the vpython specification.
This is a single script, but we (pretend) downloaded it from an external source, so modifying it to have an Embedded specification is not an option. Let's choose to create a Script-Specific specification file for this script so we can externally describe its dependencies to vpython.
We note in the Available Wheels section that a wheel for psutil is already defined. We'll use this one for the script.
Because the script is called test.py, we edit a file called test.py.vpython3 in the same directory as test.py:
python_version: "3.11" wheel: < name: "infra/python/wheels/psutil/${vpython_platform}" version: "version:5.8.0" >
Now, when we run test.py through vpython, we get:
vpython3 ./test.py Number of running processes is: 1337
This will work on developer system and bots, using the same versions and packages! We have a functional script that will behave consistently on all target platforms.
Before we finish, let's update the shebang line in test.py to mark that it should be called through vpython3 instead of python3 directly:
#!/usr/bin/env vpython3 # (...)
In summary:
vpython specification file referencing the necessary wheels.vpython3.vpython (vpython3) is a thin wrapper around a Python invocation. It accepts the same command-line options as Python, and in most cases can be used in place of a direct python command invocation. When run, vpython:
-vpython-spec command-line flag.PATH.If the VirtualEnv for a given identifier is already established, vpython operates near-instantly and completely offline, falling through to the direct invocation of the interpreter. If the VirtualEnv does not exist or is incomplete, vpython will pause at invocation to create the specified VirtualEnv. This pause is generally proportional to the number of packages in the specification, and is on the order of seconds.
The vpython specification is a text-format protocol buffer file that follows spec.proto.
Generally, a specification will be very simple, naming a Python interpreter version and the set of CIPD packages to use with it. Some environments require additional or conditional logic to setup, and may include other message fields.
For more information on how specification files are identified, see the vpython tool documentation.
All dependencies used by Chrome Operations vpython are pulled from CIPD, a Chrome Operations secure package deployment tool. Individual packages are specified as a combination of a name and version.
See Available Wheels for examples and more information.
In a CIPD vpython package:
name is a CIPD path to the wheel package. Names usually begin with infra/python/wheels, but users who build their own CIPD/Wheel packages could instruct vpython to load them from any CIPD prefix.Before thinking too much about this, it is highly likely that the package name for a package that you need is already specified in the Available Wheels section.
In order for vpython to invoke a script, it must pair it with a vpython Specification Protobuf that specifies the VirtualEnv that the script should be run in.
vpython offers a variety of options for a user to pair a specification with a script or collections of scripts. The user should choose a method that is appropriate based on their software.
Some Recommendations (see below for specifics):
.vpython3) at the root of your script collection.In all cases except Script-Specific Specification, specification will be probed only based on the realpath of the file.
If you are uncertain about which option is best for your script or project, please contact Chrome Operations.
A specification protobuf can be dropped alongside a Python script to implicitly pair vpython3 with the script.
If the script is a symbolic link, vpython will:
e.g: For symbolic link /A/B -> /C/D, vpython will first check /A/B.vpython3, then /C/D.vpython3.
For a script named foo.py, a vpython specification would appear in the same directory alongside it and be named foo.py.vpython3.
foo.py.vpython3
python_version: "3.11" wheel: < name: "infra/python/wheels/coverage/${vpython_platform}" version: "version:7.3.1" >
foo.py
#!/usr/bin/env vpython3 """This is my cool script. It does a lot of stuff. It needs "coverage" though. It runs in a VirtualEnv specified by "foo.py.vpython". """ import os import coverage print(coverage.__version__)
When this script is invoked by vpython3, its specification will be identified in the filesystem and automatically loaded.
vpython3 foo.py 4.3.4
An individual script (e.g., example.py) may include a specification in the script body, likely in a comment block or block string. The specification is read by vpython as the contents between lines containing [VPYTHON:BEGIN] and [VPYTHON:END] bookends. The contents between those bookends are interpreted as a specification text protobuf.
Comment characters are stripped from the beginning of each line.
For example:
#!/usr/bin/env vpython3 """This is my cool script. It does a lot of stuff. It needs "coverage" though. """ # [VPYTHON:BEGIN] # wheel: < # name: "infra/python/wheels/coverage/${vpython_platform}" # version: "version:7.3.1" # > # [VPYTHON:END] import os import coverage print(coverage.__version__)
When this script is invoked by vpython3, its specification will be parsed from the script content based on the presence of the bookend strings and automatically loaded.
vpython3 foo.py 4.3.4
If an individual or embedded specification cannot be found, vpython will probe walk filesystem towards root (or .gclient root) looking for a common specification file. This file must be named .vpython3 and be located in or above the directory of the invoked script.
Comment characters are stripped from the beginning of each line.
For example:
.vpython3
python_version: "3.11" wheel: < name: "infra/python/wheels/coverage/${vpython_platform}" version: "version:7.3.1" >
tools/foo.py
#!/usr/bin/env vpython3 """This is my cool script. It does a lot of stuff. It needs "coverage" though. It runs in a VirtualEnv specified by "foo.py.vpython". """ import os import coverage print(coverage.__version__)
When this script is invoked by vpython will walk up from tools/, identify .vpython3 in a parent directory, and automatically load it.
vpython3 tools/foo.py 4.3.4
An example tool which uses vpython is the Recipe Engine. It contains a fairly complicated specification that pulls in, among other things, the complete cryptography Python package.
It is available for study here.
A vpython environment can be invoked directly or with explicit support in Chrome Operations tooling.
Python scripts are invoked from recipes using the gemeric step recipe module.
Scripts can be invoked using vpython3 by replacing the Python command-line option with vpython3.
vpython3 /path/to/script.py
An explicit specification can be referenced using the -vpython-spec flag. Run vpython3 -help for more information.
If you don't provide an explicit specification (recommended), Specification Probing will be used to determine which specification your script should use.
If your script is invoking another Python script, it will likely work without modification. This is because vpython3 adds its VirtualEnv's bin/ directory to PATH during invocation, causing other python3 invocations to automatically inherit the VirtualEnv (and PATH).
However, a few standard guidelines should be followed:
sys.executable at the beginning of invocation. This prevents cases where a Python script explicitly specifies a Python interpreter in its shebang line (e.g., #! /usr/bin/python3)./usr/bin/env instead of directly referencing a Python interpreter. If you know you want to use vpython3 exclusively, you can directly reference it instead of python3.#!/usr/bin/env python3
vpython loads packages from CIPD, a Chrome Operations online package deployment service, during initial invocation. Users wishing to ensure that a checkout is usable offline should pre-instantiate that checkout's vpython virtual environments by invoking a vpython3 installation command in their gclient runhooks process.
vpython3 -vpython-spec /path/to/vpython.spec -vpython-tool install
Unlike pip or other Python tools, vpython does not perform dependency resolution. It is up to the user constructing a vpython environment specification to ensure that all immediate and transitive package dependencies are included in that specification.
This design decision was not made lightly. Package and dependency management carry a whole can of worms with them, including package expression requirements, non-linear time for package identification, and dependency version resolution requirements.
As a trade-off, we ask that users perform a one-time manual expansion of dependencies when constructing a vpython specification.
Tooling can be developed to facilitate construction of vpython specifications if this is problematic or a pain point.
A list of wheels that Chrome Operations produces using a wheel production script, dockerbuild, can be found here.
vpython wheels are stored in CIPD. The raw index of all ChOPS-provided wheel CIPD packages space can be viewed here, or via CLI:
cipd ls infra/python/wheels
If a wheel is needed, but is not in this list, please see the instructions to add it.
Below are some templates for commonly used vpython wheel bundles.
This template can be used to include requests. The largest transitive set of dependencies that requests has is derived from the PyOpenSSL package.
python_version: "3.8" wheel: < name: "infra/python/wheels/requests-py2_py3" version: "version:2.21.0" > wheel: < name: "infra/python/wheels/urllib3-py2_py3" version: "version:1.24.3" > wheel: < name: "infra/python/wheels/certifi-py2_py3" version: "version:2019.3.9" > wheel: < name: "infra/python/wheels/chardet-py2_py3" version: "version:3.0.4" > wheel: < name: "infra/python/wheels/idna-py2_py3" version: "version:2.8" > ## # BEGIN pyopenssl and its dependencies. ## wheel: < name: "infra/python/wheels/pyopenssl-py2_py3" version: "version:19.0.0" > wheel: < name: "infra/python/wheels/cryptography/${vpython_platform}" # NOTE: For Python 2.7, use version:2.9.2 instead to avoid a noisy # deprecation warning. version: "version:3.3.1.chromium.1" > wheel: < name: "infra/python/wheels/cffi/${vpython_platform}" version: "version:1.14.5.chromium.7" > wheel: < name: "infra/python/wheels/pycparser-py2_py3" version: "version:2.19" > wheel: < name: "infra/python/wheels/six-py2_py3" version: "version:1.10.0" > ## # END pyopenssl and its dependencies. ##
A Python wheel is a pre-packaged formally-specified Python distribution. For more information on wheels, see Wheel vs Egg. A wheel may be either universal, written in pure Python, or binary, including binary content specialized to a specific operating system and/or architecture.
Chrome Operations strongly prefers wheels over eggs, since the latter may include compilation steps which, in turn, result in system-specific variance and unmanaged system dependencies (compiler, headers, etc.).
TL;DR: Yes, and the overhead of sharing a VirtualEnv across N vpython instances is effectively the same as using it with a single instance!
vpython uses filesystem locking during VirtualEnv mutations to ensure that it has exclusive access. If multiple vpython instances want to operate on the same VirtualEnv, they will serialize.
This means that if you start 1000 vpython instances with the same specification, one will sieze the lock and instantiate it, while the others block. Once the VirtualEnv is created, the remainder will immediately recognize the set-up environment and start their script.
Short answer: yes. At the beginning of a vpython run, it identifies any VirtualEnv instances that haven't been used in a while and purges them.
Yes. vpython, through CIPD, caches wheel packages locally. If multiple VirtualEnv are set-up, the cache will result in any overlapping packages being downloaded once.
Feel free to reach out to Chrome Operations if you have any questions or want to discuss vpython integration design.
luci-eng@google.com