docs/python3_migration.md

# Migrating Chromium to Python3

This page describes the current status and how to migrate code.

[crbug.com/941669](https://crbug.com/941669) tracks the overall migration.

*It is now safe to write new code using Python3 as long as it isn't called
from other Python2 files. See the
[Python style guide](../styleguide/python/python.md) for the latest on this.*

## Status

As of the time of writing (2021-05-12), we're in the following state:

* depot_tools is fully Python3-compatible.
* [gclient hooks](#gclient_hooks) are being migrated to use Python3
  ([crbug.com/1208028](https://crbug.com/1208028)).
* GN is fully Python3-compatible, meaning that all the scripts invoked
  through exec_script() are using Python3.
* The [build](#gn_ninja-actions) (scripts invoked by Ninja) is using Python3 by default,
  but some actions are still run using Python2
  ([crbug.com/1112471](https://crbug.com/1112471)).
* We are updating the various test harnesses and frameworks to use
  Python3, but most still use Python2. It is possible to use
  Python3 for tests if you're ready to do so.
* [PRESUBMIT checks](#presubmit_checks) are being migrated to use Python3
  ([crbug.com/1207012](https://crbug.com/1207012)).
* Python3-compatible pylint checks are available but not yet fully
  integrated into the presubmit checks
  ([crbug.com/1157676](https://crbug.com/1157676)).


## Migrating Python code from 2 to 3

This document will not attempt to replicate the general information in
the [Python.org Porting HOWTO](https://docs.python.org/3/howto/pyporting.html)
or the many, many other guides on the Internet.

However, here are a few notes that may be helpful in a Chromium context:

* Most of our Python code is pretty straightforward and moves easily
  from Python2 to Python3, so don't stress out about this!

* When migrating code, please make the Right Changes, rather than the
  minimum needed to get things to work. For example, make sure your code
  is clear about whether something should be a byte string or a unicode
  string, rather than trying to handle both "to be compatible".
  (If you really do need the code to handle both, though, that can be okay.)

* Do not assume you can use [vpython] regardless of context! vpython has
  performance issues in some situations and so we don't want to use it yet for
  things invoked by gclient, PRESUBMITs, or Ninja. However, all tests are run
  under vpython and so you can assume it there.

* Some people find the `2to3` tool to be useful to partially or
  completely automate the migration of existing files, and the
  `six` module to shim compatibility across the two. The `six` module
  is available in vpython and in `//third_party/six/src/six.py`.

* shebang lines mostly don't matter. A "shebang line" is the line at the
  beginning of many unix scripts, like `#!/usr/bin/env python`. They are
  not that important for us because most of our python invocations come
  from tools like Ninja or Swarming and invoke python directly. So, while
  you should keep them accurate where they are useful, we can't rely
  on them to tell which code has been migrated and which hasn't.

* The major gotchas for us tend to have to do with processing output
  from a subprocess (e.g., `subprocess.check_output()`). By default
  output is returned as a binary string, so get in the habit of calling
  `.check_output().decode('utf-8')` instead. This is compatible across
  2 and 3. 'utf-8' is the default in Python3 (ASCII was the default in
  Python2), but being explicitly is probably a good idea until we have
  migrated everything.

* Be aware that `filter` and `map` return generators in Python3, which
  are one-shot objects. If you reference them inside another loop, e.g.,

      foo = [ ... ]
      bar = filter(some_function, [ ...])
      for x in foo:
          for y in bar:
              do_something(x, y)

  this won't work right, because on the second and subsequent iterations,
  bar will be an empty list.

  Best practice is to use a list comprehension instead of `map` or `filter`,
  but you can also explicitly cast the results of map or filter to a list
  if the list comprehension is too awkward.

* Some modules (like `urllib2`) were renamed and/or moved around in Python 3.
  A Google search will usually quickly tell you the new location.

* Watch out for places where you reference `basestring` or `unicode` directly.
  `six` provides some compatibility types to help here.

## Testing your migrations

Generally speaking, test your changes the same way we do everything else:
make a change locally, and rely on the CQ and CI bots to catch problems.

However, here are some specific guidelines for the different contexts
where we use Python:

### gclient hooks

To switch a gclient hook from Python2 to Python3, simply change `python`
to `python3` in the DEPS file, and make sure things still run :).

### GN/Ninja actions

Most targets in the build use Python3 now, and anything declared via an
`action()` rule in GN will use Python3.

Some targets still require Python2; they are declared as [python2_action]
targets instead. To migrate them to Python3, change the `python2_action` to
`action` and make sure things still build.

### Tests

Test targets that run by invoking python scripts (like telemetry_unittests
or blink_web_tests) should eventually migrate to using the [script_test]
GN templates. Once you do that, they will use Python3 by default. However,
some tests may specify `run_under_python2 = true` as a template variable
to use Python2, so when you're ready to test Python3, just delete that line.

Some tests still need to be migrated to `script_test()`
([crbug.com/1208648](https://crbug.com/1208648)). The process for
doing that is not yet well documented, so ask on python@chromium.org (or
ask dpranke@ directly).

There is no general mechanism for migrating tests that are C++ executables
that launch python via subprocesses, so you're on your own for dealing with
that.

### Presubmit checks

Presubmit checks are run using Python 2 by default. To run them using
Python3, add the line `USE_PYTHON3 = True` to the PRESUBMIT.py file in
question (effectively creating a global variable).

[python2_action]: https://source.chromium.org/chromium/chromium/src/+/main:build/config/python.gni;l=68?q=python2_action%20file:python.gni&ss=chromium
[script_test]: https://source.chromium.org/?q=script_test%20file:testing%2Ftest.gni&ss=chromium
[vpython]: https://chromium.googlesource.com/infra/infra/+/refs/heads/main/doc/users/vpython.md