[manifest] Explicitly terminate the multiprocessing.Pool
Python requires that a Pool object be terminated before it is GC'd
(https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool):
```
Warning: multiprocessing.pool objects have internal resources that need
to be properly managed (like any other resource) by using the pool as a
context manager or by calling close() and terminate() manually. Failure
to do this can lead to the process hanging on finalization.
Note that is not correct to rely on the garbage colletor to destroy the
pool as CPython does not assure that the finalizer of the pool will be
called (see object.__del__() for more information).
```
Chromium has been seeing hangs in 'wpt manifest' invocations since
moving to Python 3 for WPT, which sounds a lot like 'Failure to do this
can lead to the process hanging on finalization.' So let's
terminate our Pool properly :)
diff --git a/tools/manifest/manifest.py b/tools/manifest/manifest.py
index 1b4f407..123f045 100644
--- a/tools/manifest/manifest.py
+++ b/tools/manifest/manifest.py
@@ -230,6 +230,7 @@
# 25 items was derived experimentally (2020-01) to be approximately the
# point at which it is quicker to create a Pool and parallelize update.
+ pool = None
if parallel and len(to_update) > 25 and cpu_count() > 1:
# On Python 3 on Windows, using >= MAXIMUM_WAIT_OBJECTS processes
# causes a crash in the multiprocessing module. Whilst this enum
@@ -262,6 +263,11 @@
data[new_type][rel_path_parts] = manifest_items
data[new_type].hashes[rel_path_parts] = file_hash
+ # Make sure to terminate the Pool, to avoid hangs on Python 3.
+ # https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool
+ if pool is not None:
+ pool.terminate()
+
if remaining_manifest_paths:
changed = True
for rel_path_parts in remaining_manifest_paths:
