[swarming_bot] Add increase retries during Windows file tree cleanup

The Windows builders are having flaky cleanup issues when dealing with
sandboxed child processes (specifically `browser_tests`), causing ~30%
chance of having the full test suite retry.

This change tries to reduce the cleanup flakes on Windows by increasing
max_retries for `file_path.rmtree` from 4 to 6.

We'll monitor retry rates for the `browser_tests` suite on the `win-rel`
builder.

Bug: 519251254
Change-Id: Icf5027ea6f899c063bec09fa6d418e27c4786f22
Reviewed-on: https://chromium-review.googlesource.com/c/infra/luci/luci-py/+/8018202
Commit-Queue: David Hinojosa <davidhinojosa@google.com>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
NOKEYCHECK=True
GitOrigin-RevId: e34702de74ec67828ff5d7f7f9bdcbde663b5f0e

tests/file_path_test.py

diff --git a/tests/file_path_test.py b/tests/file_path_test.py
index 2c1a4bb..78457ff 100755
--- a/tests/file_path_test.py
+++ b/tests/file_path_test.py
@@ -275,9 +275,11 @@
           self.assertEqual(None, proc.poll())
         file_path.rmtree(subdir)
         # rmtree will first try to change the tree permission and sleep for
-        # 4 and 6 seconds for the 2nd and 3rd try.
-        # The last 2 seconds is for the kill_children_processes.
-        self.assertEqual([4, 6, 2], sleeps)
+        # 4, 6, 8, and 10 seconds for tries 2 through 5 on Windows, max_tries=6
+        # The last 2 seconds is for kill_children_processes.
+        expected_sleeps = [4, 6, 8, 10, 2
+                           ] if sys.platform == 'win32' else [4, 6, 2]
+        self.assertEqual(expected_sleeps, sleeps)
         # sys.stderr.getvalue() would return a fair amount of output but it is
         # not completely deterministic so we're not testing it here.
       finally:

utils/file_path.py

diff --git a/utils/file_path.py b/utils/file_path.py
index 76c0aa4..aeb0641 100644
--- a/utils/file_path.py
+++ b/utils/file_path.py
@@ -938,10 +938,10 @@
                   time.time() - start)
 
   # First try the soft way: tries 3 times to delete and sleep a bit in between.
-  # On Windows, try 4 times with a total 12 seconds of sleep.
+  # On Windows, try 6 times with a total of up to 30 seconds of sleep.
   # Retries help if test subprocesses outlive main process and try to actively
   # use or write to the directory while it is being deleted.
-  max_tries = 4 if sys.platform == 'win32' else 3
+  max_tries = 6 if sys.platform == 'win32' else 3
   has_called_change_tree_permission = False
   has_called_change_acl_for_delete = False
   for i in range(max_tries):