[config_types.Path] Refactor Path implementation.
This CL significantly simplifies the implementation of the Path
object by removing it's dependency on API, tightening the type and
lifetime of it's global variables, making Path sanitize and
normalize all incoming arguments, and removes the bogus 'config' for
the path module, and makes Path objects no longer special 'config types'.
The new implementation also correctly splits up incoming path arguments
into pieces (so `a.join('some/thing')` and `a.join('some', 'thing')`
now produce identical Paths).
This also finally makes it possible to add a way to get a Path for
an arbitrary string, known to the recipe to be an absolute string.
This is implemented as `cast_to_path` in both the API and TestAPI
subclasses here.
Recipe-Nontrivial-Roll: build_limited
Recipe-Nontrivial-Roll: fuchsia
Recipe-Nontrivial-Roll: infra
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: chromiumos
Recipe-Nontrivial-Roll: chrome_release
Recipe-Nontrivial-Roll: pigweed
Change-Id: I69c80bb40b3a62b78bb3873a807da08093d6c63f
Bug: 329113288, 40270001, 40274272
Reviewed-on: https://chromium-review.googlesource.com/c/infra/luci/recipes-py/+/5121425
Reviewed-by: Rob Mohr <mohrr@google.com>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
diff --git a/README.recipes.md b/README.recipes.md
index a25e9bd..090e394 100644
--- a/README.recipes.md
+++ b/README.recipes.md
@@ -172,7 +172,10 @@
* [milo:examples/full](#recipes-milo_examples_full)
* [nodejs:examples/full](#recipes-nodejs_examples_full)
* [path:examples/full](#recipes-path_examples_full)
+ * [path:tests/cast_to_path](#recipes-path_tests_cast_to_path)
* [path:tests/dynamic_paths](#recipes-path_tests_dynamic_paths)
+ * [path:tests/exists](#recipes-path_tests_exists)
+ * [path:tests/test_api_legacy](#recipes-path_tests_test_api_legacy) — Test to cover legacy aspects of PathTestApi.
* [placeholder](#recipes-placeholder)
* [platform:examples/full](#recipes-platform_examples_full)
* [properties:examples/full](#recipes-properties_examples_full)
@@ -237,7 +240,7 @@
[DEPS](/recipe_modules/archive/__init__.py#5): [json](#recipe_modules-json), [path](#recipe_modules-path), [platform](#recipe_modules-platform), [step](#recipe_modules-step)
-#### **class [ArchiveApi](/recipe_modules/archive/api.py#11)([RecipeApi](/recipe_engine/recipe_api.py#470)):**
+#### **class [ArchiveApi](/recipe_modules/archive/api.py#11)([RecipeApi](/recipe_engine/recipe_api.py#471)):**
Provides steps to manipulate archive files (tar, zip, etc.).
@@ -303,7 +306,7 @@
### *recipe_modules* / [assertions](/recipe_modules/assertions)
-#### **class [AssertionsApi](/recipe_modules/assertions/api.py#54)([RecipeApi](/recipe_engine/recipe_api.py#470)):**
+#### **class [AssertionsApi](/recipe_modules/assertions/api.py#54)([RecipeApi](/recipe_engine/recipe_api.py#471)):**
Provides access to the assertion methods of the python unittest module.
@@ -355,7 +358,7 @@
[DEPS](/recipe_modules/bcid_reporter/__init__.py#5): [cipd](#recipe_modules-cipd), [path](#recipe_modules-path), [properties](#recipe_modules-properties), [step](#recipe_modules-step)
-#### **class [BcidReporterApi](/recipe_modules/bcid_reporter/api.py#14)([RecipeApi](/recipe_engine/recipe_api.py#470)):**
+#### **class [BcidReporterApi](/recipe_modules/bcid_reporter/api.py#14)([RecipeApi](/recipe_engine/recipe_api.py#471)):**
API for interacting with Provenance server using the broker tool.
@@ -438,7 +441,7 @@
`build_pb2.Build` and returns a link title.
If it returns `None`, the link is not reported. Default link title is build ID.
-#### **class [BuildbucketApi](/recipe_modules/buildbucket/api.py#29)([RecipeApi](/recipe_engine/recipe_api.py#470)):**
+#### **class [BuildbucketApi](/recipe_modules/buildbucket/api.py#29)([RecipeApi](/recipe_engine/recipe_api.py#471)):**
A module for interacting with buildbucket.
@@ -920,7 +923,7 @@
API for interacting with cas client.
-#### **class [CasApi](/recipe_modules/cas/api.py#12)([RecipeApi](/recipe_engine/recipe_api.py#470)):**
+#### **class [CasApi](/recipe_modules/cas/api.py#12)([RecipeApi](/recipe_engine/recipe_api.py#471)):**
A module for interacting with cas client.
@@ -978,7 +981,7 @@
download. These can easily be download to disk with the 'download_caches'
method, and subsequently used by a recipe in whatever relevant manner.
-#### **class [CasInputApi](/recipe_modules/cas_input/api.py#20)([RecipeApi](/recipe_engine/recipe_api.py#470)):**
+#### **class [CasInputApi](/recipe_modules/cas_input/api.py#20)([RecipeApi](/recipe_engine/recipe_api.py#471)):**
A module for downloading CAS inputs to a recipe.
@@ -1016,7 +1019,7 @@
want to use this recipe module; file a ticket at:
https://bugs.chromium.org/p/chromium/issues/entry?components=Infra%3ELUCI%3EBuildService%3EPresubmit%3ECV
-#### **class [ChangeVerifierApi](/recipe_modules/change_verifier/api.py#28)([RecipeApi](/recipe_engine/recipe_api.py#470)):**
+#### **class [ChangeVerifierApi](/recipe_modules/change_verifier/api.py#28)([RecipeApi](/recipe_engine/recipe_api.py#471)):**
This module provides recipe API of LUCI Change Verifier.
@@ -1046,7 +1049,7 @@
Depends on 'cipd' binary available in PATH:
https://godoc.org/go.chromium.org/luci/cipd/client/cmd/cipd
-#### **class [CIPDApi](/recipe_modules/cipd/api.py#236)([RecipeApi](/recipe_engine/recipe_api.py#470)):**
+#### **class [CIPDApi](/recipe_modules/cipd/api.py#236)([RecipeApi](/recipe_engine/recipe_api.py#471)):**
CIPDApi provides basic support for CIPD.
@@ -1342,7 +1345,7 @@
### *recipe_modules* / [commit\_position](/recipe_modules/commit_position)
-#### **class [CommitPositionApi](/recipe_modules/commit_position/api.py#10)([RecipeApi](/recipe_engine/recipe_api.py#470)):**
+#### **class [CommitPositionApi](/recipe_modules/commit_position/api.py#10)([RecipeApi](/recipe_engine/recipe_api.py#471)):**
Recipe module providing commit position parsing and formatting.
@@ -1383,7 +1386,7 @@
api.step("cat subdir/foo", ['cat', './foo'])
```
-#### **class [ContextApi](/recipe_modules/context/api.py#77)([RecipeApi](/recipe_engine/recipe_api.py#470)):**
+#### **class [ContextApi](/recipe_modules/context/api.py#77)([RecipeApi](/recipe_engine/recipe_api.py#471)):**
  **@contextmanager**<br>— **def [\_\_call\_\_](/recipe_modules/context/api.py#109)(self, cwd=None, env_prefixes=None, env_suffixes=None, env=None, infra_steps=None, luciexe=None, realm=None, deadline=None):**
@@ -1520,7 +1523,7 @@
Wrapper for CV API.
-#### **class [CQApi](/recipe_modules/cq/api.py#18)([RecipeApi](/recipe_engine/recipe_api.py#470)):**
+#### **class [CQApi](/recipe_modules/cq/api.py#18)([RecipeApi](/recipe_engine/recipe_api.py#471)):**
This module is a thin wrapper of the cv module.
@@ -1534,7 +1537,7 @@
Recipe API for LUCI CV, the pre-commit testing system.
-#### **class [CVApi](/recipe_modules/cv/api.py#16)([RecipeApi](/recipe_engine/recipe_api.py#470)):**
+#### **class [CVApi](/recipe_modules/cv/api.py#16)([RecipeApi](/recipe_engine/recipe_api.py#471)):**
This module provides recipe API of LUCI CV, a pre-commit testing system.
@@ -1703,7 +1706,7 @@
Runs a function but defers the result until a later time.
-#### **class [DeferApi](/recipe_modules/defer/api.py#105)([RecipeApi](/recipe_engine/recipe_api.py#470)):**
+#### **class [DeferApi](/recipe_modules/defer/api.py#105)([RecipeApi](/recipe_engine/recipe_api.py#471)):**
Runs a function but defers the result until a later time.
@@ -1765,7 +1768,7 @@
File manipulation (read/write/delete/glob) methods.
-#### **class [FileApi](/recipe_modules/file/api.py#84)([RecipeApi](/recipe_engine/recipe_api.py#470)):**
+#### **class [FileApi](/recipe_modules/file/api.py#84)([RecipeApi](/recipe_engine/recipe_api.py#471)):**
— **def [chmod](/recipe_modules/file/api.py#166)(self, name, path, mode):**
@@ -2197,7 +2200,7 @@
Implements in-recipe concurrency via green threads.
-#### **class [FuturesApi](/recipe_modules/futures/api.py#43)([RecipeApi](/recipe_engine/recipe_api.py#470)):**
+#### **class [FuturesApi](/recipe_modules/futures/api.py#43)([RecipeApi](/recipe_engine/recipe_api.py#471)):**
Provides access to the Recipe concurrency primitives.
@@ -2376,7 +2379,7 @@
A simple method for running steps generated by an external script.
-#### **class [GeneratorScriptApi](/recipe_modules/generator_script/api.py#12)([RecipeApi](/recipe_engine/recipe_api.py#470)):**
+#### **class [GeneratorScriptApi](/recipe_modules/generator_script/api.py#12)([RecipeApi](/recipe_engine/recipe_api.py#471)):**
— **def [\_\_call\_\_](/recipe_modules/generator_script/api.py#71)(self, path_to_script, \*args, \*\*_):**
@@ -2421,7 +2424,7 @@
[DEPS](/recipe_modules/golang/__init__.py#5): [cipd](#recipe_modules-cipd), [context](#recipe_modules-context), [path](#recipe_modules-path), [platform](#recipe_modules-platform)
-#### **class [GolangApi](/recipe_modules/golang/api.py#10)([RecipeApi](/recipe_engine/recipe_api.py#470)):**
+#### **class [GolangApi](/recipe_modules/golang/api.py#10)([RecipeApi](/recipe_engine/recipe_api.py#471)):**
  **@contextlib.contextmanager**<br>— **def [\_\_call\_\_](/recipe_modules/golang/api.py#15)(self, version, path=None, cache=None):**
@@ -2459,7 +2462,7 @@
Methods for producing and consuming JSON.
-#### **class [JsonApi](/recipe_modules/json/api.py#132)([RecipeApi](/recipe_engine/recipe_api.py#470)):**
+#### **class [JsonApi](/recipe_modules/json/api.py#132)([RecipeApi](/recipe_engine/recipe_api.py#471)):**
  **@staticmethod**<br>— **def [dumps](/recipe_modules/json/api.py#133)(\*args, \*\*kwargs):**
@@ -2517,7 +2520,7 @@
An interface to call the led tool.
-#### **class [LedApi](/recipe_modules/led/api.py#18)([RecipeApi](/recipe_engine/recipe_api.py#470)):**
+#### **class [LedApi](/recipe_modules/led/api.py#18)([RecipeApi](/recipe_engine/recipe_api.py#471)):**
Interface to the led tool.
@@ -2616,7 +2619,7 @@
build (using the Merge Step feature from luciexe protocol). This is the
replacement for allow_subannotation feature in the legacy annotate mode.
-#### **class [LegacyAnnotationApi](/recipe_modules/legacy_annotation/api.py#23)([RecipeApi](/recipe_engine/recipe_api.py#470)):**
+#### **class [LegacyAnnotationApi](/recipe_modules/legacy_annotation/api.py#23)([RecipeApi](/recipe_engine/recipe_api.py#471)):**
— **def [\_\_call\_\_](/recipe_modules/legacy_annotation/api.py#27)(self, name, cmd, timeout=None, step_test_data=None, cost=_ResourceCost(), legacy_global_namespace=False):**
@@ -2639,7 +2642,7 @@
test results.
See go/luci-analysis for more info.
-#### **class [LuciAnalysisApi](/recipe_modules/luci_analysis/api.py#30)([RecipeApi](/recipe_engine/recipe_api.py#470)):**
+#### **class [LuciAnalysisApi](/recipe_modules/luci_analysis/api.py#30)([RecipeApi](/recipe_engine/recipe_api.py#471)):**
— **def [lookup\_bug](/recipe_modules/luci_analysis/api.py#263)(self, bug_id, system='monorail'):**
@@ -2770,7 +2773,7 @@
API for specifying Milo behavior.
-#### **class [MiloApi](/recipe_modules/milo/api.py#17)([RecipeApi](/recipe_engine/recipe_api.py#470)):**
+#### **class [MiloApi](/recipe_modules/milo/api.py#17)([RecipeApi](/recipe_engine/recipe_api.py#471)):**
A module for interacting with Milo.
@@ -2791,7 +2794,7 @@
[DEPS](/recipe_modules/nodejs/__init__.py#5): [cipd](#recipe_modules-cipd), [context](#recipe_modules-context), [path](#recipe_modules-path), [platform](#recipe_modules-platform)
-#### **class [NodeJSApi](/recipe_modules/nodejs/api.py#10)([RecipeApi](/recipe_engine/recipe_api.py#470)):**
+#### **class [NodeJSApi](/recipe_modules/nodejs/api.py#10)([RecipeApi](/recipe_engine/recipe_api.py#471)):**
  **@contextlib.contextmanager**<br>— **def [\_\_call\_\_](/recipe_modules/nodejs/api.py#15)(self, version, path=None, cache=None):**
@@ -2850,27 +2853,49 @@
should avoid 'checkout', and instead just explicitly pass paths around. This
path may be removed in the future.
-There are other anchor points which can be defined (e.g. by the
-`depot_tools/infra_paths` module). Refer to those modules for additional
-documentation.
+#### **class [PathApi](/recipe_modules/path/api.py#328)([RecipeApi](/recipe_engine/recipe_api.py#471)):**
-#### **class [PathApi](/recipe_modules/path/api.py#235)([RecipeApi](/recipe_engine/recipe_api.py#470)):**
+— **def [\_\_contains\_\_](/recipe_modules/path/api.py#588)(self, pathname: NamedBasePathsType):**
-— **def [\_\_getitem\_\_](/recipe_modules/path/api.py#531)(self, name: str):**
+This method is DEPRECATED.
+
+If `pathname` is "checkout", returns True iff checkout_dir is set.
+If you want to check if checkout_dir is set, use
+`api.path.checkout_dir is not None` or similar, instead.
+
+Returns True for all other `pathname` values in NamedBasePaths.
+Returns False for all other values.
+
+In the past, the base paths that this module knew about were extensible via
+a very complicated 'config' system. All of that has been removed, but this
+method remains for now.
+
+— **def [\_\_getitem\_\_](/recipe_modules/path/api.py#687)(self, name: NamedBasePathsType):**
Gets the base path named `name`. See module docstring for more info.
-— **def [\_\_setitem\_\_](/recipe_modules/path/api.py#483)(self, pathname: Literal[CheckoutPathName], path: config_types.Path):**
+*** note
+**DEPRECATED**: Use the following @properties on this module instead:
+ * start_dir
+ * tmp_base_dir
+ * cache_dir
+ * cleanup_dir
+ * home_dir
+ * checkout_dir (but use of checkout_dir is generally discouraged - just
+ pass the Paths around instead of using this global variable).
+***
+
+— **def [\_\_setitem\_\_](/recipe_modules/path/api.py#606)(self, pathname: CheckoutPathNameType, path: config_types.Path):**
Sets the checkout path.
*** note
-**DEPRECATED** - Use `api.path.set_checkout_dir` instead.
+**DEPRECATED** - Assign directly to `api.path.checkout_dir` instead.
***
The only valid value of `pathname` is the literal string CheckoutPathName.
-— **def [abs\_to\_path](/recipe_modules/path/api.py#423)(self, abs_string_path: str):**
+— **def [abs\_to\_path](/recipe_modules/path/api.py#525)(self, abs_string_path: str):**
Converts an absolute path string `abs_string_path` to a real Path
object, using the most appropriate known base path.
@@ -2884,7 +2909,10 @@
* recipe resource paths
* repo paths
* checkout_dir
- * base_paths
+ * home_dir
+ * start_dir
+ * tmp_base_dir
+ * cleanup_dir
Example:
```
@@ -2896,28 +2924,71 @@
Raises an ValueError if the preconditions are not met, otherwise returns the
Path object.
-— **def [abspath](/recipe_modules/path/api.py#553)(self, path: (config_types.Path | str)):**
+— **def [abspath](/recipe_modules/path/api.py#796)(self, path: (config_types.Path | str)):**
Equivalent to os.abspath.
-— **def [assert\_absolute](/recipe_modules/path/api.py#363)(self, path: (config_types.Path | str)):**
+— **def [assert\_absolute](/recipe_modules/path/api.py#465)(self, path: (config_types.Path | str)):**
Raises AssertionError if the given path is not an absolute path.
Args:
* path - The path to check.
-— **def [basename](/recipe_modules/path/api.py#557)(self, path: (config_types.Path | str)):**
+— **def [basename](/recipe_modules/path/api.py#800)(self, path: (config_types.Path | str)):**
Equivalent to os.path.basename.
-  **@checkout_dir.setter**<br>— **def [checkout\_dir](/recipe_modules/path/api.py#501)(self, path: config_types.Path):**
+  **@property**<br>— **def [cache\_dir](/recipe_modules/path/api.py#730)(self):**
+
+This directory is provided by whatever's running the recipe.
+
+When the recipe executes via Buildbucket, directories under here map to
+'named caches' which the Build has set. These caches would be preserved
+locally on the machine executing this recipe, and are restored for
+subsequent recipe exections on the same machine which request the same named
+cache.
+
+By default, Buildbucket installs a cache named 'builder' which is an
+immediate subdirectory of cache_dir, and will attempt to be persisted
+between executions of recipes on the same Buildbucket builder which use the
+same machine. So, if you are just looking for a place to put files which may
+be persisted between builds, use:
+
+ api.path.cache_dir/'builder'
+
+As the base Path.
+
+Note that directories created under here /may/ be evicted in between runs of
+the recipe (i.e. to relieve disk pressure).
+
+— **def [cast\_to\_path](/recipe_modules/path/api.py#764)(self, strpath: str):**
+
+This returns a Path for strpath which can be used anywhere a Path is
+required.
+
+If `strpath` is not an absolute path (e.g. rooted with a valid Windows drive
+or a '/' for non-Windows paths), this will raise ValueError.
+
+This implicitly tries abs_to_path prior to returning a drive-rooted Path.
+This means that if strpath is a subdirectory of a known path (say,
+cache_dir), the returned Path will be based on that known path. This is
+important for test compatibility.
+
+  **@checkout_dir.setter**<br>— **def [checkout\_dir](/recipe_modules/path/api.py#626)(self, path: config_types.Path):**
Sets the global variable `api.path.checkout_dir` to the given path.
-— **def [dirname](/recipe_modules/path/api.py#561)(self, path: (config_types.Path | str)):**
+  **@property**<br>— **def [cleanup\_dir](/recipe_modules/path/api.py#755)(self):**
+
+This directory is guaranteed to be cleaned up (eventually) after the
+execution of this recipe.
+
+This directory is guaranteed to be empty when the recipe starts.
+
+— **def [dirname](/recipe_modules/path/api.py#804)(self, path: (config_types.Path | str)):**
For "foo/bar/baz", return "foo/bar".
@@ -2930,67 +3001,77 @@
Returns dirname of path
-— **def [eq](/recipe_modules/path/api.py#727)(self, path1: config_types.Path, path2: config_types.Path):**
+— **def [eq](/recipe_modules/path/api.py#968)(self, path1: config_types.Path, path2: config_types.Path):**
Check whether path1 points to the same path as path2.
-Under most circumstances, path equality is checked via `path1 == path2`.
-However, if the paths are constructed via differently joined dirs, such as
-('foo' / 'bar') vs. ('foo/bar'), that doesn't work. This method addresses
-that problem by creating copies of the paths, and then separating them
-according to self.sep. The original paths are not modified.
+*** note
+**DEPRECATED**: Just directly compare path1 and path2 with `==`.
+***
-— **def [exists](/recipe_modules/path/api.py#665)(self, path):**
+— **def [exists](/recipe_modules/path/api.py#908)(self, path):**
Equivalent to os.path.exists.
The presence or absence of paths can be mocked during the execution of the
recipe by using the mock_* methods.
-— **def [expanduser](/recipe_modules/path/api.py#656)(self, path):**
+— **def [expanduser](/recipe_modules/path/api.py#899)(self, path):**
Do not use this, use `api.path['home']` instead.
This ONLY handles `path` == "~", and returns `str(api.path['home'])`.
-— **def [get](/recipe_modules/path/api.py#519)(self, name: str, default: (config_types.Path | None)=None):**
+— **def [get](/recipe_modules/path/api.py#655)(self, name: NamedBasePathsType):**
Gets the base path named `name`. See module docstring for more info.
-— **def [get\_config\_defaults](/recipe_modules/path/api.py#247)(self):**
+*** note
+**DEPRECATED**: Use the following @properties on this module instead:
+ * start_dir
+ * tmp_base_dir
+ * cache_dir
+ * cleanup_dir
+ * home_dir
+ * checkout_dir (but use of checkout_dir is generally discouraged - just
+ pass the Paths around instead of using this global variable).
+***
-Internal recipe implementation function.
+  **@property**<br>— **def [home\_dir](/recipe_modules/path/api.py#712)(self):**
-— **def [initialize](/recipe_modules/path/api.py#310)(self):**
+This is the path to the current $HOME directory.
-Internal recipe implementation function.
+It is generally recommended to avoid using this, because it is an indicator
+that the recipe is non-hermetic.
-— **def [is\_parent\_of](/recipe_modules/path/api.py#742)(self, parent: config_types.Path, child: config_types.Path):**
+— **def [initialize](/recipe_modules/path/api.py#428)(self):**
+
+This is called by the recipe engine immediately after __init__(), but
+with `self._paths_client` initialized.
+
+— **def [is\_parent\_of](/recipe_modules/path/api.py#975)(self, parent: config_types.Path, child: config_types.Path):**
Check whether child is contained within parent.
-Under most circumstances, this would be checked via
-`parent.is_parent_of(child)`. However, if the paths are constructed via
-differently joined dirs, such as ('foo', 'bar') vs. ('foo/bar', 'baz.txt'),
-that doesn't work. This method addresses that problem by creating copies of
-the paths, and then separating them according to self.sep. The original
-paths are not modified.
+*** note
+**DEPRECATED**: Just use `parent.is_parent_of(child)`.
+***
-— **def [isdir](/recipe_modules/path/api.py#673)(self, path):**
+— **def [isdir](/recipe_modules/path/api.py#916)(self, path):**
Equivalent to os.path.isdir.
The presence or absence of paths can be mocked during the execution of the
recipe by using the mock_* methods.
-— **def [isfile](/recipe_modules/path/api.py#681)(self, path):**
+— **def [isfile](/recipe_modules/path/api.py#924)(self, path):**
Equivalent to os.path.isfile.
The presence or absence of paths can be mocked during the execution of the
recipe by using the mock_* methods.
-— **def [join](/recipe_modules/path/api.py#580)(self, path, \*paths):**
+— **def [join](/recipe_modules/path/api.py#823)(self, path, \*paths):**
Equivalent to os.path.join.
@@ -3004,7 +3085,7 @@
retrieved with api.path[something]), then you can convert from a string path
back to a Path with the `abs_to_path` method.
-— **def [mkdtemp](/recipe_modules/path/api.py#372)(self, prefix: str=tempfile.template):**
+— **def [mkdtemp](/recipe_modules/path/api.py#474)(self, prefix: str=tempfile.template):**
Makes a new temporary directory, returns Path to it.
@@ -3013,33 +3094,38 @@
Returns a Path to the new directory.
-— **def [mkstemp](/recipe_modules/path/api.py#396)(self, prefix: str=tempfile.template):**
+— **def [mkstemp](/recipe_modules/path/api.py#497)(self, prefix: str=tempfile.template):**
Makes a new temporary file, returns Path to it.
Args:
* prefix - a tempfile template for the file name (defaults to "tmp").
-Returns a Path to the new file. Unlike tempfile.mkstemp, the file's file
-descriptor is closed.
+Returns a Path to the new file.
-— **def [mock\_add\_directory](/recipe_modules/path/api.py#699)(self, path: config_types.Path):**
+*** promo
+NOTE: Unlike tempfile.mkstemp, the file's file descriptor is closed. If you
+need the full security properties of mkstemp, please outsource this to e.g.
+either a resource script of your recipe module or recipe.
+***
+
+— **def [mock\_add\_directory](/recipe_modules/path/api.py#943)(self, path: config_types.Path):**
For testing purposes, mark that file |path| exists.
-— **def [mock\_add\_file](/recipe_modules/path/api.py#695)(self, path: config_types.Path):**
+— **def [mock\_add\_file](/recipe_modules/path/api.py#939)(self, path: config_types.Path):**
For testing purposes, mark that file |path| exists.
-— **def [mock\_add\_paths](/recipe_modules/path/api.py#689)(self, path: config_types.Path, kind: FileType=FileType.FILE):**
+— **def [mock\_add\_paths](/recipe_modules/path/api.py#932)(self, path: config_types.Path, kind: FileType=FileType.FILE):**
For testing purposes, mark that |path| exists.
-— **def [mock\_copy\_paths](/recipe_modules/path/api.py#703)(self, source: config_types.Path, dest: config_types.Path):**
+— **def [mock\_copy\_paths](/recipe_modules/path/api.py#947)(self, source: config_types.Path, dest: config_types.Path):**
For testing purposes, copy |source| to |dest|.
-— **def [mock\_remove\_paths](/recipe_modules/path/api.py#709)(self, path: config_types.Path, should_remove: Callable[([str], bool)]=(lambda p: True)):**
+— **def [mock\_remove\_paths](/recipe_modules/path/api.py#954)(self, path: config_types.Path, should_remove: Callable[([str], bool)]=(lambda p: True)):**
For testing purposes, mark that |path| doesn't exist.
@@ -3048,38 +3134,34 @@
should_remove: Called for every candidate path. Return True to remove this
path.
-— **def [normpath](/recipe_modules/path/api.py#652)(self, path):**
+— **def [normpath](/recipe_modules/path/api.py#895)(self, path):**
Equivalent to os.path.normpath.
-  **@property**<br>— **def [pardir](/recipe_modules/path/api.py#538)(self):**
+  **@property**<br>— **def [pardir](/recipe_modules/path/api.py#781)(self):**
Equivalent to os.pardir.
-  **@property**<br>— **def [pathsep](/recipe_modules/path/api.py#548)(self):**
+  **@property**<br>— **def [pathsep](/recipe_modules/path/api.py#791)(self):**
Equivalent to os.pathsep.
-— **def [realpath](/recipe_modules/path/api.py#640)(self, path: (config_types.Path | str)):**
+— **def [realpath](/recipe_modules/path/api.py#883)(self, path: (config_types.Path | str)):**
Equivalent to os.path.realpath.
-— **def [relpath](/recipe_modules/path/api.py#644)(self, path, start):**
+— **def [relpath](/recipe_modules/path/api.py#887)(self, path, start):**
Roughly equivalent to os.path.relpath.
Unlike os.path.relpath, `start` is _required_. If you want the 'current
directory', use the `recipe_engine/context` module's `cwd` property.
-  **@property**<br>— **def [sep](/recipe_modules/path/api.py#543)(self):**
+  **@property**<br>— **def [sep](/recipe_modules/path/api.py#786)(self):**
Equivalent to os.sep.
-— **def [separate](/recipe_modules/path/api.py#723)(self, path: config_types.Path):**
-
-Separate a path's pieces in-place with this platform's separator char.
-
-— **def [split](/recipe_modules/path/api.py#595)(self, path):**
+— **def [split](/recipe_modules/path/api.py#838)(self, path):**
For "foo/bar/baz", return ("foo/bar", "baz").
@@ -3093,7 +3175,7 @@
Returns (dirname(path), basename(path)).
-— **def [splitext](/recipe_modules/path/api.py#616)(self, path: (config_types.Path | str)):**
+— **def [splitext](/recipe_modules/path/api.py#859)(self, path: (config_types.Path | str)):**
For "foo/bar.baz", return ("foo/bar", ".baz").
@@ -3107,12 +3189,28 @@
Returns:
(name, extension_including_dot).
+
+  **@property**<br>— **def [start\_dir](/recipe_modules/path/api.py#701)(self):**
+
+This is the directory that the recipe started in. it's similar to `cwd`,
+except that it's constant for the duration of the entire program.
+
+If you want to modify the current working directory for a set of steps,
+See the 'recipe_engine/context' module which allows modifying the cwd safely
+via a context manager.
+
+  **@property**<br>— **def [tmp\_base\_dir](/recipe_modules/path/api.py#721)(self):**
+
+This directory is the system-configured temp dir.
+
+This is a weaker form of 'cleanup', and its use should be avoided. This may
+be removed in the future (or converted to an alias of 'cleanup').
### *recipe_modules* / [platform](/recipe_modules/platform)
Mockable system platform identity functions.
-#### **class [PlatformApi](/recipe_modules/platform/api.py#24)([RecipeApi](/recipe_engine/recipe_api.py#470)):**
+#### **class [PlatformApi](/recipe_modules/platform/api.py#24)([RecipeApi](/recipe_engine/recipe_api.py#471)):**
Provides host-platform-detection properties.
@@ -3191,7 +3289,7 @@
intentionally no API to write property values (lest they become a kind of
random-access global variable).
-#### **class [PropertiesApi](/recipe_modules/properties/api.py#27)([RecipeApi](/recipe_engine/recipe_api.py#470), collections.abc.Mapping):**
+#### **class [PropertiesApi](/recipe_modules/properties/api.py#27)([RecipeApi](/recipe_engine/recipe_api.py#471), collections.abc.Mapping):**
PropertiesApi implements all the standard Mapping functions, so you
can use it like a read-only dict.
@@ -3221,7 +3319,7 @@
Methods for producing and consuming protobuf data to/from steps and the
filesystem.
-#### **class [ProtoApi](/recipe_modules/proto/api.py#83)([RecipeApi](/recipe_engine/recipe_api.py#470)):**
+#### **class [ProtoApi](/recipe_modules/proto/api.py#83)([RecipeApi](/recipe_engine/recipe_api.py#471)):**
  **@staticmethod**<br>— **def [decode](/recipe_modules/proto/api.py#161)(data, msg_class, codec, \*\*decoding_kwargs):**
@@ -3312,7 +3410,7 @@
api.random.shuffle(my_list)
# my_list is now random!
-#### **class [RandomApi](/recipe_modules/random/api.py#30)([RecipeApi](/recipe_engine/recipe_api.py#470)):**
+#### **class [RandomApi](/recipe_modules/random/api.py#30)([RecipeApi](/recipe_engine/recipe_api.py#471)):**
— **def [\_\_getattr\_\_](/recipe_modules/random/api.py#37)(self, name):**
@@ -3324,7 +3422,7 @@
Provides objects for reading and writing raw data to and from steps.
-#### **class [RawIOApi](/recipe_modules/raw_io/api.py#305)([RecipeApi](/recipe_engine/recipe_api.py#470)):**
+#### **class [RawIOApi](/recipe_modules/raw_io/api.py#305)([RecipeApi](/recipe_engine/recipe_api.py#471)):**
  **@[returns\_placeholder](/recipe_engine/util.py#160)**<br>  **@staticmethod**<br>— **def [input](/recipe_modules/raw_io/api.py#306)(data, suffix='', name=None):**
@@ -3431,7 +3529,7 @@
Requires `rdb` command in `$PATH`:
https://godoc.org/go.chromium.org/luci/resultdb/cmd/rdb
-#### **class [ResultDBAPI](/recipe_modules/resultdb/api.py#28)([RecipeApi](/recipe_engine/recipe_api.py#470)):**
+#### **class [ResultDBAPI](/recipe_modules/resultdb/api.py#28)([RecipeApi](/recipe_engine/recipe_api.py#471)):**
A module for interacting with ResultDB.
@@ -3744,7 +3842,7 @@
### *recipe_modules* / [runtime](/recipe_modules/runtime)
-#### **class [RuntimeApi](/recipe_modules/runtime/api.py#10)([RecipeApi](/recipe_engine/recipe_api.py#470)):**
+#### **class [RuntimeApi](/recipe_modules/runtime/api.py#10)([RecipeApi](/recipe_engine/recipe_api.py#471)):**
This module assists in experimenting with production recipes.
@@ -3797,7 +3895,7 @@
RPCExplorer available at
https://luci-scheduler.appspot.com/rpcexplorer/services/scheduler.Scheduler
-#### **class [SchedulerApi](/recipe_modules/scheduler/api.py#27)([RecipeApi](/recipe_engine/recipe_api.py#470)):**
+#### **class [SchedulerApi](/recipe_modules/scheduler/api.py#27)([RecipeApi](/recipe_engine/recipe_api.py#471)):**
A module for interacting with LUCI Scheduler service.
@@ -3866,7 +3964,7 @@
Depends on luci-auth to be in PATH.
-#### **class [ServiceAccountApi](/recipe_modules/service_account/api.py#16)([RecipeApi](/recipe_engine/recipe_api.py#470)):**
+#### **class [ServiceAccountApi](/recipe_modules/service_account/api.py#16)([RecipeApi](/recipe_engine/recipe_api.py#471)):**
— **def [default](/recipe_modules/service_account/api.py#72)(self):**
@@ -3892,7 +3990,7 @@
Step is the primary API for running steps (external programs, etc.)
-#### **class [StepApi](/recipe_modules/step/api.py#28)([RecipeApi](/recipe_engine/recipe_api.py#470)):**
+#### **class [StepApi](/recipe_modules/step/api.py#28)([RecipeApi](/recipe_engine/recipe_api.py#471)):**
  **@property**<br>— **def [InfraFailure](/recipe_modules/step/api.py#147)(self):**
@@ -4283,7 +4381,7 @@
[DEPS](/recipe_modules/swarming/__init__.py#7): [buildbucket](#recipe_modules-buildbucket), [cas](#recipe_modules-cas), [cipd](#recipe_modules-cipd), [context](#recipe_modules-context), [json](#recipe_modules-json), [path](#recipe_modules-path), [properties](#recipe_modules-properties), [raw\_io](#recipe_modules-raw_io), [step](#recipe_modules-step)
-#### **class [SwarmingApi](/recipe_modules/swarming/api.py#1196)([RecipeApi](/recipe_engine/recipe_api.py#470)):**
+#### **class [SwarmingApi](/recipe_modules/swarming/api.py#1196)([RecipeApi](/recipe_engine/recipe_api.py#471)):**
API for interacting with swarming.
@@ -4407,7 +4505,7 @@
Allows mockable access to the current time.
-#### **class [TimeApi](/recipe_modules/time/api.py#87)([RecipeApi](/recipe_engine/recipe_api.py#470)):**
+#### **class [TimeApi](/recipe_modules/time/api.py#87)([RecipeApi](/recipe_engine/recipe_api.py#471)):**
— **def [exponential\_retry](/recipe_modules/time/api.py#119)(self, retries, delay, condition=None):**
@@ -4530,7 +4628,7 @@
* Recipes that accumulate comments one by one.
* Recipes that wrap other tools and parse their output.
-#### **class [TriciumApi](/recipe_modules/tricium/api.py#26)([RecipeApi](/recipe_engine/recipe_api.py#470)):**
+#### **class [TriciumApi](/recipe_modules/tricium/api.py#26)([RecipeApi](/recipe_engine/recipe_api.py#471)):**
TriciumApi provides basic support for Tricium.
@@ -4587,7 +4685,7 @@
Methods for interacting with HTTP(s) URLs.
-#### **class [UrlApi](/recipe_modules/url/api.py#15)([RecipeApi](/recipe_engine/recipe_api.py#470)):**
+#### **class [UrlApi](/recipe_modules/url/api.py#15)([RecipeApi](/recipe_engine/recipe_api.py#471)):**
— **def [get\_file](/recipe_modules/url/api.py#131)(self, url, path, step_name=None, headers=None, transient_retry=True, strip_prefix=None):**
@@ -4711,7 +4809,7 @@
Allows test-repeatable access to a random UUID.
-#### **class [UuidApi](/recipe_modules/uuid/api.py#11)([RecipeApi](/recipe_engine/recipe_api.py#470)):**
+#### **class [UuidApi](/recipe_modules/uuid/api.py#11)([RecipeApi](/recipe_engine/recipe_api.py#471)):**
— **def [random](/recipe_modules/uuid/api.py#20)(self):**
@@ -4721,7 +4819,7 @@
Thin API for parsing semver strings into comparable object.
-#### **class [VersionApi](/recipe_modules/version/api.py#13)([RecipeApi](/recipe_engine/recipe_api.py#470)):**
+#### **class [VersionApi](/recipe_modules/version/api.py#13)([RecipeApi](/recipe_engine/recipe_api.py#471)):**
  **@staticmethod**<br>— **def [parse](/recipe_modules/version/api.py#15)(version):**
@@ -4740,7 +4838,7 @@
Allows recipe modules to issue warnings in simulation test.
-#### **class [WarningApi](/recipe_modules/warning/api.py#12)([RecipeApi](/recipe_engine/recipe_api.py#470)):**
+#### **class [WarningApi](/recipe_modules/warning/api.py#12)([RecipeApi](/recipe_engine/recipe_api.py#471)):**
  **@recipe_api.escape_all_warnings**<br>— **def [issue](/recipe_modules/warning/api.py#15)(self, name):**
@@ -5625,13 +5723,33 @@
[DEPS](/recipe_modules/path/examples/full.py#5): [json](#recipe_modules-json), [path](#recipe_modules-path), [platform](#recipe_modules-platform), [properties](#recipe_modules-properties), [step](#recipe_modules-step)
-— **def [RunSteps](/recipe_modules/path/examples/full.py#17)(api):**
+— **def [RunSteps](/recipe_modules/path/examples/full.py#16)(api):**
+### *recipes* / [path:tests/cast\_to\_path](/recipe_modules/path/tests/cast_to_path.py)
+
+[DEPS](/recipe_modules/path/tests/cast_to_path.py#8): [path](#recipe_modules-path), [platform](#recipe_modules-platform)
+
+
+— **def [RunSteps](/recipe_modules/path/tests/cast_to_path.py#14)(api):**
### *recipes* / [path:tests/dynamic\_paths](/recipe_modules/path/tests/dynamic_paths.py)
[DEPS](/recipe_modules/path/tests/dynamic_paths.py#7): [path](#recipe_modules-path)
— **def [RunSteps](/recipe_modules/path/tests/dynamic_paths.py#10)(api):**
+### *recipes* / [path:tests/exists](/recipe_modules/path/tests/exists.py)
+
+[DEPS](/recipe_modules/path/tests/exists.py#7): [path](#recipe_modules-path)
+
+
+— **def [RunSteps](/recipe_modules/path/tests/exists.py#10)(api):**
+### *recipes* / [path:tests/test\_api\_legacy](/recipe_modules/path/tests/test_api_legacy.py)
+
+[DEPS](/recipe_modules/path/tests/test_api_legacy.py#8): [path](#recipe_modules-path)
+
+
+Test to cover legacy aspects of PathTestApi.
+
+— **def [RunSteps](/recipe_modules/path/tests/test_api_legacy.py#19)(api):**
### *recipes* / [placeholder](/recipes/placeholder.py)
[DEPS](/recipes/placeholder.py#5): [buildbucket](#recipe_modules-buildbucket), [properties](#recipe_modules-properties), [step](#recipe_modules-step), [swarming](#recipe_modules-swarming), [time](#recipe_modules-time)
diff --git a/recipe_engine/config_types.py b/recipe_engine/config_types.py
index f350752..38d63da 100644
--- a/recipe_engine/config_types.py
+++ b/recipe_engine/config_types.py
@@ -4,199 +4,307 @@
from __future__ import annotations
-import collections
import itertools
-from typing import Any
-import abc
-import os
+from dataclasses import dataclass, field
+from typing import ClassVar, TYPE_CHECKING
+if TYPE_CHECKING:
+ from recipe_engine.internal.recipe_deps import RecipeModule, Recipe, RecipeRepo
def ResetGlobalVariableAssignments():
- RecipeConfigType._TOSTRING_MAP.clear() # pylint: disable=W0212
- NamedBasePath._API = None
+ """This function is called from inside of the recipe test runner prior to each
+ test case executed.
-
-class RecipeConfigType:
- """Base class for custom Recipe config types, intended to be subclassed.
-
- RecipeConfigTypes are meant to be PURE data. There should be no dependency on
- any external systems (i.e. no importing sys, os, etc.).
-
- The subclasses should override default_tostring_fn. This method should
- produce a string representation of the object. This string representation
- should contain all of the data members of the subclass. This representation
- will be used during the execution of the recipe_config_tests.
-
- External entities (usually recipe modules), can override the default
- tostring_fn method by calling <RecipeConfigType
- subclass>.set_tostring_fn(<new method>). This new method will receive an
- instance of the RecipeConfigType subclass as its single argument, and is
- expected to return a string. There is no restriction on the data that the
- override tostring_fn may use. For example, the Path class in this module has
- its tostring_fn overridden by the 'path' recipe_module. This new tostring_fn
- uses data from the current recipe run, like the host os, to return platform
- specific strings using the data in the Path object.
+ See the class variables below for what they are and what sets them.
"""
- _TOSTRING_MAP = {}
-
- @property
- def tostring_fn(self):
- cls = self.__class__
- return self._TOSTRING_MAP.get(cls.__name__, cls.default_tostring_fn)
-
- @classmethod
- def set_tostring_fn(cls, new_tostring_fn):
- assert cls.__name__ not in cls._TOSTRING_MAP, (
- 'tostring_fn already installed for %s' % cls)
- cls._TOSTRING_MAP[cls.__name__] = new_tostring_fn
-
- def default_tostring_fn(self):
- raise NotImplementedError()
-
- def __str__(self):
- return self.tostring_fn(self) # pylint: disable=not-callable
+ CheckoutBasePath._resolved = None
+ Path._OS_SEP = None
-class BasePath(metaclass=abc.ABCMeta):
+@dataclass(frozen=True)
+class CheckoutBasePath:
+ """CheckoutBasePath is a placeholder base for Paths relative to
+ api.path.checkout_dir.
- @abc.abstractmethod
- def resolve(self, test_enabled: bool) -> str:
- """Returns a string representation of the path base.
+ This base is used in the following cases:
+ * Construction of Paths to be sent from GenTests to RunSteps (e.g. when
+ mocking paths with api.path.exists() from GenTests).
+ * In select circumstances when constructing Paths inside of the recipe
+ engine's "config" subsystem.
- Args:
- test_enabled: True iff this is only for recipe expectations.
+ Paths using CheckoutBasePath are 'slippery' and will try to resolve to
+ a ResolvedBasePath at almost every opportunity. Resolving a CheckoutBasePath
+ requires that the recipe has already assigned a value to checkout_dir in the
+ recipe_engine/path module, which in turn will assign to the
+ CheckoutBasePath._resolved class variable.
- Raises:
- NotImplementedError: If this method isn't overridden by a subclass.
+ If the checkout_dir has not yet been set, `resolve` on this class will raise
+ a ValueError stating as such.
+ """
+
+ # HACK: This is directly assigned to by the recipe_engine/path module in the
+ # checkout_dir setter.
+ #
+ # This is also reset by the ResetGlobalVariableAssignments() function in this
+ # file, which is called from the recipe tests prior to each test case.
+ _resolved: ClassVar[Path | None] = None
+
+ def maybe_resolve(self) -> Path | None:
+ """If CheckoutBasePath can be resolved to a real Path, return that,
+ otherwise return None."""
+ if self._resolved:
+ return self._resolved
+
+ def resolve(self) -> Path:
+ """Resolve this CheckoutBasePath, raise ValueError if it's not yet defined."""
+ checkout_dir = self.maybe_resolve()
+ if checkout_dir is None:
+ raise ValueError(
+ f'Cannot resolve CheckoutBasePath() - api.path.checkout_dir is unset.'
+ )
+ return checkout_dir
+
+ def __str__(self) -> str:
+ """Returns the resolved path as a string.
+
+ We never want to render CheckoutBasePath as anything other than the real
+ path base that it points to, which means that this will raise ValueError if
+ checkout_dir has not yet been assigned.
"""
- raise NotImplementedError()
+ return str(self.resolve())
+
+ def __repr__(self) -> str:
+ if self._resolved:
+ return str(self)
+ return 'CheckoutBasePath[UNRESOLVED]'
-class NamedBasePath(BasePath, collections.namedtuple('NamedBasePath', 'name')):
- _API = None
+@dataclass(frozen=True, order=True)
+class ResolvedBasePath:
+ """ResolvedBasePath represents a 'resolved' base path.
+
+ In tests, this will contain a string like "[START_DIR]", "[CACHE]", etc. These
+ names come from the recipe_engine/path module.
+
+ In non-tests, this will contain an actual absolute filesystem path as a string.
+ """
+ resolved: str
@classmethod
- def set_path_api(cls, api):
- cls._API = api
+ def for_recipe_module(cls, test_enabled: bool,
+ module: RecipeModule) -> ResolvedBasePath:
+ if not test_enabled:
+ return cls(module.path)
- def resolve(self, test_enabled: bool) -> str:
- if self.name == self._API.CheckoutPathName:
- checkout_dir = self._API.checkout_dir
- if checkout_dir is None:
- raise ValueError(
- f'Cannot resolve NamedBasePath({self.name!r}) - api.path.checkout_dir is unset.')
- return str(checkout_dir)
-
- if self.name in self._API.c.base_paths:
- if test_enabled:
- return repr(self)
- return self._API.join(
- *self._API.c.base_paths[self.name]) # pragma: no cover
-
- raise KeyError(
- 'Failed to resolve NamedBasePath: %s' % self.name) # pragma: no cover
-
- def __repr__(self):
- return '[%s]' % self.name.upper()
-
-
-class ModuleBasePath(BasePath, collections.namedtuple('ModuleBasePath',
- 'module')):
-
- def resolve(self, test_enabled):
- if test_enabled:
- return repr(self)
- return self.module.path # pragma: no cover
-
- def __repr__(self):
# We change python's module delimiter . to ::, since . is already used
# by expect tests.
- return f'RECIPE_MODULE[{self.module.repo.name}::{self.module.name}]'
+ return cls(f'RECIPE_MODULE[{module.repo.name}::{module.name}]')
+
+ @classmethod
+ def for_recipe_script_resources(cls, test_enabled: bool,
+ recipe: Recipe) -> ResolvedBasePath:
+ if not test_enabled:
+ return cls(recipe.resources_dir)
+ return cls(f'RECIPE[{recipe.full_name}].resources')
+
+ @classmethod
+ def for_bundled_repo(cls, test_enabled: bool,
+ repo: RecipeRepo) -> ResolvedBasePath:
+ if not test_enabled:
+ return cls(repo.path)
+ return cls(f'RECIPE_REPO[{repo.name}]')
+
+ def __repr__(self) -> str:
+ return self.resolved
-class RecipeScriptBasePath(BasePath,
- collections.namedtuple('RecipeScriptBasePath',
- 'recipe_name script_path')):
+@dataclass(frozen=True)
+class Path:
+ """Represents an absolute path which is relative to a 'base' path.
- def resolve(self, test_enabled):
- if test_enabled:
- return repr(self)
- return os.path.splitext(
- self.script_path)[0] + '.resources' # pragma: no cover
+ The `base` is either a ResolvedBasePath or a CheckoutBasePath.
- def __repr__(self):
- return 'RECIPE[%s].resources' % self.recipe_name
-
-
-class RepoBasePath(BasePath,
- collections.namedtuple('RepoBasePath',
- 'repo_name repo_root_path')):
-
- def resolve(self, test_enabled):
- if test_enabled:
- return repr(self)
- return self.repo_root_path # pragma: no cover
-
- def __repr__(self):
- return 'RECIPE_REPO[%s]' % self.repo_name
-
-
-class Path(RecipeConfigType):
- """Represents a path which is relative to a semantically-named base.
-
- Because there's a lot of platform (separator style) and runtime-specific
- context (working directory) which goes into assembling a final OS-specific
- absolute path, we only store three context-free attributes in this Path
- object.
+ This Path is made aware of the currently simulated path separator from the
+ __init__ method of the recipe_engine/path module, which assigns to this
+ class's _OS_SEP variable.
"""
+ base: CheckoutBasePath | ResolvedBasePath
+ pieces: tuple[str, ...]
- def __init__(self,
- base: BasePath,
- *pieces: str):
+ # HACK: This is directly assigned to by the recipe_engine/path module, and is
+ # populated with the current path separator character (either '/' or '\\').
+ #
+ # This is also reset by the ResetGlobalVariableAssignments() function in this
+ # file, which is called from the recipe tests prior to each test case.
+ _OS_SEP: ClassVar[str | None] = None
+
+ # This field is used to cache the output of __str__.
+ #
+ # Why not use @functools.cache on __str__? Unfortunately, this effectively
+ # creates a global variable Path.__str__.<wrapper func>.cache, which is a dict
+ # mapping Path instances to their __str__() values. This is 'fine', but it ends
+ # up capturing the value of _OS_SEP which can change multiple times per
+ # process run (see ResetGlobalVariableAssignments). This can still be used by
+ # adding Path.__str__.cache_clear() to ResetGlobalVariableAssignments, but
+ # I don't think introducing the extra global variable is necessary or
+ # desirable here, especially since we know that Path is immutable.
+ _str: str | None = field(default=None, repr=False, hash=False, compare=False)
+
+ def __init__(self, base: CheckoutBasePath | ResolvedBasePath, *pieces: str):
"""Creates a Path.
Args:
- base: The 'name' of a base path, to be filled in at recipe runtime
- by the 'path' recipe module.
- *pieces: The components of the path relative to base. These pieces must
- be non-relative (i.e. no '..' or '.', etc. as a piece).
+ base: Either a CheckoutBasePath, which represents a placeholder for
+ a ResolvedBasePath, or a ResolvedBasePath.
+ *pieces: The components of the path relative to base.
+ - If this recipe is being run on windows, pieces with '/' or '\\' will be
+ split. On non-windows, they will be split only on '/'.
+ - Split pieces equaling '..' must not go above the `base`. That is, if
+ you give `Path(ResolvedBasePath('[CACHE]'), '..', 'something')`, this will
+ raise ValueError because the '..' would bring this Path above the
+ base. However, `Path(ResolvedBasePath('[CACHE]'), 'something', '..')`
+ is OK and would be equivalent to `Path(ResolvedBasePath('[CACHE]'))`.
+ - Empty pieces and pieces which are '.' are ignored.
+ - If the recipe is not yet running (e.g. you are calling Path from
+ GenTests), and you include a '\\' in a piece, this will raise
+ ValueError (just use '/' or separate the pieces yourself in that case).
"""
super().__init__()
- assert isinstance(base, BasePath), base
- assert all(isinstance(x, str) for x in pieces), pieces
- assert not any(x in ('..', '/', '\\') for x in pieces)
+ if not isinstance(base, (CheckoutBasePath, ResolvedBasePath)):
+ raise ValueError(
+ 'First argument to Path must be CheckoutBasePath or ResolvedBasePath, '
+ f'got {base!r} ({type(base)!r})')
- self._base = base
- self._pieces = tuple(p for p in pieces if p != '.')
+ # If they gave us a CheckoutBasePath, but it's already resolvable,
+ # immediately transmute it into a ResolvedBasePath.
+ if isinstance(base, CheckoutBasePath):
+ if resolved_path := base.maybe_resolve():
+ base = resolved_path.base
+ pieces = resolved_path.pieces + tuple(pieces)
- @property
- def base(self) -> BasePath:
- return self._base
+ has_backslashes: bool = False
+ for i, piece in enumerate(pieces):
+ if not isinstance(piece, str):
+ raise ValueError('Variadic arguments to Path must only be `str`, '
+ f'argument {i} was {piece!r} ({type(piece)!r})')
+ has_backslashes = has_backslashes or '\\' in piece
- @property
- def pieces(self) -> tuple[str, ...]:
- return self._pieces
+ # NOTE: we always separate on '/', regardless of _OS_SEP, as users like to
+ # pass pieces to Path constructors and join() which contain a slash already
+ # (but almost never pass them with '\\').
+ #
+ # However, if they make a path, using backslash in pieces, during GenTests,
+ # we can't tell if these should be separated or not and so raise a ValueError.
+ if self._OS_SEP is None and has_backslashes:
+ raise ValueError(
+ f'Cannot instantiate Path({base!r}, {pieces!r}) - Pieces contain'
+ ' backslash and recipe_engine/path has not been initialized yet.'
+ ' Please use "/" (even for windows) or pass the pieces to join'
+ ' separately.')
+ need_backslash_split = has_backslashes and self._OS_SEP == '\\'
- def __eq__(self, other: Path) -> bool:
- return (self.base == other.base and
- self.pieces == other.pieces)
+ normalized_pieces = []
+ for piece in pieces:
+ slash_pieces = piece.split('/')
+ if need_backslash_split:
+ new_slash_pieces = []
+ for sp in slash_pieces:
+ new_slash_pieces.extend(sp.split(self._OS_SEP))
+ slash_pieces = new_slash_pieces
+ normalized_pieces.extend(p for p in slash_pieces if p and p != '.')
- def __hash__(self) -> int:
- return hash((
- self.base,
- self.pieces,
- ))
+ # At this point normalized_pieces is pieces but where:
+ # * All pieces have been split by / and/or \ - there are no more
+ # splittable slashes in normalized_pieces.
+ # * All empty pieces (which are '' or '.' pieces) have been removed.
- def __ne__(self, other: Any) -> bool:
- return not self == other
+ # Next, we normalize '..' passed in - This is allowed as long as it can be
+ # fully resolved within the given pieces. Otherwise we'll raise an exception
+ # if a joined '..' would take us above the base of this Path.
+ #
+ # Note that we start i at 1 and not 0: if pieces[0] == '..', this will be
+ # caught in the next check section.
+ i = 1
+ while 0 < i < len(normalized_pieces):
+ piece = normalized_pieces[i]
+ if piece == '..':
+ # At this point normalized_pieces looks like:
+ # 'previous' 'something' '..' 'other' 'things'
+ # i-2 [i-1 i ] i+1 i+2
+ #
+ # The [section] is the items in [i-1:i+1] (this syntax is a half-open
+ # range). Assigning `[]` to this range will remove the previous element,
+ # and also the '..'. This shifts the list to now be:
+ # 'previous' 'other' 'things'
+ # i-2 i-1 i
+ #
+ # Which means that we need to decrement i by one to evaluate 'other',
+ # which is the next item to analyze.
+ normalized_pieces[i - 1:i + 1] = []
+ i -= 1
+ else:
+ i += 1
- def __lt__(self, other: Path) -> bool:
- if self.base != other.base:
- # NOTE: bases all happen to extend namedtuple, which makes this comparison
- # work.
- return self.base < other.base
- return self.pieces < other.pieces
+ # Finally, check to see if any '..' was left over and raise.
+ if normalized_pieces and normalized_pieces[0] == '..':
+ raise ValueError(
+ f'Unable to compute {base!r} / {pieces!r} without going above the base.'
+ )
+
+ # This is a frozen dataclass, so we have to assign using object.__setattr__.
+ # Believe it or not, this is actually documented in
+ # https://docs.python.org/3.11/library/dataclasses.html#frozen-instances
+ object.__setattr__(self, 'base', base)
+ object.__setattr__(self, 'pieces', tuple(normalized_pieces))
+
+ def _resolve(self) -> Path:
+ """If self.base is a ResolvedBasePath, this will return self.
+
+ Otherwise, this will resolve self.base and return an equivalent path to
+ `self` but with a ResolvedBasePath base. If CheckoutBasePath is
+ unresolvable, this raises ValueError.
+ """
+ if not isinstance(self.base, CheckoutBasePath):
+ return self
+ return self.base.resolve().join(*self.pieces)
+
+ def __eq__(self, other: Path | str) -> bool:
+ if isinstance(other, str):
+ return str(self) == other
+
+ # first, if both bases are checkout, just compare pieces, since we know that
+ # CheckoutBasePath, once assigned, will always match.
+ if (isinstance(self.base, CheckoutBasePath) and
+ isinstance(other.base, CheckoutBasePath)):
+ return self.pieces == other.pieces
+
+ try:
+ spath = self._resolve()
+ opath = other._resolve()
+ return spath.base == opath.base and spath.pieces == opath.pieces
+ except Exception as ex:
+ raise ValueError('Path.__eq__ invalid for mismatched bases '
+ '(CheckoutBasePath vs ResolvedBasePath) '
+ 'before checkout_dir is set') from ex
+
+ def __lt__(self, other: Path | str) -> bool:
+ if isinstance(other, str):
+ return str(self) < other
+
+ # first, if both bases are checkout, just compare pieces, since we know that
+ # CheckoutBasePath, once assigned, will always match.
+ if (isinstance(self.base, CheckoutBasePath) and
+ isinstance(other.base, CheckoutBasePath)):
+ return self.pieces < other.pieces
+ try:
+ spath = self._resolve()
+ opath = other._resolve()
+ return (spath.base, spath.pieces) < (opath.base, opath.pieces)
+ except Exception as ex:
+ raise ValueError('Path.__lt__ invalid for mismatched bases '
+ '(CheckoutBasePath vs ResolvedBasePath) '
+ 'before checkout_dir is set') from ex
def __truediv__(self, piece: str) -> Path:
"""Adds the shorthand '/'-operator for .join(), returning a new path."""
@@ -208,8 +316,8 @@
Empty values ('', None) in pieces will be omitted.
Args:
- pieces: The components of the path relative to base. These pieces must be
- non-relative (i.e. no '..' as a piece).
+ pieces: The components of the path relative to base. The normal Path
+ __init__ rules for '..' and '.' apply.
Returns:
The new Path.
@@ -220,33 +328,58 @@
self.base,
*[p for p in itertools.chain(self.pieces, pieces) if p])
- def is_parent_of(self, child: Path) -> bool:
- """True if |child| is in a subdirectory of this path."""
- # Assumes base paths are not nested.
- # TODO(vadimsh): We should not rely on this assumption.
- if self.base != child.base:
+ def is_parent_of(self, other: Path) -> bool:
+ """True if |other| is in a subdirectory of this Path."""
+ spath = self
+ opath = other
+ # If they are BOTH CheckoutBasePath we can use them directly, otherwise we
+ # need to resolve them (which may raise)
+ if not (isinstance(self.base, CheckoutBasePath) and
+ isinstance(other.base, CheckoutBasePath)):
+ spath = self._resolve()
+ opath = other._resolve()
+
+ # NOTE: This assumes that none of the ResolvedBasePath's overlap, which is
+ # currently true, and simplifies things quite a bit.
+ if spath.base != opath.base:
return False
- # A path is not a parent to itself.
- if len(self.pieces) >= len(child.pieces):
- return False
- return child.pieces[:len(self.pieces)] == self.pieces
- def separate(self, separator: str) -> None:
- """Breaks apart any pieces of self.pieces containing the separator.
+ # They have the same base, so return True if the paths have a matching prefix.
+ #
+ # If spath.pieces is longer than opath.pieces, this will be False, which is
+ # correct.
+ return spath.pieces == opath.pieces[:len(spath.pieces)]
- Example: If self.pieces is ('foo', 'bar/baz') and separator='/', then
- self.pieces will be transformed into ('foo', 'bar', 'baz'). This allows for
- more accurate comparisons, like equality or parenthood.
-
- Args:
- separator: The file separator character for this platform: '/' for POSIX,
- '\\' for Windows. Usually fetched via api.path.sep.
- """
- self._pieces = sum((tuple(piece.split(separator)) for piece in self.pieces),
- start=())
+ def __str__(self) -> str:
+ if self._str is None:
+ if not self._OS_SEP:
+ raise ValueError('Unable to render Path to string - '
+ 'recipe_engine/path has not been initialized yet.')
+ str_val = self._OS_SEP.join(itertools.chain((str(self.base),), self.pieces))
+ object.__setattr__(self, '_str', str_val)
+ return str_val
+ return self._str
def __repr__(self) -> str:
- s = 'Path(%r' % (self.base,)
- if self.pieces:
- s += ', %s' % ', '.join(repr(x) for x in self.pieces)
+ # Try to resolve `self` - if it's rooted in CheckoutBasePath and
+ # checkout_dir is already set, display the fully resolved path, since all
+ # interactions with this Path will behave in that fashion.
+ #
+ # Otherwise, just use `self` as-is to allow repr(Path) to work in scenarios
+ # where checkout_dir hasn't been set (for example, when reporting errors
+ # involving unresolved checkout_dir paths...)
+ try:
+ spath = self._resolve()
+ except ValueError:
+ spath = self
+
+ # NOTE: It would be good to switch to the dataclass-repr instead (or just
+ # use __str__ all the time)
+ s = 'Path(%r' % (spath.base,)
+ if spath.pieces:
+ s += ', %s' % ', '.join(repr(x) for x in spath.pieces)
return s + ')'
+
+ def __hash__(self) -> int:
+ spath = self._resolve()
+ return hash(('config_types.Path', spath.base, spath.pieces))
diff --git a/recipe_engine/internal/recipe_deps.py b/recipe_engine/internal/recipe_deps.py
index bdcaba5..51d2bc5 100644
--- a/recipe_engine/internal/recipe_deps.py
+++ b/recipe_engine/internal/recipe_deps.py
@@ -52,7 +52,7 @@
from google.protobuf import json_format as jsonpb
-from ..config_types import Path, RepoBasePath, RecipeScriptBasePath
+from ..config_types import Path, ResolvedBasePath
from ..engine_types import freeze, FrozenDict
from ..recipe_api import UnresolvedRequirement, RecipeScriptApi, BoundProperty
from ..recipe_api import RecipeApi
@@ -944,10 +944,10 @@
api = RecipeScriptApi(
test_data.get_module_test_data(None),
- Path(RecipeScriptBasePath(
- self.full_name,
- os.path.splitext(self.path)[0]+".resources")),
- Path(RepoBasePath(self.repo.name, self.repo.path)),
+ Path(
+ ResolvedBasePath.for_recipe_script_resources(
+ test_data.enabled, self)),
+ Path(ResolvedBasePath.for_bundled_repo(test_data.enabled, self.repo)),
)
resolved_deps = _resolve(
self.repo.recipe_deps, self.normalized_DEPS, 'API', engine, test_data)
diff --git a/recipe_engine/recipe_api.py b/recipe_engine/recipe_api.py
index fc0bfaf..67fe28d 100644
--- a/recipe_engine/recipe_api.py
+++ b/recipe_engine/recipe_api.py
@@ -168,7 +168,8 @@
self.path_strings.append(path_string)
self.paths.append(path)
- def find_longest_prefix(self, target, sep):
+ def find_longest_prefix(self, target,
+ sep) -> tuple[str | None, config_types.Path | None]:
"""Identifies a known resource path which would contain the `target` path.
sep must be the current path separator (can vary from os.path.sep when
@@ -182,7 +183,7 @@
return (None, None) # off the end
sPath, path = self.path_strings[idx], self.paths[idx]
- if target == sPath :
+ if target == sPath:
return sPath, path
if idx > 0:
@@ -488,15 +489,14 @@
assert module
self._module = module
self._resource_directory = config_types.Path(
- config_types.ModuleBasePath(module)).join('resources')
+ config_types.ResolvedBasePath.for_recipe_module(
+ test_data.enabled, module)).join('resources')
self._repo_root = config_types.Path(
- config_types.RepoBasePath(
- module.repo.name,
- module.repo.path,
- ))
+ config_types.ResolvedBasePath.for_bundled_repo(test_data.enabled,
+ module.repo))
assert isinstance(test_data, (ModuleTestData, DisabledTestData))
- self._test_data = test_data
+ self._test_data: ModuleTestData | DisabledTestData = test_data
# If we're the 'root' api, inject directly into 'self'.
# Otherwise inject into 'self.m'
diff --git a/recipe_modules/path/api.py b/recipe_modules/path/api.py
index 717a837..e8a20b1 100644
--- a/recipe_modules/path/api.py
+++ b/recipe_modules/path/api.py
@@ -28,93 +28,189 @@
make code somewhat generic or homogeneous, but this was a mistake. New code
should avoid 'checkout', and instead just explicitly pass paths around. This
path may be removed in the future.
-
-There are other anchor points which can be defined (e.g. by the
-`depot_tools/infra_paths` module). Refer to those modules for additional
-documentation.
"""
from __future__ import annotations
import collections
-import copy
import enum
+import ntpath
import os
+import posixpath
import re
import tempfile
-import types
-from typing import Any, Callable, Literal, cast
+
+from typing import Any, Callable, Literal
from recipe_engine import recipe_api, recipe_test_api
from recipe_engine import config_types
+from . import test_api
+
class FileType(enum.Enum):
FILE = 1
DIRECTORY = 2
CheckoutPathName = 'checkout'
+CheckoutPathNameType = Literal['checkout']
+NamedBasePathsType = CheckoutPathNameType | Literal[
+ 'cache',
+ 'cleanup',
+ 'home',
+ 'start_dir',
+ 'tmp_base',
+]
-class Error(Exception):
- """Error specific to path recipe module."""
+def _cast_to_path_impl(path_mod, strpath: str) -> config_types.Path:
+ """This is the core implementation of 'cast_to_path'.
+ This exists outside of PathApi, because it's also used to rationalize
+ UnvalidatedPaths in path_set.
-def PathToString(api, test):
+ The `path_mod` argument is always effectively either the
+ ntpath or posixpath module via either fake_path.__getattr__ in the path_set
+ case, or directly via PathApi._path_mod in the testing/production case.
+ Unfortunately, this is not currently expressible as a type annotation, because
+ modules are not allowed as types (even though logically they are representable
+ as a Protocol).
- def PathToString_inner(path):
- assert isinstance(path, config_types.Path)
- return api.join(path.base.resolve(test.enabled), *path.pieces)
-
- return PathToString_inner
+ This converts the string path to a Path using the current real/simulated
+ platform's implementation of splitdrive to form a ResolvedBasePath on the
+ 'drive', with the rest of the path being split into pieces using the default
+ config_types.Path constructor logic (i.e. using platform-aware path slash).
+ """
+ drive, path = path_mod.splitdrive(strpath)
+ # NOTE(crbug.com/329113288) - this should switch to isabs when we change
+ # config_types.Path to pathlib.Path. Currently isabs does the wrong thing with
+ # testing path roots like [CACHE], and ntpath.abspath won't add a fake drive
+ # (meaning that abspath(strpath) == strpath is not a good check.
+ if path_mod.sep == '\\':
+ if not drive:
+ raise ValueError(
+ f'Cannot use {strpath!r} with cast_to_path - not absolute.')
+ else:
+ if not strpath.startswith('/'):
+ raise ValueError(
+ f'Cannot use {strpath!r} with cast_to_path - not absolute.')
+ return config_types.Path(config_types.ResolvedBasePath(drive), path)
class path_set:
"""Implements a set which contains all the parents folders of added
- folders."""
+ folders.
- # TODO(iannucci): Expand this to be a full fakey filesystem, including file
+ This all boils down to a flat, sorted, list of (strpath, kind) pairs, where kind
+ is reductively just FILE or DIRECTORY. This is a far cry from a real
+ filesystem. See crbug.com/40890779.
+
+ The initial set of paths is populated via the PathTestApi's files_exist and
+ dirs_exist module data. These can either be regular config_types.Path
+ instances, based on a ResolvedBasePath or on a CheckoutBasePath, OR they can
+ be UnvalidatedPath instances, which path_set will validate and cast into
+ a config_types.Path prior to ingestion.
+
+ Paths based on CheckoutBasePath will be held in limbo in the _checkout_paths
+ attribute until the recipe assigns a concrete Path for checkout_dir, at which
+ point these buffered Paths will now spring into existence. This is definitely
+ abstraction-breaking, but some downstream recipes depend on this behavior, so
+ it will all need to be untangled carefully.
+ """
+
+ # BUG(crbug.com/40890779): Expand this to be a full fakey filesystem, including file
# contents and file types. Coordinate with the `file` module.
- def __init__(self, path_mod: fake_path, initial_paths):
- self._path_mod: types.ModuleType = path_mod
- self._initial_paths: set[config_types.Path]|None = set(initial_paths)
+ def __init__(self, path_mod: fake_path,
+ test_data: recipe_test_api.ModuleTestData):
+
+ # path_set is only ever used in the testing paths, so we know _path_mod is
+ # always a fake_path.
+ self._path_mod: fake_path = path_mod
+
+ # _checkout_paths are buffered until `mark_checkout_dir_set` has been called,
+ # at which point we know it's acceptable to render these Paths to strings.
+ self._checkout_paths: list[tuple[config_types.Path, FileType]] = []
+
+ initial_paths: list[tuple[config_types.Path, FileType]] = []
+ for filepath in test_data.get('files_exist', ()):
+ if isinstance(filepath, test_api.UnvalidatedPath):
+ filepath = _cast_to_path_impl(path_mod,
+ filepath.base).join(*filepath.pieces)
+ assert isinstance(filepath, config_types.Path), (
+ f'path.files_exist module test data contains non-Path {type(filepath)}'
+ )
+ initial_paths.append((filepath, FileType.FILE))
+
+ for dirpath in test_data.get('dirs_exist', ()):
+ if isinstance(dirpath, test_api.UnvalidatedPath):
+ dirpath = _cast_to_path_impl(path_mod,
+ dirpath.base).join(*dirpath.pieces)
+ assert isinstance(dirpath, config_types.Path), (
+ f'path.files_exist module test data contains non-Path {type(dirpath)}'
+ )
+ initial_paths.append((dirpath, FileType.DIRECTORY))
+
# An entry in self._paths means an object exists in the mock filesystem.
# The value (either FILE or DIRECTORY) is the type of that object.
- self._paths: dict[config_types.Path, FileType] = {}
+ self._paths: dict[str, FileType] = {}
+ for path, kind in initial_paths:
+ if not isinstance(path, config_types.Path): # pragma: no cover
+ raise ValueError(
+ 'String paths to `api.path.exists` in GenTests are not allowed.'
+ ' Use one of the _dir properties on `api.path` to get a Path, or '
+ ' use `api.path.cast_to_path`.')
- def _initialize(self) -> None: # pylint: disable=method-hidden
- self._initialize: Callable[[], None] = lambda: None
- for path in self._initial_paths:
- self.add(path, FileType.FILE)
- self._initial_paths = None
- self.contains: Callable[[config_types.Path], bool] = (
- lambda path: path in self._paths
- )
+ if isinstance(path.base, config_types.CheckoutBasePath):
+ self._checkout_paths.append((path, kind))
+ else:
+ self.add(path, kind)
- @property
- def _separator(self) -> str:
- return self._path_mod.sep
+ def mark_checkout_dir_set(self) -> None:
+ """This is called by PathApi once when checkout_dir is initially assigned to
+ a concrete Path.
- def _is_contained_in(self, path: config_types.Path,
- root: config_types.Path, match_root: bool) -> bool:
+ Note that a side-effect of the assignment in PathApi is updating the
+ CheckoutBasePath._resolved class variable, which makes it possible to render
+ the Paths in _checkout_paths to strings.
+ """
+ for path, kind in self._checkout_paths:
+ self.add(path, kind)
+ self._checkout_paths.clear()
+
+ def _is_contained_in(self, path: str, root: str, match_root: bool) -> bool:
+ """Returns True iff `path` is contained in `root`.
+ Returns `match_root` if `path` == `root`.
+ """
if not path.startswith(root):
return False
if len(path) == len(root):
return match_root
- return path[len(root)] == self._separator
+ # Note - this prevents simple lexical failures such as
+ # "/a/bcdef" in "/a/b"
+ # (both have the prefix "/a/b", but "/a/bcdef" is not contained in "/a/b")
+ return path[len(root)] == self._path_mod.sep
- def add(self, path: config_types.Path, kind: FileType):
- path = str(path)
- self._initialize()
+ def add(self, path: str | config_types.Path, kind: FileType):
+ """Marks the existence of `path`.
+
+ This also implicitly marks all parent directories of `path` to also exist
+ (as type DIRECTORY).
+ """
+ sPath: str = str(path)
prev_path: str|None = None
- while path != prev_path:
- self._paths[path] = kind
- prev_path, path = path, self._path_mod.dirname(path)
+ while sPath != prev_path:
+ self._paths[sPath] = kind
+ prev_path, sPath = sPath, self._path_mod.dirname(sPath)
kind = FileType.DIRECTORY
- def copy(self, source: config_types.Path, dest: config_types.Path) -> None:
+ def copy(self, source: str | config_types.Path,
+ dest: str | config_types.Path) -> None:
+ """Copies the existence criteria of all known paths contained in `source` to `dest`.
+
+ This also implicitly marks all parent directories of `path` to also exist
+ (as type DIRECTORY).
+ """
source, dest = str(source), str(dest)
- self._initialize()
to_add: dict[str, FileType] = {}
for p in self._paths:
if self._is_contained_in(p, source, match_root=True):
@@ -122,35 +218,32 @@
for path, kind in to_add.items():
self.add(path, kind)
- def remove(self, path: config_types.Path,
- filt: Callable[[config_types.Path], bool]) -> None:
- path: str = str(path)
- self._initialize()
+ def remove(self, path: str | config_types.Path, filt: Callable[[str],
+ bool]) -> None:
+ """Removes existence criteria for `path`, and any other paths it contains.
+
+ `filt` is a required filter function. It will be called for each path
+ contained in `path`, and if it returns True, the path will be removed from
+ this path_set's existence list.
+ """
+ path = str(path)
match_root: bool = True
- if path[-1] == self._separator:
+ if path[-1] == self._path_mod.sep:
match_root = False
- path: str = path.rstrip(self._separator)
- kill_set: set[config_types.Path] = set(
+ path = path.rstrip(self._path_mod.sep)
+ kill_set: set[str] = set(
p for p in self._paths
if self._is_contained_in(p, path, match_root) and filt(p))
for entry in kill_set:
del self._paths[entry]
- # pylint: disable=method-hidden
- def contains(self, path: config_types.Path) -> bool:
- self._initialize()
- return self.contains(path)
+ def contains(self, path: str) -> bool:
+ return path in self._paths
- def kind(self, path: config_types.Path) -> FileType:
- self._initialize()
+ def kind(self, path: str) -> FileType:
return self._paths[path]
-import ntpath
-import posixpath
-PathCommonModule = Literal[ntpath, posixpath]
-
-
class fake_path:
"""Standin for os.path when we're in test mode.
@@ -160,14 +253,10 @@
the platform which is currently running.
"""
- def __init__(self, is_windows: bool, _mock_path_exists):
- if is_windows:
- import ntpath as pth
- else:
- import posixpath as pth
-
- self._pth: PathCommonModule = pth
- self._mock_path_exists = path_set(self, _mock_path_exists)
+ def __init__(self, is_windows: bool,
+ test_data: recipe_test_api.ModuleTestData):
+ self._pth = ntpath if is_windows else posixpath
+ self._mock_path_exists = path_set(self, test_data)
def __getattr__(self, name: str) -> Any:
return getattr(self._pth, name)
@@ -177,27 +266,31 @@
assert kind in FileType
self._mock_path_exists.add(path, kind)
- def mock_copy_paths(self, source: config_types.Path,
- dest: config_types.Path) -> None:
+ def mock_copy_paths(self, source: str, dest: str) -> None:
"""Duplicates a path and all of its children to another path."""
self._mock_path_exists.copy(source, dest)
- def mock_remove_paths(self, path: config_types.Path,
- filt: Callable[[config_types.Path], bool]) -> None:
+ def mock_remove_paths(self, path: str, filt: Callable[[str], bool]) -> None:
"""Removes a path and all of its children from the set of existing paths."""
self._mock_path_exists.remove(path, filt)
- def exists(self, path: config_types.Path) -> bool: # pylint: disable=E0202
+ # NOTE: These have `path: str` instead of config_types.Path because the
+ # api._path_mod type is the intersection of (os.path && fake_path) - even if
+ # these are strictly defined as config_types.Path, it will not enable better
+ # type checking, because os.path is not defined in terms of
+ # config_types.Path.
+
+ def exists(self, path: str) -> bool: # pylint: disable=method-hidden
"""Returns True if path refers to an existing path."""
return self._mock_path_exists.contains(path)
- def isdir(self, path: config_types.Path) -> bool:
- return (self.exists(path) and
- self._mock_path_exists.kind(path) == FileType.DIRECTORY)
+ def isdir(self, path: str) -> bool:
+ return self.exists(path) and self._mock_path_exists.kind(
+ path) == FileType.DIRECTORY
- def isfile(self, path: config_types.Path) -> bool:
- return (self.exists(path) and
- self._mock_path_exists.kind(path) == FileType.FILE)
+ def isfile(self, path: str) -> bool:
+ return self.exists(path) and self._mock_path_exists.kind(
+ path) == FileType.FILE
# This matches:
# [START_DIR]
@@ -206,7 +299,7 @@
# and friends at the beginning of a string.
ROOT_MATCHER = re.compile(r'^[A-Z_]*\[[^]]*\]')
- def normpath(self, path: config_types.Path) -> config_types.Path:
+ def normpath(self, path: str) -> str:
"""Normalizes the path.
This splits off a recipe base (i.e. RECIPE[...]) so that normpath is
@@ -223,17 +316,18 @@
return prefix + real_normpath(rest)
return real_normpath(path)
- def abspath(self, path: config_types.Path) -> config_types.Path:
+ def abspath(self, path: str) -> str:
"""Returns the absolute version of path."""
return self.normpath(path)
- def realpath(self, path: config_types.Path) -> config_types.Path:
+ def realpath(self, path: str) -> str:
"""Returns the canonical version of the path."""
return self.normpath(path)
class PathApi(recipe_api.RecipeApi):
- _paths_client = recipe_api.RequireClient('paths')
+ _paths_client: recipe_api.PathsClient | recipe_api.UnresolvedRequirement = recipe_api.RequireClient(
+ 'paths')
# This is the literal string 'checkout'.
#
@@ -242,34 +336,30 @@
# future. Do not use this.
#
# Use the .checkout_dir @property directly, instead.
- CheckoutPathName = CheckoutPathName
+ CheckoutPathName = 'checkout'
- def get_config_defaults(self) -> dict[str, Any]:
- """Internal recipe implementation function."""
- # TODO(iannucci): Completely remove config from path.
- return {
- 'START_DIR': self._startup_cwd,
- 'TEMP_DIR': self._temp_dir,
- 'CACHE_DIR': self._cache_dir,
- 'CLEANUP_DIR': self._cleanup_dir,
- 'HOME_DIR': self._home_dir,
- }
+ # This is a frozenset of all the named base paths that this module knows
+ # about.
+ NamedBasePaths = frozenset([
+ CheckoutPathName,
+ 'cache',
+ 'cleanup',
+ 'home',
+ 'start_dir',
+ 'tmp_base',
+ ])
def __init__(self, path_properties, **kwargs):
super().__init__(**kwargs)
- config_types.Path.set_tostring_fn(PathToString(self, self._test_data))
- config_types.NamedBasePath.set_path_api(self)
- self._path_properties = path_properties
+ self._start_dir: str
+ self._temp_dir: str
+ self._home_dir: str
- # Assigned at "initialize".
- # NT or POSIX path module, or "os.path" in prod.
- self._path_mod: ModuleType|None = None
- self._startup_cwd: config_types.Path|None = None
- self._temp_dir: config_types.Path|None = None
- self._cache_dir: config_types.Path|None = None
- self._cleanup_dir: config_types.Path|None = None
- self._home_dir: config_types.Path|None = None
+ # These are populated in __init__ OR in initialize, but the rest of the
+ # module will always see them as populated values.
+ self._cleanup_dir: str = ""
+ self._cache_dir: str = ""
# checkout_dir can be set at most once per recipe run.
self._checkout_dir: config_types.Path|None = None
@@ -277,27 +367,92 @@
# Used in mkdtemp and mkstemp when generating and checking expectations.
self._test_counter: collections.Counter = collections.Counter()
- def _read_path(self, property_name, default): # pragma: no cover
- """Reads a path from a property. If absent, returns the default.
+ if not self._test_data.enabled: # pragma: no cover
+ self._path_mod = os.path
- Validates that the path is absolute.
+ # HACK: config_types.Path._OS_SEP is a global variable.
+ # This gets reset by config_types.ResetGlobalVariableAssignments()
+ config_types.Path._OS_SEP = self._path_mod.sep
+
+ for key in ('temp_dir', 'cache_dir', 'cleanup_dir'):
+ value = path_properties.get(key)
+ if value and not os.path.isabs(value):
+ raise ValueError(
+ f'Path {value!r} in path module property {key!r} is not absolute')
+
+ # These we can compute without _paths_client.
+ self._home_dir: str = self._path_mod.expanduser('~')
+ self._temp_dir = path_properties.get('temp_dir', tempfile.gettempdir())
+
+ # These MAY be provided via the module properties - if they are, set them
+ # here, otherwise they will be populated in initialize().
+ if cache_dir := path_properties.get('cache_dir'):
+ self._cache_dir = cache_dir
+ if cleanup_dir := path_properties.get('cleanup_dir'):
+ self._cleanup_dir = cleanup_dir
+
+ else:
+ assert not isinstance(self._test_data, recipe_test_api.DisabledTestData)
+
+ for key in ('temp_dir', 'cache_dir', 'cleanup_dir'):
+ if value := path_properties.get(key): # pragma: no cover
+ raise ValueError(
+ f'Base path mocking is not supported - got {key} = {value!r}')
+
+ # HACK: The platform test_api sets platform.name specifically for the
+ # path module when users use api.platform.name(...) in their tests.
+ # This is dirty, but it avoids a LOT of interdependency complexity.
+ _test_platform = self._test_data.get('platform.name', 'linux')
+
+ self._cache_dir = '[CACHE]'
+ self._cleanup_dir = '[CLEANUP]'
+ self._home_dir = '[HOME]'
+ self._start_dir = '[START_DIR]'
+ self._temp_dir = '[TMP_BASE]'
+
+ is_windows = _test_platform == 'win'
+
+ # HACK: config_types.Path._OS_SEP is a global variable.
+ # This gets reset by config_types.ResetGlobalVariableAssignments()
+ config_types.Path._OS_SEP = '\\' if is_windows else '/'
+
+ # NOTE: This depends on _OS_SEP being set.
+ self._path_mod = fake_path(is_windows, self._test_data)
+
+ self.mock_add_directory(self.cache_dir)
+ self.mock_add_directory(self.cleanup_dir)
+ self.mock_add_directory(self.home_dir)
+ self.mock_add_directory(self.start_dir)
+ self.mock_add_directory(self.tmp_base_dir)
+
+ def initialize(self):
+ """This is called by the recipe engine immediately after __init__(), but
+ with `self._paths_client` initialized.
"""
- value = self._path_properties.get(property_name)
- if not value:
- assert os.path.isabs(default), default
- return default
- if not os.path.isabs(value):
- raise Error('Path "%s" specified by module property %s is not absolute' %
- (value, property_name))
- return value
+ if not self._test_data.enabled: # pragma: no cover
+ # These paths can only be set with _paths_client, so we do them here in
+ # initialize().
+
+ self._start_dir = self._paths_client.start_dir
+ if not self._cache_dir:
+ self._cache_dir = os.path.join(self._start_dir, 'cache')
+
+ # If no cleanup directory is specified, assume that any directory
+ # underneath of the working directory is transient and will be purged in
+ # between builds.
+ if not self._cleanup_dir:
+ self._cleanup_dir = os.path.join(self._start_dir, 'recipe_cleanup')
+
+ self._ensure_dir(self._temp_dir)
+ self._ensure_dir(self._cache_dir)
+ self._ensure_dir(self._cleanup_dir)
def _ensure_dir(self, path: str) -> None: # pragma: no cover
os.makedirs(path, exist_ok=True)
- def _split_path(self, path: config_types.Path
- ) -> tuple[str, ...]: # pragma: no cover
+ def _split_path(self, path: str) -> tuple[str, ...]: # pragma: no cover
"""Relative or absolute path -> tuple of components."""
- abs_path: list[str, ...] = os.path.abspath(path).split(self.sep)
+ abs_path: list[str] = os.path.abspath(path).split(self.sep)
# Guarantee that the first element is an absolute drive or the posix root.
if abs_path[0].endswith(':'):
abs_path[0] += '\\'
@@ -307,59 +462,6 @@
assert False, 'Got unexpected path format: %r' % abs_path
return tuple(abs_path)
- def initialize(self) -> None:
- """Internal recipe implementation function."""
- if not self._test_data.enabled: # pragma: no cover
- self._path_mod: ModuleType = os.path
- start_dir = self._paths_client.start_dir
- self._startup_cwd = self._split_path(start_dir)
- self._home_dir = self._split_path(self._path_mod.expanduser('~'))
-
- tmp_dir = self._read_path('temp_dir', tempfile.gettempdir())
- self._ensure_dir(tmp_dir)
- self._temp_dir = self._split_path(tmp_dir)
-
- cache_dir = self._read_path('cache_dir', os.path.join(start_dir, 'cache'))
- self._ensure_dir(cache_dir)
- self._cache_dir = self._split_path(cache_dir)
-
- # If no cleanup directory is specified, assume that any directory
- # underneath of the working directory is transient and will be purged in
- # between builds.
- cleanup_dir = self._read_path('cleanup_dir',
- os.path.join(start_dir, 'recipe_cleanup'))
- self._ensure_dir(cleanup_dir)
- self._cleanup_dir = self._split_path(cleanup_dir)
- else:
- tdata = cast(recipe_test_api.ModuleTestData, self._test_data)
- # HACK: The platform test_api sets platform.name specifically for the
- # path module when users use api.platform.name(...) in their tests.
- # This is dirty, but it avoids a LOT of interdependency complexity.
- #
- # In the current version of this code, we initialize _path_mod in
- # `initialize` (rather than __init__) which is already late, but we also
- # are calling the set_tostring_fn and set_path_api global variable hacks
- # in __init__ which globally modify the behavior of NamedBasePath and Path
- # across the entire process.
- #
- # In a subsequent CL, we will be able to move _path_mod initialization
- # into __init__, and remove the set_tostring_fn/set_path_api
- # interdependency, and we will also be able to return fully-encapsulated
- # Path objects from this module.
- is_windows: bool = tdata.get('platform.name', 'linux') == 'win'
-
- self._path_mod = fake_path(is_windows, tdata.get('exists', []))
-
- root: str = 'C:\\' if is_windows else '/'
- self._startup_cwd = [root, 'b', 'FakeTestingCWD']
- # Appended to placeholder '[TMP]' to get fake path in test.
- self._temp_dir = [root]
- self._cache_dir = [root, 'b', 'c']
- self._cleanup_dir = [root, 'b', 'cleanup']
- self._home_dir = [root, 'home', 'fake_user']
-
- self.set_config('BASE')
-
def assert_absolute(self, path: config_types.Path | str) -> None:
"""Raises AssertionError if the given path is not an absolute path.
@@ -377,19 +479,18 @@
Returns a Path to the new directory.
"""
+ assert isinstance(prefix, str), f'Prefix is not a string: {type(prefix)}'
+
if not self._test_data.enabled: # pragma: no cover
- # New path as str.
- new_path = tempfile.mkdtemp(prefix=prefix, dir=str(self['cleanup']))
- # Ensure it's under self._cleanup_dir, convert to Path.
- new_path = self._split_path(new_path)
- assert new_path[:len(self._cleanup_dir)] == self._cleanup_dir, (
- 'new_path: %r -- cleanup_dir: %r' % (new_path, self._cleanup_dir))
- temp_dir = self['cleanup'].join(*new_path[len(self._cleanup_dir):])
+ cleanup_dir = str(self.cleanup_dir) + self.sep
+ new_path = tempfile.mkdtemp(prefix=prefix, dir=cleanup_dir)
+ assert new_path.startswith(cleanup_dir), (
+ f'{new_path=!r} -- {cleanup_dir=!r}')
+ temp_dir = self['cleanup'].join(new_path[len(cleanup_dir):])
else:
self._test_counter[prefix] += 1
- assert isinstance(prefix, str)
- temp_dir = self['cleanup'].join('%s_tmp_%d' %
- (prefix, self._test_counter[prefix]))
+ temp_dir = self['cleanup'].join(
+ f'{prefix}_tmp_{self._test_counter[prefix]}')
self.mock_add_paths(temp_dir, FileType.DIRECTORY)
return temp_dir
@@ -399,24 +500,25 @@
Args:
* prefix - a tempfile template for the file name (defaults to "tmp").
- Returns a Path to the new file. Unlike tempfile.mkstemp, the file's file
- descriptor is closed.
+ Returns a Path to the new file.
+
+ NOTE: Unlike tempfile.mkstemp, the file's file descriptor is closed. If you
+ need the full security properties of mkstemp, please outsource this to e.g.
+ either a resource script of your recipe module or recipe.
"""
+ assert isinstance(prefix, str), f'Prefix is not a string: {type(prefix)}'
+
if not self._test_data.enabled: # pragma: no cover
- # New path as str.
- fd, new_path = tempfile.mkstemp(prefix=prefix, dir=str(self['cleanup']))
- # Ensure it's under self._cleanup_dir, convert to Path.
- split_path: list[str] = self._split_path(new_path)
- assert split_path[:len(self._cleanup_dir)] == self._cleanup_dir, (
- 'new_path: %r -- cleanup_dir: %r' % (split_path, self._cleanup_dir))
- temp_file: config_types.Path = self['cleanup'].join(
- *split_path[len(self._cleanup_dir):])
+ cleanup_dir = str(self.cleanup_dir) + self.sep
+ fd, new_path = tempfile.mkstemp(prefix=prefix, dir=cleanup_dir)
+ assert new_path.startswith(cleanup_dir), (
+ f'{new_path=!r} -- {cleanup_dir=!r}')
+ temp_file = self['cleanup'].join(new_path[len(cleanup_dir):])
os.close(fd)
else:
self._test_counter[prefix] += 1
- assert isinstance(prefix, str)
- temp_file: config_types.Path = self['cleanup'].join(
- '%s_tmp_%d' % (prefix, self._test_counter[prefix]))
+ temp_file = self['cleanup'].join(
+ f'{prefix}_tmp_{self._test_counter[prefix]}')
self.mock_add_paths(temp_file, FileType.FILE)
return temp_file
@@ -433,7 +535,10 @@
* recipe resource paths
* repo paths
* checkout_dir
- * base_paths
+ * home_dir
+ * start_dir
+ * tmp_base_dir
+ * cleanup_dir
Example:
```
@@ -456,41 +561,61 @@
sPath, path = self._paths_client.find_longest_prefix(
abs_string_path, self.sep)
if path is None:
- # try base paths now
- to_try = self.c.base_paths.keys()
- if self.checkout_dir is not None:
- to_try = [CheckoutPathName] + list(to_try)
- for path_name in to_try:
- path = self[path_name]
- sPath = str(path)
- if abs_string_path.startswith(sPath):
- break
+ to_try = [
+ self.cache_dir,
+ self.checkout_dir,
+ self.cleanup_dir,
+ self.home_dir,
+ self.start_dir,
+ self.tmp_base_dir,
+ ]
+ for path in to_try:
+ # checkout_dir can be None, skip it
+ if path:
+ sPath = str(path)
+ if abs_string_path.startswith(sPath):
+ break
else:
path = None
- if path is None:
+ if path is None or sPath is None:
raise ValueError("could not figure out a base path for %r" %
abs_string_path)
sub_path = abs_string_path[len(sPath):].strip(self.sep)
return path.join(*sub_path.split(self.sep))
- def __contains__(self, pathname: str) -> bool:
- if pathname == CheckoutPathName:
- return bool(self.checkout_dir)
- return pathname in self.c.base_paths
+ def __contains__(self, pathname: NamedBasePathsType) -> bool:
+ """This method is DEPRECATED.
- def __setitem__(self, pathname: Literal[CheckoutPathName], path: config_types.Path) -> None:
+ If `pathname` is "checkout", returns True iff checkout_dir is set.
+ If you want to check if checkout_dir is set, use
+ `api.path.checkout_dir is not None` or similar, instead.
+
+ Returns True for all other `pathname` values in NamedBasePaths.
+ Returns False for all other values.
+
+ In the past, the base paths that this module knew about were extensible via
+ a very complicated 'config' system. All of that has been removed, but this
+ method remains for now.
+ """
+ if pathname == self.CheckoutPathName:
+ return bool(self.checkout_dir)
+ return pathname in self.NamedBasePaths
+
+ def __setitem__(self, pathname: CheckoutPathNameType,
+ path: config_types.Path) -> None:
"""Sets the checkout path.
- DEPRECATED - Use `api.path.set_checkout_dir` instead.
+ DEPRECATED - Assign directly to `api.path.checkout_dir` instead.
The only valid value of `pathname` is the literal string CheckoutPathName.
"""
- if pathname != CheckoutPathName:
+ if pathname != self.CheckoutPathName:
raise ValueError(
- f'The only valid dynamic path value is `checkout`. Got {pathname!r}.'
- ' Use `api.path.checkout_dir = <path>` instead.')
+ f'The only valid dynamic path value is `{self.CheckoutPathName}`. '
+ f'Got {pathname!r}. Use `api.path.checkout_dir = <path>` instead.'
+ )
self.checkout_dir = path
@property
@@ -507,6 +632,10 @@
raise ValueError(
f'api.path.checkout_dir called with bad type: {path!r} ({type(path)})')
+ if isinstance(path.base, config_types.CheckoutBasePath):
+ raise ValueError(
+ f'api.path.checkout_dir cannot be rooted in checkout_dir: {path!r}')
+
if (current := self._checkout_dir) is not None:
if current == path:
return
@@ -515,25 +644,139 @@
f'api.path.checkout_dir can only be set once. old:{current!r} new:{path!r}')
self._checkout_dir = path
+ # HACK: config_types.CheckoutBasePath._resolved is a global variable.
+ # This gets reset by config_types.ResetGlobalVariableAssignments().
+ config_types.CheckoutBasePath._resolved = path
+ self.mock_add_directory(path)
+ if self._test_data.enabled:
+ assert isinstance(self._path_mod, fake_path)
+ self._path_mod._mock_path_exists.mark_checkout_dir_set()
- def get(self,
- name: str,
- default: config_types.Path|None = None) -> config_types.Path:
- """Gets the base path named `name`. See module docstring for more info."""
- if name == CheckoutPathName:
- return config_types.Path(config_types.NamedBasePath(CheckoutPathName))
+ def get(self, name: NamedBasePathsType) -> config_types.Path:
+ """Gets the base path named `name`. See module docstring for more info.
- if name in self.c.base_paths:
- return config_types.Path(config_types.NamedBasePath(name))
+ DEPRECATED: Use the following @properties on this module instead:
+ * start_dir
+ * tmp_base_dir
+ * cache_dir
+ * cleanup_dir
+ * home_dir
+ * checkout_dir (but use of checkout_dir is generally discouraged - just
+ pass the Paths around instead of using this global variable).
+ """
+ match name:
+ case 'cache':
+ return self.cache_dir
+ case 'checkout':
+ if cdir := self.checkout_dir:
+ # If the checkout_dir is already set, just return it directly.
+ return cdir
+ # In this case, the checkout_dir is not yet set, but it could be later.
+ return config_types.Path(config_types.CheckoutBasePath())
+ case 'cleanup':
+ return self.cleanup_dir
+ case 'home':
+ return self.home_dir
+ case 'start_dir':
+ return self.start_dir
+ case 'tmp_base':
+ return self.tmp_base_dir
- return default
+ raise ValueError(f'Unable to api.path.get({name!r}) - unknown base path.')
- def __getitem__(self, name: str) -> config_types.Path:
- """Gets the base path named `name`. See module docstring for more info."""
- result = self.get(name)
- if not result:
- raise KeyError('Unknown path: %s' % name)
- return result
+ def __getitem__(self, name: NamedBasePathsType) -> config_types.Path:
+ """Gets the base path named `name`. See module docstring for more info.
+
+ DEPRECATED: Use the following @properties on this module instead:
+ * start_dir
+ * tmp_base_dir
+ * cache_dir
+ * cleanup_dir
+ * home_dir
+ * checkout_dir (but use of checkout_dir is generally discouraged - just
+ pass the Paths around instead of using this global variable).
+ """
+ return self.get(name)
+
+ @property
+ def start_dir(self) -> config_types.Path:
+ """This is the directory that the recipe started in. it's similar to `cwd`,
+ except that it's constant for the duration of the entire program.
+
+ If you want to modify the current working directory for a set of steps,
+ See the 'recipe_engine/context' module which allows modifying the cwd safely
+ via a context manager.
+ """
+ return config_types.Path(config_types.ResolvedBasePath(self._start_dir))
+
+ @property
+ def home_dir(self) -> config_types.Path:
+ """This is the path to the current $HOME directory.
+
+ It is generally recommended to avoid using this, because it is an indicator
+ that the recipe is non-hermetic.
+ """
+ return config_types.Path(config_types.ResolvedBasePath(self._home_dir))
+
+ @property
+ def tmp_base_dir(self) -> config_types.Path:
+ """This directory is the system-configured temp dir.
+
+ This is a weaker form of 'cleanup', and its use should be avoided. This may
+ be removed in the future (or converted to an alias of 'cleanup').
+ """
+ return config_types.Path(config_types.ResolvedBasePath(self._temp_dir))
+
+ @property
+ def cache_dir(self) -> config_types.Path:
+ """This directory is provided by whatever's running the recipe.
+
+ When the recipe executes via Buildbucket, directories under here map to
+ 'named caches' which the Build has set. These caches would be preserved
+ locally on the machine executing this recipe, and are restored for
+ subsequent recipe exections on the same machine which request the same named
+ cache.
+
+ By default, Buildbucket installs a cache named 'builder' which is an
+ immediate subdirectory of cache_dir, and will attempt to be persisted
+ between executions of recipes on the same Buildbucket builder which use the
+ same machine. So, if you are just looking for a place to put files which may
+ be persisted between builds, use:
+
+ api.path.cache_dir/'builder'
+
+ As the base Path.
+
+ Note that directories created under here /may/ be evicted in between runs of
+ the recipe (i.e. to relieve disk pressure).
+ """
+ return config_types.Path(config_types.ResolvedBasePath(self._cache_dir))
+
+ @property
+ def cleanup_dir(self) -> config_types.Path:
+ """This directory is guaranteed to be cleaned up (eventually) after the
+ execution of this recipe.
+
+ This directory is guaranteed to be empty when the recipe starts.
+ """
+ return config_types.Path(config_types.ResolvedBasePath(self._cleanup_dir))
+
+ def cast_to_path(self, strpath: str) -> config_types.Path:
+ """This returns a Path for strpath which can be used anywhere a Path is
+ required.
+
+ If `strpath` is not an absolute path (e.g. rooted with a valid Windows drive
+ or a '/' for non-Windows paths), this will raise ValueError.
+
+ This implicitly tries abs_to_path prior to returning a drive-rooted Path.
+ This means that if strpath is a subdirectory of a known path (say,
+ cache_dir), the returned Path will be based on that known path. This is
+ important for test compatibility.
+ """
+ try:
+ return self.abs_to_path(strpath)
+ except ValueError:
+ return _cast_to_path_impl(self._path_mod, strpath)
@property
def pardir(self) -> str:
@@ -590,7 +833,7 @@
retrieved with api.path[something]), then you can convert from a string path
back to a Path with the `abs_to_path` method.
"""
- return self._path_mod.join(str(path), *map(str, paths))
+ return self._path_mod.join(str(path), *[str(p) for p in paths])
def split(self, path):
"""For "foo/bar/baz", return ("foo/bar", "baz").
@@ -690,6 +933,7 @@
kind: FileType = FileType.FILE) -> None:
"""For testing purposes, mark that |path| exists."""
if self._test_data.enabled:
+ assert isinstance(self._path_mod, fake_path)
self._path_mod.mock_add_paths(path, kind)
def mock_add_file(self, path: config_types.Path) -> None:
@@ -704,6 +948,7 @@
dest: config_types.Path) -> None:
"""For testing purposes, copy |source| to |dest|."""
if self._test_data.enabled:
+ assert isinstance(self._path_mod, fake_path)
self._path_mod.mock_copy_paths(source, dest)
def mock_remove_paths(
@@ -720,38 +965,17 @@
if self._test_data.enabled:
self._path_mod.mock_remove_paths(path, should_remove)
- def separate(self, path: config_types.Path) -> None:
- """Separate a path's pieces in-place with this platform's separator char."""
- path.separate(self.sep)
-
def eq(self, path1: config_types.Path, path2: config_types.Path) -> bool:
"""Check whether path1 points to the same path as path2.
- Under most circumstances, path equality is checked via `path1 == path2`.
- However, if the paths are constructed via differently joined dirs, such as
- ('foo' / 'bar') vs. ('foo/bar'), that doesn't work. This method addresses
- that problem by creating copies of the paths, and then separating them
- according to self.sep. The original paths are not modified.
+ DEPRECATED: Just directly compare path1 and path2 with `==`.
"""
- path1_copy = copy.deepcopy(path1)
- path2_copy = copy.deepcopy(path2)
- self.separate(path1_copy)
- self.separate(path2_copy)
- return path1_copy == path2_copy
+ return path1 == path2
def is_parent_of(self, parent: config_types.Path,
child: config_types.Path) -> bool:
"""Check whether child is contained within parent.
- Under most circumstances, this would be checked via
- `parent.is_parent_of(child)`. However, if the paths are constructed via
- differently joined dirs, such as ('foo', 'bar') vs. ('foo/bar', 'baz.txt'),
- that doesn't work. This method addresses that problem by creating copies of
- the paths, and then separating them according to self.sep. The original
- paths are not modified.
+ DEPRECATED: Just use `parent.is_parent_of(child)`.
"""
- parent_copy = copy.deepcopy(parent)
- child_copy = copy.deepcopy(child)
- self.separate(parent_copy)
- self.separate(child_copy)
- return parent_copy.is_parent_of(child_copy)
+ return parent.is_parent_of(child)
diff --git a/recipe_modules/path/config.py b/recipe_modules/path/config.py
deleted file mode 100644
index 45c67d3..0000000
--- a/recipe_modules/path/config.py
+++ /dev/null
@@ -1,37 +0,0 @@
-# Copyright 2013 The LUCI Authors. All rights reserved.
-# Use of this source code is governed under the Apache License, Version 2.0
-# that can be found in the LICENSE file.
-
-from recipe_engine.config import config_item_context, ConfigGroup, Dict, Static
-
-
-def BaseConfig(START_DIR, TEMP_DIR, CACHE_DIR, CLEANUP_DIR, HOME_DIR,
- **_kwargs):
- assert START_DIR[0].endswith(('\\', '/')), START_DIR
- assert TEMP_DIR[0].endswith(('\\', '/')), TEMP_DIR
- assert CACHE_DIR[0].endswith(('\\', '/')), CACHE_DIR
- assert CLEANUP_DIR[0].endswith(('\\', '/')), CLEANUP_DIR
- assert HOME_DIR[0].endswith(('\\', '/')), HOME_DIR
- return ConfigGroup(
- # base path name -> [tokenized absolute path]
- base_paths=Dict(value_type=tuple),
-
- # dynamic path name -> Path object (referencing one of the base_paths)
- START_DIR=Static(tuple(START_DIR)),
- TEMP_DIR=Static(tuple(TEMP_DIR)),
- CACHE_DIR=Static(tuple(CACHE_DIR)),
- CLEANUP_DIR=Static(tuple(CLEANUP_DIR)),
- HOME_DIR=Static(tuple(HOME_DIR)),
- )
-
-
-config_ctx = config_item_context(BaseConfig)
-
-
-@config_ctx(is_root=True)
-def BASE(c):
- c.base_paths['start_dir'] = c.START_DIR
- c.base_paths['tmp_base'] = c.TEMP_DIR
- c.base_paths['cache'] = c.CACHE_DIR
- c.base_paths['cleanup'] = c.CLEANUP_DIR
- c.base_paths['home'] = c.HOME_DIR
diff --git a/recipe_modules/path/examples/full.expected/linux.json b/recipe_modules/path/examples/full.expected/linux.json
index 431bfe6..3d6e9b9 100644
--- a/recipe_modules/path/examples/full.expected/linux.json
+++ b/recipe_modules/path/examples/full.expected/linux.json
@@ -139,21 +139,6 @@
{
"cmd": [
"echo",
- "[CACHE]",
- "[CLEANUP]",
- "[HOME]",
- "[START_DIR]",
- "[TMP_BASE]"
- ],
- "name": "base paths",
- "~followup_annotations": [
- "@@@STEP_LOG_LINE@result@base_paths: {\"cache\": [\"/\", \"b\", \"c\"], \"cleanup\": [\"/\", \"b\", \"cleanup\"], \"home\": [\"/\", \"home\", \"fake_user\"], \"start_dir\": [\"/\", \"b\", \"FakeTestingCWD\"], \"tmp_base\": [\"/\"]}@@@",
- "@@@STEP_LOG_END@result@@@"
- ]
- },
- {
- "cmd": [
- "echo",
"[START_DIR]/some/thing"
],
"name": "converted path [START_DIR]/some/thing"
@@ -168,6 +153,20 @@
{
"cmd": [
"echo",
+ "[CACHE]/a file"
+ ],
+ "name": "converted path [CACHE]/a file"
+ },
+ {
+ "cmd": [
+ "echo",
+ "[HOME]/another file"
+ ],
+ "name": "converted path [HOME]/another file"
+ },
+ {
+ "cmd": [
+ "echo",
"RECIPE_MODULE[recipe_engine::path]/resources/module_resource.py"
],
"name": "converted path RECIPE_MODULE[recipe_engine::path]/resources/module_resource.py"
diff --git a/recipe_modules/path/examples/full.expected/linux_kitchen.json b/recipe_modules/path/examples/full.expected/linux_kitchen.json
deleted file mode 100644
index 431bfe6..0000000
--- a/recipe_modules/path/examples/full.expected/linux_kitchen.json
+++ /dev/null
@@ -1,199 +0,0 @@
-[
- {
- "cmd": [
- "/bin/echo",
- "[TMP_BASE]/foo"
- ],
- "name": "step1"
- },
- {
- "cmd": [
- "echo",
- "RECIPE_MODULE[recipe_engine::path]/resources/dir/file.py"
- ],
- "name": "print resource"
- },
- {
- "cmd": [
- "echo",
- "RECIPE_REPO[recipe_engine]/dir/file.py"
- ],
- "name": "print package dir"
- },
- {
- "cmd": [
- "/bin/echo",
- "[TMP_BASE]/checkout/jerky"
- ],
- "name": "checkout path"
- },
- {
- "cmd": [
- "touch",
- "[TMP_BASE]/new_file"
- ],
- "name": "touch me"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]/directory"
- ],
- "name": "rm directory (initial)"
- },
- {
- "cmd": [
- "mkdir",
- "-p",
- "[START_DIR]/directory"
- ],
- "name": "mkdir directory"
- },
- {
- "cmd": [
- "touch",
- "[START_DIR]/directory/filepath"
- ],
- "name": "touch filepath"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]/directory"
- ],
- "name": "rm directory"
- },
- {
- "cmd": [
- "mkdir",
- "-p",
- "[START_DIR]/directory"
- ],
- "name": "mkdir directory (2)"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]/copy2"
- ],
- "name": "rm copy2 (initial)"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]/copy20"
- ],
- "name": "rm copy20 (initial)"
- },
- {
- "cmd": [
- "mkdir",
- "-p",
- "[START_DIR]/copy1/foo/bar"
- ],
- "name": "mkdirs"
- },
- {
- "cmd": [
- "touch",
- "[START_DIR]/copy10"
- ],
- "name": "touch copy10"
- },
- {
- "cmd": [
- "cp",
- "-a",
- "[START_DIR]/copy1",
- "[START_DIR]/copy2"
- ],
- "name": "cp copy1 copy2"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]/copy2/foo"
- ],
- "name": "rm copy2/foo"
- },
- {
- "cmd": [
- "touch",
- "[START_DIR]/copy20"
- ],
- "name": "touch copy20"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]/copy2"
- ],
- "name": "rm copy2"
- },
- {
- "cmd": [
- "echo",
- "[CACHE]",
- "[CLEANUP]",
- "[HOME]",
- "[START_DIR]",
- "[TMP_BASE]"
- ],
- "name": "base paths",
- "~followup_annotations": [
- "@@@STEP_LOG_LINE@result@base_paths: {\"cache\": [\"/\", \"b\", \"c\"], \"cleanup\": [\"/\", \"b\", \"cleanup\"], \"home\": [\"/\", \"home\", \"fake_user\"], \"start_dir\": [\"/\", \"b\", \"FakeTestingCWD\"], \"tmp_base\": [\"/\"]}@@@",
- "@@@STEP_LOG_END@result@@@"
- ]
- },
- {
- "cmd": [
- "echo",
- "[START_DIR]/some/thing"
- ],
- "name": "converted path [START_DIR]/some/thing"
- },
- {
- "cmd": [
- "echo",
- "[START_DIR]"
- ],
- "name": "converted path [START_DIR]"
- },
- {
- "cmd": [
- "echo",
- "RECIPE_MODULE[recipe_engine::path]/resources/module_resource.py"
- ],
- "name": "converted path RECIPE_MODULE[recipe_engine::path]/resources/module_resource.py"
- },
- {
- "cmd": [
- "echo",
- "RECIPE_MODULE[recipe_engine::path]/resources"
- ],
- "name": "converted path RECIPE_MODULE[recipe_engine::path]/resources"
- },
- {
- "cmd": [
- "echo",
- "RECIPE[recipe_engine::path:examples/full].resources/recipe_resource.py"
- ],
- "name": "converted path RECIPE[recipe_engine::path:examples/full].resources/recipe_resource.py"
- },
- {
- "cmd": [
- "echo",
- "RECIPE[recipe_engine::path:examples/full].resources"
- ],
- "name": "converted path RECIPE[recipe_engine::path:examples/full].resources"
- },
- {
- "name": "$result"
- }
-]
\ No newline at end of file
diff --git a/recipe_modules/path/examples/full.expected/linux_luci.json b/recipe_modules/path/examples/full.expected/linux_luci.json
deleted file mode 100644
index 431bfe6..0000000
--- a/recipe_modules/path/examples/full.expected/linux_luci.json
+++ /dev/null
@@ -1,199 +0,0 @@
-[
- {
- "cmd": [
- "/bin/echo",
- "[TMP_BASE]/foo"
- ],
- "name": "step1"
- },
- {
- "cmd": [
- "echo",
- "RECIPE_MODULE[recipe_engine::path]/resources/dir/file.py"
- ],
- "name": "print resource"
- },
- {
- "cmd": [
- "echo",
- "RECIPE_REPO[recipe_engine]/dir/file.py"
- ],
- "name": "print package dir"
- },
- {
- "cmd": [
- "/bin/echo",
- "[TMP_BASE]/checkout/jerky"
- ],
- "name": "checkout path"
- },
- {
- "cmd": [
- "touch",
- "[TMP_BASE]/new_file"
- ],
- "name": "touch me"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]/directory"
- ],
- "name": "rm directory (initial)"
- },
- {
- "cmd": [
- "mkdir",
- "-p",
- "[START_DIR]/directory"
- ],
- "name": "mkdir directory"
- },
- {
- "cmd": [
- "touch",
- "[START_DIR]/directory/filepath"
- ],
- "name": "touch filepath"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]/directory"
- ],
- "name": "rm directory"
- },
- {
- "cmd": [
- "mkdir",
- "-p",
- "[START_DIR]/directory"
- ],
- "name": "mkdir directory (2)"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]/copy2"
- ],
- "name": "rm copy2 (initial)"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]/copy20"
- ],
- "name": "rm copy20 (initial)"
- },
- {
- "cmd": [
- "mkdir",
- "-p",
- "[START_DIR]/copy1/foo/bar"
- ],
- "name": "mkdirs"
- },
- {
- "cmd": [
- "touch",
- "[START_DIR]/copy10"
- ],
- "name": "touch copy10"
- },
- {
- "cmd": [
- "cp",
- "-a",
- "[START_DIR]/copy1",
- "[START_DIR]/copy2"
- ],
- "name": "cp copy1 copy2"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]/copy2/foo"
- ],
- "name": "rm copy2/foo"
- },
- {
- "cmd": [
- "touch",
- "[START_DIR]/copy20"
- ],
- "name": "touch copy20"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]/copy2"
- ],
- "name": "rm copy2"
- },
- {
- "cmd": [
- "echo",
- "[CACHE]",
- "[CLEANUP]",
- "[HOME]",
- "[START_DIR]",
- "[TMP_BASE]"
- ],
- "name": "base paths",
- "~followup_annotations": [
- "@@@STEP_LOG_LINE@result@base_paths: {\"cache\": [\"/\", \"b\", \"c\"], \"cleanup\": [\"/\", \"b\", \"cleanup\"], \"home\": [\"/\", \"home\", \"fake_user\"], \"start_dir\": [\"/\", \"b\", \"FakeTestingCWD\"], \"tmp_base\": [\"/\"]}@@@",
- "@@@STEP_LOG_END@result@@@"
- ]
- },
- {
- "cmd": [
- "echo",
- "[START_DIR]/some/thing"
- ],
- "name": "converted path [START_DIR]/some/thing"
- },
- {
- "cmd": [
- "echo",
- "[START_DIR]"
- ],
- "name": "converted path [START_DIR]"
- },
- {
- "cmd": [
- "echo",
- "RECIPE_MODULE[recipe_engine::path]/resources/module_resource.py"
- ],
- "name": "converted path RECIPE_MODULE[recipe_engine::path]/resources/module_resource.py"
- },
- {
- "cmd": [
- "echo",
- "RECIPE_MODULE[recipe_engine::path]/resources"
- ],
- "name": "converted path RECIPE_MODULE[recipe_engine::path]/resources"
- },
- {
- "cmd": [
- "echo",
- "RECIPE[recipe_engine::path:examples/full].resources/recipe_resource.py"
- ],
- "name": "converted path RECIPE[recipe_engine::path:examples/full].resources/recipe_resource.py"
- },
- {
- "cmd": [
- "echo",
- "RECIPE[recipe_engine::path:examples/full].resources"
- ],
- "name": "converted path RECIPE[recipe_engine::path:examples/full].resources"
- },
- {
- "name": "$result"
- }
-]
\ No newline at end of file
diff --git a/recipe_modules/path/examples/full.expected/linux_swarming.json b/recipe_modules/path/examples/full.expected/linux_swarming.json
deleted file mode 100644
index 431bfe6..0000000
--- a/recipe_modules/path/examples/full.expected/linux_swarming.json
+++ /dev/null
@@ -1,199 +0,0 @@
-[
- {
- "cmd": [
- "/bin/echo",
- "[TMP_BASE]/foo"
- ],
- "name": "step1"
- },
- {
- "cmd": [
- "echo",
- "RECIPE_MODULE[recipe_engine::path]/resources/dir/file.py"
- ],
- "name": "print resource"
- },
- {
- "cmd": [
- "echo",
- "RECIPE_REPO[recipe_engine]/dir/file.py"
- ],
- "name": "print package dir"
- },
- {
- "cmd": [
- "/bin/echo",
- "[TMP_BASE]/checkout/jerky"
- ],
- "name": "checkout path"
- },
- {
- "cmd": [
- "touch",
- "[TMP_BASE]/new_file"
- ],
- "name": "touch me"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]/directory"
- ],
- "name": "rm directory (initial)"
- },
- {
- "cmd": [
- "mkdir",
- "-p",
- "[START_DIR]/directory"
- ],
- "name": "mkdir directory"
- },
- {
- "cmd": [
- "touch",
- "[START_DIR]/directory/filepath"
- ],
- "name": "touch filepath"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]/directory"
- ],
- "name": "rm directory"
- },
- {
- "cmd": [
- "mkdir",
- "-p",
- "[START_DIR]/directory"
- ],
- "name": "mkdir directory (2)"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]/copy2"
- ],
- "name": "rm copy2 (initial)"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]/copy20"
- ],
- "name": "rm copy20 (initial)"
- },
- {
- "cmd": [
- "mkdir",
- "-p",
- "[START_DIR]/copy1/foo/bar"
- ],
- "name": "mkdirs"
- },
- {
- "cmd": [
- "touch",
- "[START_DIR]/copy10"
- ],
- "name": "touch copy10"
- },
- {
- "cmd": [
- "cp",
- "-a",
- "[START_DIR]/copy1",
- "[START_DIR]/copy2"
- ],
- "name": "cp copy1 copy2"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]/copy2/foo"
- ],
- "name": "rm copy2/foo"
- },
- {
- "cmd": [
- "touch",
- "[START_DIR]/copy20"
- ],
- "name": "touch copy20"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]/copy2"
- ],
- "name": "rm copy2"
- },
- {
- "cmd": [
- "echo",
- "[CACHE]",
- "[CLEANUP]",
- "[HOME]",
- "[START_DIR]",
- "[TMP_BASE]"
- ],
- "name": "base paths",
- "~followup_annotations": [
- "@@@STEP_LOG_LINE@result@base_paths: {\"cache\": [\"/\", \"b\", \"c\"], \"cleanup\": [\"/\", \"b\", \"cleanup\"], \"home\": [\"/\", \"home\", \"fake_user\"], \"start_dir\": [\"/\", \"b\", \"FakeTestingCWD\"], \"tmp_base\": [\"/\"]}@@@",
- "@@@STEP_LOG_END@result@@@"
- ]
- },
- {
- "cmd": [
- "echo",
- "[START_DIR]/some/thing"
- ],
- "name": "converted path [START_DIR]/some/thing"
- },
- {
- "cmd": [
- "echo",
- "[START_DIR]"
- ],
- "name": "converted path [START_DIR]"
- },
- {
- "cmd": [
- "echo",
- "RECIPE_MODULE[recipe_engine::path]/resources/module_resource.py"
- ],
- "name": "converted path RECIPE_MODULE[recipe_engine::path]/resources/module_resource.py"
- },
- {
- "cmd": [
- "echo",
- "RECIPE_MODULE[recipe_engine::path]/resources"
- ],
- "name": "converted path RECIPE_MODULE[recipe_engine::path]/resources"
- },
- {
- "cmd": [
- "echo",
- "RECIPE[recipe_engine::path:examples/full].resources/recipe_resource.py"
- ],
- "name": "converted path RECIPE[recipe_engine::path:examples/full].resources/recipe_resource.py"
- },
- {
- "cmd": [
- "echo",
- "RECIPE[recipe_engine::path:examples/full].resources"
- ],
- "name": "converted path RECIPE[recipe_engine::path:examples/full].resources"
- },
- {
- "name": "$result"
- }
-]
\ No newline at end of file
diff --git a/recipe_modules/path/examples/full.expected/mac.json b/recipe_modules/path/examples/full.expected/mac.json
deleted file mode 100644
index 431bfe6..0000000
--- a/recipe_modules/path/examples/full.expected/mac.json
+++ /dev/null
@@ -1,199 +0,0 @@
-[
- {
- "cmd": [
- "/bin/echo",
- "[TMP_BASE]/foo"
- ],
- "name": "step1"
- },
- {
- "cmd": [
- "echo",
- "RECIPE_MODULE[recipe_engine::path]/resources/dir/file.py"
- ],
- "name": "print resource"
- },
- {
- "cmd": [
- "echo",
- "RECIPE_REPO[recipe_engine]/dir/file.py"
- ],
- "name": "print package dir"
- },
- {
- "cmd": [
- "/bin/echo",
- "[TMP_BASE]/checkout/jerky"
- ],
- "name": "checkout path"
- },
- {
- "cmd": [
- "touch",
- "[TMP_BASE]/new_file"
- ],
- "name": "touch me"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]/directory"
- ],
- "name": "rm directory (initial)"
- },
- {
- "cmd": [
- "mkdir",
- "-p",
- "[START_DIR]/directory"
- ],
- "name": "mkdir directory"
- },
- {
- "cmd": [
- "touch",
- "[START_DIR]/directory/filepath"
- ],
- "name": "touch filepath"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]/directory"
- ],
- "name": "rm directory"
- },
- {
- "cmd": [
- "mkdir",
- "-p",
- "[START_DIR]/directory"
- ],
- "name": "mkdir directory (2)"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]/copy2"
- ],
- "name": "rm copy2 (initial)"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]/copy20"
- ],
- "name": "rm copy20 (initial)"
- },
- {
- "cmd": [
- "mkdir",
- "-p",
- "[START_DIR]/copy1/foo/bar"
- ],
- "name": "mkdirs"
- },
- {
- "cmd": [
- "touch",
- "[START_DIR]/copy10"
- ],
- "name": "touch copy10"
- },
- {
- "cmd": [
- "cp",
- "-a",
- "[START_DIR]/copy1",
- "[START_DIR]/copy2"
- ],
- "name": "cp copy1 copy2"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]/copy2/foo"
- ],
- "name": "rm copy2/foo"
- },
- {
- "cmd": [
- "touch",
- "[START_DIR]/copy20"
- ],
- "name": "touch copy20"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]/copy2"
- ],
- "name": "rm copy2"
- },
- {
- "cmd": [
- "echo",
- "[CACHE]",
- "[CLEANUP]",
- "[HOME]",
- "[START_DIR]",
- "[TMP_BASE]"
- ],
- "name": "base paths",
- "~followup_annotations": [
- "@@@STEP_LOG_LINE@result@base_paths: {\"cache\": [\"/\", \"b\", \"c\"], \"cleanup\": [\"/\", \"b\", \"cleanup\"], \"home\": [\"/\", \"home\", \"fake_user\"], \"start_dir\": [\"/\", \"b\", \"FakeTestingCWD\"], \"tmp_base\": [\"/\"]}@@@",
- "@@@STEP_LOG_END@result@@@"
- ]
- },
- {
- "cmd": [
- "echo",
- "[START_DIR]/some/thing"
- ],
- "name": "converted path [START_DIR]/some/thing"
- },
- {
- "cmd": [
- "echo",
- "[START_DIR]"
- ],
- "name": "converted path [START_DIR]"
- },
- {
- "cmd": [
- "echo",
- "RECIPE_MODULE[recipe_engine::path]/resources/module_resource.py"
- ],
- "name": "converted path RECIPE_MODULE[recipe_engine::path]/resources/module_resource.py"
- },
- {
- "cmd": [
- "echo",
- "RECIPE_MODULE[recipe_engine::path]/resources"
- ],
- "name": "converted path RECIPE_MODULE[recipe_engine::path]/resources"
- },
- {
- "cmd": [
- "echo",
- "RECIPE[recipe_engine::path:examples/full].resources/recipe_resource.py"
- ],
- "name": "converted path RECIPE[recipe_engine::path:examples/full].resources/recipe_resource.py"
- },
- {
- "cmd": [
- "echo",
- "RECIPE[recipe_engine::path:examples/full].resources"
- ],
- "name": "converted path RECIPE[recipe_engine::path:examples/full].resources"
- },
- {
- "name": "$result"
- }
-]
\ No newline at end of file
diff --git a/recipe_modules/path/examples/full.expected/mac_kitchen.json b/recipe_modules/path/examples/full.expected/mac_kitchen.json
deleted file mode 100644
index 431bfe6..0000000
--- a/recipe_modules/path/examples/full.expected/mac_kitchen.json
+++ /dev/null
@@ -1,199 +0,0 @@
-[
- {
- "cmd": [
- "/bin/echo",
- "[TMP_BASE]/foo"
- ],
- "name": "step1"
- },
- {
- "cmd": [
- "echo",
- "RECIPE_MODULE[recipe_engine::path]/resources/dir/file.py"
- ],
- "name": "print resource"
- },
- {
- "cmd": [
- "echo",
- "RECIPE_REPO[recipe_engine]/dir/file.py"
- ],
- "name": "print package dir"
- },
- {
- "cmd": [
- "/bin/echo",
- "[TMP_BASE]/checkout/jerky"
- ],
- "name": "checkout path"
- },
- {
- "cmd": [
- "touch",
- "[TMP_BASE]/new_file"
- ],
- "name": "touch me"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]/directory"
- ],
- "name": "rm directory (initial)"
- },
- {
- "cmd": [
- "mkdir",
- "-p",
- "[START_DIR]/directory"
- ],
- "name": "mkdir directory"
- },
- {
- "cmd": [
- "touch",
- "[START_DIR]/directory/filepath"
- ],
- "name": "touch filepath"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]/directory"
- ],
- "name": "rm directory"
- },
- {
- "cmd": [
- "mkdir",
- "-p",
- "[START_DIR]/directory"
- ],
- "name": "mkdir directory (2)"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]/copy2"
- ],
- "name": "rm copy2 (initial)"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]/copy20"
- ],
- "name": "rm copy20 (initial)"
- },
- {
- "cmd": [
- "mkdir",
- "-p",
- "[START_DIR]/copy1/foo/bar"
- ],
- "name": "mkdirs"
- },
- {
- "cmd": [
- "touch",
- "[START_DIR]/copy10"
- ],
- "name": "touch copy10"
- },
- {
- "cmd": [
- "cp",
- "-a",
- "[START_DIR]/copy1",
- "[START_DIR]/copy2"
- ],
- "name": "cp copy1 copy2"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]/copy2/foo"
- ],
- "name": "rm copy2/foo"
- },
- {
- "cmd": [
- "touch",
- "[START_DIR]/copy20"
- ],
- "name": "touch copy20"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]/copy2"
- ],
- "name": "rm copy2"
- },
- {
- "cmd": [
- "echo",
- "[CACHE]",
- "[CLEANUP]",
- "[HOME]",
- "[START_DIR]",
- "[TMP_BASE]"
- ],
- "name": "base paths",
- "~followup_annotations": [
- "@@@STEP_LOG_LINE@result@base_paths: {\"cache\": [\"/\", \"b\", \"c\"], \"cleanup\": [\"/\", \"b\", \"cleanup\"], \"home\": [\"/\", \"home\", \"fake_user\"], \"start_dir\": [\"/\", \"b\", \"FakeTestingCWD\"], \"tmp_base\": [\"/\"]}@@@",
- "@@@STEP_LOG_END@result@@@"
- ]
- },
- {
- "cmd": [
- "echo",
- "[START_DIR]/some/thing"
- ],
- "name": "converted path [START_DIR]/some/thing"
- },
- {
- "cmd": [
- "echo",
- "[START_DIR]"
- ],
- "name": "converted path [START_DIR]"
- },
- {
- "cmd": [
- "echo",
- "RECIPE_MODULE[recipe_engine::path]/resources/module_resource.py"
- ],
- "name": "converted path RECIPE_MODULE[recipe_engine::path]/resources/module_resource.py"
- },
- {
- "cmd": [
- "echo",
- "RECIPE_MODULE[recipe_engine::path]/resources"
- ],
- "name": "converted path RECIPE_MODULE[recipe_engine::path]/resources"
- },
- {
- "cmd": [
- "echo",
- "RECIPE[recipe_engine::path:examples/full].resources/recipe_resource.py"
- ],
- "name": "converted path RECIPE[recipe_engine::path:examples/full].resources/recipe_resource.py"
- },
- {
- "cmd": [
- "echo",
- "RECIPE[recipe_engine::path:examples/full].resources"
- ],
- "name": "converted path RECIPE[recipe_engine::path:examples/full].resources"
- },
- {
- "name": "$result"
- }
-]
\ No newline at end of file
diff --git a/recipe_modules/path/examples/full.expected/mac_luci.json b/recipe_modules/path/examples/full.expected/mac_luci.json
deleted file mode 100644
index 431bfe6..0000000
--- a/recipe_modules/path/examples/full.expected/mac_luci.json
+++ /dev/null
@@ -1,199 +0,0 @@
-[
- {
- "cmd": [
- "/bin/echo",
- "[TMP_BASE]/foo"
- ],
- "name": "step1"
- },
- {
- "cmd": [
- "echo",
- "RECIPE_MODULE[recipe_engine::path]/resources/dir/file.py"
- ],
- "name": "print resource"
- },
- {
- "cmd": [
- "echo",
- "RECIPE_REPO[recipe_engine]/dir/file.py"
- ],
- "name": "print package dir"
- },
- {
- "cmd": [
- "/bin/echo",
- "[TMP_BASE]/checkout/jerky"
- ],
- "name": "checkout path"
- },
- {
- "cmd": [
- "touch",
- "[TMP_BASE]/new_file"
- ],
- "name": "touch me"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]/directory"
- ],
- "name": "rm directory (initial)"
- },
- {
- "cmd": [
- "mkdir",
- "-p",
- "[START_DIR]/directory"
- ],
- "name": "mkdir directory"
- },
- {
- "cmd": [
- "touch",
- "[START_DIR]/directory/filepath"
- ],
- "name": "touch filepath"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]/directory"
- ],
- "name": "rm directory"
- },
- {
- "cmd": [
- "mkdir",
- "-p",
- "[START_DIR]/directory"
- ],
- "name": "mkdir directory (2)"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]/copy2"
- ],
- "name": "rm copy2 (initial)"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]/copy20"
- ],
- "name": "rm copy20 (initial)"
- },
- {
- "cmd": [
- "mkdir",
- "-p",
- "[START_DIR]/copy1/foo/bar"
- ],
- "name": "mkdirs"
- },
- {
- "cmd": [
- "touch",
- "[START_DIR]/copy10"
- ],
- "name": "touch copy10"
- },
- {
- "cmd": [
- "cp",
- "-a",
- "[START_DIR]/copy1",
- "[START_DIR]/copy2"
- ],
- "name": "cp copy1 copy2"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]/copy2/foo"
- ],
- "name": "rm copy2/foo"
- },
- {
- "cmd": [
- "touch",
- "[START_DIR]/copy20"
- ],
- "name": "touch copy20"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]/copy2"
- ],
- "name": "rm copy2"
- },
- {
- "cmd": [
- "echo",
- "[CACHE]",
- "[CLEANUP]",
- "[HOME]",
- "[START_DIR]",
- "[TMP_BASE]"
- ],
- "name": "base paths",
- "~followup_annotations": [
- "@@@STEP_LOG_LINE@result@base_paths: {\"cache\": [\"/\", \"b\", \"c\"], \"cleanup\": [\"/\", \"b\", \"cleanup\"], \"home\": [\"/\", \"home\", \"fake_user\"], \"start_dir\": [\"/\", \"b\", \"FakeTestingCWD\"], \"tmp_base\": [\"/\"]}@@@",
- "@@@STEP_LOG_END@result@@@"
- ]
- },
- {
- "cmd": [
- "echo",
- "[START_DIR]/some/thing"
- ],
- "name": "converted path [START_DIR]/some/thing"
- },
- {
- "cmd": [
- "echo",
- "[START_DIR]"
- ],
- "name": "converted path [START_DIR]"
- },
- {
- "cmd": [
- "echo",
- "RECIPE_MODULE[recipe_engine::path]/resources/module_resource.py"
- ],
- "name": "converted path RECIPE_MODULE[recipe_engine::path]/resources/module_resource.py"
- },
- {
- "cmd": [
- "echo",
- "RECIPE_MODULE[recipe_engine::path]/resources"
- ],
- "name": "converted path RECIPE_MODULE[recipe_engine::path]/resources"
- },
- {
- "cmd": [
- "echo",
- "RECIPE[recipe_engine::path:examples/full].resources/recipe_resource.py"
- ],
- "name": "converted path RECIPE[recipe_engine::path:examples/full].resources/recipe_resource.py"
- },
- {
- "cmd": [
- "echo",
- "RECIPE[recipe_engine::path:examples/full].resources"
- ],
- "name": "converted path RECIPE[recipe_engine::path:examples/full].resources"
- },
- {
- "name": "$result"
- }
-]
\ No newline at end of file
diff --git a/recipe_modules/path/examples/full.expected/mac_swarming.json b/recipe_modules/path/examples/full.expected/mac_swarming.json
deleted file mode 100644
index 431bfe6..0000000
--- a/recipe_modules/path/examples/full.expected/mac_swarming.json
+++ /dev/null
@@ -1,199 +0,0 @@
-[
- {
- "cmd": [
- "/bin/echo",
- "[TMP_BASE]/foo"
- ],
- "name": "step1"
- },
- {
- "cmd": [
- "echo",
- "RECIPE_MODULE[recipe_engine::path]/resources/dir/file.py"
- ],
- "name": "print resource"
- },
- {
- "cmd": [
- "echo",
- "RECIPE_REPO[recipe_engine]/dir/file.py"
- ],
- "name": "print package dir"
- },
- {
- "cmd": [
- "/bin/echo",
- "[TMP_BASE]/checkout/jerky"
- ],
- "name": "checkout path"
- },
- {
- "cmd": [
- "touch",
- "[TMP_BASE]/new_file"
- ],
- "name": "touch me"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]/directory"
- ],
- "name": "rm directory (initial)"
- },
- {
- "cmd": [
- "mkdir",
- "-p",
- "[START_DIR]/directory"
- ],
- "name": "mkdir directory"
- },
- {
- "cmd": [
- "touch",
- "[START_DIR]/directory/filepath"
- ],
- "name": "touch filepath"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]/directory"
- ],
- "name": "rm directory"
- },
- {
- "cmd": [
- "mkdir",
- "-p",
- "[START_DIR]/directory"
- ],
- "name": "mkdir directory (2)"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]/copy2"
- ],
- "name": "rm copy2 (initial)"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]/copy20"
- ],
- "name": "rm copy20 (initial)"
- },
- {
- "cmd": [
- "mkdir",
- "-p",
- "[START_DIR]/copy1/foo/bar"
- ],
- "name": "mkdirs"
- },
- {
- "cmd": [
- "touch",
- "[START_DIR]/copy10"
- ],
- "name": "touch copy10"
- },
- {
- "cmd": [
- "cp",
- "-a",
- "[START_DIR]/copy1",
- "[START_DIR]/copy2"
- ],
- "name": "cp copy1 copy2"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]/copy2/foo"
- ],
- "name": "rm copy2/foo"
- },
- {
- "cmd": [
- "touch",
- "[START_DIR]/copy20"
- ],
- "name": "touch copy20"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]/copy2"
- ],
- "name": "rm copy2"
- },
- {
- "cmd": [
- "echo",
- "[CACHE]",
- "[CLEANUP]",
- "[HOME]",
- "[START_DIR]",
- "[TMP_BASE]"
- ],
- "name": "base paths",
- "~followup_annotations": [
- "@@@STEP_LOG_LINE@result@base_paths: {\"cache\": [\"/\", \"b\", \"c\"], \"cleanup\": [\"/\", \"b\", \"cleanup\"], \"home\": [\"/\", \"home\", \"fake_user\"], \"start_dir\": [\"/\", \"b\", \"FakeTestingCWD\"], \"tmp_base\": [\"/\"]}@@@",
- "@@@STEP_LOG_END@result@@@"
- ]
- },
- {
- "cmd": [
- "echo",
- "[START_DIR]/some/thing"
- ],
- "name": "converted path [START_DIR]/some/thing"
- },
- {
- "cmd": [
- "echo",
- "[START_DIR]"
- ],
- "name": "converted path [START_DIR]"
- },
- {
- "cmd": [
- "echo",
- "RECIPE_MODULE[recipe_engine::path]/resources/module_resource.py"
- ],
- "name": "converted path RECIPE_MODULE[recipe_engine::path]/resources/module_resource.py"
- },
- {
- "cmd": [
- "echo",
- "RECIPE_MODULE[recipe_engine::path]/resources"
- ],
- "name": "converted path RECIPE_MODULE[recipe_engine::path]/resources"
- },
- {
- "cmd": [
- "echo",
- "RECIPE[recipe_engine::path:examples/full].resources/recipe_resource.py"
- ],
- "name": "converted path RECIPE[recipe_engine::path:examples/full].resources/recipe_resource.py"
- },
- {
- "cmd": [
- "echo",
- "RECIPE[recipe_engine::path:examples/full].resources"
- ],
- "name": "converted path RECIPE[recipe_engine::path:examples/full].resources"
- },
- {
- "name": "$result"
- }
-]
\ No newline at end of file
diff --git a/recipe_modules/path/examples/full.expected/win.json b/recipe_modules/path/examples/full.expected/win.json
index b72f6da..cb2fdc1 100644
--- a/recipe_modules/path/examples/full.expected/win.json
+++ b/recipe_modules/path/examples/full.expected/win.json
@@ -139,21 +139,6 @@
{
"cmd": [
"echo",
- "[CACHE]",
- "[CLEANUP]",
- "[HOME]",
- "[START_DIR]",
- "[TMP_BASE]"
- ],
- "name": "base paths",
- "~followup_annotations": [
- "@@@STEP_LOG_LINE@result@base_paths: {\"cache\": [\"C:\\\\\", \"b\", \"c\"], \"cleanup\": [\"C:\\\\\", \"b\", \"cleanup\"], \"home\": [\"C:\\\\\", \"home\", \"fake_user\"], \"start_dir\": [\"C:\\\\\", \"b\", \"FakeTestingCWD\"], \"tmp_base\": [\"C:\\\\\"]}@@@",
- "@@@STEP_LOG_END@result@@@"
- ]
- },
- {
- "cmd": [
- "echo",
"[START_DIR]\\some\\thing"
],
"name": "converted path [START_DIR]\\some\\thing"
@@ -168,6 +153,20 @@
{
"cmd": [
"echo",
+ "[CACHE]\\a file"
+ ],
+ "name": "converted path [CACHE]\\a file"
+ },
+ {
+ "cmd": [
+ "echo",
+ "[HOME]\\another file"
+ ],
+ "name": "converted path [HOME]\\another file"
+ },
+ {
+ "cmd": [
+ "echo",
"RECIPE_MODULE[recipe_engine::path]\\resources\\module_resource.py"
],
"name": "converted path RECIPE_MODULE[recipe_engine::path]\\resources\\module_resource.py"
diff --git a/recipe_modules/path/examples/full.expected/win_kitchen.json b/recipe_modules/path/examples/full.expected/win_kitchen.json
deleted file mode 100644
index b72f6da..0000000
--- a/recipe_modules/path/examples/full.expected/win_kitchen.json
+++ /dev/null
@@ -1,199 +0,0 @@
-[
- {
- "cmd": [
- "/bin/echo",
- "[TMP_BASE]\\foo"
- ],
- "name": "step1"
- },
- {
- "cmd": [
- "echo",
- "RECIPE_MODULE[recipe_engine::path]\\resources\\dir\\file.py"
- ],
- "name": "print resource"
- },
- {
- "cmd": [
- "echo",
- "RECIPE_REPO[recipe_engine]\\dir\\file.py"
- ],
- "name": "print package dir"
- },
- {
- "cmd": [
- "/bin/echo",
- "[TMP_BASE]\\checkout\\jerky"
- ],
- "name": "checkout path"
- },
- {
- "cmd": [
- "touch",
- "[TMP_BASE]\\new_file"
- ],
- "name": "touch me"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]\\directory"
- ],
- "name": "rm directory (initial)"
- },
- {
- "cmd": [
- "mkdir",
- "-p",
- "[START_DIR]\\directory"
- ],
- "name": "mkdir directory"
- },
- {
- "cmd": [
- "touch",
- "[START_DIR]\\directory\\filepath"
- ],
- "name": "touch filepath"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]\\directory"
- ],
- "name": "rm directory"
- },
- {
- "cmd": [
- "mkdir",
- "-p",
- "[START_DIR]\\directory"
- ],
- "name": "mkdir directory (2)"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]\\copy2"
- ],
- "name": "rm copy2 (initial)"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]\\copy20"
- ],
- "name": "rm copy20 (initial)"
- },
- {
- "cmd": [
- "mkdir",
- "-p",
- "[START_DIR]\\copy1\\foo\\bar"
- ],
- "name": "mkdirs"
- },
- {
- "cmd": [
- "touch",
- "[START_DIR]\\copy10"
- ],
- "name": "touch copy10"
- },
- {
- "cmd": [
- "cp",
- "-a",
- "[START_DIR]\\copy1",
- "[START_DIR]\\copy2"
- ],
- "name": "cp copy1 copy2"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]\\copy2\\foo"
- ],
- "name": "rm copy2/foo"
- },
- {
- "cmd": [
- "touch",
- "[START_DIR]\\copy20"
- ],
- "name": "touch copy20"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]\\copy2"
- ],
- "name": "rm copy2"
- },
- {
- "cmd": [
- "echo",
- "[CACHE]",
- "[CLEANUP]",
- "[HOME]",
- "[START_DIR]",
- "[TMP_BASE]"
- ],
- "name": "base paths",
- "~followup_annotations": [
- "@@@STEP_LOG_LINE@result@base_paths: {\"cache\": [\"C:\\\\\", \"b\", \"c\"], \"cleanup\": [\"C:\\\\\", \"b\", \"cleanup\"], \"home\": [\"C:\\\\\", \"home\", \"fake_user\"], \"start_dir\": [\"C:\\\\\", \"b\", \"FakeTestingCWD\"], \"tmp_base\": [\"C:\\\\\"]}@@@",
- "@@@STEP_LOG_END@result@@@"
- ]
- },
- {
- "cmd": [
- "echo",
- "[START_DIR]\\some\\thing"
- ],
- "name": "converted path [START_DIR]\\some\\thing"
- },
- {
- "cmd": [
- "echo",
- "[START_DIR]"
- ],
- "name": "converted path [START_DIR]"
- },
- {
- "cmd": [
- "echo",
- "RECIPE_MODULE[recipe_engine::path]\\resources\\module_resource.py"
- ],
- "name": "converted path RECIPE_MODULE[recipe_engine::path]\\resources\\module_resource.py"
- },
- {
- "cmd": [
- "echo",
- "RECIPE_MODULE[recipe_engine::path]\\resources"
- ],
- "name": "converted path RECIPE_MODULE[recipe_engine::path]\\resources"
- },
- {
- "cmd": [
- "echo",
- "RECIPE[recipe_engine::path:examples/full].resources\\recipe_resource.py"
- ],
- "name": "converted path RECIPE[recipe_engine::path:examples/full].resources\\recipe_resource.py"
- },
- {
- "cmd": [
- "echo",
- "RECIPE[recipe_engine::path:examples/full].resources"
- ],
- "name": "converted path RECIPE[recipe_engine::path:examples/full].resources"
- },
- {
- "name": "$result"
- }
-]
\ No newline at end of file
diff --git a/recipe_modules/path/examples/full.expected/win_luci.json b/recipe_modules/path/examples/full.expected/win_luci.json
deleted file mode 100644
index b72f6da..0000000
--- a/recipe_modules/path/examples/full.expected/win_luci.json
+++ /dev/null
@@ -1,199 +0,0 @@
-[
- {
- "cmd": [
- "/bin/echo",
- "[TMP_BASE]\\foo"
- ],
- "name": "step1"
- },
- {
- "cmd": [
- "echo",
- "RECIPE_MODULE[recipe_engine::path]\\resources\\dir\\file.py"
- ],
- "name": "print resource"
- },
- {
- "cmd": [
- "echo",
- "RECIPE_REPO[recipe_engine]\\dir\\file.py"
- ],
- "name": "print package dir"
- },
- {
- "cmd": [
- "/bin/echo",
- "[TMP_BASE]\\checkout\\jerky"
- ],
- "name": "checkout path"
- },
- {
- "cmd": [
- "touch",
- "[TMP_BASE]\\new_file"
- ],
- "name": "touch me"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]\\directory"
- ],
- "name": "rm directory (initial)"
- },
- {
- "cmd": [
- "mkdir",
- "-p",
- "[START_DIR]\\directory"
- ],
- "name": "mkdir directory"
- },
- {
- "cmd": [
- "touch",
- "[START_DIR]\\directory\\filepath"
- ],
- "name": "touch filepath"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]\\directory"
- ],
- "name": "rm directory"
- },
- {
- "cmd": [
- "mkdir",
- "-p",
- "[START_DIR]\\directory"
- ],
- "name": "mkdir directory (2)"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]\\copy2"
- ],
- "name": "rm copy2 (initial)"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]\\copy20"
- ],
- "name": "rm copy20 (initial)"
- },
- {
- "cmd": [
- "mkdir",
- "-p",
- "[START_DIR]\\copy1\\foo\\bar"
- ],
- "name": "mkdirs"
- },
- {
- "cmd": [
- "touch",
- "[START_DIR]\\copy10"
- ],
- "name": "touch copy10"
- },
- {
- "cmd": [
- "cp",
- "-a",
- "[START_DIR]\\copy1",
- "[START_DIR]\\copy2"
- ],
- "name": "cp copy1 copy2"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]\\copy2\\foo"
- ],
- "name": "rm copy2/foo"
- },
- {
- "cmd": [
- "touch",
- "[START_DIR]\\copy20"
- ],
- "name": "touch copy20"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]\\copy2"
- ],
- "name": "rm copy2"
- },
- {
- "cmd": [
- "echo",
- "[CACHE]",
- "[CLEANUP]",
- "[HOME]",
- "[START_DIR]",
- "[TMP_BASE]"
- ],
- "name": "base paths",
- "~followup_annotations": [
- "@@@STEP_LOG_LINE@result@base_paths: {\"cache\": [\"C:\\\\\", \"b\", \"c\"], \"cleanup\": [\"C:\\\\\", \"b\", \"cleanup\"], \"home\": [\"C:\\\\\", \"home\", \"fake_user\"], \"start_dir\": [\"C:\\\\\", \"b\", \"FakeTestingCWD\"], \"tmp_base\": [\"C:\\\\\"]}@@@",
- "@@@STEP_LOG_END@result@@@"
- ]
- },
- {
- "cmd": [
- "echo",
- "[START_DIR]\\some\\thing"
- ],
- "name": "converted path [START_DIR]\\some\\thing"
- },
- {
- "cmd": [
- "echo",
- "[START_DIR]"
- ],
- "name": "converted path [START_DIR]"
- },
- {
- "cmd": [
- "echo",
- "RECIPE_MODULE[recipe_engine::path]\\resources\\module_resource.py"
- ],
- "name": "converted path RECIPE_MODULE[recipe_engine::path]\\resources\\module_resource.py"
- },
- {
- "cmd": [
- "echo",
- "RECIPE_MODULE[recipe_engine::path]\\resources"
- ],
- "name": "converted path RECIPE_MODULE[recipe_engine::path]\\resources"
- },
- {
- "cmd": [
- "echo",
- "RECIPE[recipe_engine::path:examples/full].resources\\recipe_resource.py"
- ],
- "name": "converted path RECIPE[recipe_engine::path:examples/full].resources\\recipe_resource.py"
- },
- {
- "cmd": [
- "echo",
- "RECIPE[recipe_engine::path:examples/full].resources"
- ],
- "name": "converted path RECIPE[recipe_engine::path:examples/full].resources"
- },
- {
- "name": "$result"
- }
-]
\ No newline at end of file
diff --git a/recipe_modules/path/examples/full.expected/win_swarming.json b/recipe_modules/path/examples/full.expected/win_swarming.json
deleted file mode 100644
index b72f6da..0000000
--- a/recipe_modules/path/examples/full.expected/win_swarming.json
+++ /dev/null
@@ -1,199 +0,0 @@
-[
- {
- "cmd": [
- "/bin/echo",
- "[TMP_BASE]\\foo"
- ],
- "name": "step1"
- },
- {
- "cmd": [
- "echo",
- "RECIPE_MODULE[recipe_engine::path]\\resources\\dir\\file.py"
- ],
- "name": "print resource"
- },
- {
- "cmd": [
- "echo",
- "RECIPE_REPO[recipe_engine]\\dir\\file.py"
- ],
- "name": "print package dir"
- },
- {
- "cmd": [
- "/bin/echo",
- "[TMP_BASE]\\checkout\\jerky"
- ],
- "name": "checkout path"
- },
- {
- "cmd": [
- "touch",
- "[TMP_BASE]\\new_file"
- ],
- "name": "touch me"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]\\directory"
- ],
- "name": "rm directory (initial)"
- },
- {
- "cmd": [
- "mkdir",
- "-p",
- "[START_DIR]\\directory"
- ],
- "name": "mkdir directory"
- },
- {
- "cmd": [
- "touch",
- "[START_DIR]\\directory\\filepath"
- ],
- "name": "touch filepath"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]\\directory"
- ],
- "name": "rm directory"
- },
- {
- "cmd": [
- "mkdir",
- "-p",
- "[START_DIR]\\directory"
- ],
- "name": "mkdir directory (2)"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]\\copy2"
- ],
- "name": "rm copy2 (initial)"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]\\copy20"
- ],
- "name": "rm copy20 (initial)"
- },
- {
- "cmd": [
- "mkdir",
- "-p",
- "[START_DIR]\\copy1\\foo\\bar"
- ],
- "name": "mkdirs"
- },
- {
- "cmd": [
- "touch",
- "[START_DIR]\\copy10"
- ],
- "name": "touch copy10"
- },
- {
- "cmd": [
- "cp",
- "-a",
- "[START_DIR]\\copy1",
- "[START_DIR]\\copy2"
- ],
- "name": "cp copy1 copy2"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]\\copy2\\foo"
- ],
- "name": "rm copy2/foo"
- },
- {
- "cmd": [
- "touch",
- "[START_DIR]\\copy20"
- ],
- "name": "touch copy20"
- },
- {
- "cmd": [
- "rm",
- "-rf",
- "[START_DIR]\\copy2"
- ],
- "name": "rm copy2"
- },
- {
- "cmd": [
- "echo",
- "[CACHE]",
- "[CLEANUP]",
- "[HOME]",
- "[START_DIR]",
- "[TMP_BASE]"
- ],
- "name": "base paths",
- "~followup_annotations": [
- "@@@STEP_LOG_LINE@result@base_paths: {\"cache\": [\"C:\\\\\", \"b\", \"c\"], \"cleanup\": [\"C:\\\\\", \"b\", \"cleanup\"], \"home\": [\"C:\\\\\", \"home\", \"fake_user\"], \"start_dir\": [\"C:\\\\\", \"b\", \"FakeTestingCWD\"], \"tmp_base\": [\"C:\\\\\"]}@@@",
- "@@@STEP_LOG_END@result@@@"
- ]
- },
- {
- "cmd": [
- "echo",
- "[START_DIR]\\some\\thing"
- ],
- "name": "converted path [START_DIR]\\some\\thing"
- },
- {
- "cmd": [
- "echo",
- "[START_DIR]"
- ],
- "name": "converted path [START_DIR]"
- },
- {
- "cmd": [
- "echo",
- "RECIPE_MODULE[recipe_engine::path]\\resources\\module_resource.py"
- ],
- "name": "converted path RECIPE_MODULE[recipe_engine::path]\\resources\\module_resource.py"
- },
- {
- "cmd": [
- "echo",
- "RECIPE_MODULE[recipe_engine::path]\\resources"
- ],
- "name": "converted path RECIPE_MODULE[recipe_engine::path]\\resources"
- },
- {
- "cmd": [
- "echo",
- "RECIPE[recipe_engine::path:examples/full].resources\\recipe_resource.py"
- ],
- "name": "converted path RECIPE[recipe_engine::path:examples/full].resources\\recipe_resource.py"
- },
- {
- "cmd": [
- "echo",
- "RECIPE[recipe_engine::path:examples/full].resources"
- ],
- "name": "converted path RECIPE[recipe_engine::path:examples/full].resources"
- },
- {
- "name": "$result"
- }
-]
\ No newline at end of file
diff --git a/recipe_modules/path/examples/full.py b/recipe_modules/path/examples/full.py
index af26df9..9851278 100644
--- a/recipe_modules/path/examples/full.py
+++ b/recipe_modules/path/examples/full.py
@@ -12,7 +12,6 @@
from builtins import range, zip
-from recipe_engine.config_types import Path
def RunSteps(api):
api.step('step1', ['/bin/echo', str(api.path['tmp_base'].join('foo'))])
@@ -27,16 +26,22 @@
assert 'start_dir' in api.path
assert api.path['start_dir'].join('.') == api.path['start_dir']
+
assert 'checkout' not in api.path
api.path['checkout'] = api.path['tmp_base'].join('checkout')
assert 'checkout' in api.path
# Test missing/default value.
assert 'nonexistent' not in api.path
- assert api.path.get('nonexistent') is None
+ try:
+ api.path.get('nonexistent')
+ assert False, "We should never get here" # pragma: no cover
+ except ValueError as ex:
+ assert 'unknown base path' in str(ex), str(ex)
+
try:
raise Exception('Should never raise: %s' % (api.path['nonexistent'],))
- except KeyError:
+ except ValueError:
pass
# Global dynamic paths (see config.py example for declaration):
@@ -164,22 +169,17 @@
assert not api.path.exists(copy2)
assert api.path.exists(copy20)
- result = api.step('base paths', ['echo'] + [
- api.path[name] for name in sorted(api.path.c.base_paths.keys())
- ])
- result.presentation.logs['result'] = [
- 'base_paths: %s' % (api.json.dumps(api.path.c.base_paths.as_jsonish()),),
- ]
-
# Convert strings to Paths.
def _mk_paths():
return [
- api.path['start_dir'].join('some', 'thing'),
- api.path['start_dir'],
- api.path.resource("module_resource.py"),
- api.path.resource(),
- api.resource("recipe_resource.py"),
- api.resource(),
+ api.path['start_dir'].join('some', 'thing'),
+ api.path['start_dir'],
+ api.path['cache'] / 'a file',
+ api.path['home'] / 'another file',
+ api.path.resource("module_resource.py"),
+ api.path.resource(),
+ api.resource("recipe_resource.py"),
+ api.resource(),
]
static_paths = _mk_paths()
for p in static_paths:
@@ -215,54 +215,30 @@
assert "could not figure out" in str(ex), ex
start_dir = api.path['start_dir']
- tmp_base = api.path['tmp_base']
- assert start_dir < tmp_base
- assert not (tmp_base < start_dir)
-
- a = start_dir.join('a')
- b = start_dir.join('b')
+ a = start_dir / 'a'
+ b = start_dir / 'b'
assert a < b
+ assert b > a
assert not (b < a)
+ # there is also a join method on the path module
+ assert start_dir.join('a') == api.path.join(start_dir, 'a')
+
slashy_path = api.path['start_dir'].join(f'foo{api.path.sep}bar')
separated_path = api.path['start_dir'].join('foo', 'bar')
assert str(slashy_path) == str(separated_path)
- assert slashy_path != separated_path # Not desired, but true.
+ assert slashy_path == separated_path
assert api.path.eq(slashy_path, separated_path)
- assert slashy_path != separated_path # eq() didn't modify the paths.
slashy_file = api.path['start_dir'].join(
f'foo{api.path.sep}bar{api.path.sep}baz.txt')
- assert not separated_path.is_parent_of(slashy_file) # Again, not desired.
+ assert separated_path.is_parent_of(slashy_file)
assert api.path.is_parent_of(separated_path, slashy_file)
- assert not separated_path.is_parent_of(slashy_file) # Again, not modified.
def GenTests(api):
- for platform in ('linux', 'win', 'mac'):
- yield (api.test(platform) + api.platform.name(platform) +
- # Test when a file already exists
- api.path.exists(api.path['tmp_base']))
-
- # We have support for chromium swarming built in to the engine for some
- # reason. TODO(phajdan.jr) remove it.
- yield (api.test('%s_swarming' % platform) +
- api.platform.name(platform) +
- api.properties(path_config='swarming') +
- api.path.exists(api.path['tmp_base']))
-
- yield (api.test('%s_kitchen' % platform) +
- api.platform.name(platform) +
- api.properties(path_config='kitchen') +
- api.path.exists(api.path['tmp_base']))
-
- yield (api.test('%s_luci' % platform) +
- api.platform.name(platform) +
- api.properties(**{
- '$recipe_engine/path': {
- 'cache_dir': '/c',
- 'temp_dir': '/t',
- 'cleanup_dir': '/build.dead',
- },
- }) +
- api.path.exists(api.path['tmp_base']))
+ for platform in ('linux', 'win'):
+ yield api.test(
+ platform,
+ api.platform.name(platform),
+ )
diff --git a/recipe_modules/path/test_api.py b/recipe_modules/path/test_api.py
index b9d39e1..877495b 100644
--- a/recipe_modules/path/test_api.py
+++ b/recipe_modules/path/test_api.py
@@ -2,15 +2,131 @@
# Use of this source code is governed under the Apache License, Version 2.0
# that can be found in the LICENSE file.
+from __future__ import annotations
+from dataclasses import dataclass
+
+from typing import TYPE_CHECKING
+
from recipe_engine import recipe_test_api
-from recipe_engine.config_types import Path, NamedBasePath
+from recipe_engine.config_types import CheckoutBasePath, Path, ResolvedBasePath
+
+# Avoid circular import.
+if TYPE_CHECKING: # pragma: no cover
+ from .api import NamedBasePathsType
+
+
+@dataclass(frozen=True)
+class UnvalidatedPath:
+ base: str
+ pieces: tuple[str, ...]
+
+ def join(self, *pieces: str) -> UnvalidatedPath:
+ return UnvalidatedPath(self.base, self.pieces + pieces)
+
class PathTestApi(recipe_test_api.RecipeTestApi):
+
+ def exists(self, *paths: Path):
+ """This is an alias for `files_exist`."""
+ return self.files_exist(*paths)
+
@recipe_test_api.mod_test_data
@staticmethod
- def exists(*paths):
- assert all(isinstance(p, Path) for p in paths)
- return paths
+ def files_exist(*paths: Path | UnvalidatedPath):
+ """This mocks the path module to believe that the given `paths` exist as
+ FILES prior to the start of the recipe.
- def __getitem__(self, name):
- return Path(NamedBasePath(name))
+ To mock the existence of paths which are generated DURING the execution of
+ the recipe, use recipe_engine/path.mock_* functions.
+
+ This sets the type of paths to be 'FILE'. If you want to mock the existence
+ of a directory, use dirs_exist().
+ """
+ assert all(isinstance(p, (Path, UnvalidatedPath)) for p in paths)
+ return list(paths)
+
+ @recipe_test_api.mod_test_data
+ @staticmethod
+ def dirs_exist(*paths: Path | UnvalidatedPath):
+ """This mocks the path module to believe that the given `paths` exist as
+ DIRECTORIES prior to the start of the recipe.
+
+ To mock the existence of paths which are generated DURING the execution of
+ the recipe, use recipe_engine/path.mock_* functions.
+
+ This sets the type of paths to be 'DIRECTORY'. If you want to mock the
+ existence of a file, use exists().
+ """
+ assert all(isinstance(p, (Path, UnvalidatedPath)) for p in paths)
+ return list(paths)
+
+ def __getitem__(self, name: NamedBasePathsType) -> Path:
+ """This gets a Path based on `name` for passing to the `exists`,
+ `files_exist` or `dirs_exist` methods on this test API.
+
+ DEPRECATED: Use the following @properties on this module instead:
+ * start_dir
+ * tmp_base_dir
+ * cache_dir
+ * cleanup_dir
+ * home_dir
+ * checkout_dir (but use of checkout_dir is generally discouraged - just
+ pass the Paths around instead of using this global variable).
+ """
+ match name:
+ case 'cache':
+ return self.cache_dir
+ case 'cleanup':
+ return self.cleanup_dir
+ case 'home':
+ return self.home_dir
+ case 'start_dir':
+ return self.start_dir
+ case 'tmp_base':
+ return self.tmp_base_dir
+ case 'checkout':
+ return self.checkout_dir
+
+ # Avoid circular import.
+ from .api import PathApi
+ if name not in PathApi.NamedBasePaths:
+ raise ValueError(
+ f'Unknown base path {name!r} - allowed names are {PathApi.NamedBasePaths!r}'
+ )
+
+ @property
+ def start_dir(self) -> Path:
+ return Path(ResolvedBasePath('[START_DIR]'))
+
+ @property
+ def tmp_base_dir(self) -> Path:
+ return Path(ResolvedBasePath('[TMP_BASE]'))
+
+ @property
+ def cache_dir(self) -> Path:
+ return Path(ResolvedBasePath('[CACHE]'))
+
+ @property
+ def cleanup_dir(self) -> Path:
+ return Path(ResolvedBasePath('[CLEANUP]'))
+
+ @property
+ def home_dir(self) -> Path:
+ return Path(ResolvedBasePath('[HOME]'))
+
+ @property
+ def checkout_dir(self) -> Path:
+ return Path(CheckoutBasePath())
+
+ def cast_to_path(self, strpath: str) -> UnvalidatedPath:
+ """Allows an absolute path to be used to mock the existence.
+
+ This path will be validated to be absolute by the path module when it loads
+ the mocked data. This will validate and split `strpath` according to the
+ mocked OS (e.g. using '\\' and '/' and parsing Windows drive on Windows,
+ using '/' on *nix).
+
+ These paths are ONLY good for .exists, .files_exist and .dirs_exist on this
+ test API.
+ """
+ return UnvalidatedPath(strpath, ())
diff --git a/recipe_modules/path/tests/cast_to_path.py b/recipe_modules/path/tests/cast_to_path.py
new file mode 100644
index 0000000..0242e15
--- /dev/null
+++ b/recipe_modules/path/tests/cast_to_path.py
@@ -0,0 +1,67 @@
+# Copyright 2023 The LUCI Authors. All rights reserved.
+# Use of this source code is governed under the Apache License, Version 2.0
+# that can be found in the LICENSE file.
+
+from recipe_engine.post_process import DropExpectation
+from recipe_engine.config_types import Path, ResolvedBasePath
+
+DEPS = [
+ 'path',
+ 'platform',
+]
+
+
+def RunSteps(api):
+ if api.platform.is_win:
+ arbitrary = api.path.cast_to_path(r'c:\some\random/path')
+ assert arbitrary.base == ResolvedBasePath(r'c:'), f'"{arbitrary.base!r}"'
+ assert arbitrary.pieces == ('some', 'random',
+ 'path'), f'{arbitrary.pieces!r}'
+ assert str(arbitrary) == r'c:\some\random\path'
+
+ arbitrary2 = api.path.cast_to_path(r'c:\\\\\hey/there')
+ assert arbitrary2 == Path(ResolvedBasePath(r'c:'), 'hey', 'there')
+
+ try:
+ # non-absolute paths fail
+ api.path.cast_to_path(r'hey\there')
+ assert False # pragma: no cover
+ except ValueError:
+ pass
+
+ assert api.path.isfile(r'd:\legit\file')
+
+ else:
+ arbitrary = api.path.cast_to_path('/some/random/path')
+ assert arbitrary.base == ResolvedBasePath(''), f'"{arbitrary.base!r}"'
+ assert arbitrary.pieces == ('some', 'random',
+ 'path'), f'{arbitrary.pieces!r}'
+ assert str(arbitrary) == '/some/random/path'
+
+ try:
+ # non-absolute paths fail
+ api.path.cast_to_path(r'cool/beans')
+ assert False # pragma: no cover
+ except ValueError:
+ pass
+
+ assert api.path.isdir(r'/legit/dir')
+ assert api.path.isdir(r'/legit/dir/etc')
+
+
+def GenTests(api):
+ yield api.test(
+ 'win',
+ api.platform.name('win'),
+ api.path.exists(api.path.cast_to_path(r'd:\legit\file')),
+ api.post_process(DropExpectation),
+ )
+
+ base = api.path.cast_to_path('/legit/dir')
+
+ yield api.test(
+ 'linux',
+ api.platform.name('linux'),
+ api.path.dirs_exist(base.join('etc')),
+ api.post_process(DropExpectation),
+ )
diff --git a/recipe_modules/path/tests/dynamic_paths.py b/recipe_modules/path/tests/dynamic_paths.py
index 5722bff..1ee69cc 100644
--- a/recipe_modules/path/tests/dynamic_paths.py
+++ b/recipe_modules/path/tests/dynamic_paths.py
@@ -15,6 +15,14 @@
assert 'called with bad type' in str(ex), str(ex)
try:
+ # Note - legacy api.path.get('checkout') is the only way to get a dynamic
+ # checkout path before assignment to checkout_dir.
+ api.path.checkout_dir = api.path.get('checkout').join('something')
+ assert False, 'able to assign string to path?' # pragma: no cover
+ except ValueError as ex:
+ assert 'cannot be rooted in checkout_dir' in str(ex), str(ex)
+
+ try:
api.path['something'] = api.path['start_dir'].join('coolstuff')
assert False, 'able to assign path to non-dynamic path?' # pragma: no cover
except ValueError as ex:
diff --git a/recipe_modules/path/tests/exists.py b/recipe_modules/path/tests/exists.py
new file mode 100644
index 0000000..76feb02
--- /dev/null
+++ b/recipe_modules/path/tests/exists.py
@@ -0,0 +1,39 @@
+# Copyright 2023 The LUCI Authors. All rights reserved.
+# Use of this source code is governed under the Apache License, Version 2.0
+# that can be found in the LICENSE file.
+
+from recipe_engine.post_process import DropExpectation
+
+DEPS = ['path']
+
+
+def RunSteps(api):
+ assert not api.path.exists(api.path.start_dir / 'does not exist')
+ assert not api.path.isfile(api.path.start_dir / 'does not exist')
+ assert not api.path.isdir(api.path.start_dir / 'does not exist')
+
+ assert api.path.exists(api.path.start_dir / 'a file')
+ assert api.path.isfile(api.path.start_dir / 'a file')
+ assert not api.path.isdir(api.path.start_dir / 'a file')
+
+ assert api.path.exists(api.path.start_dir / 'a dir')
+ assert not api.path.isfile(api.path.start_dir / 'a dir')
+ assert api.path.isdir(api.path.start_dir / 'a dir')
+
+ # Our PathTestApi allows us to mock the existence of paths in the checkout
+ # directory. However, the checkout directory still must be set before this
+ # check is done.
+ api.path.checkout_dir = api.path.cache_dir / 'builder' / 'src'
+ assert api.path.exists(api.path['checkout'] / 'somefile')
+
+
+def GenTests(api):
+ yield api.test(
+ 'basic',
+ api.path.files_exist(
+ api.path.start_dir / 'a file',
+ api.path.checkout_dir / 'somefile',
+ ),
+ api.path.dirs_exist(api.path.start_dir / 'a dir'),
+ api.post_process(DropExpectation),
+ )
diff --git a/recipe_modules/path/tests/test_api_legacy.py b/recipe_modules/path/tests/test_api_legacy.py
new file mode 100644
index 0000000..6b076cb
--- /dev/null
+++ b/recipe_modules/path/tests/test_api_legacy.py
@@ -0,0 +1,44 @@
+# Copyright 2023 The LUCI Authors. All rights reserved.
+# Use of this source code is governed under the Apache License, Version 2.0
+# that can be found in the LICENSE file.
+"""Test to cover legacy aspects of PathTestApi."""
+
+from recipe_engine.post_process import DropExpectation
+
+DEPS = ['path']
+
+GETITEM_NAMES = [
+ 'cache',
+ 'cleanup',
+ 'home',
+ 'start_dir',
+ 'tmp_base',
+]
+
+
+def RunSteps(api):
+ for name in GETITEM_NAMES:
+ p = api.path.get(name) / 'file'
+ assert api.path.exists(p), p
+
+ api.path.checkout_dir = api.path.start_dir / 'somedir'
+ assert api.path.exists(api.path.get('checkout') / 'file')
+
+
+def GenTests(api):
+ paths = [api.path[name].join('file') for name in GETITEM_NAMES]
+ paths.append(api.path['checkout'].join('file'))
+
+ # This is for coverage - we need to make sure that api.path[typo] raises an
+ # exception.
+ try:
+ api.path['chekout'] # note the typo
+ assert False, 'PathTestApi did not catch typo' # pragma: no cover
+ except ValueError:
+ pass
+
+ yield api.test(
+ 'basic',
+ api.path.exists(*paths),
+ api.post_process(DropExpectation),
+ )
diff --git a/unittests/config_types_test.py b/unittests/config_types_test.py
index f5edcd5..e24e07f 100755
--- a/unittests/config_types_test.py
+++ b/unittests/config_types_test.py
@@ -3,51 +3,173 @@
# Use of this source code is governed under the Apache License, Version 2.0
# that can be found in the LICENSE file.
-import test_env
+import unittest
-from recipe_engine import config_types
+import test_env # for sys.path manipulation
+
+from recipe_engine.config_types import Path, ResolvedBasePath, CheckoutBasePath
+from recipe_engine.config_types import ResetGlobalVariableAssignments
-class TestPaths(test_env.RecipeEngineUnitTest):
- """Test case for config_types.Path."""
- base_path = config_types.Path(config_types.NamedBasePath('base'))
+class TestPathsPreGlobalInit(unittest.TestCase):
+ """Test case for config_types.Path prior to recipe_engine/path module
+ initialization.
+ """
- def test_path_join(self) -> None:
+ def tearDown(self) -> None:
+ CheckoutBasePath._resolved = None
+ return super().tearDown()
+
+ def test_path_construction_resolved(self):
+ # Doesn't raise any errors
+ cachePath = Path(ResolvedBasePath('[CACHE]'))
+ assert isinstance(cachePath.base, ResolvedBasePath)
+ self.assertEqual(cachePath.base.resolved, '[CACHE]')
+ self.assertEqual(cachePath.pieces, ())
+
+ def test_path_construction_checkout(self):
+ checkoutPath = Path(CheckoutBasePath())
+ assert isinstance(checkoutPath.base, CheckoutBasePath)
+
+ def test_path_construction_error_base(self):
+ with self.assertRaisesRegex(ValueError, 'First argument'):
+ Path('yo') # type: ignore
+
+ def test_path_construction_error_pieces(self):
+ with self.assertRaisesRegex(ValueError, 'must only be `str`'):
+ Path(ResolvedBasePath('[CACHE]'), 100) # type: ignore
+
+ def test_path_construction_error_backslash(self):
+ with self.assertRaisesRegex(ValueError, 'contain backslash'):
+ Path(ResolvedBasePath('[CACHE]'), 'bad\\path')
+
+ def test_path_construction_resolved_pieces(self):
+ a = Path(ResolvedBasePath('[CACHE]'), 'hello', 'world')
+ self.assertEqual(a.pieces, ('hello', 'world'))
+
+ b = Path(ResolvedBasePath('[CACHE]'), 'hello/world')
+ self.assertEqual(b.pieces, ('hello', 'world'))
+
+ self.assertEqual(a, b)
+
+ def test_path_construction_checkout_pieces(self):
+ a = Path(CheckoutBasePath(), 'hello', 'world')
+ self.assertEqual(a.pieces, ('hello', 'world'))
+
+ b = Path(CheckoutBasePath(), 'hello/world')
+ self.assertEqual(b.pieces, ('hello', 'world'))
+
+ # Note that these can be compared when they are both based on
+ # CheckoutBasePath.
+ self.assertEqual(a, b)
+
+ def test_path_inequality_resolved(self):
+ p = Path(ResolvedBasePath('[CACHE]'))
+ self.assertLess(p / 'a', p / 'b')
+ self.assertLess(p / 'a', p / 'b' / 'c')
+ self.assertLess(p / 'a' / 'c', p / 'b' / 'c')
+
+ def test_path_inequality_checkout(self):
+ p = Path(CheckoutBasePath())
+ self.assertLess(p / 'a', p / 'b')
+ self.assertLess(p / 'a', p / 'b' / 'c')
+ self.assertLess(p / 'a' / 'c', p / 'b' / 'c')
+
+ def test_path_inequality_mismatch(self):
+ a = Path(CheckoutBasePath())
+ b = Path(ResolvedBasePath('[CACHE]'))
+ with self.assertRaisesRegex(ValueError, 'before checkout_dir is set'):
+ self.assertLess(a, b)
+
+ def test_path_equality_mismatch(self):
+ a = Path(CheckoutBasePath())
+ b = Path(ResolvedBasePath('[CACHE]'))
+ with self.assertRaisesRegex(ValueError, 'before checkout_dir is set'):
+ self.assertEqual(a, b)
+
+ def test_path_dots_removal(self):
+ p = Path(ResolvedBasePath('[CACHE]'))
+
+ self.assertEqual(p / 'hello', p / '.' / 'hello' / '.' / '.')
+
+ self.assertEqual(p, p / '.')
+
+ self.assertEqual(
+ p / 'some/hello',
+ # Note that no one would ever construct a path with all these styles,
+ # however it's important that all the various joinery/embedded slash
+ # styles result in the same Path because recipe code passes Paths around
+ # and joins to them in multiple methods, so while we would never see
+ # such a construction all in one line like this, it's possible that
+ # a Path is logically constructed in multiple places in this fashion.
+ (p / 'some/path/to/stuff' / '../..').join('etc', '..////.', '..',
+ 'hello'))
+
+ def test_path_dots_removal_error(self):
+ p = Path(ResolvedBasePath('[CACHE]'))
+
+ with self.assertRaisesRegex(ValueError, 'going above the base'):
+ print(repr(p / '..'))
+
+ with self.assertRaisesRegex(ValueError, 'going above the base'):
+ print(repr(p / 'something' / '..///./..'))
+
+ def test_path_join(self):
"""Tests for Path.join()."""
- reference_path = self.base_path.join('foo').join('bar')
- self.assertEqual(self.base_path / 'foo' / 'bar', reference_path)
+ base_path = Path(ResolvedBasePath('[START_DIR]'))
+ reference_path = base_path.join('foo').join('bar')
+ self.assertEqual(base_path / 'foo' / 'bar', reference_path)
- def test_equality_after_separate(self) -> None:
- """Test that separating paths makes equality work.
+ def test_is_parent_of(self):
+ p = Path(ResolvedBasePath('[CACHE]'))
- Config types don't know what platform they're running on. Thus, Paths don't
- know what their separator character is. Until their pieces are explicitly
- separated, two Paths representing identical locations might present as
- unequal.
- """
- path_with_slashes = self.base_path.join('foo/bar')
- path_with_multiple_pieces = self.base_path.join('foo', 'bar')
- # This first assertion isn't desired behavior, but it demonstrates the
- # problem being solved.
- self.assertNotEqual(path_with_slashes, path_with_multiple_pieces)
- path_with_slashes.separate('/')
- self.assertEqual(path_with_slashes, path_with_multiple_pieces)
+ self.assertTrue(p.is_parent_of(p / 'a'))
+ self.assertTrue(p.is_parent_of(p / 'a' / 'b' / 'c'))
+ self.assertTrue((p / 'a').is_parent_of(p / 'a' / 'b' / 'c'))
- def test_parenthood_after_separate(self) -> None:
- """Test that separating paths makes parenthood checks work.
+ def test_is_parent_of_mismatch(self):
+ p1 = Path(ResolvedBasePath('[CACHE]'))
+ p2 = Path(ResolvedBasePath('[CLEANUP]'))
- Config types don't know what platform they're running on. Thus, Paths don't
- know what their separator character is. Until their pieces are explicitly
- separated, one path might represent a parent of another, but is_parent_of
- might not agree.
- """
- my_file = self.base_path.join('foo/bar.txt')
- my_dir = self.base_path.join('foo')
- # This first assertion isn't desired behavior, but it demonstrates the
- # problem being solved.
- self.assertFalse(my_dir.is_parent_of(my_file))
- my_file.separate('/')
- self.assertTrue(my_dir.is_parent_of(my_file))
+ self.assertFalse(p1.is_parent_of(p2))
+ self.assertFalse(p2.is_parent_of(p1))
+
+ def test_is_parent_of_checkout(self):
+ p1 = Path(CheckoutBasePath(), 'some')
+ p2 = Path(ResolvedBasePath('[CACHE]'), 'builder', 'src', 'some', 'thing')
+
+ with self.assertRaisesRegex(ValueError, 'checkout_dir is unset'):
+ p1.is_parent_of(p2)
+ with self.assertRaisesRegex(ValueError, 'checkout_dir is unset'):
+ p2.is_parent_of(p1)
+
+ CheckoutBasePath._resolved = Path(
+ ResolvedBasePath('[CACHE]'), 'builder', 'src')
+
+ self.assertTrue(p1.is_parent_of(p2))
+ self.assertFalse(p2.is_parent_of(p1))
+
+ def test_is_parent_of_checkout_mismatch(self):
+ p1 = Path(CheckoutBasePath(), 'some')
+ p2 = Path(ResolvedBasePath('[CLEANUP]'), 'unrelated')
+
+ CheckoutBasePath._resolved = Path(
+ ResolvedBasePath('[CACHE]'), 'builder', 'src')
+
+ self.assertFalse(p1.is_parent_of(p2))
+ self.assertFalse(p2.is_parent_of(p1))
+
+ def test_is_parent_of_sanity(self):
+ p = Path(ResolvedBasePath('[CLEANUP]'))
+ self.assertFalse((p / 'a').is_parent_of(p / 'ab'))
+
+
+class TestPathsPostGlobalInit(unittest.TestCase):
+ """Test case for config_types.Path."""
+
+ def tearDown(self):
+ ResetGlobalVariableAssignments()
+ return super().tearDown()
if __name__ == '__main__':
diff --git a/unittests/errors_test.py b/unittests/errors_test.py
index 9d8a58b..b25faa9 100755
--- a/unittests/errors_test.py
+++ b/unittests/errors_test.py
@@ -101,16 +101,19 @@
yield api.test('basic')
''')
- def _assert_keyerror(output):
- self.assertRegex(output,
- "KeyError.{1,3}Unknown path: bippityboppityboo.{1,3}")
+ def _assert_error(output):
+ self.assertRegex(output, "bippityboppityboo.*unknown base path")
- self._test_cmd(deps, ['test', 'train', '--filter', 'missing_path'],
- asserts=_assert_keyerror, retcode=1)
- self._test_cmd(deps, ['test', 'run', '--filter', 'missing_path'],
- asserts=_assert_keyerror, retcode=1)
- self._test_cmd(deps, ['run', 'missing_path'],
- asserts=_assert_keyerror, retcode=1)
+ self._test_cmd(
+ deps, ['test', 'train', '--filter', 'missing_path'],
+ asserts=_assert_error,
+ retcode=1)
+ self._test_cmd(
+ deps, ['test', 'run', '--filter', 'missing_path'],
+ asserts=_assert_error,
+ retcode=1)
+ self._test_cmd(
+ deps, ['run', 'missing_path'], asserts=_assert_error, retcode=1)
def test_engine_failure(self):
deps = self.FakeRecipeDeps()
