[warning] Records import warnings when initialize api
R=iannucci
Change-Id: Id9674c785ee0da2b26a3ed056f99ad2f8e470203
Reviewed-on: https://chromium-review.googlesource.com/c/infra/luci/recipes-py/+/2363284
Commit-Queue: Yiwei Zhang <yiwzhang@google.com>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
diff --git a/README.recipes.md b/README.recipes.md
index 1be0317..ce2802b 100644
--- a/README.recipes.md
+++ b/README.recipes.md
@@ -165,7 +165,7 @@
[DEPS](/recipe_modules/archive/__init__.py#5): [json](#recipe_modules-json), [path](#recipe_modules-path), [platform](#recipe_modules-platform), [python](#recipe_modules-python), [step](#recipe_modules-step)
-#### **class [ArchiveApi](/recipe_modules/archive/api.py#8)([RecipeApi](/recipe_engine/recipe_api.py#873)):**
+#### **class [ArchiveApi](/recipe_modules/archive/api.py#8)([RecipeApi](/recipe_engine/recipe_api.py#877)):**
Provides steps to manipulate archive files (tar, zip, etc.).
@@ -225,7 +225,7 @@
Package object.
### *recipe_modules* / [assertions](/recipe_modules/assertions)
-#### **class [AssertionsApi](/recipe_modules/assertions/api.py#56)([RecipeApi](/recipe_engine/recipe_api.py#873)):**
+#### **class [AssertionsApi](/recipe_modules/assertions/api.py#56)([RecipeApi](/recipe_engine/recipe_api.py#877)):**
Provides access to the assertion methods of the python unittest module.
@@ -283,7 +283,7 @@
`build_pb2.Build` and returns a link title.
If it returns `None`, the link is not reported. Default link title is build id.
-#### **class [BuildbucketApi](/recipe_modules/buildbucket/api.py#29)([RecipeApi](/recipe_engine/recipe_api.py#873)):**
+#### **class [BuildbucketApi](/recipe_modules/buildbucket/api.py#29)([RecipeApi](/recipe_engine/recipe_api.py#877)):**
A module for interacting with buildbucket.
@@ -664,7 +664,7 @@
API for interacting with cas client.
-#### **class [CasApi](/recipe_modules/cas/api.py#13)([RecipeApi](/recipe_engine/recipe_api.py#873)):**
+#### **class [CasApi](/recipe_modules/cas/api.py#13)([RecipeApi](/recipe_engine/recipe_api.py#877)):**
A module for interacting with cas client.
@@ -706,7 +706,7 @@
Depends on 'cipd' binary available in PATH:
https://godoc.org/go.chromium.org/luci/cipd/client/cmd/cipd
-#### **class [CIPDApi](/recipe_modules/cipd/api.py#202)([RecipeApi](/recipe_engine/recipe_api.py#873)):**
+#### **class [CIPDApi](/recipe_modules/cipd/api.py#202)([RecipeApi](/recipe_engine/recipe_api.py#877)):**
CIPDApi provides basic support for CIPD.
@@ -974,7 +974,7 @@
Returns the CIPDApi.Pin instance.
### *recipe_modules* / [commit\_position](/recipe_modules/commit_position)
-#### **class [CommitPositionApi](/recipe_modules/commit_position/api.py#10)([RecipeApi](/recipe_engine/recipe_api.py#873)):**
+#### **class [CommitPositionApi](/recipe_modules/commit_position/api.py#10)([RecipeApi](/recipe_engine/recipe_api.py#877)):**
Recipe module providing commit position parsing and formatting.
@@ -1014,7 +1014,7 @@
api.step("cat subdir/foo", ['cat', './foo'])
```
-#### **class [ContextApi](/recipe_modules/context/api.py#78)([RecipeApi](/recipe_engine/recipe_api.py#873)):**
+#### **class [ContextApi](/recipe_modules/context/api.py#78)([RecipeApi](/recipe_engine/recipe_api.py#877)):**
  **@contextmanager**<br>— **def [\_\_call\_\_](/recipe_modules/context/api.py#108)(self, cwd=None, env_prefixes=None, env_suffixes=None, env=None, infra_steps=None, luciexe=None, realm=None):**
@@ -1133,7 +1133,7 @@
[DEPS](/recipe_modules/cq/__init__.py#7): [buildbucket](#recipe_modules-buildbucket), [properties](#recipe_modules-properties), [step](#recipe_modules-step)
-#### **class [CQApi](/recipe_modules/cq/api.py#14)([RecipeApi](/recipe_engine/recipe_api.py#873)):**
+#### **class [CQApi](/recipe_modules/cq/api.py#14)([RecipeApi](/recipe_engine/recipe_api.py#877)):**
This module provides recipe API of LUCI CQ, aka pre-commit testing system.
@@ -1263,7 +1263,7 @@
File manipulation (read/write/delete/glob) methods.
-#### **class [FileApi](/recipe_modules/file/api.py#83)([RecipeApi](/recipe_engine/recipe_api.py#873)):**
+#### **class [FileApi](/recipe_modules/file/api.py#83)([RecipeApi](/recipe_engine/recipe_api.py#877)):**
— **def [compute\_hash](/recipe_modules/file/api.py#182)(self, name, paths, base_path, test_data=''):**
@@ -1653,7 +1653,7 @@
Implements in-recipe concurrency via green threads.
-#### **class [FuturesApi](/recipe_modules/futures/api.py#42)([RecipeApi](/recipe_engine/recipe_api.py#873)):**
+#### **class [FuturesApi](/recipe_modules/futures/api.py#42)([RecipeApi](/recipe_engine/recipe_api.py#877)):**
Provides access to the Recipe concurrency primitives.
@@ -1802,7 +1802,7 @@
another repo. It is not recommended to use this, and it will be removed in the
near future.
-#### **class [GeneratorScriptApi](/recipe_modules/generator_script/api.py#16)([RecipeApi](/recipe_engine/recipe_api.py#873)):**
+#### **class [GeneratorScriptApi](/recipe_modules/generator_script/api.py#16)([RecipeApi](/recipe_engine/recipe_api.py#877)):**
— **def [\_\_call\_\_](/recipe_modules/generator_script/api.py#44)(self, path_to_script, \*args):**
@@ -1843,7 +1843,7 @@
[DEPS](/recipe_modules/isolated/__init__.py#1): [cipd](#recipe_modules-cipd), [context](#recipe_modules-context), [json](#recipe_modules-json), [path](#recipe_modules-path), [properties](#recipe_modules-properties), [raw\_io](#recipe_modules-raw_io), [runtime](#recipe_modules-runtime), [step](#recipe_modules-step)
-#### **class [IsolatedApi](/recipe_modules/isolated/api.py#15)([RecipeApi](/recipe_engine/recipe_api.py#873)):**
+#### **class [IsolatedApi](/recipe_modules/isolated/api.py#15)([RecipeApi](/recipe_engine/recipe_api.py#877)):**
API for interacting with isolated.
@@ -1902,7 +1902,7 @@
Methods for producing and consuming JSON.
-#### **class [JsonApi](/recipe_modules/json/api.py#95)([RecipeApi](/recipe_engine/recipe_api.py#873)):**
+#### **class [JsonApi](/recipe_modules/json/api.py#95)([RecipeApi](/recipe_engine/recipe_api.py#877)):**
  **@staticmethod**<br>— **def [dumps](/recipe_modules/json/api.py#96)(\*args, \*\*kwargs):**
@@ -1943,7 +1943,7 @@
[DEPS](/recipe_modules/led/__init__.py#5): [cipd](#recipe_modules-cipd), [context](#recipe_modules-context), [json](#recipe_modules-json), [path](#recipe_modules-path), [proto](#recipe_modules-proto), [step](#recipe_modules-step)
-#### **class [LedApi](/recipe_modules/led/api.py#15)([RecipeApi](/recipe_engine/recipe_api.py#873)):**
+#### **class [LedApi](/recipe_modules/led/api.py#15)([RecipeApi](/recipe_engine/recipe_api.py#877)):**
Interface to the led tool.
@@ -2011,7 +2011,7 @@
build (using the Merge Step feature from luciexe protocol). This is the
replacement for allow_subannotation feature in the legacy annotate mode.
-#### **class [LegacyAnnotationApi](/recipe_modules/legacy_annotation/api.py#24)([RecipeApiPlain](/recipe_engine/recipe_api.py#725)):**
+#### **class [LegacyAnnotationApi](/recipe_modules/legacy_annotation/api.py#24)([RecipeApiPlain](/recipe_engine/recipe_api.py#729)):**
— **def [\_\_call\_\_](/recipe_modules/legacy_annotation/api.py#28)(self, name, cmd, timeout=None, step_test_data=None, cost=_ResourceCost()):**
@@ -2026,7 +2026,7 @@
API for specifying Milo behavior.
-#### **class [MiloApi](/recipe_modules/milo/api.py#17)([RecipeApi](/recipe_engine/recipe_api.py#873)):**
+#### **class [MiloApi](/recipe_modules/milo/api.py#17)([RecipeApi](/recipe_engine/recipe_api.py#877)):**
A module for interacting with Milo.
@@ -2077,7 +2077,7 @@
`depot_tools/infra_paths` module). Refer to those modules for additional
documentation.
-#### **class [PathApi](/recipe_modules/path/api.py#206)([RecipeApi](/recipe_engine/recipe_api.py#873)):**
+#### **class [PathApi](/recipe_modules/path/api.py#206)([RecipeApi](/recipe_engine/recipe_api.py#877)):**
— **def [\_\_getitem\_\_](/recipe_modules/path/api.py#438)(self, name):**
@@ -2243,7 +2243,7 @@
Mockable system platform identity functions.
-#### **class [PlatformApi](/recipe_modules/platform/api.py#24)([RecipeApi](/recipe_engine/recipe_api.py#873)):**
+#### **class [PlatformApi](/recipe_modules/platform/api.py#24)([RecipeApi](/recipe_engine/recipe_api.py#877)):**
Provides host-platform-detection properties.
@@ -2328,7 +2328,7 @@
intentionally no API to write property values (lest they become a kind of
random-access global variable).
-#### **class [PropertiesApi](/recipe_modules/properties/api.py#28)([RecipeApiPlain](/recipe_engine/recipe_api.py#725), collections.Mapping):**
+#### **class [PropertiesApi](/recipe_modules/properties/api.py#28)([RecipeApiPlain](/recipe_engine/recipe_api.py#729), collections.Mapping):**
PropertiesApi implements all the standard Mapping functions, so you
can use it like a read-only dict.
@@ -2355,7 +2355,7 @@
Methods for producing and consuming protobuf data to/from steps and the
filesystem.
-#### **class [ProtoApi](/recipe_modules/proto/api.py#75)([RecipeApi](/recipe_engine/recipe_api.py#873)):**
+#### **class [ProtoApi](/recipe_modules/proto/api.py#75)([RecipeApi](/recipe_engine/recipe_api.py#877)):**
  **@staticmethod**<br>— **def [decode](/recipe_modules/proto/api.py#151)(data, msg_class, codec, \*\*decoding_kwargs):**
@@ -2432,7 +2432,7 @@
correctly for bots (e.g. ensuring that python is working on Windows, passing the
unbuffered flag, etc.)
-#### **class [PythonApi](/recipe_modules/python/api.py#17)([RecipeApi](/recipe_engine/recipe_api.py#873)):**
+#### **class [PythonApi](/recipe_modules/python/api.py#17)([RecipeApi](/recipe_engine/recipe_api.py#877)):**
— **def [\_\_call\_\_](/recipe_modules/python/api.py#18)(self, name, script, args=None, unbuffered=True, venv=None, \*\*kwargs):**
@@ -2516,7 +2516,7 @@
api.random.shuffle(my_list)
# my_list is now random!
-#### **class [RandomApi](/recipe_modules/random/api.py#31)([RecipeApi](/recipe_engine/recipe_api.py#873)):**
+#### **class [RandomApi](/recipe_modules/random/api.py#31)([RecipeApi](/recipe_engine/recipe_api.py#877)):**
— **def [\_\_getattr\_\_](/recipe_modules/random/api.py#38)(self, name):**
@@ -2527,7 +2527,7 @@
Provides objects for reading and writing raw data to and from steps.
-#### **class [RawIOApi](/recipe_modules/raw_io/api.py#297)([RecipeApi](/recipe_engine/recipe_api.py#873)):**
+#### **class [RawIOApi](/recipe_modules/raw_io/api.py#297)([RecipeApi](/recipe_engine/recipe_api.py#877)):**
  **@[returns\_placeholder](/recipe_engine/util.py#151)**<br>  **@staticmethod**<br>— **def [input](/recipe_modules/raw_io/api.py#298)(data, suffix='', name=None):**
@@ -2622,7 +2622,7 @@
Requires `rdb` command in `$PATH`:
https://godoc.org/go.chromium.org/luci/resultdb/cmd/rdb
-#### **class [ResultDBAPI](/recipe_modules/resultdb/api.py#19)([RecipeApi](/recipe_engine/recipe_api.py#873)):**
+#### **class [ResultDBAPI](/recipe_modules/resultdb/api.py#19)([RecipeApi](/recipe_engine/recipe_api.py#877)):**
A module for interacting with ResultDB.
@@ -2741,7 +2741,7 @@
file name with a relative path. The value must start with "//".
### *recipe_modules* / [runtime](/recipe_modules/runtime)
-#### **class [RuntimeApi](/recipe_modules/runtime/api.py#8)([RecipeApi](/recipe_engine/recipe_api.py#873)):**
+#### **class [RuntimeApi](/recipe_modules/runtime/api.py#8)([RecipeApi](/recipe_engine/recipe_api.py#877)):**
This module assists in experimenting with production recipes.
@@ -2770,7 +2770,7 @@
RPCExplorer available at
https://luci-scheduler.appspot.com/rpcexplorer/services/scheduler.Scheduler
-#### **class [SchedulerApi](/recipe_modules/scheduler/api.py#26)([RecipeApi](/recipe_engine/recipe_api.py#873)):**
+#### **class [SchedulerApi](/recipe_modules/scheduler/api.py#26)([RecipeApi](/recipe_engine/recipe_api.py#877)):**
A module for interacting with LUCI Scheduler service.
@@ -2826,7 +2826,7 @@
Depends on luci-auth to be in PATH.
-#### **class [ServiceAccountApi](/recipe_modules/service_account/api.py#16)([RecipeApi](/recipe_engine/recipe_api.py#873)):**
+#### **class [ServiceAccountApi](/recipe_modules/service_account/api.py#16)([RecipeApi](/recipe_engine/recipe_api.py#877)):**
— **def [default](/recipe_modules/service_account/api.py#57)(self):**
@@ -2852,7 +2852,7 @@
Step is the primary API for running steps (external programs, scripts,
etc.).
-#### **class [StepApi](/recipe_modules/step/api.py#26)([RecipeApiPlain](/recipe_engine/recipe_api.py#725)):**
+#### **class [StepApi](/recipe_modules/step/api.py#26)([RecipeApiPlain](/recipe_engine/recipe_api.py#729)):**
  **@property**<br>— **def [InfraFailure](/recipe_modules/step/api.py#138)(self):**
@@ -3159,7 +3159,7 @@
[DEPS](/recipe_modules/swarming/__init__.py#8): [buildbucket](#recipe_modules-buildbucket), [cipd](#recipe_modules-cipd), [context](#recipe_modules-context), [isolated](#recipe_modules-isolated), [json](#recipe_modules-json), [path](#recipe_modules-path), [properties](#recipe_modules-properties), [raw\_io](#recipe_modules-raw_io), [runtime](#recipe_modules-runtime), [step](#recipe_modules-step)
-#### **class [SwarmingApi](/recipe_modules/swarming/api.py#1161)([RecipeApi](/recipe_engine/recipe_api.py#873)):**
+#### **class [SwarmingApi](/recipe_modules/swarming/api.py#1161)([RecipeApi](/recipe_engine/recipe_api.py#877)):**
API for interacting with swarming.
@@ -3261,7 +3261,7 @@
Allows mockable access to the current time.
-#### **class [TimeApi](/recipe_modules/time/api.py#12)([RecipeApi](/recipe_engine/recipe_api.py#873)):**
+#### **class [TimeApi](/recipe_modules/time/api.py#12)([RecipeApi](/recipe_engine/recipe_api.py#877)):**
— **def [ms\_since\_epoch](/recipe_modules/time/api.py#49)(self):**
@@ -3289,7 +3289,7 @@
API for Tricium analyzers to use.
-#### **class [TriciumApi](/recipe_modules/tricium/api.py#15)([RecipeApi](/recipe_engine/recipe_api.py#873)):**
+#### **class [TriciumApi](/recipe_modules/tricium/api.py#15)([RecipeApi](/recipe_engine/recipe_api.py#877)):**
TriciumApi provides basic support for Tricium.
@@ -3311,7 +3311,7 @@
Methods for interacting with HTTP(s) URLs.
-#### **class [UrlApi](/recipe_modules/url/api.py#15)([RecipeApi](/recipe_engine/recipe_api.py#873)):**
+#### **class [UrlApi](/recipe_modules/url/api.py#15)([RecipeApi](/recipe_engine/recipe_api.py#877)):**
— **def [get\_file](/recipe_modules/url/api.py#131)(self, url, path, step_name=None, headers=None, transient_retry=True, strip_prefix=None, timeout=None):**
@@ -3416,7 +3416,7 @@
Allows test-repeatable access to a random UUID.
-#### **class [UuidApi](/recipe_modules/uuid/api.py#11)([RecipeApi](/recipe_engine/recipe_api.py#873)):**
+#### **class [UuidApi](/recipe_modules/uuid/api.py#11)([RecipeApi](/recipe_engine/recipe_api.py#877)):**
— **def [random](/recipe_modules/uuid/api.py#20)(self):**
@@ -3425,7 +3425,7 @@
Thin API for parsing semver strings into comparable object.
-#### **class [VersionApi](/recipe_modules/version/api.py#12)([RecipeApi](/recipe_engine/recipe_api.py#873)):**
+#### **class [VersionApi](/recipe_modules/version/api.py#12)([RecipeApi](/recipe_engine/recipe_api.py#877)):**
  **@staticmethod**<br>— **def [parse](/recipe_modules/version/api.py#14)(version):**
@@ -3441,7 +3441,7 @@
Allows recipe modules to issue warnings in simulation test.
-#### **class [WarningApi](/recipe_modules/warning/api.py#12)([RecipeApiPlain](/recipe_engine/recipe_api.py#725)):**
+#### **class [WarningApi](/recipe_modules/warning/api.py#12)([RecipeApiPlain](/recipe_engine/recipe_api.py#729)):**
  **@recipe_api.escape_all_warnings**<br>— **def [issue](/recipe_modules/warning/api.py#15)(self, name):**
diff --git a/recipe_engine/internal/engine.py b/recipe_engine/internal/engine.py
index 9155289..3f3c01c 100644
--- a/recipe_engine/internal/engine.py
+++ b/recipe_engine/internal/engine.py
@@ -208,6 +208,10 @@
"""
self._clients['paths']._initialize_with_recipe_api(root_api)
+ def record_import_warning(self, warning, importer):
+ """Records an import warning."""
+ self._clients['warning'].record_import_warning(warning, importer)
+
def close_non_parent_step(self):
"""Closes the tip of the _step_stack if it's not a parent nesting step."""
try:
diff --git a/recipe_engine/internal/recipe_deps.py b/recipe_engine/internal/recipe_deps.py
index d70b23a..e026fbb 100644
--- a/recipe_engine/internal/recipe_deps.py
+++ b/recipe_engine/internal/recipe_deps.py
@@ -62,10 +62,8 @@
from .exceptions import UnknownRecipeModule
from .simple_cfg import SimpleRecipesCfg, RECIPES_CFG_LOCATION_REL
from .test.test_util import filesystem_safe
-from .warn.definition import (
- parse_warning_definitions,
- RECIPE_WARNING_DEFINITIONS_REL,
-)
+from .warn.definition import (parse_warning_definitions,
+ RECIPE_WARNING_DEFINITIONS_REL)
LOG = logging.getLogger(__name__)
@@ -408,6 +406,19 @@
"""
return parse_deps_spec(self.repo.name, self.do_import().DEPS)
+ @cached_property
+ def warnings(self):
+ """Returns a tuple of warnings issued against this recipe module."""
+ return tuple(self.do_import().WARNINGS)
+
+ @cached_property
+ def _cumulative_import_warnings(self):
+ """Returns all import warnings as a tuple that this module and its
+ dependent modules hit. Each element of the tuple is a tuple of
+ (fully-qualified warning name, importer RecipeModule).
+ """
+ return tuple(_collect_import_warnings(self))
+
def do_import(self):
"""Imports the raw recipe module (i.e. python module).
@@ -675,6 +686,8 @@
test_data=test_data.get_module_test_data(None))
resolved_deps = _resolve(
self.repo.recipe_deps, self.normalized_DEPS, 'API', engine, test_data)
+ for _, (warning, importer) in enumerate(_collect_import_warnings(self)):
+ engine.record_import_warning(warning, importer)
api.__dict__.update({
local_name: resolved_dep
for local_name, resolved_dep in resolved_deps.iteritems()
@@ -801,6 +814,24 @@
return deps
+def _collect_import_warnings(root):
+ """Traverses the dependency tree from root and collects all import warnings.
+
+ Returns a set of (fully-qualified warning name, importing recipe or
+ recipe module) tuple.
+ """
+ ret = set()
+ recipe_deps = root.repo.recipe_deps
+ for _, (repo_name, module_name) in root.normalized_DEPS.iteritems():
+ module = recipe_deps.repos[repo_name].modules[module_name]
+ for warning in module.warnings:
+ if '/' not in warning:
+ warning = '/'.join((repo_name, warning))
+ ret.add((warning, root))
+ ret.update(module._cumulative_import_warnings)
+ return ret
+
+
def _instantiate_test_api(imported_module, resolved_deps):
"""Instantiates the RecipeTestApi class from the given imported recipe module.
diff --git a/recipe_engine/internal/recipe_module_importer.py b/recipe_engine/internal/recipe_module_importer.py
index 0a9d885..4591f72 100644
--- a/recipe_engine/internal/recipe_module_importer.py
+++ b/recipe_engine/internal/recipe_module_importer.py
@@ -181,6 +181,7 @@
module, or None if no config.py exists.
* `DEPS`: Sets a default DEPS value of `()` so that other code in the
engine can assume that there's ALWAYS a DEPS object for a module.
+ * `WARNINGS`: A list of warnings issued against this recipe module.
* `DISABLE_STRICT_COVERAGE`: Sets a default value of False.
Args:
@@ -196,6 +197,7 @@
mod.REPO_ROOT = Path(RepoBasePath(repo_name, repo_root))
mod.CONFIG_CTX = getattr(mod, 'CONFIG_CTX', None)
mod.DEPS = getattr(mod, 'DEPS', ())
+ mod.WARNINGS = getattr(mod, 'WARNINGS', ())
# TODO(iannucci, probably): remove DISABLE_STRICT_COVERAGE (crbug/693058).
mod.DISABLE_STRICT_COVERAGE = getattr(mod, 'DISABLE_STRICT_COVERAGE', False)
diff --git a/recipe_engine/recipe_api.py b/recipe_engine/recipe_api.py
index bf05de0..391544d 100644
--- a/recipe_engine/recipe_api.py
+++ b/recipe_engine/recipe_api.py
@@ -310,6 +310,10 @@
[frame_tup[0] for frame_tup in inspect.stack()],
)
+ def record_import_warning(self, name, importer):
+ """Records import warning during DEPS resolution."""
+ self._recorder.record_import_warning(name, importer)
+
def resolve_warning(self, name, issuer_file):
"""Returns the fully-qualified warning name for the given warning.
