[swarming] Refactor "tasks" package and how it is tested.

The primary motivation is to do all changes to state of
tasks via a single interface tasks.Manager, which is
passed to other server subsystems (botapi, rpcs, scan,
etc). It encapsulates all necessary dependencies to
change tasks (removing the need for callbacks soup added
in https://chromium-review.googlesource.com/c/6427869).

Tests for botapi and other server parts can just mock it
out by using &tasks.MockedManager{}.

But some RPC tests actually rely on implementation of tasks
logic. To avoid breaking (or significantly changing) them,
we need to use the "production" implementation, but mock
some of its bits, in particular all TQ tasks.

To avoid adding back the need to pass extra fields other
than tasks.Manager, and to avoid excessively polluting
tasks implementation with mocks, use tqtesting.Scheduler
infrastructure in tests. This would also eventually allow
to run more high-level integration-like tests (since
tqtesting.Scheduler can actually run full workflows).

Unfortunately, it has its own can of worms: tq.AddTask can
only add tasks that have been registered in the dispatcher,
and there are a bunch of different packages that register
tasks.

To avoid coupling between all these packages, move TQ task
definitions into a dedicated "tqtasks" package that just
exposes all known TQ tasks, without their implementations.

Finally, make API of various existing task operations more
consistent. In particular, cancellation now always require
the caller to open a transaction, just like ClaimOp and
AbandonOp.

To simplify this transactional API, use txndefer.Defer(...)
to register post-transactional work, like metrics reporting,
instead of making callers to call post-txn functions
explicitly.

txndefer.Defer is already used by server/tq implementation,
so it is already a significantly dependency of the server.

R=chanli@chromium.org

Change-Id: Ie016a416d9b0eec10bdd7e96d6a1a39476d6287c
Reviewed-on: https://chromium-review.googlesource.com/c/infra/luci/luci-go/+/6433706
Reviewed-by: Chan Li <chanli@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
41 files changed
tree: 0f408c515b279c0973b0d522a427cd45a2cd3003
  1. analysis/
  2. appengine/
  3. auth/
  4. auth_service/
  5. bisection/
  6. build/
  7. buildbucket/
  8. casviewer/
  9. cipd/
  10. cipkg/
  11. client/
  12. common/
  13. config/
  14. config_service/
  15. cv/
  16. deploy/
  17. examples/
  18. gae/
  19. gce/
  20. grpc/
  21. hardcoded/
  22. led/
  23. logdog/
  24. luci_notify/
  25. lucicfg/
  26. lucictx/
  27. luciexe/
  28. mailer/
  29. milo/
  30. mmutex/
  31. provenance/
  32. recipes_py/
  33. resultdb/
  34. scheduler/
  35. scripts/
  36. server/
  37. source_index/
  38. standalone/
  39. starlark/
  40. swarming/
  41. teams/
  42. third_party/
  43. tokenserver/
  44. tools/
  45. tree_status/
  46. vpython/
  47. web/
  48. .gitallowed
  49. .gitignore
  50. .go-lintable
  51. AUTHORS
  52. codereview.settings
  53. CONTRIBUTING.md
  54. CONTRIBUTORS
  55. go.mod
  56. go.sum
  57. LICENSE
  58. OWNERS
  59. PRESUBMIT.py
  60. README.md
  61. staticcheck.conf
  62. tools.go
  63. WATCHLISTS
README.md

luci-go: LUCI services and tools in Go

GoReference

Installing

LUCI Go code is meant to be worked on from an Chromium infra.git checkout, which enforces packages versions and Go toolchain version. First get fetch via depot_tools.git then run:

fetch infra
cd infra/go
eval `./env.py`
cd src/go.chromium.org/luci

It is now possible to directly install tools with go install:

go install go.chromium.org/luci/auth/client/cmd/...@latest
go install go.chromium.org/luci/buildbucket/cmd/...@latest
go install go.chromium.org/luci/cipd/client/cmd/...@latest
go install go.chromium.org/luci/client/cmd/...@latest
go install go.chromium.org/luci/cv/cmd/...@latest
go install go.chromium.org/luci/gce/cmd/...@latest
go install go.chromium.org/luci/grpc/cmd/...@latest
go install go.chromium.org/luci/logdog/client/cmd/...@latest
go install go.chromium.org/luci/luci_notify/cmd/...@latest
go install go.chromium.org/luci/lucicfg/cmd/...@latest
go install go.chromium.org/luci/luciexe/legacy/cmd/...@latest
go install go.chromium.org/luci/mailer/cmd/...@latest
go install go.chromium.org/luci/mmutex/cmd/...@latest
go install go.chromium.org/luci/resultdb/cmd/...@latest
go install go.chromium.org/luci/server/cmd/...@latest
go install go.chromium.org/luci/swarming/cmd/...@latest
go install go.chromium.org/luci/tokenserver/cmd/...@latest
go install go.chromium.org/luci/tools/cmd/...@latest

Contributing

Contributing uses the same flow as Chromium contributions.