Document execution engine
diff --git a/platforms/core-execution/execution/.editorconfig b/platforms/core-execution/execution/.editorconfig new file mode 100644 index 0000000..72257b6 --- /dev/null +++ b/platforms/core-execution/execution/.editorconfig
@@ -0,0 +1,2 @@ +[*.adoc] +indent_size = 2
diff --git a/platforms/core-execution/execution/README.adoc b/platforms/core-execution/execution/README.adoc new file mode 100644 index 0000000..7d3401b --- /dev/null +++ b/platforms/core-execution/execution/README.adoc
@@ -0,0 +1,69 @@ +// Copyright (C) 2023 Gradle, Inc. +// +// Licensed under the Creative Commons Attribution-Noncommercial-ShareAlike 4.0 International License.; +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// https://creativecommons.org/licenses/by-nc-sa/4.0/ +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + += Execution Engine + +The role of the **execution engine** is to take some work with well-defined inputs and outputs, and using the inputs manifest the outputs as efficiently as possible. + +The engine is implemented as a network of steps. +During execution the engine passes a _context_ object along that accumulates information about the work as it gets passed deeper in the hierarchy of steps. +When the outputs have been manifested, a _result_ object gets passed back along the same path, accumulating metadata about the manifested outputs along the way. + +== Semantic structure of pipeline + +[mermaid] +.... +flowchart TD + Identify[Identify work] + IdentityCache{{In identity cache?}} + ChoosePipeline{{Work type?}} + + Identify -->|identity| IdentityCache + IdentityCache -->|hit| Done + IdentityCache -->|miss| ChoosePipeline + Done[Done] + + ChoosePipeline -->|"non-incremental work\n(excluding tasks)"| ImmutablePipeline + subgraph ImmutablePipeline[Immutable pipeline] + LoadImmutableHistory["Load history\n(check if workspace exists)"] + + LoadImmutableHistory -->|hit| CheckImmutableWorkspace[Check workspace] + LoadImmutableHistory -->|miss| AllocateTemporaryWorkspace[Allocate temporary workspace] + AllocateTemporaryWorkspace --> CheckIfImmutableEmpty{{Empty sources?}} + CheckIfImmutableEmpty -->|no| ExecuteNonIncrementalWork["Execute work\n(or load from cache)"] + ExecuteNonIncrementalWork -->|success| StoreImmutableHistory + CheckIfImmutableEmpty -->|yes| StoreImmutableHistory + StoreImmutableHistory["Store history"] --> MoveWorkspaceToImmutableArea[Move workspace\nto immutable area] + end + CheckImmutableWorkspace --> Done + ExecuteNonIncrementalWork -->|failure| Done + MoveWorkspaceToImmutableArea --> Done + + ChoosePipeline --->|"tasks and incremental work"| MutablePipeline + subgraph MutablePipeline["Mutable pipeline"] + AllocateMutableWorkspace[Allocate mutable workspace\nunder lock] + + AllocateMutableWorkspace --> LoadMutableHistory[Load history] + LoadMutableHistory --> CleanStaleOutputs[Clean stale outputs] + CleanStaleOutputs --> SkipIfSourcesEmpty{{Empty sources?}} + SkipIfSourcesEmpty -->|no| SkipUpToDate{{Is up-to-date?}} + SkipIfSourcesEmpty -->|yes| DeletePreviousOutputs[Delete previous outputs] + SkipUpToDate -->|no| ExecuteIncrementalWork["Execute work\n(or load from cache)"] + DeletePreviousOutputs ----> StoreMutableHistory[Store history] + ExecuteIncrementalWork --> RecordOutputs["Record outputs\n(for stale output cleanup later)"] + RecordOutputs --> StoreMutableHistory + end + StoreMutableHistory --> Done + SkipUpToDate -->|yes| Done +....
diff --git a/platforms/core-execution/execution/Steps.md b/platforms/core-execution/execution/Steps.md new file mode 100644 index 0000000..5f17682 --- /dev/null +++ b/platforms/core-execution/execution/Steps.md
@@ -0,0 +1,94 @@ +```mermaid +flowchart TD + Identify[Identify work] <--> MutablePipeline[Mutable pipeline] + Identify <--> ImmutablePipeline[Immutable pipeline] + MutablePipeline <--> Execution[Execution pipeline] + ImmutablePipeline <--> Execution +``` + +## Identification of the work + +```mermaid +sequenceDiagram + Execution engine ->> Identify: ExecutionRequestContext + Identify -> Identify: Capture identity inputs + Identify ->> IdentityCache: IdentityContext + alt Not in identity cache + IdentityCache -->> ChoosePipeline: IdentityContext + alt Mutable work + ChoosePipeline ->> Mutable pipeline: IdentityContext + Mutable pipeline ->> ChoosePipeline: Result + else Immutable work + ChoosePipeline ->> Immutable pipeline: IdentityContext + Immutable pipeline ->> ChoosePipeline: Result + end + ChoosePipeline -->> IdentityCache: Result + end + IdentityCache ->> Identify: Result + Identify ->> Execution engine: Result +``` + +## Immutable pipeline + +```mermaid +sequenceDiagram + AssignImmutableWorkspace ->> AssignTemporaryWorkspace: ImmutableWorkspaceContext + alt Immutable workspace missing + AssignTemporaryWorkspace ->> Validate: TemporaryWorkspaceContext + Validate ->> ResolveCachingState: ValidationFinishedContext + ResolveCachingState ->> SkipEmptyImmutableWork: CachingContext + alt Non-empty sources + SkipEmptyImmutableWork ->> BuildCache: CachingContext + alt Cache miss + BuildCache ->> BroadcastChangingOutputs: CachingContext + BroadcastChangingOutputs ->> Execution pipeline: ChangingOutputsContext + Execution pipeline ->> BroadcastChangingOutputs: Result + BroadcastChangingOutputs ->> BuildCache: Result + end + BuildCache ->> SkipEmptyImmutableWork: CachingResult + end + SkipEmptyImmutableWork ->> ResolveCachingState: CachingResult + ResolveCachingState ->> Validate: CachingResult + Validate ->> AssignTemporaryWorkspace: CachingResult + AssignTemporaryWorkspace ->> AssignTemporaryWorkspace: Store history + AssignTemporaryWorkspace ->> AssignTemporaryWorkspace: Move workspace to immutable location + end + AssignTemporaryWorkspace ->> AssignImmutableWorkspace: CachingResult +``` + +## Mutable pipeline + +```mermaid +sequenceDiagram + + AssignPersistentWorkspace ->> HandleStaleOutputs: WorkspaceContext + HandleStaleOutputs ->> LoadPreviousExecutionState: WorkspaceContext + LoadPreviousExecutionState ->> SkipEmptyIncrementalWork: PreviousExecutionContext + SkipEmptyIncrementalWork ->> CaptureStateBeforeExecution: PreviousExecutionContext + CaptureStateBeforeExecution ->> Validate: BeforeExecutionState + Validate ->> ResolveCachingState: ValidationFinishedContext + ResolveCachingState ->> ResolveChanges: CachingContext + ResolveChanges ->> SkipUpToDate: IncrementalChangesContext + SkipUpToDate ->> StoreExecutionState: IncrementalChangesContext + StoreExecutionState ->> BuildCache: PreviousExecutionContext + BuildCache ->> ResolveInputChanges: IncrementalChangesContext + ResolveInputChanges ->> CaptureStateAfterExecution: InputChangesContext + CaptureStateAfterExecution ->> BroadcastChangingOutputs: BeforeExecutionState + BroadcastChangingOutputs ->> RemovePreviousOutputs: ChangingOutputsContext + RemovePreviousOutputs ->> Execution pipeline: ChangingOutputsContext + Execution pipeline ->> RemovePreviousOutputs: Result + RemovePreviousOutputs ->> BroadcastChangingOutputs: Result + BroadcastChangingOutputs ->> CaptureStateAfterExecution: Result + CaptureStateAfterExecution ->> ResolveInputChanges: AfterExecutionResult + ResolveInputChanges ->> BuildCache: Result + BuildCache ->> StoreExecutionState: AfterExecutionResult + StoreExecutionState ->> SkipUpToDate: AfterExecutionResult + SkipUpToDate ->> ResolveChanges: UpToDateResult + ResolveChanges ->> ResolveCachingState: Result + ResolveCachingState ->> Validate: CachingResult + Validate ->> CaptureStateBeforeExecution: CachingResult + CaptureStateBeforeExecution ->> SkipEmptyIncrementalWork: CachingResult + SkipEmptyIncrementalWork ->> LoadPreviousExecutionState: AfterExecutionResult + LoadPreviousExecutionState ->> HandleStaleOutputs: AfterExecutionResult + HandleStaleOutputs ->> AssignPersistentWorkspace: AfterExecutionResult +```