turboci/graph/orchestrator/v1/stage_attempt_state.proto

// Copyright 2025 The Chromium Authors
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.

syntax = "proto3";

package turboci.graph.orchestrator.v1;

option go_package = "go.chromium.org/turboci/proto/go/graph/orchestrator/v1;orchestratorpb";
option java_multiple_files = true;

// NOTE - This is StageAttemptState with prefixed enum values instead of
// Stage Attempt.State so that field_options.proto can import this without making
// a circular import with stage.proto.

// StageAttemptState describes the current state of a Stage Attempt.
//
// Possible initial states are PENDING, THROTTLED and AWAITING_RETRY.
//
// This state evolves like:
//
//   PENDING -> THROTTLED
//   PENDING -> RUNNING
//   PENDING -> COMPLETE
//   PENDING -> INCOMPLETE
//   PENDING -> SCHEDULED
//
//   THROTTLED -> INCOMPLETE
//   THROTTLED -> PENDING
//
//   SCHEDULED -> RUNNING
//   SCHEDULED -> CANCELLING
//   SCHEDULED -> COMPLETE
//   SCHEDULED -> INCOMPLETE
//
//   RUNNING -> CANCELLING
//   RUNNING -> TEARING_DOWN
//   RUNNING -> COMPLETE
//   RUNNING -> INCOMPLETE
//
//   CANCELLING -> TEARING_DOWN
//   CANCELLING -> COMPLETE
//   CANCELLING -> INCOMPLETE
//
//   TEARING_DOWN -> COMPLETE
//   TEARING_DOWN -> INCOMPLETE
//
//   INCOMPLETE and COMPLETE are terminal states.
//
// The Nth Stage Attempt (i.e. any Stage Attempt after the first) may also be
// created in an AWAITING_RETRY state rather than PENDING.
//
//   AWAITING_RETRY -> PENDING
//   AWAITING_RETRY -> INCOMPLETE
//
// The Orchestrator manages Stage Attempt state evolution in conjunction with
// the Executor. The only state transition which can be initiated by a third
// party (and actuated by the Orchestrator) would be transitions to INCOMPLETE
// or CANCELLING done when the Stage is cancelled.
//
// These states have enum values in multiples of 10 in case we need to add more
// states later which fall between these initial states.
enum StageAttemptState {
  // UNKNOWN is the default, invalid, state.
  STAGE_ATTEMPT_STATE_UNKNOWN = 0;

  // This state indicates that the Stage Attempt is being sent to be executed
  // now.
  //
  // Either the attempt is queued to start running soon or executors' RunStage
  // is already running but the Executor hasn't updated the attempt's state yet.
  //
  // This is usually the initial state for an attempt. Two other possible
  // initial states are THROTTLED (if the execution policy has
  // `throttle_first_attempt_until` set and this is the first attempt or if
  // a previous attempt finished as INCOMPLETE and `throttle_next_attempt_until`
  // was set) and AWAITING_RETRY (if this attempt follows an INCOMPLETE previous
  // attempt and no `throttle_next_attempt_until` was set).
  //
  // The Executor transitions this state into:
  //  * RUNNING - to mark the stage attempt as being executed now.
  //  * SCHEDULED - to acknowledge that the stage attempt is scheduled for a
  //    asynchronous execution.
  //  * THROTTLED - to purposefully delay running the stage attempt.
  //  * COMPLETE - to immediately complete the stage.
  //  * INCOMPLETE - to immediately fail the stage attempt.
  //
  // The Orchestrator transitions this state into:
  //  * INCOMPLETE - on reaching `pending_throttled` timeout.
  STAGE_ATTEMPT_STATE_PENDING = 10;

  // This state indicates that the Stage Attempt is currently throttled.
  //
  // It means the stage attempt's execution is purposefully delayed until some
  // specified time. An attempt either starts in this state (if it was throttled
  // when the stage was submitted or by the previous attempt, see below) or it
  // transitions into this state from PENDING.
  //
  // A similar state is AWAITING_RETRY, with the primary difference being that
  // AWAITING_RETRY indicates that a previous attempt could not complete due to
  // some unexpected reason. The delay time for AWAITING_RETRY is calculated by
  // the Orchestrator itself using exponential backoff.
  //
  // Throttling can be activated via 3 mechanisms:
  //   * Via StageExecutionPolicy's `throttle_first_attempt_until` field (set
  //     either by whoever submits the stage or by executors' ValidateStage).
  //     If the stage gets unblocked before this time, the first stage attempt
  //     will start in THROTTLED state (instead of PENDING). This is useful for
  //     throttling decisions that happen when the stage is inserted.
  //   * By the executor in its RunStage by using CurrentAttemptWrite.Throttled.
  //     The current PENDING attempt will become THROTTLED. This is useful for
  //     throttling decisions that happen when the stage is about to run (which
  //     can be much later than the time the stage was inserted).
  //   * By the executor in its RunStage by using CurrentAttemptWrite.Incomplete
  //     and setting `throttle_next_attempt_until`. The next attempt will start
  //     in THROTTLED state. This is useful if the executors discovers it needs
  //     to slow down after already starting the stage execution.
  //
  // The Orchestrator transitions this state into:
  //  * PENDING - when it is time to launch the attempt.
  //  * INCOMPLETE - on reaching `pending_throttled` timeout.
  STAGE_ATTEMPT_STATE_THROTTLED = 20;

  // This state indicates that the Stage Attempt was PENDING, but is now picked
  // up for future execution by an Executor.
  //
  // The Executor transitions this state into:
  //  * RUNNING - to mark the stage attempt as being executed now.
  //  * COMPLETE - to immediately complete the stage.
  //  * INCOMPLETE - to immediately fail the stage attempt.
  //
  // The Orchestrator transitions this state into:
  //  * CANCELLING - if the stage was cancelled.
  //  * INCOMPLETE - on reaching `scheduled` timeout or missing `scheduled`
  //    heartbeat, or if the stage was cancelled and the attempt execution
  //    policy has no `cancelling` timeout set.
  STAGE_ATTEMPT_STATE_SCHEDULED = 30;

  // This state indicates that the Stage Attempt is now actually being executed
  // by an Executor.
  //
  // The Executor transitions this state into:
  //  * COMPLETE - to complete the stage.
  //  * INCOMPLETE - to fail the stage attempt.
  //  * TEARING_DOWN - to signal the stage attempt is about to be finished soon.
  //
  // The Orchestrator transitions this state into:
  //  * CANCELLING - if the stage was cancelled.
  //  * INCOMPLETE - on reaching `running` timeout or missing `running`
  //    heartbeat, or if the stage was cancelled and the attempt execution
  //    policy has no `cancelling` timeout set
  STAGE_ATTEMPT_STATE_RUNNING = 40;

  // This state indicates that the stage has been cancelled, but this has not
  // yet been acknowledged by the Executor.
  //
  // While the attempt is in this state, The Orchestrator will keep calling
  // executor's CancelStage RPC (with exponential backoff). The Executor is
  // supposed to acknowledge the cancellation by moving the attempt into
  // TEARING_DOWN or COMPLETE/INCOMPLETE state.
  //
  // This state will be skipped if the attempt execution policy has no
  // `cancelling` timeout set. In that case the cancelled attempt will
  // transition into INCOMPLETE immediately.
  //
  // The Executor transitions this state into:
  //  * COMPLETE - to complete the stage.
  //  * INCOMPLETE - to fail the stage attempt.
  //  * TEARING_DOWN - to acknowledge the cancellation, but keep working.
  //
  // The Orchestrator transitions this state into:
  //  * INCOMPLETE - on reaching `cancelling` timeout or missing a heartbeat.
  //    The heartbeat deadline that is being checked depends on the attempt
  //    history: it is either `scheduled` if the attempt was SCHEDULED prior to
  //    being cancelled, or `running` if the attempt was RUNNING prior to
  //    being cancelled.
  STAGE_ATTEMPT_STATE_CANCELLING = 50;

  // This state indicates that the Stage Attempt is doing some work after the
  // RUNNING state, in particular (but not necessarily) in response to
  // cancellation.
  //
  // This is meant to model things like:
  //   * Doing best-effort cleanup.
  //   * Exporting logs or other state to other systems.
  //
  // It's desirable for stage attempts to explicitly transition to TEARING_DOWN
  // for two reasons:
  //  * To signal that the attempt is shutting down (e.g. for debugging UIs).
  //  * To set a much shorter cleanup timeout when they know their primary work
  //    is done. Otherwise they would only timeout on their possibly very-long
  //    `running` timeout.
  //
  // Note that cancellation of an attempt in TEARING_DOWN state does nothing
  // (the executor's CancelStage RPC won't be called), because the attempt is
  // assumed do be done soon anyway (as enforced by `tearing_down` timeout).
  //
  // The Executor transitions this state into:
  //  * COMPLETE - to complete the stage.
  //  * INCOMPLETE - to fail the stage attempt.
  //
  // The Orchestrator transitions this state into:
  //  * INCOMPLETE - on reaching `tearing_down` timeout or missing
  //    `tearing_down` heartbeat.
  STAGE_ATTEMPT_STATE_TEARING_DOWN = 60;

  // This is a final state which indicates that this Stage Attempt finished
  // everything the Executor intended it to do.
  //
  // NOTE: 'finished everything the Executor intended it to do' does not imply
  // workflow-level success criteria such as "all tests passed" or "the build
  // successfully compiled". This is more like "the test harness ran all of the
  // tests (regardless of their outcome)" or "the build started and all possible
  // compilation actions were done". These outcomes are highy varied and nuanced
  // (e.g. tests may be graded on flakiness, not simply pass/fail, build may
  // have failed on an optional target but successfully completed the required
  // targets). These workflow-level outcomes are reflected in the Check Results
  // which this Stage Attempt would have written during its execution.
  //
  // This state is terminal.
  STAGE_ATTEMPT_STATE_COMPLETE = 70;

  // This is a final state which indicates that this Stage Attempt was not able
  // to complete everything that the Executor intended it to do.
  //
  // The Orchestrator itself can transition the Stage Attempt to this state in
  // the event of a heartbeat timeout. This state can also be entered by
  // explicit external actions (e.g. the Executor marking the stage as
  // INCOMPLETE, or external cancellation signals).
  //
  // The Orchestrator will consult the StageExecutionPolicy, and if allowed,
  // will make a new Attempt. This can be blocked when using WriteNodes to
  // mark a Stage Attempt as INCOMPLETE by setting the `block_new_attempts`
  //
  // This state is terminal (but if a retry was requested, a new Stage Attempt
  // may be created for this Stage).
  STAGE_ATTEMPT_STATE_INCOMPLETE = 80;

  // This is an initial state and indicates that a previous Stage Attempt went
  // to INCOMPLETE and the Stage's execution policy permitted a retry.
  //
  // Transition to PENDING is automatic after the exponential backoff delay
  // computed based on the number of previous incomplete attempts.
  //
  // The Orchestrator transitions this state into:
  //  * PENDING - when it is time to launch the attempt.
  //  * INCOMPLETE - on reaching `pending_throttled` timeout.
  STAGE_ATTEMPT_STATE_AWAITING_RETRY = 90;
}