siso starlark config

siso uses starlark for configuration to apply to ninja build rules.

siso ninja subcommand will load starlark file specified by -load flag at startup time. Default is @config//main.star (//build/config/siso/main.star).

The starlark script should provide an init function to register handlers and step configs. In the starlark script, load can load from current directory, @builtin or @config.

Adding a top level strict_config = True variable will add extra validity checks on the config.

The starlark script will be evaluated at initialization phase and per-step phase

Example

load("@builtin//json.star", "json")
load("@builtin//runtime.star", "runtime")
load("@builtin//struct.star", "module")

def _stamp(ctx, cmd):
  out = cmd.outputs[0]
  ctx.actions.write(out)
  ctx.actions.exit(exit_status = 0)

def init(ctx):
  print("runtime os:%s arch:%s num:%d " % (runtime.os, runtime.arch, runtime.num_cpu)

  step_config = {}
  step_config["platforms"] = {
    "default": {
        "OSFamily": "Linux",
        "container-image": "docker://gcr.io/chops-private-images-prod/rbe/siso-chromium/linux@sha256:d4fcda628ebcdb3dd79b166619c56da08d5d7bd43d1a7b1f69734904cc7a1bb2",
    },
  }
  input_deps = {
    "some/toolchain/clang": [
       "some/toolchain/libclang.so",
       "some/toolchain/extra-inputs",
    ],
  }
  step_config["input_deps"] = input_deps
  rules = [
     {
       "name": "cxx/compile",
       "action": "cxx",
       "inputs": [
         "some/toolchain/clang",
       ],
       "remote": True,
     },
     {
       "name": "stamp",
       "action": "stamp",
       "handler": "stamp",
     },
  ]
  step_config["rules"] = rules
  return module(
    "config",
    step_config = json.encode(step_config),
    filegroups = {
       # "base:headers" is set in input_deps by using glob.
       # and it is used as precomputed tree for include dir "base",
       # when command line uses the directory as include dir.
       # e.g.
       #  dir="out/Default" command="clang -I../../base"
       "base:headers": {
          "type": "glob",
          "includes": ["*.h"],
       },
    },
    handlers = {
      "stamp": _stamp,
    }
 )

Initialization

Before starting build process, siso calls init function with ctx to register handlers and step configs.

ctx provides

• actions module
  • metadata update metadata.
• metadata dict
  • args.gn: contents of args.gn
  • build.manifest: digest of ninja build manifest
• flags dict
  • command line flags. key doesn't have prefix - of flag. “project” is set even if it is specified by SISO_PROJECT_ID. “is_terminal” is set to indicate terminal or not, which would change some default value of flags. -C uses “dir” as key. targets (non-flags) uses “target” as key.
• fs: path is workspace relative.
  • read: read contents.
    • fname: filename
  • is_dir check path is dir.
    • fname: filename
  • exists check path exists.
    • fname: filename
  • size get file size.
    • fname: filename
  • canonpath: convert ninja dir relative to workspace relative
    • fname: filename

print will print a message to log file. fail will abort the process.

init should return a module.

• filegroups dict
  • filegroups is used to create input_deps at initialization phase by using glob. It is recorded in .siso_filegroups and reuse without running glob if build graph and config is not changed.
  • key: filegroup label (i.e. dir:name).
  • value: dict of a filegroup generator
    • “type” specifies filegroup generator type.
      • “glob”:
        • Recursively traverses files under the target directory using glob patterns. Directories themselves are not included in the result.
        • “dir”: (optional) root directory to search. If omitted, defaults to the directory part before : in the filegroup label key.
        • “includes”: list of glob patterns to include in the glob. e.g. ["*"] to include all files recursively.
        • “excludes”: (optional) list of glob patterns to exclude from the glob.
        • A file is included if it matches at least one pattern in includes and does not match any pattern in excludes.
      • TODO(b/266759797): add other type, like “filelist”.
• handlers dict
  • key: handler name
  • value: handler function used in per-step config.
• step_config json string StepConfig. It will be stored in .siso_config in working directory.
  • properties
    • a dict for config properties. used for resultstore if enabled.
    • key: property key
    • value: property value
  • platforms
    • key: platform reference name. “default” is used by default.
    • value: a dict of platform properties
  • sandbox
    • a dict for sandbox. key “backend” specifies sandbox backend.
    • backend = nsjail
      • nsjail_path: a path to nsjail binary.
      • nsjail_workdir: a top directory for nsjail work dir (on same device with nsjail_outdir)
      • nsjail_outdir: a top directory of output directory (to make it writable)
  • input_deps
    • key: input path, or label (label contains ‘:’). If the key is a path, it will be included in the expanded inputs. If the key is a label, it won't be included in the expanded inputs. :headers label would be used for c++ scandeps for include dirs or sysroots. e.g. dir=“out/Default” and command=“../../third_party/llvm-build/Release+Asserts/bin/clang -I../../base ...” then, “third_party/llvm-build/Release+Asserts:headers” “base:headers” are used as precomputed tree in the remote exec inputs.
    • values: other files or labels needed for the key. <target>:inputs label would be expanded to inputs of <target>'s inputs, if <target>:inputs is not explicitly defined in input_deps.
  • case_sensitive_inputs
    • a list of filenames for case sensitive filesystem. if “a.txt” and “A.txt” are in this, and “a.txt” is used as input, “A.txt” will be added as input too.
  • inputs_requiring_clang_scandeps a list of filename that requires clang scandeps. deprecated: use scandeps.inputs_requiring_clang instead.
  • clang_scandeps clang scandeps mode. deprecated: use scandeps.use_clang instead.
  • scandeps:
    • inputs_requiring_clang: a list of filename that requires clang scandeps.
    • use_clang:
      • "": don't use clang scandeps
      • “unsupported-macro”: use clang scandeps when unsupported macro detected.
      • “scandeps-err”: use clang scandeps when builtin scandeps failed.
    • step_inputs: path_filter specify what inputs from the ninja graph are used in addition to tool_inputs, scandeps results.
  • bad_deps
    • key: output target known to have bad deps
    • value: annotation (usually bug link)
  • executables
    • (Windows only) a list of filenames for executables. This is used to send Linux executables from Windows machine. e.g. node binary for typescript action.
  • rules list of StepRule. path is workspace relative, or ninja dir relative if it starts with “./”
    • identifier
      • name: unique name of the rule. required.
    • rule selector
      • action: regexp of action name. i.e. ninja rule name.
      • action_outs: output of the action.
      • command_prefix: prefix of step's command.
    • rule to apply
      • inputs: additional inputs
      • exclude_input_patterns: glob pattern to exclude from inputs
        • if it contains ‘/’, full match to input path
        • otherwise, match basename of input path.
      • indirect_inputs: path_filter specify what inputs from the previous steps to use as inputs of this step. The matched indirect input are included recursively. Sibling outputs are also included. e.g. gen/{foo.stamp, foo.h, foo.cc} -> a step depending on gen/foo.stamp may need foo.h/foo.cc. Consider using replace or accumulate first, and use indirect_inputs only when they are not sufficient.
      • outputs: additional outputs. note: ignored in cleandead.
      • outputs_map: different deps based on outputs[0]
        • key: outputs[0]
        • value
          • inputs: additional inputs
          • outputs: additional outputs. note: ignored in cleandead.
          • platform: additional platform properties
          • platform_ref: overrides reference to platform properties
      • auxiliary_log_output_files: additional output files, that siso explicitly logs digest of. e.g. crash reports when RBE is used. siso logs the digest of the file for debugging (console or siso_output file), but doesn't download the file.
      • auxiliary_log_output_dirs: additional output dirs, that siso explicitly logs digest of. e.g. crash reports when RBE is used. siso logs the digest of the directory for debugging (console or siso_output file), but doesn't download files in the directory.
      • restat: true if step cmd reads output file and not write it (when no update needed), and considers output is clean if mtime is not changed (same as ninja's restat).
      • restat_content: true if step cmd considers output is clean if content is not changed (like ninja's restat, but not use mtime).
      • platform_ref: reference to platform properties
      • platform: additional platform properties
      • remote: use remote exec or not
      • strict_remote: force strict remote execution (i.e. no fastlocal, no local fallback) for the step.
      • remote_wrapper: a wrapper command used in remote execution
      • remote_command: args[0] will be replaced with remote_command.
      • remote_inputs
        • key: path of input of remote action
        • value: path of file (local) for the key path.
      • input_root_absolute_path: need InputRootAbsolutePath or not.
      • canonicalize_dir: ok to canonicalize work dir or not. enable by default, but disable if input_root_absolute_path is set.
      • use_system_input: ok to use input outside of workspace as it assumes those are platform container image.
      • use_remote_exec_wrapper: true if gomacc/rewrapper is used, so it runs locally without using deps/file trace.
      • timeout: duration of the step remote execution. See also Timeout field on StepRule.
      • exec_timeout: duration of the action timeout of the step remote execution. See also ExecTimeout field on StepRule.
      • handler: handler name to use for the step
      • deps: deps overrides
        • gcc: use gcc -M
        • msvc: use clang-cl /showIncludes
        • depfile: depfile variable of the step
        • none: ignore deps variable in ninja
      • output_local: force download/outputs to local disk
      • ignore_extra_input_pattern: regexp to allow if it is used, but not listed in inputs.
      • ignore_extra_output_pattern: regexp to allow if it is generated, but not listed in outputs.
      • impure: mark it as not pure. i.e. not check inputs/outputs
      • replace: if any output of this step is used in other steps, those steps will use the inputs of this step as inputs instead of the outputs of this step. used for stamp step or so.
      • accumulate: if any output of this step is used in other steps, those steps will use the inputs and outputs of this step as inputs. used for thin archive or so. Not recursively accumulated.
      • debug: enable debug log in this step.

path_filter

path_filter is used for scandeps.step_inputs and indirect_inputs.

• patterns: A list of glob patterns. If pattern starts with ‘!’, handle it as excludes. otherwise incldues. If patterns is set, it will ignore excludes and includes.

• excludes: A list of glob patterns. Any input path that matches a pattern in this list is excluded. excludes are processed before includes.

• includes: An optional list of glob patterns that specifies which inputs to include.

If includes is not specified or is empty, all inputs not excluded by excludes are included.

If includes is specified, an input is included only if it matches a pattern inincludes (and is not excluded by excludes).

glob pattern

Glob patterns are used in filegroups and path_filter. The matching logic is based on Go's path.Match.

• If a pattern contains / (e.g. "sub/*.h"), it is matched against the full or relative path of an input. Note that * in path.Match does not cross path separators (/).
• Otherwise (if the pattern contains no /, e.g. "*.h"), the pattern is matched against the basename of the input's path, matching files at any directory depth. For example, ["*"] matches all files under the directory recursively in filegroups.
• In path_filter (or patterns list):
  • If a pattern starts with !, it works as an exclude pattern.
  • If a pattern ends with /** (e.g. "dir/**"), it is matched against the directory prefix (matching any files under the directory). Note that ** glob is only supported in path_filter and does not work in filegroups.

per-step config

for each step, siso will apply the first matched rule. If the rule has handler, it runs handler with ctx and cmd when siso decides to run the step. All inputs should be ready to use (via ctx.fs).

ctx provides

• actions
  • fix: fix step
    • inputs: input pathnames
    • tool_inputs: input pathnames (not modified by deps) deprecated: use scandeps.step_inputs to filter step inputs with deps.
    • outputs: output pathnames. note: ignored in cleandead.
    • args: args for the step
    • rspfile_content: rspfile_content for the step.
    • reconcile_outputdirs: reconcile directories to detect file removals in dirs after local step executions.
    • auxiliary_log_output_files: additional output files, that siso explicitly logs digest of.
    • auxiliary_log_output_dirs: additional output dirs, that siso explicitly logs digest of.
  • write: write file
    • fname: filename
    • content: content
    • is_executable: is_executable flag
  • copy: copy file
    • src: src filename
    • dst: dst filename
    • recursive: recursive copy?
    • ignore_missing_intermediates: ignore missing intermediates.
  • symlink: create symlink
    • target: symlnk target
    • linkpath: symlink path.
  • exit: exit. don't run locally nor remotely.
    • exit_status: exit status
    • stdout: stdout
    • stderr: stderr
• fs: same as fs in ctx.fs used in init.

cmd provides

• args tuple: command line args
• envs dict: environment variables
• dir string: working directory
• workspace_root string: absolute path to workspace
• deps string: deps type
• inputs list: input pathnames
• tool_inputs list: input pathnames (rule's inputs). deprecated: use scandeps.step_inputs to filter inputs with deps.
• expanded_inputs: func returns list: expanded input pathnames
• rspfile_content: bytes: rspfile content
• outputs list: output pathnames

print will print in log.

@builtin

@builtin provides builtin modules/functions.

@builtin//encoding.star

provide json

@builtin//lib/gn.star

provide gn.

note: deprecated. parsing args.gn doesn't work well. emit computed value and parse it, like gn_logs.txt. https://crbug.com/427333789

• args: returns a dict for content of args.gn. value is not interpreted, but just as string. e.g.
  • foo=true -> gn.args(ctx)["foo"]=="true"
  • foo="data" -> gn.args(ctx)["foo"]=="\"data\""

@builtin//path.star

provide path.

• base: returns base name.
  • fname: filename
• dir: return dir name.
  • fname: filename
• join: join path elements.
  • args: path elements
• rel: return relative path.
  • basepath: base path
  • targetpath: target path
• isabs: check path is abs.
  • fname: filename

@builtin//runtime.star

provide runtime.

• num_cpu int: number of cpus
• os string: GOOS string.
• arch string: GOARCH string

@builtin//struct.star

provide struct and module

@config

@config points //build/config/siso by default (changed by --config). In chromium, @config is //build/config/siso

@config_overrides

@config_overrides provides access to local starlark files, in $workspace/.siso_remote. It is expected to have a module with name (basename of starlark) that has rules, input_deps functions. If file doesn't exist, it provides a None for the name (basename of starlark).

References

• starlark
• ninja build rules