Add source manifest proto to recipe_engine.

This adds the initial form of the "source manifest" proto which will
allow recipe_engine to communicate structured data about
stuff-on-disk-that-we-got-from-remote-servers to LUCI services like Milo,
and will also give us a better option than 'got_revision' properties for
exporting this data from recipes (i.e. we'll be able to export all the
data and then query/read it instead of hard-coding a few hacks to export
specific repos).

This only covers git, cipd and isolated for now, but could be expanded
to other data sources, if needed. We don't currently anticipate a need
for that, but the proto is hopefully obviously extensible for this.

Clients consuming this proto (like Milo) may have specialized support
when comparing two manifest (e.g. two entries sharing the same git repo
may render as a git log), but it should also be somewhat easy to show a
useful generic diff when comparing two Manifest protos, even when the
deployment type changes (i.e. local path "foo" used to be a git checkout,
but is now multiple CIPD packages).

R=dnj@chromium.org, hinoka@chromium.org, machenbach@chromium.org
BUG=468053

Review-Url: https://codereview.chromium.org/2998523002
diff --git a/recipe_engine/source_manifest.proto b/recipe_engine/source_manifest.proto
new file mode 100644
index 0000000..d599069
--- /dev/null
+++ b/recipe_engine/source_manifest.proto
@@ -0,0 +1,128 @@
+// Copyright 2016 The LUCI Authors. All rights reserved.
+// Use of this source code is governed under the Apache License, Version 2.0
+// that can be found in the LICENSE file.
+
+// Recompile with protoc 3.1.0+:
+// cd recipe_engine && protoc deployment_manifest.proto --python_out=.
+
+syntax = "proto3";
+
+// A Manifest attempts to make an accurate accounting of source/data directories
+// during the execution of a LUCI task.
+//
+// These directories are primarially in the form of e.g. git checkouts of
+// source, but also include things like isolated hashes and CIPD package
+// deployments. In the future, other deployment forms may be supported (like
+// other SCMs).
+//
+// The purpose of this manifest is so that other parts of the LUCI stack (e.g.
+// Milo) can work with the descripitons of this deployed data as a first-class
+// citizen. Initially this Manifest will be used to allow Milo to display diffs
+// between jobs, but it will also be useful for tools and humans to get a
+// record of exactly what data went into this LUCI task.
+//
+// Source Manifests can be emitted from recipes using the
+// 'recipe_engine/manifest' module.
+message Manifest {
+  // Version will increment on backwards-incompatible changes only. Backwards
+  // compatible changes will not alter this version number.
+  //
+  // Currently, the only valid version number is 0.
+  int32 version = 1;
+
+  message GitCheckout {
+    // The canonicalized URL of the original repo that is considered the “source
+    // of truth” for the source code. Ex.
+    //   https://chromium.googlesource.com/chromium/tools/build.git
+    //   https://github.com/luci/recipes-py
+    string repo_url = 1;
+
+    // If different from repo_url, this can be the URL of the repo that the source
+    // was actually fetched from (i.e. a mirror). Ex.
+    //   https://chromium.googlesource.com/external/github.com/luci/recipes-py
+    //
+    // If this is empty, it's presumed to be equal to repo_url.
+    string fetch_url = 2;
+
+    // The fully resolved revision (commit hash) of the source. Ex.
+    //   3617b0eea7ec74b8e731a23fed2f4070cbc284c4
+    //
+    // Note that this is the raw revision bytes, not their hex-encoded form.
+    bytes revision = 3;
+
+    // The ref that the task used to resolve the revision of the source (if any). Ex.
+    //   refs/heads/master
+    //   refs/changes/04/511804/4
+    //
+    // This should always be a ref on the hosted repo (not any local alias
+    // like 'refs/remotes/...').
+    //
+    // This should always be an absolute ref (i.e. starts with 'refs/'). An
+    // example of a non-absolute ref would be 'master'.
+    string tracking_ref = 4;
+  }
+
+  message CIPDPackage {
+    // The canonicalized URL of the CIPD server which provided the package. Ex.
+    //   https://chrome-infra-packages.appspot.com
+    string cipd_server_url = 1;
+
+    // The fully resolved CIPD package name that was deployed. Ex.
+    //   infra/tools/luci/led/linux-amd64
+    string cipd_package_name = 2;
+
+    // The package pattern that was given to the CIPD client (if known). Ex.
+    //   infra/tools/luci/led/${platform}
+    string cipd_package_pattern = 3;
+
+    // The fully resolved instance ID of the deployed package. Ex.
+    //   0cfafb3a705bd8f05f86c6444ff500397fbb711c
+    //
+    // Note that this is the raw instance_id bytes, not their hex-encoded form.
+    bytes cipd_instance_id = 4;
+
+    // The unresolved version ID of the deployed package (if known). Ex.
+    //   git_revision:aaf3a2cfccc227b5141caa1b6b3502c9907d7420
+    //   latest
+    string cipd_version = 5;
+  }
+
+  message Isolated {
+    // The canonicalized URL of the isolated server which hosts the isolated.
+    // Ex.
+    //   https://isolateserver.appspot.com
+    string isolated_server_url = 1;
+
+    // The namespace of the isolated document. Ex.
+    //   default-gzip
+    string namespace = 2;
+
+    // The hash of the isolated document. Ex.
+    //   62a7df62ea122380afb306bb4d9cdac1bc7e9a96
+    //
+    // Note that this is the raw hash bytes, not their hex-encoded form.
+    bytes hash = 3;
+  }
+
+  // A Directory contains one or more descriptions of deployed artifacts. Note
+  // that due to the practical nature of jobs on bots, it may be the case that
+  // a given directory contains e.g. a git checkout and multiple cipd packages.
+  message Directory {
+    GitCheckout git_checkout = 1;
+
+    repeated CIPDPackage cipd_package = 2;
+    repeated Isolated isolated = 3;
+  }
+
+  // Map of local file system directory path (with forward slashes) to
+  // a Directory message containing one or more deployments.
+  //
+  // The local path is relative to some job-specific root. This should be used
+  // for informational/display/organization purposes, and should not be used as
+  // a global primary key. i.e. if you depend on chromium/src.git being in
+  // a folder called “src”, I will find you and make really angry faces at you
+  // until you change it...(╬ಠ益ಠ). Instead, implementations should consider
+  // indexing by e.g. git repository URL or cipd package name as more better
+  // primary keys.
+  map<string, Directory> directories = 2;
+}
diff --git a/recipe_engine/source_manifest_pb2.py b/recipe_engine/source_manifest_pb2.py
new file mode 100644
index 0000000..a8c4533
--- /dev/null
+++ b/recipe_engine/source_manifest_pb2.py
@@ -0,0 +1,362 @@
+# Generated by the protocol buffer compiler.  DO NOT EDIT!
+# source: source_manifest.proto
+
+import sys
+_b=sys.version_info[0]<3 and (lambda x:x) or (lambda x:x.encode('latin1'))
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+from google.protobuf import descriptor_pb2
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+
+
+DESCRIPTOR = _descriptor.FileDescriptor(
+  name='source_manifest.proto',
+  package='',
+  syntax='proto3',
+  serialized_pb=_b('\n\x15source_manifest.proto\"\xdb\x04\n\x08Manifest\x12\x0f\n\x07version\x18\x01 \x01(\x05\x12/\n\x0b\x64irectories\x18\x02 \x03(\x0b\x32\x1a.Manifest.DirectoriesEntry\x1aZ\n\x0bGitCheckout\x12\x10\n\x08repo_url\x18\x01 \x01(\t\x12\x11\n\tfetch_url\x18\x02 \x01(\t\x12\x10\n\x08revision\x18\x03 \x01(\x0c\x12\x14\n\x0ctracking_ref\x18\x04 \x01(\t\x1a\x8f\x01\n\x0b\x43IPDPackage\x12\x17\n\x0f\x63ipd_server_url\x18\x01 \x01(\t\x12\x19\n\x11\x63ipd_package_name\x18\x02 \x01(\t\x12\x1c\n\x14\x63ipd_package_pattern\x18\x03 \x01(\t\x12\x18\n\x10\x63ipd_instance_id\x18\x04 \x01(\x0c\x12\x14\n\x0c\x63ipd_version\x18\x05 \x01(\t\x1aH\n\x08Isolated\x12\x1b\n\x13isolated_server_url\x18\x01 \x01(\t\x12\x11\n\tnamespace\x18\x02 \x01(\t\x12\x0c\n\x04hash\x18\x03 \x01(\x0c\x1a\x8b\x01\n\tDirectory\x12+\n\x0cgit_checkout\x18\x01 \x01(\x0b\x32\x15.Manifest.GitCheckout\x12+\n\x0c\x63ipd_package\x18\x02 \x03(\x0b\x32\x15.Manifest.CIPDPackage\x12$\n\x08isolated\x18\x03 \x03(\x0b\x32\x12.Manifest.Isolated\x1aG\n\x10\x44irectoriesEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\"\n\x05value\x18\x02 \x01(\x0b\x32\x13.Manifest.Directory:\x02\x38\x01\x62\x06proto3')
+)
+_sym_db.RegisterFileDescriptor(DESCRIPTOR)
+
+
+
+
+_MANIFEST_GITCHECKOUT = _descriptor.Descriptor(
+  name='GitCheckout',
+  full_name='Manifest.GitCheckout',
+  filename=None,
+  file=DESCRIPTOR,
+  containing_type=None,
+  fields=[
+    _descriptor.FieldDescriptor(
+      name='repo_url', full_name='Manifest.GitCheckout.repo_url', index=0,
+      number=1, type=9, cpp_type=9, label=1,
+      has_default_value=False, default_value=_b("").decode('utf-8'),
+      message_type=None, enum_type=None, containing_type=None,
+      is_extension=False, extension_scope=None,
+      options=None),
+    _descriptor.FieldDescriptor(
+      name='fetch_url', full_name='Manifest.GitCheckout.fetch_url', index=1,
+      number=2, type=9, cpp_type=9, label=1,
+      has_default_value=False, default_value=_b("").decode('utf-8'),
+      message_type=None, enum_type=None, containing_type=None,
+      is_extension=False, extension_scope=None,
+      options=None),
+    _descriptor.FieldDescriptor(
+      name='revision', full_name='Manifest.GitCheckout.revision', index=2,
+      number=3, type=12, cpp_type=9, label=1,
+      has_default_value=False, default_value=_b(""),
+      message_type=None, enum_type=None, containing_type=None,
+      is_extension=False, extension_scope=None,
+      options=None),
+    _descriptor.FieldDescriptor(
+      name='tracking_ref', full_name='Manifest.GitCheckout.tracking_ref', index=3,
+      number=4, type=9, cpp_type=9, label=1,
+      has_default_value=False, default_value=_b("").decode('utf-8'),
+      message_type=None, enum_type=None, containing_type=None,
+      is_extension=False, extension_scope=None,
+      options=None),
+  ],
+  extensions=[
+  ],
+  nested_types=[],
+  enum_types=[
+  ],
+  options=None,
+  is_extendable=False,
+  syntax='proto3',
+  extension_ranges=[],
+  oneofs=[
+  ],
+  serialized_start=104,
+  serialized_end=194,
+)
+
+_MANIFEST_CIPDPACKAGE = _descriptor.Descriptor(
+  name='CIPDPackage',
+  full_name='Manifest.CIPDPackage',
+  filename=None,
+  file=DESCRIPTOR,
+  containing_type=None,
+  fields=[
+    _descriptor.FieldDescriptor(
+      name='cipd_server_url', full_name='Manifest.CIPDPackage.cipd_server_url', index=0,
+      number=1, type=9, cpp_type=9, label=1,
+      has_default_value=False, default_value=_b("").decode('utf-8'),
+      message_type=None, enum_type=None, containing_type=None,
+      is_extension=False, extension_scope=None,
+      options=None),
+    _descriptor.FieldDescriptor(
+      name='cipd_package_name', full_name='Manifest.CIPDPackage.cipd_package_name', index=1,
+      number=2, type=9, cpp_type=9, label=1,
+      has_default_value=False, default_value=_b("").decode('utf-8'),
+      message_type=None, enum_type=None, containing_type=None,
+      is_extension=False, extension_scope=None,
+      options=None),
+    _descriptor.FieldDescriptor(
+      name='cipd_package_pattern', full_name='Manifest.CIPDPackage.cipd_package_pattern', index=2,
+      number=3, type=9, cpp_type=9, label=1,
+      has_default_value=False, default_value=_b("").decode('utf-8'),
+      message_type=None, enum_type=None, containing_type=None,
+      is_extension=False, extension_scope=None,
+      options=None),
+    _descriptor.FieldDescriptor(
+      name='cipd_instance_id', full_name='Manifest.CIPDPackage.cipd_instance_id', index=3,
+      number=4, type=12, cpp_type=9, label=1,
+      has_default_value=False, default_value=_b(""),
+      message_type=None, enum_type=None, containing_type=None,
+      is_extension=False, extension_scope=None,
+      options=None),
+    _descriptor.FieldDescriptor(
+      name='cipd_version', full_name='Manifest.CIPDPackage.cipd_version', index=4,
+      number=5, type=9, cpp_type=9, label=1,
+      has_default_value=False, default_value=_b("").decode('utf-8'),
+      message_type=None, enum_type=None, containing_type=None,
+      is_extension=False, extension_scope=None,
+      options=None),
+  ],
+  extensions=[
+  ],
+  nested_types=[],
+  enum_types=[
+  ],
+  options=None,
+  is_extendable=False,
+  syntax='proto3',
+  extension_ranges=[],
+  oneofs=[
+  ],
+  serialized_start=197,
+  serialized_end=340,
+)
+
+_MANIFEST_ISOLATED = _descriptor.Descriptor(
+  name='Isolated',
+  full_name='Manifest.Isolated',
+  filename=None,
+  file=DESCRIPTOR,
+  containing_type=None,
+  fields=[
+    _descriptor.FieldDescriptor(
+      name='isolated_server_url', full_name='Manifest.Isolated.isolated_server_url', index=0,
+      number=1, type=9, cpp_type=9, label=1,
+      has_default_value=False, default_value=_b("").decode('utf-8'),
+      message_type=None, enum_type=None, containing_type=None,
+      is_extension=False, extension_scope=None,
+      options=None),
+    _descriptor.FieldDescriptor(
+      name='namespace', full_name='Manifest.Isolated.namespace', index=1,
+      number=2, type=9, cpp_type=9, label=1,
+      has_default_value=False, default_value=_b("").decode('utf-8'),
+      message_type=None, enum_type=None, containing_type=None,
+      is_extension=False, extension_scope=None,
+      options=None),
+    _descriptor.FieldDescriptor(
+      name='hash', full_name='Manifest.Isolated.hash', index=2,
+      number=3, type=12, cpp_type=9, label=1,
+      has_default_value=False, default_value=_b(""),
+      message_type=None, enum_type=None, containing_type=None,
+      is_extension=False, extension_scope=None,
+      options=None),
+  ],
+  extensions=[
+  ],
+  nested_types=[],
+  enum_types=[
+  ],
+  options=None,
+  is_extendable=False,
+  syntax='proto3',
+  extension_ranges=[],
+  oneofs=[
+  ],
+  serialized_start=342,
+  serialized_end=414,
+)
+
+_MANIFEST_DIRECTORY = _descriptor.Descriptor(
+  name='Directory',
+  full_name='Manifest.Directory',
+  filename=None,
+  file=DESCRIPTOR,
+  containing_type=None,
+  fields=[
+    _descriptor.FieldDescriptor(
+      name='git_checkout', full_name='Manifest.Directory.git_checkout', index=0,
+      number=1, type=11, cpp_type=10, label=1,
+      has_default_value=False, default_value=None,
+      message_type=None, enum_type=None, containing_type=None,
+      is_extension=False, extension_scope=None,
+      options=None),
+    _descriptor.FieldDescriptor(
+      name='cipd_package', full_name='Manifest.Directory.cipd_package', index=1,
+      number=2, type=11, cpp_type=10, label=3,
+      has_default_value=False, default_value=[],
+      message_type=None, enum_type=None, containing_type=None,
+      is_extension=False, extension_scope=None,
+      options=None),
+    _descriptor.FieldDescriptor(
+      name='isolated', full_name='Manifest.Directory.isolated', index=2,
+      number=3, type=11, cpp_type=10, label=3,
+      has_default_value=False, default_value=[],
+      message_type=None, enum_type=None, containing_type=None,
+      is_extension=False, extension_scope=None,
+      options=None),
+  ],
+  extensions=[
+  ],
+  nested_types=[],
+  enum_types=[
+  ],
+  options=None,
+  is_extendable=False,
+  syntax='proto3',
+  extension_ranges=[],
+  oneofs=[
+  ],
+  serialized_start=417,
+  serialized_end=556,
+)
+
+_MANIFEST_DIRECTORIESENTRY = _descriptor.Descriptor(
+  name='DirectoriesEntry',
+  full_name='Manifest.DirectoriesEntry',
+  filename=None,
+  file=DESCRIPTOR,
+  containing_type=None,
+  fields=[
+    _descriptor.FieldDescriptor(
+      name='key', full_name='Manifest.DirectoriesEntry.key', index=0,
+      number=1, type=9, cpp_type=9, label=1,
+      has_default_value=False, default_value=_b("").decode('utf-8'),
+      message_type=None, enum_type=None, containing_type=None,
+      is_extension=False, extension_scope=None,
+      options=None),
+    _descriptor.FieldDescriptor(
+      name='value', full_name='Manifest.DirectoriesEntry.value', index=1,
+      number=2, type=11, cpp_type=10, label=1,
+      has_default_value=False, default_value=None,
+      message_type=None, enum_type=None, containing_type=None,
+      is_extension=False, extension_scope=None,
+      options=None),
+  ],
+  extensions=[
+  ],
+  nested_types=[],
+  enum_types=[
+  ],
+  options=_descriptor._ParseOptions(descriptor_pb2.MessageOptions(), _b('8\001')),
+  is_extendable=False,
+  syntax='proto3',
+  extension_ranges=[],
+  oneofs=[
+  ],
+  serialized_start=558,
+  serialized_end=629,
+)
+
+_MANIFEST = _descriptor.Descriptor(
+  name='Manifest',
+  full_name='Manifest',
+  filename=None,
+  file=DESCRIPTOR,
+  containing_type=None,
+  fields=[
+    _descriptor.FieldDescriptor(
+      name='version', full_name='Manifest.version', index=0,
+      number=1, type=5, cpp_type=1, label=1,
+      has_default_value=False, default_value=0,
+      message_type=None, enum_type=None, containing_type=None,
+      is_extension=False, extension_scope=None,
+      options=None),
+    _descriptor.FieldDescriptor(
+      name='directories', full_name='Manifest.directories', index=1,
+      number=2, type=11, cpp_type=10, label=3,
+      has_default_value=False, default_value=[],
+      message_type=None, enum_type=None, containing_type=None,
+      is_extension=False, extension_scope=None,
+      options=None),
+  ],
+  extensions=[
+  ],
+  nested_types=[_MANIFEST_GITCHECKOUT, _MANIFEST_CIPDPACKAGE, _MANIFEST_ISOLATED, _MANIFEST_DIRECTORY, _MANIFEST_DIRECTORIESENTRY, ],
+  enum_types=[
+  ],
+  options=None,
+  is_extendable=False,
+  syntax='proto3',
+  extension_ranges=[],
+  oneofs=[
+  ],
+  serialized_start=26,
+  serialized_end=629,
+)
+
+_MANIFEST_GITCHECKOUT.containing_type = _MANIFEST
+_MANIFEST_CIPDPACKAGE.containing_type = _MANIFEST
+_MANIFEST_ISOLATED.containing_type = _MANIFEST
+_MANIFEST_DIRECTORY.fields_by_name['git_checkout'].message_type = _MANIFEST_GITCHECKOUT
+_MANIFEST_DIRECTORY.fields_by_name['cipd_package'].message_type = _MANIFEST_CIPDPACKAGE
+_MANIFEST_DIRECTORY.fields_by_name['isolated'].message_type = _MANIFEST_ISOLATED
+_MANIFEST_DIRECTORY.containing_type = _MANIFEST
+_MANIFEST_DIRECTORIESENTRY.fields_by_name['value'].message_type = _MANIFEST_DIRECTORY
+_MANIFEST_DIRECTORIESENTRY.containing_type = _MANIFEST
+_MANIFEST.fields_by_name['directories'].message_type = _MANIFEST_DIRECTORIESENTRY
+DESCRIPTOR.message_types_by_name['Manifest'] = _MANIFEST
+
+Manifest = _reflection.GeneratedProtocolMessageType('Manifest', (_message.Message,), dict(
+
+  GitCheckout = _reflection.GeneratedProtocolMessageType('GitCheckout', (_message.Message,), dict(
+    DESCRIPTOR = _MANIFEST_GITCHECKOUT,
+    __module__ = 'source_manifest_pb2'
+    # @@protoc_insertion_point(class_scope:Manifest.GitCheckout)
+    ))
+  ,
+
+  CIPDPackage = _reflection.GeneratedProtocolMessageType('CIPDPackage', (_message.Message,), dict(
+    DESCRIPTOR = _MANIFEST_CIPDPACKAGE,
+    __module__ = 'source_manifest_pb2'
+    # @@protoc_insertion_point(class_scope:Manifest.CIPDPackage)
+    ))
+  ,
+
+  Isolated = _reflection.GeneratedProtocolMessageType('Isolated', (_message.Message,), dict(
+    DESCRIPTOR = _MANIFEST_ISOLATED,
+    __module__ = 'source_manifest_pb2'
+    # @@protoc_insertion_point(class_scope:Manifest.Isolated)
+    ))
+  ,
+
+  Directory = _reflection.GeneratedProtocolMessageType('Directory', (_message.Message,), dict(
+    DESCRIPTOR = _MANIFEST_DIRECTORY,
+    __module__ = 'source_manifest_pb2'
+    # @@protoc_insertion_point(class_scope:Manifest.Directory)
+    ))
+  ,
+
+  DirectoriesEntry = _reflection.GeneratedProtocolMessageType('DirectoriesEntry', (_message.Message,), dict(
+    DESCRIPTOR = _MANIFEST_DIRECTORIESENTRY,
+    __module__ = 'source_manifest_pb2'
+    # @@protoc_insertion_point(class_scope:Manifest.DirectoriesEntry)
+    ))
+  ,
+  DESCRIPTOR = _MANIFEST,
+  __module__ = 'source_manifest_pb2'
+  # @@protoc_insertion_point(class_scope:Manifest)
+  ))
+_sym_db.RegisterMessage(Manifest)
+_sym_db.RegisterMessage(Manifest.GitCheckout)
+_sym_db.RegisterMessage(Manifest.CIPDPackage)
+_sym_db.RegisterMessage(Manifest.Isolated)
+_sym_db.RegisterMessage(Manifest.Directory)
+_sym_db.RegisterMessage(Manifest.DirectoriesEntry)
+
+
+_MANIFEST_DIRECTORIESENTRY.has_options = True
+_MANIFEST_DIRECTORIESENTRY._options = _descriptor._ParseOptions(descriptor_pb2.MessageOptions(), _b('8\001'))
+# @@protoc_insertion_point(module_scope)