The LogDog Archivist is tasked by the Coordinator, and has the job of collecting log stream data from Intermediate Storage (where it was deposited by the Collector) and loading it into Archival Storage. It does this by scanning through Intermediate Storage for consecutive log entries and constructing archive files:
LogStreamDescriptorprotobuf followed by every
LogEntryprotobuf in the stream.
LogStreamDescriptorRecordIO entry followed by a
These files are written into Archival Storage by the Archivist during archival. After archival is complete, the Archivist notifies the Coordinator and the log stream's state is updated.
Archivist microservices are designed to operate cooperatively as part of a scalable cluster. Deploying additional Archivist instances will linearly increase the archival throughput.
Archival is initially written to a staging storage location. After the archival successfully completes, the staged files are moved to permanent location using an inexpensive rename operation.
It is possible for log streams to be missing data at the time of archival.