)]}'
{
  "commit": "7c116cf2a16352306a37b41453be1bb27293202b",
  "tree": "630e1ad4139cc5e0e25b3589b808e51c7789aeae",
  "parents": [
    "721e4aca6aa11124a192ee0dd1abb84e4478589c"
  ],
  "author": {
    "name": "Sergey Shevchenko",
    "email": "sergeygs@google.com",
    "time": "Thu Mar 21 15:44:13 2024"
  },
  "committer": {
    "name": "Copybara-Service",
    "email": "copybara-worker@google.com",
    "time": "Thu Mar 21 15:44:55 2024"
  },
  "message": "#Centipede More RAM/CPU-efficient `ReadShard()`, p.2 (measured \u003e 20x RSS reduction)\n\nBEFORE:\n\n- Read all the features into a hash map in RAM (hash -\u003e feature vec).\n- Read the inputs. For every input, find a matching feature in the hash map and report the pair to the callback.\n\nAFTER:\n\n- Read all the inputs top a hash map in RAM (hash -\u003e input).\n- Read the features. For every feature, find a matching input in the hash map, report the pair to the callback, remove the input from the hash map.\n- Report the remaining featureless inputs in the hash map to the callback.\n- Additional benefit: the new code automatically deduplicates inputs (the old code would report duplicates repeatedly).\n\nTESTED:\n\nFor a sample shard taken from a real run:\n\n- The RSS peak decreased from 23 GB to less than 1 GB.\n- The VSS peak decreased from 25 GB to 3.4 GB.\n- The profiles for both RSS and VSS changed from very peaky to virtually flat: after a quick ramp up to the mentioned numbers, they stayed at almost the same level (actually, slightly decreasing). Prior to this change, both were steadily increasing to the peak during the first 90% of the function, then RSS decreased rapidly to the baseline.\n- The wall clock time decreased from 810 sec to 720 sec.\n- The system time decreased from 64 sec to 8 sec.\n\nPiperOrigin-RevId: 617855333\n",
  "tree_diff": [
    {
      "type": "modify",
      "old_id": "81a80a6b78cde473b5ad848966603bc473a8a0e2",
      "old_mode": 33188,
      "old_path": "centipede/BUILD",
      "new_id": "d280b26f492100db8ad42201c7bfa331c7763794",
      "new_mode": 33188,
      "new_path": "centipede/BUILD"
    },
    {
      "type": "modify",
      "old_id": "2f99a2c019d0a5ffeae6bd7bee05d02a568b0be9",
      "old_mode": 33188,
      "old_path": "centipede/corpus_io.cc",
      "new_id": "9cf3ee85c80b8e446a3bb7a42615022cb9f7c07b",
      "new_mode": 33188,
      "new_path": "centipede/corpus_io.cc"
    }
  ]
}