Security Shepherding is a rotational assignment for security bug triage (Primary Shepherd) and managing the flow of incoming inquiries and progressing security issues (Secondary Shepherd). The Shepherding rota pool is made up of people actively working on security in Chrome.
All Security Shepherds are Googlers; therefore, some links on this page may not be externally accessible or even further restricted to just Chrome Security Googlers.
There is a Primary and Secondary Shepherd scheduled each rotation, with two rotations each week, one Tuesday - Thursday and Friday - Monday.
Security Shepherding is not an on-call rotation. There’s no pager duty, nor are you expected to perform Shepherding duties outside of your usual working hours, such as overnight or on holidays, weekends, or other off time.
Shepherds are only responsible for triage of security bugs during your shift. You are not responsible for bug triage or updating partially triaged bugs past your shift, unless you have specifically taken ownership of an issue, such as due to a complicated or OS-specific reproduction, and arranged that with the oncoming shepherd. All shepherds should use the handoff doc to communicate items for handover; however, the oncoming primary shepherd should operate on the premise all new or under-triaged issues are your responsibility. Please do not leave any unaddressed red cells in the dashboard at the end of your shift.
(“I’m Primary Shepherd, what do I do???”)
The Primary Security Shepherd is the front line of security bug triage during their shift. The goal is to triage incoming security issues accurately, thoroughly, and quickly (as quickly as realistically possible).
Your PRIMARY DIRECTIVE as PRIMARY SHEPHERD is to tackle all the red cells on the security bug dashboard.
For every new incoming security bug:
For every new incoming security bug that is S2 or more severe:
All of the above should be completed as soon as possible during your shift, and at least, by the shift-handoff.
One or more of the above actions may be necessary to complete the triage of an under-triaged bug, i.e. covering any of the open red cells in the dashboard that were not completed from ClusterFuzz auto-triage or previous work on the bug.
All this is hard, so please remember to ask for help. Yell if you must!
(“I’m Secondary Shepherd, what do I do???”)
Here are some of the important references and resources you need or may need during your shepherding shift:
Monitor the Chrome open security bugs dashboard. Tackle all the empty red cells. New bugs populate at the top of the sheet and will need full triage. Partially triaged bugs, such as those triaged by ClusterFuzz or ones pending updates from a prior shift, may be lower in the sheet. Please check the sheet for any red cells and do your best to get any bugs to a fully triaged state.
We aim to have every bug triaged and assigned within two business days, preferably one. This does not include weekends, but please ensure you leave a clear queue before the weekend or a holiday.
There should be one complete, self-contained report, per root cause. To ensure this is the case when assigning security bugs to engineering teams, you may need to take some specific actions here:
blocked on each. The parent bug should be set to the severity of the full chain. Each child bug may have a lower severity.We expect engineering teams to address security bugs promptly. In order to do that, our goal is to pass them actionable reports with little ambiguity.
For each bug, please take the appropriate action, either:
WontFix it as invalid (Many recurring types of invalid reports are covered by the Security FAQ, such as those related physically local attacks or inputting JavaScript in the URL bar or running Javascript directly in DevTools not being an indication of an XSS vulnerability. Mark as WontFix and update the ‘Issue access level’ to Default access so the issue is publicly visible.
Mark as duplicate – we want exactly one bug per root cause problem. Please check for duplicate issues of a given issue from that or other reporters / sources (such as ClusterFuzz).
Mark as Duplicate button at the upper right of the report pane. This will provide a pop-up to input the bug number of the canonical report that you are merging this report into as a duplicate of.Mark as Duplicate is the best practice for merging issues as duplicates.Convert functional bugs to Type=Bug For example, many reports are for crashes of a functional nature, rather than an exploitable security condition, such as most null pointer dereferences. Convert such reports from Type=Vulnerability to Type=Bug. Do NOT remove security@chromium.org from collaborators first (as this will result in orphaning the bug), but update the ‘Issue Access Level’ to the appropriate visibility. You may consider adding other visibility restrictions, such as Limited visibility + Googlers and add edit-bug-access@chromium.org to CC (this is similar to ‘Restrict-View-EditAccess’ in the legacy issue tracker) if the immediate disclosure could result in potential abuse (e.g. denial of service issue).
Convert to a privacy bug - privacy issues (such as issues with incognito) are not considered security bugs, but functional privacy issues. Convert to Type=Bug and add the Privacy component. Add yourself and any other security team members who may potentially need access to the cc: line. Update the ‘Issue access level’ to Limited visibility + Googlers and deselect / remove security@chromium.org from the ‘add collaborator groups’.
Add the Needs-Feedback hotlist (hotlistID: 5433459) and set a Next Action date of 24-48 hours for more information if there is no response, close the issue as WontFix.
Determine issue to be theoretical - and follow the process for such issues – theoretical issues are ones that appear to be potentially real bugs, but the report is lacking evidence of exploitability or reachability. These cases can be shared with engineering teams with a very clear message conveying the speculative nature of the issue. These reports should generally not be prioritized as a Pri-1 as they do not warrant disruption to the engineering teams to investigate and prioritize without more or new information to demonstrate conditions of exploitability.
None of these apply? Great – this means the bug may be valid and actionable! It can take multiple discussions with a reporter to understand a bug. Try really hard to reach a conclusion by the end of your shift. If this isn’t possible, please discuss outstanding cases with the next shepherd and don’t let bugs fall through the cracks. You are responsible for any bug reported or in an un-triaged state during your shift.
The best way to determine the validity of a security bug is to reproduce it. It’s helpful to remember that reporters invested time and energy in their bug reports:
If you have to close it, please give an explanation as to why.
Reproducing the bug isn’t always required, but often it’s needed and the only way to:
These things must be done correctly, so as Security Shepherd, you’ll spend a lot of time reproducing bugs. Here are some tips in doing so:
Use the Security Severity Guidelines.
If you can, reproduce it using ClusterFuzz, as the severity is usually set automatically.
For V8 issues, you can tentatively set the issue as High (S1) severity – see Assign,below.
Please adjust severity as your understanding of the bug evolves - but please always add a comment explaining the change. Higher severity bugs involve significant disruption for multiple teams; lower severity issues may not be fixed and a fix released to users as quickly as may be warranted. That’s why it’s important to get the severity as correct as possible.
We do not release severe security regressions, so we need to know the earliest impacted Chrome release branch.
First, if an issue doesn’t impact Chrome users by default (such as be being behind a disabled feature or a command line flag), add the hotlist Security_Impact-None; otherwise, set a Found In milestone in the Found In field as follows:
Check ChromiumDash for the earliest relevant milestone number (Extended Stable or Stable – sometimes they are the same).
Found In field to, to the appropriate milestone number.Found In field to the oldest impacted branch you find.There is no general reason to test versions older than the current Extended Stable milestone. If you can reproduce using ClusterFuzz the Found In field can often be set automatically if ClusterFuzz can identify the culprit CL.
Otherwise, you may need to reproduce the bug manually to determine the impacted branches.
If you have a bisection or other convincing evidence, that’s sufficient. You can manually check which milestone has a given commit in ChromiumDash commits check.
Please do not base Found In- on the Chrome version number provided in the original report. This is often based on the version number the individual is using when discovering this issue or is automatically set in the report by the tracker’s report wizard and is not correct in terms of coverage of all active release channels.
For V8 bugs, you can set Found In as the current extended stable milestone unless you have reproduced the issue or an accurate bisection has been provided. (See Assign, below.)
Set the OS field as best you can based on these guidelines. You do not need to reproduce the bug on each platform, but it really helps if you set this field roughly right to ensure the bug has the attention of the different desktop and mobile release teams.
Some issues may be specific to a particular platform, if you need to reproduce a bug that is platform specific and you do not have access to a device with that OS, please ask for help, there is likely someone on the team that does and can help you.
ChromeOS is in the Google issue tracker. VRP reports for ChromeOS should be directly reported to ChromeOS. Please request the reporter submit reports directly to ChromeOS in the future. For VRP and other human-submitted security bug reports specific to ChromeOS, please move the report corresponding component (componentid:1335705) in the Google issue tracker. Since this bug is being moved between trackers you will need to use your google.com account to move the bug into that tracker component.
Some machine-discovered (Clusterfuzz, Crash AutoBugFiler, GWP-ASAN) may be specific to ChromeOS. If this is determined to be the case after investigation (please remember some GWP-ASAN or crash bug auto-filer bugs may have come from a ChromeOS crash, but the issue may not be specific to ChromeOS), move the bug to the appropriate ChromeOS component (componentid:1214738) in the Google issue tracker for these reports. Again, you will need to use your google.com account to move this bug into that component.
Security bugs are not automatically visible, so you must add people to get them fixed. For each bug, set:
git blame or look for similar past bugs in the tracker.It’s okay if you cannot determine or know the exact right assignee, but please pass it along to / include someone who can direct it more precisely.
Some types of bugs have specific assignment needs:
Found In of the current Extended Stable.Found In are provisional. Note that V8 CHECK failure crashes can have security implications, so don't triage it yourself.Security_Impact-None hotlist (hotlistID:5433277).V8 Sandbox hotlist (hotlistID:4802478).Component to: Chromium > BoringSSL.third_party/boringssl/OWNERS; Add owners to cc: on the bug to ensure visibility.Security_Impact-None hotlist; owner will update if this issue does impact Chrome.//third_party or elsewhere://third_party package owner to file it.//third_party package.Status_ExternalDependency (hotlistID: 5438152)As you work through the queue each day, please manage your time and ensure you have addressed all red rows and cells in the sheet to the best of your ability. Make sure there are no red cells at the top of your sheet before the end of your shift. It’s not okay to leave a backlog for the next oncoming security shepherd.
Please fill out the Shepherding Handoff Log to communicate issues from your shift that may be helpful to the oncoming shift.
Security bug triage is hard. We receive around 75 bug reports per week on average. If you are ever stuck or in doubt, please ask for help from the Chrome Security Shepherds chat or the Chrome Security Chat. During some shifts, there are just too many incoming bugs. It’s okay to ask for help, please do!
You may also like the classic HOWTO: Be a Security Shepherd deck
Because shepherding is fun. You like fun. Don't you? Fun is great.
Review open security bug reports and check that progress is occurring. This does not apply to the new bug reports as these are handled by the primary shepherd. The rule of thumb is if there is any red cell on the dashboard, it needs your attention: that especially includes the last updated column. Our severity guidelines contain the expected duration for shipping fixes, but it’s important to remember that to get a fix to all users in 60 days or so, this may require us to land a fix in a week or two.
Suggestions for cultivating progress on security bugs:
Security_Impact-None bugs in unlaunched features, where the response has been that there are no plans to launch that feature? Perhaps inquire as to if that code can be removed rather than keeping vulnerable code production code base. (Removing code that is not being used is a win!)You can’t possibly usher all bugs toward meaningful progress during your shift. As a general rule, expect to spend a solid two hours each day ushering bugs toward progress during your shift. Use the last updated column to avoid duplicating the work of the previous secondary.
Ensure that all incoming inquiries to the security@chromium.org, security-dev@chromium.org, and chrome-security@google.com lists get a reply (by someone; not necessarily you). See go/chrome-security-emails for a dashboard.
What do I do here?
You are not responsible for handling merges or approving a fix for backmerge. If the issue is resolved and there is a landed CL, please ensure the bug is closed as Fixed. Please also make sure the bug has a severity and FoundIn set. This will allow the bot (Sheriffbot) to add the appropriately update the Merge custom field with the appropriate request-MMM or review-MMM labels, where MMM = the milestones for backmerge consideration (based on rules driven by severity (and security_impact, derived from Found In). See security merge triage for more information.
That issue will be visible to the security merge review queue. There are designated members of the security team who have the hefty responsibility of reviewing security issues for backmerge. Merge approvals will be handled by them after at least the fix has had sufficient bake time on Canary.
(e.g. how and when does a VRP bug get to the Chrome VRP Panel?) See Life of a Security Issue.
Chrome VRP policies and rewards page and Chrome VRP News and FAQs. You can also reach out directly to the Chrome VRP TL or ask questions in the Chrome Security Shepherds chat, all VRP Panel members are also members of that chat.
For cases of PII, simply delete the attachment or comment that contains PII within the issue tracker. If PII is contained in the text of the original description of the report, simply choose the Edit description option and remove any PII.
For cases in which we are just delaying public disclosure (such as when a security issue impacts other products or vendors), please add the SecurityEmbargo hotlist (hotlistID: 5432549) and set a date in the Next Action field so that disclosure can be re-evaluated at that time.
Many researchers report security issues under a pseudonym and from a specific email address pertaining to that pseudonym. Please do not refer to the researcher by the email username directly in any comments of the report. When reports are publicly disclosed, that becomes visible to all and we have to delete those comments to protect that information. To direct a comment at an external security researcher, please use “OP”, “reporter”, or "researcher”.
You may come across some reports in the security bug triage queue with a red banner, “The issue has been deleted. Reason: ABUSE,” this is generally due to the overactive spam filtering in the issue tracker. Just click Undelete in the right side of the banner, and triage the report as you normally would.
Sometimes you’ll need to handle a security emergency, such as a critical severity bug or bug known or under active exploitation in the wild. In such cases:
That‘s a lot of stuff! You have this resource and your peers to lean on for questions and expertise. Hopefully this doc helps. You’re gonna do great!