310fe88994249a5a02e20d1211b8fc067e34aa78 - external/github.com/python/cpython

commit: 310fe88994249a5a02e20d1211b8fc067e34aa78
[log]
author: Serhiy Storchaka <storchaka@gmail.com>
Thu May 07 19:06:57 2026
committer: GitHub <noreply@github.com>
Thu May 07 19:06:57 2026
tree: b63c3500395b737850b61091c2a5b1217ef95200
parent: fbba343622c9e4a38c8ef0f0b0e311164394d76a [diff]

gh-79638: Treat an unreachable robots.txt as "disallow all" (GH-138555)

Disallow all access in urllib.robotparser if the robots.txt file is
unreachable due to server or network errors.

Lib/test/test_robotparser.py[diff]
Lib/urllib/robotparser.py[diff]
Misc/NEWS.d/next/Library/2025-09-05-20-50-35.gh-issue-79638.Y-JfaH.rst[Added - diff]

3 files changed