Add support for Cache-Control: only-if-cached and corresponding options for request() and send()
diff --git a/HISTORY.md b/HISTORY.md
index d496e03..8e5b086 100644
--- a/HISTORY.md
+++ b/HISTORY.md
@@ -4,10 +4,12 @@
[See all issues and PRs for 0.10](https://github.com/reclosedev/requests-cache/milestone/5?closed=1)
**Expiration & Headers:**
-* Add `refresh` option to `CachedSession.request()` and `send()` to make (and cache) an new request regardless of existing cache contents
-* Add `revalidate` option to `CachedSession.request()` and `send()` to send conditional request (if possible) before using a cached response
+* Add support for `Cache-Control: only-if-cached`
* Revalidate for `Cache-Control: no-cache` request or response header
* Revalidate for `Cache-Control: max-age=0, must-revalidate` response headers
+* Add `only_if_cached` option to `CachedSession.request()` and `send()` to return only cached results without sending real requests
+* Add `refresh` option to `CachedSession.request()` and `send()` to make (and cache) an new request regardless of existing cache contents
+* Add `revalidate` option to `CachedSession.request()` and `send()` to send conditional request (if possible) before using a cached response
### 0.9.3 (2022-02-22)
* Fix handling BSON serializer differences between pymongo's `bson` and standalone `bson` codec.
diff --git a/docs/user_guide/expiration.md b/docs/user_guide/expiration.md
index ef0e6b3..5c81ef0 100644
--- a/docs/user_guide/expiration.md
+++ b/docs/user_guide/expiration.md
@@ -117,3 +117,57 @@
```python
>>> session.remove_expired_responses(expire_after=timedelta(days=30))
```
+
+## Request Options
+In addition to the base arguments for {py:func}`requests.request`, requests-cache adds some extra
+cache-related arguments. These apply to {py:meth}`.CachedSession.request`,
+{py:meth}`.CachedSession.send`, and all HTTP method-specific functions (`get()`, `post()`, etc.).
+
+### Per-Request Expiration
+The `expire_after` argument can be used to override the session's expiration for a single request.
+```python
+>>> session = CachedSession(expire_after=300)
+>>> # This request will be cached for 60 seconds, not 300
+>>> session.get('http://httpbin.org/get', expire_after=60)
+```
+
+### Manual Refresh
+If you want to manually refresh a response before it expires, you can use the `refresh` argument.
+This will always send a new request, and ignore and overwrite any previously cached response. The
+response will be saved with a new expiration time, according to the normal expiration rules described above.
+```python
+>>> response = session.get('http://httpbin.org/get')
+>>> response = session.get('http://httpbin.org/get', refresh=True)
+>>> assert response.from_cache is False
+```
+
+A related argument is `revalidate`, which is basically a "soft refresh." It will send a quick
+{ref}`conditional request <conditional-requests>` to the server, and use the cached response if the
+remote content has not changed. If the cached response does not contain validation headers, this
+option will have no effect.
+
+### Cache-Only Requests
+If you want to only use cached responses without making any real requests, you can use the
+`only_if_cached` option. This essentially uses your cache in "offline mode". If a response isn't
+cached or is expired, you will get a `504 Not Cached` response instead.
+```python
+>>> session = CachedSession()
+>>> session.cache.clear()
+>>> response = session.get('http://httpbin.org/get', only_if_cached=True)
+>>> print(response.status_code)
+504
+>>> response.raise_for_status()
+HTTPError: 504 Server Error: Not Cached for url: http://httpbin.org/get
+```
+
+You can also combine this with `stale_if_error` to return cached responses even if they are expired.
+```python
+>>> session = CachedSession(expire_after=1, stale_if_error=True)
+>>> session.get('http://httpbin.org/get')
+>>> time.sleep(1)
+
+>>> # The response will be cached but expired by this point
+>>> response = session.get('http://httpbin.org/get', only_if_cached=True)
+>>> print(response.status_code)
+200
+```
diff --git a/docs/user_guide/headers.md b/docs/user_guide/headers.md
index ecc95c1..1dccde6 100644
--- a/docs/user_guide/headers.md
+++ b/docs/user_guide/headers.md
@@ -11,6 +11,7 @@
would like to see, please create an issue to request it.
```
+(conditional-requests)=
## Conditional Requests
[Conditional requests](https://developer.mozilla.org/en-US/docs/Web/HTTP/Conditional_requests) are
automatically sent for any servers that support them. Once a cached response expires, it will only
@@ -49,6 +50,8 @@
- `Cache-Control: max-age`: Used as the expiration time in seconds
- `Cache-Control: no-cache`: Revalidate with the server before using a cached response
- `Cache-Control: no-store`: Skip reading from and writing to the cache
+- `Cache-Control: only-if-cached`: Only return results from the cache. If not cached, return a 504
+ response instead of sending a new request. Note that this may return a stale response.
- `If-None-Match`: Automatically added for revalidation, if an `ETag` is available
- `If-Modified-Since`: Automatically added for revalidation, if `Last-Modified` is available
diff --git a/requests_cache/cache_control.py b/requests_cache/cache_control.py
index 10e9ed1..278ed29 100644
--- a/requests_cache/cache_control.py
+++ b/requests_cache/cache_control.py
@@ -1,5 +1,6 @@
"""Internal utilities for determining cache expiration and other cache actions. This module defines
-the caching policy, and resulting actions are applied in the :py:mod:`requests_cache.session` module.
+the majority of the caching policy, and resulting actions are handled in
+:py:meth:`CachedSession.send`.
.. automodsumm:: requests_cache.cache_control
:classes-only:
@@ -65,6 +66,7 @@
cache_control: bool = field(default=False)
cache_key: str = field(default=None)
expire_after: ExpirationTime = field(default=None)
+ only_if_cached: bool = field(default=False)
request_directives: Dict[str, CacheDirective] = field(factory=dict)
revalidate: bool = field(default=False)
skip_read: bool = field(default=False)
@@ -80,6 +82,7 @@
session_expire_after: ExpirationTime = None,
urls_expire_after: ExpirationPatterns = None,
request_expire_after: ExpirationTime = None,
+ only_if_cached: bool = False,
refresh: bool = False,
revalidate: bool = False,
**kwargs,
@@ -107,17 +110,22 @@
)
# Check conditions for cache read and write based on args and request headers
- check_expiration = directives.get('max-age') if cache_control else expire_after
refresh_temp_header = request.headers.pop('requests-cache-refresh', False)
+ check_expiration = directives.get('max-age') if cache_control else expire_after
skip_write = check_expiration == DO_NOT_CACHE or 'no-store' in directives
+
+ # These behaviors may be set by either request headers or keyword arguments
+ only_if_cached = only_if_cached or 'only-if-cached' in directives
+ revalidate = revalidate or 'no-cache' in directives
skip_read = skip_write or refresh or bool(refresh_temp_header)
return cls(
cache_control=cache_control,
cache_key=cache_key,
expire_after=expire_after,
+ only_if_cached=only_if_cached,
request_directives=directives,
- revalidate=revalidate or 'no-cache' in directives,
+ revalidate=revalidate,
skip_read=skip_read,
skip_write=skip_write,
)
@@ -232,6 +240,17 @@
return kv_directives
+def get_504_response(request: PreparedRequest) -> Response:
+ from .models import CachedResponse
+
+ return CachedResponse(
+ url=request.url or '',
+ status_code=504,
+ reason='Not Cached',
+ request=request, # type: ignore
+ )
+
+
def get_url_expiration(
url: Optional[str], urls_expire_after: ExpirationPatterns = None
) -> ExpirationTime:
diff --git a/requests_cache/session.py b/requests_cache/session.py
index 6292bbb..f86307c 100644
--- a/requests_cache/session.py
+++ b/requests_cache/session.py
@@ -25,7 +25,13 @@
from ._utils import get_valid_kwargs
from .backends import BackendSpecifier, init_backend
-from .cache_control import CacheActions, ExpirationTime, append_directive, get_expiration_seconds
+from .cache_control import (
+ CacheActions,
+ ExpirationTime,
+ append_directive,
+ get_504_response,
+ get_expiration_seconds,
+)
from .models import AnyResponse, CachedResponse, set_response_defaults
__all__ = ['ALL_METHODS', 'CachedSession', 'CacheMixin']
@@ -78,8 +84,9 @@
method: str,
url: str,
*args,
- expire_after: ExpirationTime = None,
headers: MutableMapping[str, str] = None,
+ expire_after: ExpirationTime = None,
+ only_if_cached: bool = False,
refresh: bool = False,
revalidate: bool = False,
**kwargs,
@@ -95,6 +102,8 @@
expire_after: Expiration time to set only for this request; see details below.
Overrides ``CachedSession.expire_after``. Accepts all the same values as
``CachedSession.expire_after``. Use ``-1`` to disable expiration.
+ only_if_cached: Only return results from the cache. If not cached, return a 504 response
+ instead of sending a new request.
refresh: Always make a new request, and overwrite any previously cached response
revalidate: Revalidate with the server before using a cached response (e.g., a "soft refresh")
@@ -115,6 +124,8 @@
headers = headers or {}
if expire_after is not None:
headers = append_directive(headers, f'max-age={get_expiration_seconds(expire_after)}')
+ if only_if_cached:
+ headers = append_directive(headers, 'only-if-cached')
if revalidate:
headers = append_directive(headers, 'no-cache')
if refresh:
@@ -128,6 +139,7 @@
self,
request: PreparedRequest,
expire_after: ExpirationTime = None,
+ only_if_cached: bool = False,
refresh: bool = False,
revalidate: bool = False,
**kwargs,
@@ -137,6 +149,8 @@
Args:
expire_after: Expiration time to set only for this request
+ only_if_cached: Only return results from the cache. If not cached, return a 504 response
+ instead of sending a new request.
refresh: Always make a new request, and overwrite any previously cached response
revalidate: Revalidate with the server before using a cached response (e.g., a "soft refresh")
"""
@@ -149,6 +163,7 @@
session_expire_after=self.expire_after,
urls_expire_after=self.urls_expire_after,
cache_control=self.cache_control,
+ only_if_cached=only_if_cached,
refresh=refresh,
revalidate=revalidate,
**kwargs,
@@ -161,9 +176,13 @@
actions.update_from_cached_response(cached_response)
is_expired = getattr(cached_response, 'is_expired', False)
- # If the response is expired or missing, or the cache is disabled, then fetch a new response
- if cached_response is None or actions.revalidate:
+ # Handle missing and expired responses based on settings and headers
+ if (cached_response is None or is_expired) and actions.only_if_cached:
+ response: AnyResponse = get_504_response(request)
+ elif cached_response is None or actions.revalidate:
response = self._send_and_cache(request, actions, cached_response, **kwargs)
+ elif is_expired and self.stale_if_error and actions.only_if_cached:
+ response = cached_response
elif is_expired and self.stale_if_error:
response = self._resend_and_ignore(request, actions, cached_response, **kwargs)
elif is_expired:
diff --git a/tests/unit/test_session.py b/tests/unit/test_session.py
index 1a33779..b7246d1 100644
--- a/tests/unit/test_session.py
+++ b/tests/unit/test_session.py
@@ -10,7 +10,7 @@
import pytest
import requests
-from requests import Request, RequestException
+from requests import HTTPError, Request, RequestException
from requests.structures import CaseInsensitiveDict
from requests_cache import ALL_METHODS, CachedSession
@@ -28,32 +28,14 @@
MOCKED_URL_REDIRECT_TARGET,
)
-
-def test_init_unregistered_backend():
- with pytest.raises(ValueError):
- CachedSession(backend='nonexistent')
-
-
-def test_cache_path_expansion():
- session = CachedSession('~', backend='filesystem')
- assert session.cache.cache_dir == Path("~").expanduser()
-
-
-@patch.dict(BACKEND_CLASSES, {'mongo': get_placeholder_class()})
-def test_init_missing_backend_dependency():
- """Test that the correct error is thrown when a user does not have a dependency installed"""
- with pytest.raises(ImportError):
- CachedSession(backend='mongo')
+# Basic initialization
+# -----------------------------------------------------
class MyCache(BaseCache):
pass
-# Basic initialization
-# -----------------------------------------------------
-
-
def test_init_backend_instance():
backend = MyCache()
session = CachedSession(backend=backend)
@@ -80,6 +62,23 @@
assert session.cache.cache_name == 'test_cache'
+def test_init_unregistered_backend():
+ with pytest.raises(ValueError):
+ CachedSession(backend='nonexistent')
+
+
+def test_init_cache_path_expansion():
+ session = CachedSession('~', backend='filesystem')
+ assert session.cache.cache_dir == Path("~").expanduser()
+
+
+@patch.dict(BACKEND_CLASSES, {'mongo': get_placeholder_class()})
+def test_init_missing_backend_dependency():
+ """Test that the correct error is thrown when a user does not have a dependency installed"""
+ with pytest.raises(ImportError):
+ CachedSession(backend='mongo')
+
+
def test_repr(mock_session):
"""Test session and cache string representations"""
mock_session.expire_after = 11
@@ -534,10 +533,10 @@
mock_session.mock_adapter.register_uri(
'GET', unexpired_url, status_code=200, text='mock response'
)
- mock_session.expire_after = timedelta(seconds=0.2)
+ mock_session.expire_after = 1
mock_session.get(MOCKED_URL)
mock_session.get(MOCKED_URL_JSON)
- time.sleep(0.2)
+ time.sleep(1)
mock_session.get(unexpired_url)
# At this point we should have 1 unexpired response and 2 expired responses
@@ -548,7 +547,7 @@
assert cached_response.url == unexpired_url
# Now the last response should be expired as well
- time.sleep(0.2)
+ time.sleep(1)
mock_session.remove_expired_responses()
assert len(mock_session.cache.responses) == 0
@@ -657,6 +656,51 @@
assert response.from_cache is False
+def test_request_only_if_cached__cached(mock_session):
+ """only_if_cached has no effect if the response is already cached"""
+ mock_session.get(MOCKED_URL)
+ response = mock_session.get(MOCKED_URL, only_if_cached=True)
+ assert response.from_cache is True
+ assert response.is_expired is False
+
+
+def test_request_only_if_cached__uncached(mock_session):
+ """only_if_cached should return a 504 response if it is not already cached"""
+ response = mock_session.get(MOCKED_URL, only_if_cached=True)
+ assert response.status_code == 504
+ with pytest.raises(HTTPError):
+ response.raise_for_status()
+
+
+def test_request_only_if_cached__expired(mock_session):
+ """By default, only_if_cached will not return an expired response"""
+ mock_session.get(MOCKED_URL, expire_after=1)
+ time.sleep(1)
+
+ response = mock_session.get(MOCKED_URL, only_if_cached=True)
+ assert response.status_code == 504
+
+
+def test_request_only_if_cached__stale_if_error__expired(mock_session):
+ """only_if_cached *will* return an expired response if stale_if_error is also set"""
+ mock_session.get(MOCKED_URL, expire_after=1)
+ mock_session.stale_if_error = True
+ time.sleep(1)
+
+ response = mock_session.get(MOCKED_URL, only_if_cached=True)
+ assert response.from_cache is True
+ assert response.is_expired is False
+
+
+def test_request_only_if_cached__prepared_request(mock_session):
+ """The only_if_cached option should also work for PreparedRequests with CachedSession.send()"""
+ request = Request(method='GET', url=MOCKED_URL, headers={}).prepare()
+ response = mock_session.send(request, only_if_cached=True)
+ assert response.status_code == 504
+ with pytest.raises(HTTPError):
+ response.raise_for_status()
+
+
def test_request_refresh(mock_session):
"""The refresh option should send and cache a new request. Any expire_after value provided
should overwrite the previous value."""
