Merge pull request #568 from JWCook/refresh
Revise behavior for refresh, force_refresh, and DO_NOT_CACHE
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index 232736a..0d60297 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -1,6 +1,6 @@
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
- rev: v4.1.0
+ rev: v4.2.0
hooks:
- id: check-toml
- id: check-yaml
diff --git a/HISTORY.md b/HISTORY.md
index d2ad014..a9aea84 100644
--- a/HISTORY.md
+++ b/HISTORY.md
@@ -1,15 +1,17 @@
# History
-## 0.10.0 (Unreleased)
-[See all issues and PRs for 0.10](https://github.com/reclosedev/requests-cache/milestone/5?closed=1)
+## Unreleased
+[See all unreleased issues and PRs](https://github.com/reclosedev/requests-cache/milestone/5?closed=1)
**Expiration & Headers:**
* Add support for `Cache-Control: only-if-cached`
* Revalidate for `Cache-Control: no-cache` request or response header
* Revalidate for `Cache-Control: max-age=0, must-revalidate` response headers
* Add `only_if_cached` option to `CachedSession.request()` and `send()` to return only cached results without sending real requests
-* Add `refresh` option to `CachedSession.request()` and `send()` to make (and cache) an new request regardless of existing cache contents
-* Add `revalidate` option to `CachedSession.request()` and `send()` to send conditional request (if possible) before using a cached response
+* Add `refresh` option to `CachedSession.request()` and `send()` to revalidate with the server before using a cached response
+* Add `force_refresh` option to `CachedSession.request()` and `send()` to awlays make and cache a new request regardless of existing cache contents
+* Make behavior for `expire_after=0` consistent with `Cache-Control: max-age=0`: if the response has a validator, save it to the cache but revalidate on use.
+* The constant `requests_cache.DO_NOT_CACHE` may be used to completely disable caching for a request
**Backends:**
* Add `wal` parameter for SQLite backend to enable write-ahead logging
@@ -24,6 +26,7 @@
* `is_expired`
* Populate `cache_key` and `expires` for new (non-cached) responses, if it was written to the cache
* Add return type hints for all `CachedSession` request methods (`get()`, `post()`, etc.)
+* Always skip both cache read and write for requests excluded by `allowable_methods` (previously only skipped write)
**Dependencies:**
* Replace `appdirs` with `platformdirs`
diff --git a/docs/user_guide/expiration.md b/docs/user_guide/expiration.md
index 24b9c8a..f6a90aa 100644
--- a/docs/user_guide/expiration.md
+++ b/docs/user_guide/expiration.md
@@ -23,38 +23,56 @@
5. Per-session expiration (`expire_after` argument for {py:class}`.CacheBackend`)
## Expiration Values
-`expire_after` can be any of the following:
-- `-1` (to never expire)
-- `0` (to "expire immediately," e.g. bypass the cache)
+`expire_after` can be any of the following time values:
- A positive number (in seconds)
- A {py:class}`~datetime.timedelta`
- A {py:class}`~datetime.datetime`
+Or one of the following special values:
+- `DO_NOT_CACHE`: Skip both reading from and writing to the cache
+- `EXPIRE_IMMEDIATELY`: Consider the response already expired, but potentially usable
+- `NEVER_EXPIRE`: Store responses indefinitely
+
+```{note}
+A value of 0 or `EXPIRE_IMMEDIATELY` will behave the same as
+[`Cache-Control: max-age=0`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control#response_directives).
+Depending on other settings and headers, an expired response may either be cached and require
+revalidation for each use, or not be cached at all. See {ref}`conditional-requests` for more details.
+```
+
Examples:
```python
->>> # To specify a unit of time other than seconds, use a timedelta
>>> from datetime import timedelta
+>>> from requests_cache import DO_NOT_CACHE, NEVER_EXPIRE, EXPIRE_IMMEDIATELY, CachedSession
+
+>>> # Specify a simple expiration value in seconds
+>>> session = CachedSession(expire_after=60)
+
+>>> # To specify a unit of time other than seconds, use a timedelta
>>> session = CachedSession(expire_after=timedelta(days=30))
->>> # Update an existing session to disable expiration (i.e., store indefinitely)
->>> session.settings.expire_after = -1
+>>> # Or expire on a specific date and time
+>>> session = CachedSession(expire_after=datetime(2023, 1, 1, 0, 0))
+
+>>> # Update an existing session to store new responses indefinitely
+>>> session.settings.expire_after = NEVER_EXPIRE
>>> # Disable caching by default, unless enabled by other settings
->>> session = CachedSession(expire_after=0)
+>>> session = CachedSession(expire_after=DO_NOT_CACHE)
+
+>>> # Override for a single request: cache the response if it can be revalidated
+>>> session.request(expire_after=EXPIRE_IMMEDIATELY)
```
(url-patterns)=
## Expiration With URL Patterns
-You can use `urls_expire_after` to set different expiration values based on URL glob patterns.
-This allows you to customize caching based on what you know about the resources you're requesting
-or how you intend to use them. For example, you might request one resource that gets updated
-frequently, another that changes infrequently, and another that never changes. Example:
+You can use `urls_expire_after` to set different expiration values based on URL glob patterns:
```python
>>> urls_expire_after = {
... '*.site_1.com': 30,
... 'site_2.com/resource_1': 60 * 2,
... 'site_2.com/resource_2': 60 * 60 * 24,
-... 'site_2.com/static': -1,
+... 'site_2.com/static': NEVER_EXPIRE,
... }
>>> session = CachedSession(urls_expire_after=urls_expire_after)
```
@@ -66,6 +84,7 @@
is equivalent to `http*://site.com/resource/**`
- If there is more than one match, the first match will be used in the order they are defined
- If no patterns match a request, `CachedSession.settings.expire_after` will be used as a default
+- See {ref}`selective-caching` for an example of using `urls_expire_after` as an allowlist
(request-errors)=
## Expiration and Error Handling
@@ -133,18 +152,21 @@
### Manual Refresh
If you want to manually refresh a response before it expires, you can use the `refresh` argument.
-This will always send a new request, and ignore and overwrite any previously cached response. The
-response will be saved with a new expiration time, according to the normal expiration rules described above.
-```python
->>> response = session.get('http://httpbin.org/get')
->>> response = session.get('http://httpbin.org/get', refresh=True)
->>> assert response.from_cache is False
-```
-A related argument is `revalidate`, which is basically a "soft refresh." It will send a quick
-{ref}`conditional request <conditional-requests>` to the server, and use the cached response if the
-remote content has not changed. If the cached response does not contain validation headers, this
-option will have no effect.
+* This is equivalent to **F5** in most browsers.
+* The response will be saved with a new expiration time, according to the normal expiration rules
+described above.
+* If possible, this will {ref}`revalidate <conditional-requests>` with the server to potentially
+ avoid re-downloading an unchanged response.
+* To force a refresh (e.g., skip revalidation and always send a new request), use the
+ `force_refresh` argument. This is equivalent to **Ctrl-F5** in most browsers.
+
+Example:
+```python
+>>> response_1 = session.get('http://httpbin.org/get')
+>>> response_2 = session.get('http://httpbin.org/get', refresh=True)
+>>> assert response_2.from_cache is False
+```
### Cache-Only Requests
If you want to only use cached responses without making any real requests, you can use the
diff --git a/docs/user_guide/filtering.md b/docs/user_guide/filtering.md
index 68e65f0..e2afa70 100644
--- a/docs/user_guide/filtering.md
+++ b/docs/user_guide/filtering.md
@@ -11,7 +11,7 @@
```
(http-methods)=
-## Cached HTTP Methods
+## Filter by HTTP Methods
To cache additional HTTP methods, specify them with `allowable_methods`:
```python
>>> session = CachedSession(allowable_methods=('GET', 'POST'))
@@ -22,7 +22,7 @@
requests that may exceed the max size of a `GET` request. You may also want to cache `POST` requests
to ensure you don't send the exact same data multiple times.
-## Cached Status Codes
+## Filter by Status Codes
To cache additional status codes, specify them with `allowable_codes`
```python
>>> session = CachedSession(allowable_codes=(200, 418))
@@ -30,15 +30,14 @@
```
(selective-caching)=
-## Cached URLs
+## Filter by URLs
You can use {ref}`URL patterns <url-patterns>` to define an allowlist for selective caching, by
-using a expiration value of `0` (or `requests_cache.DO_NOT_CACHE`, to be more explicit) for
-non-matching request URLs:
+using a expiration value of `requests_cache.DO_NOT_CACHE` for non-matching request URLs:
```python
->>> from requests_cache import DO_NOT_CACHE, CachedSession
+>>> from requests_cache import DO_NOT_CACHE, NEVER_EXPIRE, CachedSession
>>> urls_expire_after = {
... '*.site_1.com': 30,
-... 'site_2.com/static': -1,
+... 'site_2.com/static': NEVER_EXPIRE,
... '*': DO_NOT_CACHE,
... }
>>> session = CachedSession(urls_expire_after=urls_expire_after)
@@ -51,6 +50,7 @@
>>> session = CachedSession(urls_expire_after=urls_expire_after, expire_after=0)
```
+(custom-filtering)=
## Custom Cache Filtering
If you need more advanced behavior for choosing what to cache, you can provide a custom filtering
function via the `filter_fn` param. This can by any function that takes a
diff --git a/examples/url_patterns.py b/examples/url_patterns.py
index 0fddb86..c948fef 100755
--- a/examples/url_patterns.py
+++ b/examples/url_patterns.py
@@ -5,13 +5,13 @@
"""
from datetime import timedelta
-from requests_cache import CachedSession
+from requests_cache import DO_NOT_CACHE, NEVER_EXPIRE, CachedSession
default_expire_after = 60 * 60 # By default, cached responses expire in an hour
urls_expire_after = {
'httpbin.org/image': timedelta(days=7), # Requests for this base URL will expire in a week
- '*.fillmurray.com': -1, # Requests matching this pattern will never expire
- '*.placeholder.com/*': 0, # Requests matching this pattern will not be cached
+ '*.fillmurray.com': NEVER_EXPIRE, # Requests matching this pattern will never expire
+ '*.placeholder.com/*': DO_NOT_CACHE, # Requests matching this pattern will not be cached
}
urls = [
'https://httpbin.org/get', # Will expire in an hour
diff --git a/requests_cache/__init__.py b/requests_cache/__init__.py
index 81da8e9..30ffb55 100644
--- a/requests_cache/__init__.py
+++ b/requests_cache/__init__.py
@@ -1,8 +1,6 @@
# flake8: noqa: E402,F401
from logging import getLogger
-logger = getLogger('requests_cache')
-
# Version is defined in pyproject.toml.
# It's copied here to make it easier for client code to check the installed version.
__version__ = '0.10.0'
@@ -11,6 +9,7 @@
from .backends import *
from .cache_control import *
from .cache_keys import *
+ from .expiration import *
from .models import *
from .patcher import *
from .serializers import *
@@ -18,4 +17,4 @@
from .settings import *
# Log and ignore ImportErrors, if imported outside a virtualenv (e.g., just to check __version__)
except ImportError as e:
- logger.warning(e, exc_info=True)
+ getLogger('requests_cache').warning(e, exc_info=True)
diff --git a/requests_cache/_utils.py b/requests_cache/_utils.py
index edbf4c6..a57f4ea 100644
--- a/requests_cache/_utils.py
+++ b/requests_cache/_utils.py
@@ -1,8 +1,9 @@
"""Minor internal utility functions that don't really belong anywhere else"""
from inspect import signature
from logging import getLogger
-from typing import Any, Callable, Dict, Iterable, Iterator, List, Optional
+from typing import Any, Callable, Dict, Iterable, Iterator, List, Optional, Tuple
+KwargDict = Dict[str, Any]
logger = getLogger('requests_cache')
@@ -53,11 +54,23 @@
return Placeholder
-def get_valid_kwargs(func: Callable, kwargs: Dict, extras: Iterable[str] = None) -> Dict:
- """Get the subset of non-None ``kwargs`` that are valid params for ``func``"""
+def get_valid_kwargs(func: Callable, kwargs: Dict, extras: Iterable[str] = None) -> KwargDict:
+ """Get the subset of non-None ``kwargs`` that are valid arguments for ``func``"""
+ kwargs, _ = split_kwargs(func, kwargs, extras)
+ return {k: v for k, v in kwargs.items() if v is not None}
+
+
+def split_kwargs(
+ func: Callable, kwargs: Dict, extras: Iterable[str] = None
+) -> Tuple[KwargDict, KwargDict]:
+ """Split ``kwargs`` into two dicts: those that are valid arguments for ``func``, and those that
+ are not
+ """
params = list(signature(func).parameters)
params.extend(extras or [])
- return {k: v for k, v in kwargs.items() if k in params and v is not None}
+ valid_kwargs = {k: v for k, v in kwargs.items() if k in params}
+ invalid_kwargs = {k: v for k, v in kwargs.items() if k not in params}
+ return valid_kwargs, invalid_kwargs
def try_int(value: Any) -> Optional[int]:
diff --git a/requests_cache/backends/base.py b/requests_cache/backends/base.py
index f85c175..472ba9f 100644
--- a/requests_cache/backends/base.py
+++ b/requests_cache/backends/base.py
@@ -4,12 +4,14 @@
:classes-only:
:nosignatures:
"""
-import pickle
+from __future__ import annotations
+
from abc import ABC
from collections import UserDict
from collections.abc import MutableMapping
from datetime import datetime
from logging import getLogger
+from pickle import PickleError
from typing import Iterable, Iterator, Optional, Tuple, Union
from requests import PreparedRequest, Response
@@ -21,7 +23,7 @@
from ..settings import DEFAULT_CACHE_NAME, CacheSettings
# Specific exceptions that may be raised during deserialization
-DESERIALIZE_ERRORS = (AttributeError, ImportError, TypeError, ValueError, pickle.PickleError)
+DESERIALIZE_ERRORS = (AttributeError, ImportError, PickleError, TypeError, ValueError)
ResponseOrKey = Union[CachedResponse, str]
logger = getLogger(__name__)
diff --git a/requests_cache/cache_control.py b/requests_cache/cache_control.py
index ed4e9ce..a13da45 100644
--- a/requests_cache/cache_control.py
+++ b/requests_cache/cache_control.py
@@ -8,8 +8,6 @@
:functions-only:
:nosignatures:
"""
-from __future__ import annotations
-
from datetime import datetime
from logging import getLogger
from typing import Dict, MutableMapping, Optional, Tuple, Union
@@ -21,23 +19,24 @@
from ._utils import coalesce, try_int
from .expiration import (
DO_NOT_CACHE,
+ EXPIRE_IMMEDIATELY,
NEVER_EXPIRE,
ExpirationTime,
get_expiration_datetime,
+ get_expiration_seconds,
get_url_expiration,
)
from .models import CachedResponse
-from .settings import CacheSettings, RequestSettings
+from .settings import CacheSettings
__all__ = ['CacheActions']
-# Temporary header only used to support the 'refresh' option in CachedSession.request()
-REFRESH_TEMP_HEADER = 'requests-cache-refresh'
-
-CacheDirective = Union[None, int, bool]
+CacheDirective = Union[None, bool, int, str]
+HeaderDict = MutableMapping[str, str]
logger = getLogger(__name__)
+# TODO: Add custom __rich_repr__ to exclude default values to make logs cleaner (w/ RichHandler)
@define
class CacheActions:
"""Translates cache settings and headers into specific actions to take for a given cache item.
@@ -70,54 +69,57 @@
skip_write: bool = field(default=False)
# Inputs/internal attributes
- _settings: RequestSettings = field(default=None, repr=False, init=False)
+ _settings: CacheSettings = field(default=None, repr=False, init=False)
_validation_headers: Dict[str, str] = field(factory=dict, repr=False, init=False)
+ # TODO: It would be nice to not need these temp variables
+ _only_if_cached: bool = field(default=False)
+ _refresh: bool = field(default=False)
@classmethod
- def from_request(
- cls,
- cache_key: str,
- request: PreparedRequest,
- settings: CacheSettings,
- **kwargs,
- ):
- """Initialize from request info and cache settings"""
- request.headers = request.headers or CaseInsensitiveDict()
+ def from_request(cls, cache_key: str, request: PreparedRequest, settings: CacheSettings = None):
+ """Initialize from request info and cache settings.
+
+ Note on refreshing: `must-revalidate` isn't a standard request header, but is used here to
+ indicate a user-requested refresh. Typically that's only used in response headers, and
+ `max-age=0` would be used by a client to request a refresh. However, this would conflict
+ with the `expire_after` option provided in :py:meth:`.CachedSession.request`.
+ """
directives = get_cache_directives(request.headers)
logger.debug(f'Cache directives from request headers: {directives}')
- # Merge session settings and request settings
- settings = RequestSettings(settings, skip_invalid=True, **kwargs)
- settings.only_if_cached = settings.only_if_cached or 'only-if-cached' in directives
- settings.refresh = settings.refresh or bool(request.headers.pop(REFRESH_TEMP_HEADER, False))
- settings.revalidate = settings.revalidate or 'no-cache' in directives
+ # Merge relevant headers with session + request settings
+ settings = settings or CacheSettings()
+ only_if_cached = settings.only_if_cached or 'only-if-cached' in directives
+ expire_immediately = directives.get('max-age') == EXPIRE_IMMEDIATELY
+ refresh = expire_immediately or 'must-revalidate' in directives
+ force_refresh = 'no-cache' in directives
# Check expiration values in order of precedence
expire_after = coalesce(
directives.get('max-age'),
- settings.request_expire_after,
get_url_expiration(request.url, settings.urls_expire_after),
settings.expire_after,
)
- # Check conditions for reading from the cache
- skip_read = any(
- [
- settings.refresh,
- settings.disabled,
- 'no-store' in directives,
- try_int(expire_after) == DO_NOT_CACHE,
- ]
- )
+ # Check and log conditions for reading from the cache
+ read_criteria = {
+ 'disabled cache': settings.disabled,
+ 'disabled method': str(request.method) not in settings.allowable_methods,
+ 'disabled by headers': 'no-store' in directives,
+ 'disabled by refresh': force_refresh,
+ 'disabled by expiration': expire_after == DO_NOT_CACHE,
+ }
+ _log_cache_criteria('read', read_criteria)
actions = cls(
cache_key=cache_key,
expire_after=expire_after,
- skip_read=skip_read,
+ only_if_cached=only_if_cached,
+ refresh=refresh,
+ skip_read=any(read_criteria.values()),
skip_write='no-store' in directives,
)
actions._settings = settings
- actions._validation_headers = {}
return actions
@property
@@ -134,35 +136,38 @@
# Determine if we need to send a new request or respond with an error
is_expired = getattr(cached_response, 'is_expired', False)
invalid_response = cached_response is None or is_expired
- if invalid_response and self._settings.only_if_cached and not self._settings.stale_if_error:
+ if invalid_response and self._only_if_cached and not self._settings.stale_if_error:
self.error_504 = True
elif cached_response is None:
self.send_request = True
- elif is_expired and not (self._settings.only_if_cached and self._settings.stale_if_error):
+ elif is_expired and not (self._only_if_cached and self._settings.stale_if_error):
self.resend_request = True
if cached_response is not None:
self._update_validation_headers(cached_response)
+ logger.debug(f'Post-read cache actions: {self}')
def _update_validation_headers(self, response: CachedResponse):
- # Revalidation may be triggered by either stale response or request/cached response headers
+ """If needed, get validation headers based on a cached response. Revalidation may be
+ triggered by a stale response, request headers, or cached response headers.
+ """
directives = get_cache_directives(response.headers)
- revalidate = _has_validator(response.headers) and any(
- [
- response.is_expired,
- self._settings.revalidate,
- 'no-cache' in directives,
- 'must-revalidate' in directives and directives.get('max-age') == 0,
- ]
+ revalidate = _has_validator(response.headers) and (
+ response.is_expired
+ or self._refresh
+ or 'no-cache' in directives
+ or 'must-revalidate' in directives
+ and directives.get('max-age') == 0
)
# Add the appropriate validation headers, if needed
if revalidate:
- self.send_request = True
if response.headers.get('ETag'):
self._validation_headers['If-None-Match'] = response.headers['ETag']
if response.headers.get('Last-Modified'):
self._validation_headers['If-Modified-Since'] = response.headers['Last-Modified']
+ self.send_request = True
+ self.resend_request = False
def update_from_response(self, response: Response):
"""Update expiration + actions based on headers and other details from a new response.
@@ -173,24 +178,25 @@
self._update_from_response_headers(response)
# If "expired" but there's a validator, save it to the cache and revalidate on use
- expire_immediately = try_int(self.expire_after) == DO_NOT_CACHE
+ do_not_cache = self.expire_after == DO_NOT_CACHE
+ expire_immediately = self.expire_after == EXPIRE_IMMEDIATELY
has_validator = _has_validator(response.headers)
- self.skip_write = self.skip_write or (expire_immediately and not has_validator)
# Apply filter callback, if any
callback = self._settings.filter_fn
filtered_out = callback is not None and not callback(response)
- # Apply and log remaining checks needed to determine if the response should be cached
- cache_criteria = {
+ # Check and log conditions for writing to the cache
+ write_criteria = {
'disabled cache': self._settings.disabled,
'disabled method': str(response.request.method) not in self._settings.allowable_methods,
'disabled status': response.status_code not in self._settings.allowable_codes,
'disabled by filter': filtered_out,
- 'disabled by headers or expiration params': self.skip_write,
+ 'disabled by headers': self.skip_write,
+ 'disabled by expiration': do_not_cache or (expire_immediately and not has_validator),
}
- logger.debug(f'Pre-cache checks for response from {response.url}: {cache_criteria}')
- self.skip_write = any(cache_criteria.values())
+ self.skip_write = any(write_criteria.values())
+ _log_cache_criteria('write', write_criteria)
def _update_from_response_headers(self, response: Response):
"""Check response headers for expiration and other cache directives"""
@@ -217,7 +223,8 @@
) -> CachedResponse:
"""After revalidation, update the cached response's headers and reset its expiration"""
logger.debug(
- f'Response for URL {response.request.url} has not been modified; updating and using cached response'
+ f'Response for URL {response.request.url} has not been modified; '
+ 'updating and using cached response'
)
cached_response.expires = self.expires
cached_response.headers.update(response.headers)
@@ -225,25 +232,23 @@
return cached_response
-def append_directive(
- headers: Optional[MutableMapping[str, str]], directive: str
-) -> MutableMapping[str, str]:
+def append_directive(headers: HeaderDict, directive: str) -> HeaderDict:
"""Append a Cache-Control directive to existing headers (if any)"""
- headers = CaseInsensitiveDict(headers)
directives = headers['Cache-Control'].split(',') if headers.get('Cache-Control') else []
directives.append(directive)
headers['Cache-Control'] = ','.join(directives)
return headers
-def get_cache_directives(headers: MutableMapping) -> Dict[str, CacheDirective]:
- """Get all Cache-Control directives as a dict. Handle duplicate headers and comma-separated
- lists. Key-only directives are returned as ``{key: True}``.
+def get_cache_directives(headers: HeaderDict) -> Dict[str, CacheDirective]:
+ """Get all Cache-Control directives as a dict. Handles duplicate headers (with
+ CaseInsensitiveDict) and comma-separated lists.
+ Key-only directives are returned as ``{key: True}``.
"""
if not headers:
return {}
- kv_directives = {}
+ kv_directives: Dict[str, CacheDirective] = {}
if headers.get('Cache-Control'):
cache_directives = headers['Cache-Control'].split(',')
kv_directives = dict([_split_kv_directive(value) for value in cache_directives])
@@ -265,5 +270,30 @@
return header_value, True
-def _has_validator(headers: MutableMapping) -> bool:
+def set_request_headers(
+ headers: Optional[HeaderDict], expire_after, only_if_cached, refresh, force_refresh
+):
+ """Translate keyword arguments into equivalent request headers, to be handled in CacheActions"""
+ headers = CaseInsensitiveDict(headers)
+ if expire_after is not None:
+ headers = append_directive(headers, f'max-age={get_expiration_seconds(expire_after)}')
+ if only_if_cached:
+ headers = append_directive(headers, 'only-if-cached')
+ if refresh:
+ headers = append_directive(headers, 'must-revalidate')
+ if force_refresh:
+ headers = append_directive(headers, 'no-cache')
+ return headers
+
+
+def _has_validator(headers: HeaderDict) -> bool:
return bool(headers.get('ETag') or headers.get('Last-Modified'))
+
+
+def _log_cache_criteria(operation: str, criteria: Dict):
+ """Log details on any failed checks for cache read or write"""
+ if any(criteria.values()):
+ status = ', '.join([k for k, v in criteria.items() if v])
+ else:
+ status = 'Passed'
+ logger.debug(f'Pre-{operation} cache checks: {status}')
diff --git a/requests_cache/expiration.py b/requests_cache/expiration.py
index e70cae4..eca86be 100644
--- a/requests_cache/expiration.py
+++ b/requests_cache/expiration.py
@@ -8,10 +8,11 @@
from ._utils import try_int
-__all__ = ['DO_NOT_CACHE', 'NEVER_EXPIRE', 'get_expiration_datetime']
+__all__ = ['DO_NOT_CACHE', 'EXPIRE_IMMEDIATELY', 'NEVER_EXPIRE', 'get_expiration_datetime']
-# May be set by either headers or expire_after param to disable caching or disable expiration
-DO_NOT_CACHE = 0
+# Special expiration values that may be set by either headers or keyword args
+DO_NOT_CACHE = 0x0D0E0200020704 # Per RFC 4824
+EXPIRE_IMMEDIATELY = 0
NEVER_EXPIRE = -1
ExpirationTime = Union[None, int, float, str, datetime, timedelta]
@@ -22,11 +23,11 @@
def get_expiration_datetime(expire_after: ExpirationTime) -> Optional[datetime]:
"""Convert an expiration value in any supported format to an absolute datetime"""
- # Never expire
- if expire_after is None or expire_after == NEVER_EXPIRE:
+ # Never expire (or do not cache, in which case expiration won't be used)
+ if expire_after is None or expire_after in [NEVER_EXPIRE, DO_NOT_CACHE]:
return None
# Expire immediately
- elif try_int(expire_after) == DO_NOT_CACHE:
+ elif try_int(expire_after) == EXPIRE_IMMEDIATELY:
return datetime.utcnow()
# Already a datetime or datetime str
if isinstance(expire_after, str):
@@ -42,6 +43,8 @@
def get_expiration_seconds(expire_after: ExpirationTime) -> int:
"""Convert an expiration value in any supported format to an expiration time in seconds"""
+ if expire_after == DO_NOT_CACHE:
+ return DO_NOT_CACHE
expires = get_expiration_datetime(expire_after)
return ceil((expires - datetime.utcnow()).total_seconds()) if expires else NEVER_EXPIRE
diff --git a/requests_cache/session.py b/requests_cache/session.py
index 6abd360..3a1b1a3 100644
--- a/requests_cache/session.py
+++ b/requests_cache/session.py
@@ -13,7 +13,7 @@
.. autoclass:: requests_cache.session.CacheMixin
"""
-from contextlib import contextmanager
+from contextlib import contextmanager, nullcontext
from logging import getLogger
from threading import RLock
from typing import TYPE_CHECKING, Dict, Iterable, MutableMapping, Optional, Union
@@ -25,8 +25,8 @@
from ._utils import get_valid_kwargs
from .backends import BackendSpecifier, init_backend
-from .cache_control import REFRESH_TEMP_HEADER, CacheActions, append_directive
-from .expiration import ExpirationTime, get_expiration_seconds
+from .cache_control import CacheActions, set_request_headers
+from .expiration import ExpirationTime
from .models import AnyResponse, CachedResponse, OriginalResponse
from .serializers import SerializerPipeline
from .settings import (
@@ -70,7 +70,7 @@
**kwargs,
):
self.cache = init_backend(cache_name, backend, serializer=serializer, **kwargs)
- self.settings = CacheSettings(
+ self.settings = CacheSettings.from_kwargs(
expire_after=expire_after,
urls_expire_after=urls_expire_after,
cache_control=cache_control,
@@ -81,17 +81,16 @@
filter_fn=filter_fn,
key_fn=key_fn,
stale_if_error=stale_if_error,
- skip_invalid=True,
**kwargs,
)
self._lock = RLock()
- # If the mixin superclass is custom Session, pass along any valid kwargs
+ # If the mixin superclass is a custom Session, pass along any valid kwargs
super().__init__(**get_valid_kwargs(super().__init__, kwargs)) # type: ignore
@property
def settings(self) -> CacheSettings:
- """Settings that affect cache behavior, and can be changed at any time"""
+ """Settings that affect cache behavior"""
return self.cache._settings
@settings.setter
@@ -141,7 +140,7 @@
expire_after: ExpirationTime = None,
only_if_cached: bool = False,
refresh: bool = False,
- revalidate: bool = False,
+ force_refresh: bool = False,
**kwargs,
) -> AnyResponse:
"""This method prepares and sends a request while automatically performing any necessary
@@ -152,49 +151,50 @@
See :py:meth:`requests.Session.request` for base parameters. Additional parameters:
Args:
- expire_after: Expiration time to set only for this request; see details below.
- Overrides ``CachedSession.expire_after``. Accepts all the same values as
- ``CachedSession.expire_after``. Use ``-1`` to disable expiration.
+ expire_after: Expiration time to set only for this request. See :ref:`expiration` for
+ details.
only_if_cached: Only return results from the cache. If not cached, return a 504 response
instead of sending a new request.
- refresh: Always make a new request, and overwrite any previously cached response
- revalidate: Revalidate with the server before using a cached response (e.g., a "soft refresh")
+ refresh: Revalidate with the server before using a cached response, and refresh if needed
+ (e.g., a "soft refresh," like F5 in a browser)
+ force_refresh: Always make a new request, and overwrite any previously cached response
+ (e.g., a "hard refresh", like Ctrl-F5 in a browser))
Returns:
Either a new or cached response
"""
- # Set extra options as headers to be handled in send(), since we can't pass args directly
- headers = headers or {}
- if expire_after is not None:
- headers = append_directive(headers, f'max-age={get_expiration_seconds(expire_after)}')
- if only_if_cached:
- headers = append_directive(headers, 'only-if-cached')
- if revalidate:
- headers = append_directive(headers, 'no-cache')
- if refresh:
- headers[REFRESH_TEMP_HEADER] = 'true'
- kwargs['headers'] = headers
+ headers = set_request_headers(headers, expire_after, only_if_cached, refresh, force_refresh)
+ with patch_form_boundary() if kwargs.get('files') else nullcontext():
+ return super().request(method, url, *args, headers=headers, **kwargs) # type: ignore
- with patch_form_boundary(**kwargs):
- return super().request(method, url, *args, **kwargs) # type: ignore
-
- def send(self, request: PreparedRequest, **kwargs) -> AnyResponse:
+ def send(
+ self,
+ request: PreparedRequest,
+ expire_after: ExpirationTime = None,
+ only_if_cached: bool = False,
+ refresh: bool = False,
+ force_refresh: bool = False,
+ **kwargs,
+ ) -> AnyResponse:
"""Send a prepared request, with caching. See :py:meth:`requests.Session.send` for base
parameters, and see :py:meth:`.request` for extra parameters.
**Order of operations:** For reference, a request will pass through the following methods:
- 1. :py:func:`requests.get`/:py:meth:`requests.Session.get` or other method-specific functions (optional)
+ 1. :py:func:`requests.get`, :py:meth:`CachedSession.get`, etc. (optional)
2. :py:meth:`.CachedSession.request`
3. :py:meth:`requests.Session.request`
4. :py:meth:`.CachedSession.send`
5. :py:meth:`.BaseCache.get_response`
- 6. :py:meth:`requests.Session.send` (if not previously cached)
- 7. :py:meth:`.BaseCache.save_response` (if not previously cached)
+ 6. :py:meth:`requests.Session.send` (if not using a cached response)
+ 7. :py:meth:`.BaseCache.save_response` (if not using a cached response)
"""
# Determine which actions to take based on settings and request info
+ request.headers = set_request_headers(
+ request.headers, expire_after, only_if_cached, refresh, force_refresh
+ )
actions = CacheActions.from_request(
- self.cache.create_key(request, **kwargs), request, self.settings, **kwargs
+ self.cache.create_key(request, **kwargs), request, self.settings
)
# Attempt to fetch a cached response
@@ -328,7 +328,7 @@
``['sqlite', 'filesystem', 'mongodb', 'gridfs', 'redis', 'dynamodb', 'memory']``
serializer: Serializer name or instance; name may be one of
``['pickle', 'json', 'yaml', 'bson']``.
- expire_after: Time after which cached items will expire
+ expire_after: Time after which cached items will expire. See :ref:`expiration` for details.
urls_expire_after: Expiration times to apply for different URL patterns
cache_control: Use Cache-Control and other response headers to set expiration
allowable_codes: Only cache responses with one of these status codes
@@ -338,8 +338,9 @@
ignored_parameters: List of request parameters to not match against, and exclude from the cache
stale_if_error: Return stale cache data if a new request raises an exception
filter_fn: Response filtering function that indicates whether or not a given response should
- be cached.
- key_fn: Request matching function for generating custom cache keys
+ be cached. See :ref:`custom-filtering` for details.
+ key_fn: Request matching function for generating custom cache keys. See
+ :ref:`custom-matching` for details.
"""
@@ -354,15 +355,12 @@
@contextmanager
-def patch_form_boundary(**request_kwargs):
+def patch_form_boundary():
"""If the ``files`` param is present, patch the form boundary used to separate multipart
uploads. ``requests`` does not provide a way to pass a custom boundary to urllib3, so this just
monkey-patches it instead.
"""
- if request_kwargs.get('files'):
- original_boundary = filepost.choose_boundary
- filepost.choose_boundary = lambda: '##requests-cache-form-boundary##'
- yield
- filepost.choose_boundary = original_boundary
- else:
- yield
+ original_boundary = filepost.choose_boundary
+ filepost.choose_boundary = lambda: '##requests-cache-form-boundary##'
+ yield
+ filepost.choose_boundary = original_boundary
diff --git a/requests_cache/settings.py b/requests_cache/settings.py
index e9c6006..5a7cc6f 100644
--- a/requests_cache/settings.py
+++ b/requests_cache/settings.py
@@ -1,6 +1,6 @@
from typing import Callable, Dict, Iterable, Union
-from attr import asdict, define, field
+from attr import define, field
from requests import Response
from ._utils import get_valid_kwargs
@@ -16,15 +16,12 @@
KeyCallback = Callable[..., str]
-@define(init=False)
+@define
class CacheSettings:
"""Class used internally to store settings that affect caching behavior. This allows settings
to be used across multiple modules, but exposed to the user in a single property
(:py:attr:`.CachedSession.settings`). These values can safely be modified after initialization. See
:py:class:`.CachedSession` and :ref:`user-guide` for usage details.
-
- Args:
- skip_invalid: Ignore invalid settings for easier initialization from mixed ``**kwargs``
"""
allowable_codes: Iterable[int] = field(default=DEFAULT_STATUS_CODES)
@@ -40,34 +37,21 @@
stale_if_error: bool = field(default=False)
urls_expire_after: Dict[str, ExpirationTime] = field(factory=dict)
- def __init__(self, **kwargs):
- kwargs = self._rename_kwargs(kwargs)
- if kwargs.pop('skip_invalid', False) is True:
- kwargs = get_valid_kwargs(self.__attrs_init__, kwargs)
- self.__attrs_init__(**kwargs)
+ @classmethod
+ def from_kwargs(cls, **kwargs):
+ """Constructor with some additional steps:
+
+ * Handle some deprecated argument names
+ * Ignore invalid settings, for easier initialization from mixed ``**kwargs``
+ """
+ kwargs = cls._rename_kwargs(kwargs)
+ kwargs = get_valid_kwargs(cls.__init__, kwargs)
+ return cls(**kwargs)
@staticmethod
def _rename_kwargs(kwargs):
- """Handle some deprecated argument names"""
if 'old_data_on_error' in kwargs:
kwargs['stale_if_error'] = kwargs.pop('old_data_on_error')
if 'include_get_headers' in kwargs:
kwargs['match_headers'] = kwargs.pop('include_get_headers')
return kwargs
-
-
-@define(init=False)
-class RequestSettings(CacheSettings):
- """Cache settings that may be set for an individual request"""
-
- # Additional settings that may be set for an individual request; not used at session level
- refresh: bool = field(default=False)
- revalidate: bool = field(default=False)
- request_expire_after: ExpirationTime = field(default=None)
-
- def __init__(self, session_settings: CacheSettings = None, **kwargs):
- """Start with session-level cache settings and append/override with request-level settings"""
- session_kwargs = asdict(session_settings) if session_settings else {}
- # request-level expiration needs to be stored separately
- kwargs['request_expire_after'] = kwargs.pop('expire_after', None)
- super().__init__(**{**session_kwargs, **kwargs})
diff --git a/tests/conftest.py b/tests/conftest.py
index a21b3fc..7de9b6d 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -20,6 +20,7 @@
import requests
from requests_mock import ANY as ANY_METHOD
from requests_mock import Adapter
+from rich.logging import RichHandler
from timeout_decorator import timeout
from requests_cache import ALL_METHODS, CachedSession, install_cache, uninstall_cache
@@ -62,6 +63,7 @@
MOCKED_URL_REDIRECT = 'http+mock://requests-cache.com/redirect'
MOCKED_URL_REDIRECT_TARGET = 'http+mock://requests-cache.com/redirect_target'
MOCKED_URL_404 = 'http+mock://requests-cache.com/nonexistent'
+MOCKED_URL_500 = 'http+mock://requests-cache.com/answer?q=this-statement-is-false'
MOCK_PROTOCOLS = ['mock://', 'http+mock://', 'https+mock://']
PROJECT_DIR = abspath(dirname(dirname(__file__)))
@@ -77,7 +79,12 @@
# Configure logging to show log output when tests fail (or with pytest -s)
-basicConfig(level='INFO')
+basicConfig(
+ level='INFO',
+ format='%(message)s',
+ datefmt='[%m-%d %H:%M:%S]',
+ handlers=[RichHandler(rich_tracebacks=True, markup=True)],
+)
# getLogger('requests_cache').setLevel('DEBUG')
logger = getLogger(__name__)
@@ -206,11 +213,8 @@
text='mock redirected response',
status_code=200,
)
- adapter.register_uri(
- ANY_METHOD,
- MOCKED_URL_404,
- status_code=404,
- )
+ adapter.register_uri(ANY_METHOD, MOCKED_URL_404, status_code=404)
+ adapter.register_uri(ANY_METHOD, MOCKED_URL_500, status_code=500)
return adapter
diff --git a/tests/unit/test_cache_control.py b/tests/unit/test_cache_control.py
index 8847bc2..5bfa2ad 100644
--- a/tests/unit/test_cache_control.py
+++ b/tests/unit/test_cache_control.py
@@ -2,12 +2,12 @@
from unittest.mock import MagicMock, patch
import pytest
-from requests import PreparedRequest
+from requests import PreparedRequest, Request
-from requests_cache.cache_control import DO_NOT_CACHE, CacheActions
+from requests_cache.cache_control import EXPIRE_IMMEDIATELY, CacheActions
from requests_cache.models import CachedResponse
-from requests_cache.settings import CacheSettings, RequestSettings
-from tests.conftest import ETAG, HTTPDATE_STR, LAST_MODIFIED, get_mock_response
+from requests_cache.settings import CacheSettings
+from tests.conftest import ETAG, HTTPDATE_STR, LAST_MODIFIED, MOCKED_URL, get_mock_response
IGNORED_DIRECTIVES = [
'no-transform',
@@ -44,12 +44,7 @@
get_url_expiration.return_value = url_expire_after
settings = CacheSettings(cache_control=True, expire_after=1)
- settings = RequestSettings(settings, expire_after=request_expire_after)
- actions = CacheActions.from_request(
- cache_key='key',
- request=request,
- settings=settings,
- )
+ actions = CacheActions.from_request(cache_key='key', request=request, settings=settings)
assert actions.expire_after == expected_expiration
@@ -60,18 +55,17 @@
({'Expires': HTTPDATE_STR}, None), # Only valid for response headers
({'Cache-Control': 'max-age=60'}, 60),
({'Cache-Control': 'public, max-age=60'}, 60),
- ({'Cache-Control': 'max-age=0'}, DO_NOT_CACHE),
+ ({'Cache-Control': 'max-age=0'}, EXPIRE_IMMEDIATELY),
],
)
def test_init_from_headers(headers, expected_expiration):
"""Test with Cache-Control request headers"""
- settings = RequestSettings(cache_control=True)
- actions = CacheActions.from_request('key', MagicMock(headers=headers), settings)
+ settings = CacheSettings(cache_control=True)
+ request = Request(method='GET', url=MOCKED_URL, headers=headers).prepare()
+ actions = CacheActions.from_request('key', request, settings)
assert actions.cache_key == 'key'
- if expected_expiration == DO_NOT_CACHE:
- assert actions.skip_read is True
- else:
+ if expected_expiration != EXPIRE_IMMEDIATELY:
assert actions.expire_after == expected_expiration
assert actions.skip_read is False
assert actions.skip_write is False
@@ -79,10 +73,9 @@
def test_init_from_headers__no_store():
"""Test with Cache-Control request headers"""
- settings = RequestSettings(cache_control=True)
- actions = CacheActions.from_request(
- 'key', MagicMock(headers={'Cache-Control': 'no-store'}), settings
- )
+ settings = CacheSettings(cache_control=True)
+ request = Request(method='GET', url=MOCKED_URL, headers={'Cache-Control': 'no-store'}).prepare()
+ actions = CacheActions.from_request('key', request, settings)
assert actions.skip_read is True
assert actions.skip_write is True
@@ -91,19 +84,19 @@
@pytest.mark.parametrize(
'url, request_expire_after, expected_expiration',
[
- ('img.site_1.com', None, timedelta(hours=12)),
- ('img.site_1.com', 60, 60),
- ('http://img.site.com/base/', None, 1),
+ ('https://img.site_1.com', None, timedelta(hours=12)),
+ ('https://img.site_1.com', 60, 60),
+ ('https://img.site.com/base/', None, 1),
('https://img.site.com/base/img.jpg', None, 1),
- ('site_2.com/resource_1', None, timedelta(hours=20)),
- ('http://site_2.com/resource_1/index.html', None, timedelta(hours=20)),
+ ('http://site_2.com/resource_1', None, timedelta(hours=20)),
+ ('ftp://site_2.com/resource_1/index.html', None, timedelta(hours=20)),
('http://site_2.com/resource_2/', None, timedelta(days=7)),
('http://site_2.com/static/', None, -1),
('http://site_2.com/static/img.jpg', None, -1),
- ('site_2.com', None, 1),
- ('site_2.com', 60, 60),
- ('some_other_site.com', None, 1),
- ('some_other_site.com', 60, 60),
+ ('http://site_2.com', None, 1),
+ ('http://site_2.com', 60, 60),
+ ('https://some_other_site.com', None, 1),
+ ('https://some_other_site.com', 60, 60),
],
)
def test_init_from_settings(url, request_expire_after, expected_expiration):
@@ -117,11 +110,11 @@
'site_2.com/static': -1,
},
)
- request = MagicMock(url=url)
+ request = Request(method='GET', url=url)
if request_expire_after:
request.headers = {'Cache-Control': f'max-age={request_expire_after}'}
- actions = CacheActions.from_request('key', request, RequestSettings(settings))
+ actions = CacheActions.from_request('key', request.prepare(), settings)
assert actions.expire_after == expected_expiration
@@ -130,20 +123,20 @@
[
({'Cache-Control': 'max-age=60'}, 1, 60, False),
({}, 1, 1, False),
- ({}, 0, 0, True),
+ ({}, 0, 0, False),
({'Cache-Control': 'max-age=60'}, 1, 60, False),
- ({'Cache-Control': 'max-age=0'}, 1, 0, True),
+ ({'Cache-Control': 'max-age=0'}, 1, 0, False),
({'Cache-Control': 'no-store'}, 1, 1, True),
- ({'Cache-Control': 'no-cache'}, 1, 1, False),
+ ({'Cache-Control': 'no-cache'}, 1, 1, True),
],
)
def test_init_from_settings_and_headers(
headers, expire_after, expected_expiration, expected_skip_read
):
"""Test behavior with both cache settings and request headers."""
- request = get_mock_response(headers=headers)
+ request = Request(method='GET', url=MOCKED_URL, headers=headers)
settings = CacheSettings(expire_after=expire_after)
- actions = CacheActions.from_request('key', request, RequestSettings(settings))
+ actions = CacheActions.from_request('key', request, settings)
assert actions.expire_after == expected_expiration
assert actions.skip_read == expected_skip_read
@@ -164,9 +157,7 @@
def test_update_from_cached_response(response_headers, expected_validation_headers):
"""Conditional request headers should be added if the cached response is expired"""
actions = CacheActions.from_request(
- 'key',
- MagicMock(url='https://img.site.com/base/img.jpg', headers={}),
- RequestSettings(),
+ 'key', MagicMock(url='https://img.site.com/base/img.jpg', headers={})
)
cached_response = CachedResponse(
headers=response_headers, expires=datetime.now() - timedelta(1)
@@ -179,7 +170,6 @@
@pytest.mark.parametrize(
'request_headers, response_headers',
[
- ({'Cache-Control': 'no-cache'}, {}),
({}, {'Cache-Control': 'no-cache'}),
({}, {'Cache-Control': 'max-age=0,must-revalidate'}),
],
@@ -188,9 +178,7 @@
"""Conditional request headers should be added if requested by headers (even if the response
is not expired)"""
actions = CacheActions.from_request(
- 'key',
- MagicMock(url='https://img.site.com/base/img.jpg', headers=request_headers),
- RequestSettings(),
+ 'key', MagicMock(url='https://img.site.com/base/img.jpg', headers=request_headers)
)
cached_response = CachedResponse(headers={'ETag': ETAG, **response_headers}, expires=None)
@@ -202,9 +190,7 @@
"""Conditional request headers should NOT be added if the cached response is not expired and
revalidation is otherwise not requested"""
actions = CacheActions.from_request(
- 'key',
- MagicMock(url='https://img.site.com/base/img.jpg', headers={}),
- RequestSettings(),
+ 'key', MagicMock(url='https://img.site.com/base/img.jpg', headers={})
)
cached_response = CachedResponse(
headers={'ETag': ETAG, 'Last-Modified': LAST_MODIFIED}, expires=None
@@ -233,18 +219,18 @@
"""Test with Cache-Control response headers"""
url = 'https://img.site.com/base/img.jpg'
actions = CacheActions.from_request(
- 'key', MagicMock(url=url), RequestSettings(cache_control=True)
+ 'key', MagicMock(url=url), CacheSettings(cache_control=True)
)
actions.update_from_response(get_mock_response(headers=headers))
assert actions.expire_after == expected_expiration
- assert actions.skip_write is (expected_expiration == DO_NOT_CACHE)
+ assert actions.skip_write is (expected_expiration == EXPIRE_IMMEDIATELY)
def test_update_from_response__no_store():
url = 'https://img.site.com/base/img.jpg'
actions = CacheActions.from_request(
- 'key', MagicMock(url=url), RequestSettings(cache_control=True)
+ 'key', MagicMock(url=url), CacheSettings(cache_control=True)
)
actions.update_from_response(get_mock_response(headers={'Cache-Control': 'no-store'}))
assert actions.skip_write is True
@@ -253,7 +239,7 @@
def test_update_from_response__ignored():
url = 'https://img.site.com/base/img.jpg'
actions = CacheActions.from_request(
- 'key', MagicMock(url=url), RequestSettings(cache_control=False)
+ 'key', MagicMock(url=url), CacheSettings(cache_control=False)
)
actions.update_from_response(get_mock_response(headers={'Cache-Control': 'max-age=5'}))
assert actions.expire_after is None
@@ -268,7 +254,7 @@
"""
url = 'https://img.site.com/base/img.jpg'
actions = CacheActions.from_request(
- 'key', MagicMock(url=url), RequestSettings(cache_control=True)
+ 'key', MagicMock(url=url), CacheSettings(cache_control=True)
)
response = get_mock_response(headers={**cache_headers, **validator_headers})
actions.update_from_response(response)
@@ -280,11 +266,12 @@
@pytest.mark.parametrize('directive', IGNORED_DIRECTIVES)
def test_ignored_headers(directive):
"""Ensure that currently unimplemented Cache-Control headers do not affect behavior"""
- request = PreparedRequest()
- request.url = 'https://img.site.com/base/img.jpg'
- request.headers = {'Cache-Control': directive}
+ request = Request(
+ method='GET', url='https://img.site.com/base/img.jpg', headers={'Cache-Control': directive}
+ ).prepare()
+
settings = CacheSettings(expire_after=1, cache_control=True)
- actions = CacheActions.from_request('key', request, RequestSettings(settings))
+ actions = CacheActions.from_request('key', request, settings)
assert actions.expire_after == 1
assert actions.skip_read is False
diff --git a/tests/unit/test_expiration.py b/tests/unit/test_expiration.py
index 05490dd..037444d 100644
--- a/tests/unit/test_expiration.py
+++ b/tests/unit/test_expiration.py
@@ -3,7 +3,11 @@
import pytest
-from requests_cache.expiration import DO_NOT_CACHE, get_expiration_datetime, get_url_expiration
+from requests_cache.expiration import (
+ EXPIRE_IMMEDIATELY,
+ get_expiration_datetime,
+ get_url_expiration,
+)
from tests.conftest import HTTPDATE_DATETIME, HTTPDATE_STR
@@ -11,7 +15,7 @@
def test_get_expiration_datetime__no_expiration(mock_datetime):
assert get_expiration_datetime(None) is None
assert get_expiration_datetime(-1) is None
- assert get_expiration_datetime(DO_NOT_CACHE) == mock_datetime.utcnow()
+ assert get_expiration_datetime(EXPIRE_IMMEDIATELY) == mock_datetime.utcnow()
@pytest.mark.parametrize(
diff --git a/tests/unit/test_patcher.py b/tests/unit/test_patcher.py
index 1004dd8..6043c86 100644
--- a/tests/unit/test_patcher.py
+++ b/tests/unit/test_patcher.py
@@ -73,6 +73,14 @@
assert original_request.call_count == 0
+def test_is_installed():
+ assert requests_cache.is_installed() is False
+ requests_cache.install_cache(name=CACHE_NAME, use_temp=True)
+ assert requests_cache.is_installed() is True
+ requests_cache.uninstall_cache()
+ assert requests_cache.is_installed() is False
+
+
@patch.object(BaseCache, 'remove_expired_responses')
def test_remove_expired_responses(remove_expired_responses, tempfile_path):
requests_cache.install_cache(tempfile_path, expire_after=360)
diff --git a/tests/unit/test_session.py b/tests/unit/test_session.py
index 6a20107..4fa1221 100644
--- a/tests/unit/test_session.py
+++ b/tests/unit/test_session.py
@@ -18,9 +18,11 @@
from requests_cache.backends import BACKEND_CLASSES, BaseCache, SQLiteDict, SQLitePickleDict
from requests_cache.backends.base import DESERIALIZE_ERRORS
from requests_cache.cache_keys import create_key
+from requests_cache.expiration import DO_NOT_CACHE, EXPIRE_IMMEDIATELY
from tests.conftest import (
MOCKED_URL,
MOCKED_URL_404,
+ MOCKED_URL_500,
MOCKED_URL_ETAG,
MOCKED_URL_HTTPS,
MOCKED_URL_JSON,
@@ -52,11 +54,11 @@
assert session.cache.cache_dir == Path("~").expanduser()
-@patch.dict(BACKEND_CLASSES, {'mongo': get_placeholder_class()})
+@patch.dict(BACKEND_CLASSES, {'mongodb': get_placeholder_class()})
def test_init_missing_backend_dependency():
"""Test that the correct error is thrown when a user does not have a dependency installed"""
with pytest.raises(ImportError):
- CachedSession(backend='mongo')
+ CachedSession(backend='mongodb')
def test_repr(mock_session):
@@ -412,13 +414,46 @@
# -----------------------------------------------------
+def test_allowable_codes(mock_session):
+ mock_session.settings.allowable_codes = (200, 404)
+
+ # This request should be cached
+ mock_session.get(MOCKED_URL_404)
+ assert mock_session.cache.has_url(MOCKED_URL_404)
+ assert mock_session.get(MOCKED_URL_404).from_cache is True
+
+ # This request should be filtered out on both read and write
+ mock_session.get(MOCKED_URL_500)
+ assert not mock_session.cache.has_url(MOCKED_URL_500)
+ assert mock_session.get(MOCKED_URL_500).from_cache is False
+
+
+def test_allowable_methods(mock_session):
+ mock_session.settings.allowable_methods = ['GET', 'OPTIONS']
+
+ # This request should be cached
+ mock_session.options(MOCKED_URL)
+ assert mock_session.cache.has_url(MOCKED_URL, method='OPTIONS')
+ assert mock_session.options(MOCKED_URL).from_cache is True
+
+ # This request should be filtered out on both read and write
+ mock_session.put(MOCKED_URL)
+ assert not mock_session.cache.has_url(MOCKED_URL, method='PUT')
+ assert mock_session.put(MOCKED_URL).from_cache is False
+
+
def test_filter_fn(mock_session):
mock_session.settings.filter_fn = lambda r: r.request.url != MOCKED_URL_JSON
- mock_session.get(MOCKED_URL)
- mock_session.get(MOCKED_URL_JSON)
+ # This request should be cached
+ mock_session.get(MOCKED_URL)
assert mock_session.cache.has_url(MOCKED_URL)
+ assert mock_session.get(MOCKED_URL).from_cache is True
+
+ # This request should be filtered out on both read and write
+ mock_session.get(MOCKED_URL_JSON)
assert not mock_session.cache.has_url(MOCKED_URL_JSON)
+ assert mock_session.get(MOCKED_URL_JSON).from_cache is False
def test_filter_fn__retroactive(mock_session):
@@ -426,7 +461,6 @@
mock_session.get(MOCKED_URL_JSON)
mock_session.settings.filter_fn = lambda r: r.request.url != MOCKED_URL_JSON
mock_session.get(MOCKED_URL_JSON)
-
assert not mock_session.cache.has_url(MOCKED_URL_JSON)
@@ -457,19 +491,41 @@
assert state[hook] == 5
+def test_expire_after_alias(mock_session):
+ """CachedSession has an `expire_after` property for backwards-compatibility"""
+ mock_session.expire_after = 60
+ assert mock_session.expire_after == mock_session.settings.expire_after == 60
+
+
def test_do_not_cache(mock_session):
- """expire_after=0 should bypass the cache on both read and write"""
- # Skip read
+ """DO_NOT_CACHE should bypass the cache on both read and write"""
mock_session.get(MOCKED_URL)
assert mock_session.cache.has_url(MOCKED_URL)
- assert mock_session.get(MOCKED_URL, expire_after=0).from_cache is False
+
+ # Skip read
+ response = mock_session.get(MOCKED_URL, expire_after=DO_NOT_CACHE)
+ assert response.from_cache is False
# Skip write
- mock_session.settings.expire_after = 0
+ mock_session.settings.expire_after = DO_NOT_CACHE
mock_session.get(MOCKED_URL_JSON)
assert not mock_session.cache.has_url(MOCKED_URL_JSON)
+def test_expire_immediately(mock_session):
+ """EXPIRE_IMMEDIATELY should save a response only if it has a validator"""
+ # Without validator
+ mock_session.settings.expire_after = EXPIRE_IMMEDIATELY
+ mock_session.get(MOCKED_URL)
+ response = mock_session.get(MOCKED_URL)
+ assert not mock_session.cache.has_url(MOCKED_URL)
+ assert response.from_cache is False
+
+ # With validator
+ mock_session.get(MOCKED_URL_ETAG)
+ response = mock_session.get(MOCKED_URL_ETAG)
+
+
@pytest.mark.parametrize(
'response_code, cache_hit, cache_expired, expected_from_cache',
[
@@ -502,12 +558,13 @@
"""If the default is 0, only URLs matching patterns in urls_expire_after should be cached"""
mock_session.settings.urls_expire_after = {
MOCKED_URL_JSON: 60,
- '*': 0,
+ '*': DO_NOT_CACHE,
}
mock_session.get(MOCKED_URL_JSON)
assert mock_session.get(MOCKED_URL_JSON).from_cache is True
mock_session.get(MOCKED_URL)
assert mock_session.get(MOCKED_URL).from_cache is False
+ assert not mock_session.cache.has_url(MOCKED_URL)
def test_remove_expired_responses(mock_session):
@@ -685,55 +742,32 @@
def test_request_refresh(mock_session):
- """The refresh option should send and cache a new request. Any expire_after value provided
- should overwrite the previous value."""
- response_1 = mock_session.get(MOCKED_URL, expire_after=-1)
- response_2 = mock_session.get(MOCKED_URL, expire_after=360, refresh=True)
- response_3 = mock_session.get(MOCKED_URL)
-
- assert response_1.from_cache is False
- assert response_2.from_cache is False
- assert response_3.from_cache is True
- assert response_3.expires is not None
-
-
-def test_request_refresh__prepared_request(mock_session):
- """The refresh option should also work for PreparedRequests with CachedSession.send()"""
- request = Request(method='GET', url=MOCKED_URL, headers={}, data=None).prepare()
- response_1 = mock_session.send(request)
- response_2 = mock_session.send(request, expire_after=360, refresh=True)
- response_3 = mock_session.send(request)
-
- assert response_1.from_cache is False
- assert response_2.from_cache is False
- assert response_3.from_cache is True
- assert response_3.expires is not None
-
-
-def test_request_revalidate(mock_session):
- """The revalidate option should immediately send a conditional request, if possible"""
+ """The refresh option should send a conditional request, if possible"""
response_1 = mock_session.get(MOCKED_URL_ETAG, expire_after=60)
response_2 = mock_session.get(MOCKED_URL_ETAG)
mock_session.mock_adapter.register_uri('GET', MOCKED_URL_ETAG, status_code=304)
- response_3 = mock_session.get(MOCKED_URL_ETAG, revalidate=True, expire_after=60)
+ response_3 = mock_session.get(MOCKED_URL_ETAG, refresh=True, expire_after=60)
response_4 = mock_session.get(MOCKED_URL_ETAG)
assert response_1.from_cache is False
assert response_2.from_cache is True
assert response_3.from_cache is True
+ assert response_4.from_cache is True
# Expect expiration to get reset after revalidation
assert response_2.expires < response_4.expires
-def test_request_revalidate__no_validator(mock_session):
- """The revalidate option should have no effect if the cached response has no validator"""
+def test_request_refresh__no_validator(mock_session):
+ """The refresh option should result in a new (unconditional) request if the cached response has
+ no validator
+ """
response_1 = mock_session.get(MOCKED_URL, expire_after=60)
response_2 = mock_session.get(MOCKED_URL)
mock_session.mock_adapter.register_uri('GET', MOCKED_URL, status_code=304)
- response_3 = mock_session.get(MOCKED_URL, revalidate=True, expire_after=60)
+ response_3 = mock_session.get(MOCKED_URL, refresh=True, expire_after=60)
response_4 = mock_session.get(MOCKED_URL)
assert response_1.from_cache is False
@@ -742,14 +776,15 @@
assert response_2.expires == response_4.expires
-def test_request_revalidate__prepared_request(mock_session):
- """The revalidate option should also work for PreparedRequests with CachedSession.send()"""
+def test_request_refresh__prepared_request(mock_session):
+ """The refresh option should also work for PreparedRequests with CachedSession.send()"""
+ mock_session.settings.expire_after = 60
request = Request(method='GET', url=MOCKED_URL_ETAG, headers={}, data=None).prepare()
- response_1 = mock_session.send(request, expire_after=60)
+ response_1 = mock_session.send(request)
response_2 = mock_session.send(request)
mock_session.mock_adapter.register_uri('GET', MOCKED_URL_ETAG, status_code=304)
- response_3 = mock_session.send(request, revalidate=True, expire_after=60)
+ response_3 = mock_session.send(request, refresh=True)
response_4 = mock_session.send(request)
assert response_1.from_cache is False
@@ -758,3 +793,30 @@
# Expect expiration to get reset after revalidation
assert response_2.expires < response_4.expires
+
+
+def test_request_force_refresh(mock_session):
+ """The force_refresh option should send and cache a new request. Any expire_after value provided
+ should overwrite the previous value."""
+ response_1 = mock_session.get(MOCKED_URL, expire_after=-1)
+ response_2 = mock_session.get(MOCKED_URL, expire_after=360, force_refresh=True)
+ response_3 = mock_session.get(MOCKED_URL)
+
+ assert response_1.from_cache is False
+ assert response_2.from_cache is False
+ assert response_3.from_cache is True
+ assert response_3.expires is not None
+
+
+def test_request_force_refresh__prepared_request(mock_session):
+ """The force_refresh option should also work for PreparedRequests with CachedSession.send()"""
+ mock_session.settings.expire_after = 60
+ request = Request(method='GET', url=MOCKED_URL, headers={}, data=None)
+ response_1 = mock_session.send(request.prepare())
+ response_2 = mock_session.send(request.prepare(), force_refresh=True)
+ response_3 = mock_session.send(request.prepare())
+
+ assert response_1.from_cache is False
+ assert response_2.from_cache is False
+ assert response_3.from_cache is True
+ assert response_3.expires is not None
