Repository navigation
Replies: 1 comment
|
Do not add a preliminary HEAD/exists request for every miss. It turns one S3 request into two requests for every object that does exist, and a missing key still requires a request to S3 either way. The normal pattern is exactly (1): issue GetObject and handle NoSuchKey/404. That is both race-free and minimum-request-count. A HEAD followed by GET also has a time-of-check/time-of-use gap: the object can change between the two calls. There are still costs to misses. S3 requests are billable according to the bucket/storage pricing, and each miss consumes network latency, connection capacity and application work. Exception construction in Python is generally not the part worth optimizing here; the remote round trip dominates. For a documentation website with bots probing thousands of nonexistent paths, the bigger optimization boundary is usually before S3:
Be careful with a permanent negative cache if documents can appear dynamically; give misses a TTL or invalidate them on deploy. So between your options, use GetObject + handle NoSuchKey. HEAD-before-GET only makes sense when you actually need metadata before deciding whether to fetch the body, not as an existence optimization. |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Hi,
We have a website with 1000's of pages of documentation and it seems many bots on the internet request pages that don't exist.
boto3 calls S3, looking for the doc, client.get_object, and hits NoSuchKey.
That's fine, everything works.
The question I have is - are such errors "expensive" in any way?
What's the fastest, quickest, easiest, way to deal with large numbers of errors?
or
or
All reactions