Retry bulk import one document per request on 413 - #39
Merged
Conversation
When a bulk request returns RequestEntityTooLargeError after the existing size-balancing pass, retry one document per request as a last resort. Only raise when a single-document bulk still exceeds the limit. Also falls back to per-document retry when the 413 message has no parseable byte limit, and sends oversized single docs in their own request instead of silently discarding them. Bumps version to 0.5.0.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When a bulk import receives
RequestEntityTooLargeError(HTTP 413),Esse::Import::Bulk#each_requestalready retries once with size-balanced chunks (balance_requests_size). If that retry still 413s, the error currently bubbles up and aborts the whole import — even though one or two oversized documents in the batch may be the only problem.This PR adds a third recovery tier: after the balanced retry, the bulk is split one document per request as a last resort. The error is only raised once a single-document bulk still exceeds the limit (which is the only case where the data is genuinely unimportable).
Changes:
balance_requests_size, with a distinct warning so the fallback is observable.balance_requests_sizere-raised immediately, defeating retry).last_retry_per_document: truekeyword arg oneach_requestto opt out.0.5.0; allGemfile.lockfiles refreshed.Test plan
bundle exec rspec spec/esse/import/bulk_spec.rb— 11 examples, 0 failures (includes 4 new examples + 1 updated for the changed discard behavior)STUB_STACK=elasticsearch-7.17.0 BUNDLE_GEMFILE=gemfiles/Gemfile.elasticsearch-7.x bundle exec rspec --exclude-pattern 'spec/esse/integrations/**/*') — 1017 examples, 0 failures