[6.x] Improve performance of data reference updates - #15089
Open
daun wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TLDR
Speed up asset & term reference updates with a cheap string check on each item's raw data, performed before resolving the blueprint and iterating over any fields.
3x faster in benchmarks + memory stays flat.
The problem
On larger sites (20k+ entries) the
UpdateAssetReferences/UpdateTermReferencesjobs regularly exceed queue worker timeouts and memory limits. Once a job exceeds the timeout it gets discarded and the references silently never get updated :(The listener already iterates lazily over items, so that part is efficient. The expensive part is that for every item it resolves the blueprint and instantiates all fields to figure out which fields could realistically hold a reference. On an asset rename, 99% of items don't reference the asset at all, but they all pay the full cost of resolving the blueprint.
The fix
An early exit in
DataReferenceUpdater: json-encode the item's raw data, check if the resulting string contains the value being replaced, and bail before touching the blueprint if it's not found.Is it safe?
This sounds too dumb to be correct, I thought, but it works out because the updater itself never augments anything: it reads and writes raw stored data, i.e. the blueprint is only used to decide which keys to look at. Every replacement path requires the old value to appear verbatim in the data: assets compare using
===, including in Bard, markdown searches exactstatamic://urls. So if the old value isn't a substring of the encoded data, there is nothing the updater could ever rewrite.Numbers
The more complex the blueprints, the bigger the speedup, since the skipped work grows while the string check stays constant. Tge production site in question went from exceeding a 128MB memory limit to finishing within it.
On Eloquent
On Eloquent sites there's an even better early exit available: a
whereon thedatacolumn before hydrating anything. We run that in production with a subclassed listener and it's really effective but tied directly to the Eloquent driver using thedatacolumn setup. The check in this PR is driver agnostic. It runs on hydrated data, so flat files, a data column or hoisted columns all behave the same.