Skip to main content

Deduplication

When two systems request the same set of assets at the same time, AsyncLoad Toolkit merges them into a single native streaming request instead of issuing two identical loads.

How it works​

Every RequestLoad call computes a hash of the sorted path set. Before issuing a new streaming request, the subsystem checks whether an identical request is already in flight.

  • Match found: the new caller's callbacks are registered on the existing handle. Both callers receive On Loaded when the single request completes. No duplicate I/O.
  • No match: a new streaming request is created normally.
System A: Load [/Game/Meshes/Rock, /Game/Meshes/Tree]
System B: Load [/Game/Meshes/Rock, /Game/Meshes/Tree]   ← same paths

Result: ONE streaming request. Both A and B get On Loaded.

Enabling / disabling​

Deduplication is enabled by default via FAsyncKitLoadParams::bDeduplicate.

To disable it for a specific call (e.g., you intentionally want an independent request):

Params.bDeduplicate = false

You can also change the project-wide default in Project Settings → AsyncLoad Toolkit → Default Deduplicate.

Edge cases​

  • Dedup only applies to requests with the exact same set of paths (order-independent). Adding or removing even one path creates a different hash and does not deduplicate.
  • If a previous request for the same paths has already completed, a new request is issued normally (the completed handle is no longer in the in-flight map).
  • If a previous request was cancelled, a new request is issued normally.
note

The dedup key is computed from the full set of paths, not individual assets. Requesting [A, B] and then requesting [A] separately does not deduplicate — [A] is a different set than [A, B].