Do redirected requests get the same uniqueKey? #1359
-
The site I want to crawl contains a lot of shortened URLs that are getting redirected when requested. Additionally there are multiple short URLs for the same destination. Example:
So naturally with |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 1 reply
-
Only the original URL is considered for request deduplication, so the |
Beta Was this translation helpful? Give feedback.
Only the original URL is considered for request deduplication, so the
handlePageFunction
will be executed multiple times even if the redirects point to the same URL in the end.