[core] Networking enhancements

[dirkf]

In #33045, we found an AWS server that rejected HTTP/1.1.

Originally posted by @dirkf in #33045 (comment)

I confirmed that adding --http1.1 to the working curl command causes the same 464 error.

Other sites are deploying web-breaking technology via Cloudflare, Fastly, etc, that are primarily intended to protect against large-scale DDoS attacks but have a possibly not unwelcome effect on tools like yt-dl. The magic words include "Enable JavaScript to continue", "Verifying your browser" (I'm at your site, so it's a browser, what is there to verify?), and so on. Some of the main video sites (Vimeo, eg) are fingerprinting requests to deny access.

yt-dlp has re-implemented networking so as to take advantage of the more modern standard and external library modules available to it. This allows it to support, eg, curl-impersonate to make network requests, and so satisfy the door policy of some of the more choosy sites. While that might be impractical for some of yt-dl's target platforms, it would be nice to have the option.

In the linked comment I continued:

One way of looking at this is that Python's urllib2, which doesn't support HTTP/2, is ancient: yt-dl should get with the programme and join the third decade of the 21st century. OTOH, it's the best we can do with the standard library of supported Python versions.
...
Should we really need to support HTTP/2, we can wrap the curl command-line. There is a potentially better technical solution in which libcurl is called directly from Python, but only the 5-year-old version 7.43.0.5 of the pycurl PyPi package that would avoid having to re-implement that is compatible with our Python platforms. Wrapping the curl command-line would also allow for versions with additional functionality, such as curl-impersonate.

This issue invites suggestions of other potential approaches and especially of features that should be supported.

These issues were linked as resolved by yt-dlp's network rework:

• when youtube-dl sends too many dns request to find video quality, dns server blocks the host running youtube-dl temporarily #13734
• Support connection reuse / keep-alive #14523
• Whether https proxy --proxy=https://user:[email protected] is not supported? [it isn't] #32549

Related:

• socks5h proxy support.