Does the scraper actually support basic auth?

[comatory]

The documentation mentions two env variables:

• DOCSEARCH_BASICAUTH_USERNAME
• DOCSEARCH_BASICAUTH_PASSWORD

I was looking through the code to figure out how they are encoded into Authorization headers so I can set up my private internal site correctly. However, the only mention I see is documentation_spider.py file.

This file reads the environment variables but does not seem to do anything with them. I see these are class properties assigned to http_user and http_pass so I tried searching the codebase for them, but did not find anything.

Am I right to assume that this does not actually work? Is the documentation lying or did I miss some important piece?

Any clarification would help. Meanwhile, I hope #76 gets merged, that way I could specify Authorization header directly without relying on implementation.

[comatory]

Basically I'd expect that the scraper would:

1. Dispatch all requests with Authorization header in case these two variables are provided.
2. The Authorization for basic HTTP auth would use username:password encoded as base64

I would also hope redirects would be respected. For example, the domain I'm trying to reach is internally redirecting to opaque domain (Cloudflare Worker). So I'd need for these headers to be sent there as well.