[FIX] Online Search Broken

[jakobd04]

Describe the bug

A clear and concise description of what the bug is. Please include what you were expecting to happen vs. what actually happened.

Whenever Khoj tries to use online search, I get the following in my logs:

[16:42:53.830754] INFO khoj.processor.tools.online_searc online_search.py:81
h: 🌐 Searching the Internet for
['2024 presidential election
news', 'latest updates on 2024
presidential candidates', '2024
presidential election key issues
and debates']
[16:42:54.098720] ERROR khoj.processor.tools.online_sear online_search.py:328
ch:
{"data":null,"code":401,"name":"
AuthenticationRequiredError","st
atus":40103,"message":"Absence
of bearer
token","readableMessage":"Authen
ticationRequiredError: Absence
of bearer token"}
[16:42:54.110862] ERROR khoj.processor.tools.online_sear online_search.py:328
ch:
{"data":null,"code":401,"name":"
AuthenticationRequiredError","st
atus":40103,"message":"Absence
of bearer
token","readableMessage":"Authen
ticationRequiredError: Absence
of bearer token"}
[16:42:54.349986] ERROR khoj.processor.tools.online_sear online_search.py:328
ch:
{"data":null,"code":401,"name":"
AuthenticationRequiredError","st
atus":40103,"message":"Absence
of bearer
token","readableMessage":"Authen
ticationRequiredError: Absence
of bearer token"}

To Reproduce

Steps to reproduce the behavior:
I do not know how, this just started happening for me after a week of working completely fine.

Screenshots

If applicable, add screenshots to help explain your problem.

Platform

• Server:
  • Cloud-Hosted (https://app.khoj.dev)
  • Self-Hosted Docker
  • Self-Hosted Python package
  • Self-Hosted source code
• Client:
  • Obsidian
  • Emacs
  • Desktop app
  • Web browser
  • WhatsApp
• OS:
  • Windows
  • macOS
  • Linux
  • Android
  • iOS

If self-hosted

• Server Version [e.g. 1.0.1]: 1.26.2

Additional context

Add any other context about the problem here.

[debanjum]

That's strange. Just to confirm:

1. Did you upgrade Khoj between when it was working and when it stopped working?
2. Are you using the default setup for online search or did you setup SERPER_DEV_API_KEY or JINA_API_KEY to do online search?

[jakobd04]

Didn't upgrade and I use the default online search it would

[debanjum]

Found the issue. We use Jina's reader API for online search by default because they didn't require an API key, which made the self-hosting setup easy. But it seems like they've turned off the ability to do online search without an API key 🙈.

The two options to mitigate this issue and use online search currently would be:

1. Get a free API key from https://jina.ai/reader and set JINA_API_KEY environment variable in your Khoj docker_compose.yml before restarting your Khoj container
2. Get an API key from https://serper.dev/ (sign-in required) and set SERPER_DEV_API_KEY environment variable in your Khoj docker_compose.yml before restarting your Khoj container.

Serper.dev is faster than Jina in our testing but you can choose whatever is most convenient for you.

I'll push an update to Khoj to either bypass using Jina by default or disable online search if no online search API key is provided in a few days. Let me know if that works for you?

[jakobd04]

Used the free Jina API Key and it works again now, thanks a lot for the fast help. This is such an amazing project!