Stuck on not being able to build a wheel for tokenizers

[daxmonson]

I've upgraded pip and double checked Rust as that was in the error message too. Any ideas?

[daxmonson]

Actually figured it out. Downgraded the Python version and it built the wheel. Hope that helps someone.

[debanjum]

@daxmonson, good to know that downgrading python resolved your issue!
Which python version were you using before? And which python version did you downgrade to?

[daxmonson]

@debanjum was on 3.11 and down to 3.10. Now stuck on the the other issue you saw. Will get back to you there. Khoj seems pretty great, basically haven't messed around with tons of YAML etc since I saw this coming in the future. Thanks

[daxmonson]

One more try this weekend. I'm trying to run khoj and get get the long error message below which looks like it might have something to do with Huggingface or Numpy. When I try to upgrade Numpy, I get this error. Any ideas?

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
khoj-assistant 0.2.5 requires numpy==1.22.4, but you have numpy 1.24.2 which is incompatible.

RuntimeError: module compiled against API version 0x10 but this version of numpy is 0xf
RuntimeError: module compiled against API version 0x10 but this version of numpy is 0xf
ImportError: numpy.core._multiarray_umath failed to import
ImportError: numpy.core.umath failed to import
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1002, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/transformers/optimization.py", line 26, in <module>
    from .trainer_utils import SchedulerType
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/transformers/trainer_utils.py", line 47, in <module>
    import tensorflow as tf
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/tensorflow/__init__.py", line 37, in <module>
    from tensorflow.python.tools import module_util as _module_util
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/tensorflow/python/__init__.py", line 42, in <module>
    from tensorflow.python import data
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/tensorflow/python/data/__init__.py", line 21, in <module>
    from tensorflow.python.data import experimental
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/tensorflow/python/data/experimental/__init__.py", line 96, in <module>
    from tensorflow.python.data.experimental import service
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/tensorflow/python/data/experimental/service/__init__.py", line 419, in <module>
    from tensorflow.python.data.experimental.ops.data_service_ops import distribute
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/tensorflow/python/data/experimental/ops/data_service_ops.py", line 22, in <module>
    from tensorflow.python.data.experimental.ops import compression_ops
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/tensorflow/python/data/experimental/ops/compression_ops.py", line 16, in <module>
    from tensorflow.python.data.util import structure
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/tensorflow/python/data/util/structure.py", line 22, in <module>
    from tensorflow.python.data.util import nest
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/tensorflow/python/data/util/nest.py", line 34, in <module>
    from tensorflow.python.framework import sparse_tensor as _sparse_tensor
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/tensorflow/python/framework/sparse_tensor.py", line 24, in <module>
    from tensorflow.python.framework import constant_op
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/tensorflow/python/framework/constant_op.py", line 25, in <module>
    from tensorflow.python.eager import execute
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/tensorflow/python/eager/execute.py", line 21, in <module>
    from tensorflow.python.framework import dtypes
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/tensorflow/python/framework/dtypes.py", line 34, in <module>
    _np_bfloat16 = _pywrap_bfloat16.TF_bfloat16_type()
TypeError: Unable to convert function return value to a Python type! The signature was
	() -> handle

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/bin/khoj", line 5, in <module>
    from src.main import run
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/src/main.py", line 23, in <module>
    from src.configure import configure_server
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/src/configure.py", line 13, in <module>
    from src.search_type import image_search, text_search
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/src/search_type/image_search.py", line 10, in <module>
    from sentence_transformers import SentenceTransformer, util
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sentence_transformers/__init__.py", line 3, in <module>
    from .datasets import SentencesDataset, ParallelSentencesDataset
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sentence_transformers/datasets/__init__.py", line 3, in <module>
    from .ParallelSentencesDataset import ParallelSentencesDataset
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sentence_transformers/datasets/ParallelSentencesDataset.py", line 4, in <module>
    from .. import SentenceTransformer
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sentence_transformers/SentenceTransformer.py", line 33, in <module>
    class SentenceTransformer(nn.Sequential):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sentence_transformers/SentenceTransformer.py", line 564, in SentenceTransformer
    optimizer_class: Type[Optimizer] = transformers.AdamW,
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 992, in __getattr__
    module = self._get_module(self._class_to_module[name])
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1004, in _get_module
    raise RuntimeError(
RuntimeError: Failed to import transformers.optimization because of the following error (look up to see its traceback):
Unable to convert function return value to a Python type! The signature was
	() -> handle
djm@DJs-MacBook-Pro polymath % 

[debanjum]

I just loosened the dependency requirements for khoj earlier today in c156b3e.
To resolve the conflicting numpy dependency issue upgrade to the latest pre-release version of khoj with:

pip install --upgrade --pip khoj-assistant

[daxmonson]

Hi, here's what I get with that:

djm@DJs-MacBook-Pro ~ % pip install --upgrade --pip khoj-assistant

Usage:   
  pip install [options] <requirement specifier> [package-index-options] ...
  pip install [options] -r <requirements file> [package-index-options] ...
  pip install [options] [-e] <vcs project url> ...
  pip install [options] [-e] <local project path> ...
  pip install [options] <archive url/path> ...

no such option: --pip
djm@DJs-MacBook-Pro ~ %

[debanjum]

Woops, that was a typo on my end! You need to run pip install with the --pre flag (not --pip) to ensure you install the pre-release version of khoj-assistant.

pip install --upgrade --pre khoj-assistant

[debanjum]

Converting this into a q&a discussion topic for now, as this discussion is in the general troubleshooting phase to install/use khoj. Will create issues when any bugs, features come out of this discussion

[daxmonson]

Thanks. Installed fine with that one. Running khoj after that I get the below in Terminal, but the UI doesn't pop up. Rebooted and same. Python is open and is in taskbar, so it seems khoj is running, just no UI. Any other dependencies I need to install or maybe something that needs to be run again to get the UI back?

INFO:     Started server process [1442]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)

[debanjum]

What do you mean by UI doesn't pop-up?
If you started the khoj backend with khoj --no-gui the desktop UI is not expected to pop-up.
For using the khoj obsidian plugin the desktop UI is not necessary as

1. The plugin automatically configures the backend
2. Search etc is provided from within obsidian itself

Of course, if you want to see the desktop UI, you can just start khoj without the --no-gui flag, i.e just run khoj

[daxmonson]

Thanks, that makes sense, at the same time I am entering khoj (only) in Terminal and the GUI doesn't open. Also the Obsidian plugin isn't connecting now: "! Disconnected from Khoj backend. Ensure Khoj backend is running and Khoj URL is correctly set below." Pretty sure it just needs a completely fresh install after trying to work all the kinks out. Will trash the original files, install, and get back if it doesn't go right.