Jedi features fail across notebook cells

[tmchartrand]

Type: Bug

Behaviour

When using the Jedi language server, all LS features work properly in a single cell, but don't recognize variables defined in one cell and used in another. Go to def gives a "no definition found" and hover previews are not available (autocomplete does still work).

It seems to me that the fix for #23897 worked partially but not completely - I think I missed this when I first tested that fix due to only testing features in a single cell initially.

Steps to reproduce:

1. Change the python language server setting to jedi.
2. Create a jupyter notebook with two cells, run import os in the first cell and type os in the second cell.
3. Note that LS features work for os in the first cell and not the second.
4. Add import os additionally to the second cell - features work there again.

Diagnostic data

Nothing seen in the logs, but I'm not sure how to change the Python extension logging level

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

2024-10-25 13:17:37.279 [info] Starting Jedi language server for aind-data-schema.
2024-10-25 13:27:56.724 [info] > . ./.venv/bin/activate && echo 'e8b39361-0157-4923-80e1-22d70d46dee6' && python ~/.vscode/extensions/ms-python.python-2024.16.1-linux-x64/python_files/printEnvVariables.py
2024-10-25 13:27:56.724 [info] shell: bash

Extension version: 2024.16.1
VS Code version: Code 1.94.2 (384ff7382de624fb94dbaf6da11977bba1ecd427, 2024-10-09T16:08:44.566Z)
OS version: Linux x64 5.14.0-427.31.1.el9_4.x86_64
Modes:

• Python version (& distribution if applicable, e.g. Anaconda): 3.9.18
• Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Venv
• Value of the python.languageServer setting: Jedi

User Settings

condaPath: "<placeholder>"

languageServer: "Jedi"

Installed Extensions

Extension Name	Extension Id	Version
autoDocstring - Python Docstring Generator	njpwerner.autodocstring	0.6.1
Black Formatter	ms-python.black-formatter	2024.2.0
Data Wrangler	ms-toolsai.datawrangler	1.0.2
Git Graph	mhutchie.git-graph	1.30.0
GitHub Codespaces	GitHub.codespaces	1.16.19
GitHub Copilot	GitHub.copilot	1.241.0
GitHub Copilot Chat	GitHub.copilot-chat	0.21.2
GitHub Pull Requests	GitHub.vscode-pull-request-github	0.98.0
GitLens — Git supercharged	eamodio.gitlens	15.6.2
isort	ms-python.isort	2023.10.1
JavaScript Debugger	ms-vscode.js-debug	1.94.0
JavaScript Debugger Companion Extension	ms-vscode.js-debug-companion	1.1.3
Jupyter	ms-toolsai.jupyter	2024.9.1
Jupyter Cell Tags	ms-toolsai.vscode-jupyter-cell-tags	0.1.9
Jupyter Keymap	ms-toolsai.jupyter-keymap	1.1.2
Jupyter Notebook Renderers	ms-toolsai.jupyter-renderers	1.0.19
Jupyter Slide Show	ms-toolsai.vscode-jupyter-slideshow	0.1.6
Live Share	ms-vsliveshare.vsliveshare	1.0.5941
Markdown All in One	yzhang.markdown-all-in-one	3.6.2
Pylance	ms-python.vscode-pylance	2024.10.1
Python	ms-python.python	2024.16.1
Python Debugger	ms-python.debugpy	2024.12.0
Python Environment Manager	donjayamanne.python-environment-manager	1.2.4
Rainbow CSV	mechatroner.rainbow-csv	3.11.0
Remote - SSH	ms-vscode-remote.remote-ssh	0.107.1
Remote - SSH: Editing Configuration Files	ms-vscode-remote.remote-ssh-edit	0.86.0
Remote Explorer	ms-vscode.remote-explorer	0.4.1
Table Visualizer for JavaScript Profiles	ms-vscode.vscode-js-profile-table	1.0.10
YAML	redhat.vscode-yaml	1.14.0

System Info

Item	Value
CPUs	Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz (8 x 0)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: enabled_on direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok skia_graphite: disabled_off video_decode: enabled video_encode: disabled_software vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: disabled_off webnn: disabled_off
Load (avg)	0, 1, 1
Memory (System)	30.98GB (19.81GB free)
Process Argv	--crash-reporter-id 549ec349-1818-4235-911a-668a77cf10fe
Screen Reader	no
VM	0%
DESKTOP_SESSION	gnome
XDG_CURRENT_DESKTOP	GNOME
XDG_SESSION_DESKTOP	gnome
XDG_SESSION_TYPE	wayland

A/B Experiments

vsliv368cf:30146710
vspor879:30202332
vspor708:30202333
vspor363:30204092
vscod805:30301674
binariesv615:30325510
vsaa593cf:30376535
py29gd2263:31024239
c4g48928:30535728
azure-dev_surveyone:30548225
2i9eh265:30646982
962ge761:30959799
pythongtdpath:30769146
pythonnoceb:30805159
asynctok:30898717
pythonmypyd1:30879173
2e7ec940:31000449
pythontbext0:30879054
cppperfnew:31000557
dsvsc020:30976470
pythonait:31006305
dsvsc021:30996838
0ee40948:31013168
dvdeprecation:31068756
dwnewjupytercf:31046870
2f103344:31071589
impr_priority:31102340
nativerepl1:31139838
refactort:31108082
pythonrstrctxt:31112756
wkspc-onlycs-t:31132770
wkspc-ranged-t:31151552
cf971741:31144450
autoexpandse:31146404
iacca2:31156134
notype1cf:31157160
5fd0e150:31155592
dwcopilot:31164048
iconenabled:31158251

[tmchartrand]

Also note that as in #23897, this issue is not present in older extension versions: jupyter v2023.9.100, python v2024.2.1

[karthiknadig]

This has not changed for a while in the python extension. Moving this to Jupyter for further investigation.

[tmchartrand]

@DonJayamanne curious for your thoughts on how this relates to the recent "middleware addon component" change - perhaps unrelated, but that issue was covering up this one? As before though, this actually affects all non-Pylance LS features in the notebook context, and would be great to have fixed.

[DonJayamanne]

@karthiknadig this is LSP related, not Jupyter, given jupyter doesn't own LSP for Jedi, moving back to Python extension.
If you read the user comments, it works in the first cell and not the second cell, meaning LSP isn't able to determine the fact that second cell is part of the notebook or the like.

We already had some other issues in the past in this same space, #23898

@tmchartrand
Sorry you are running into this,

"middleware addon component"

What changes are those?
Please can you try a different version of Jupyter and or Python extension, perhaps we can find out where this went wrong.
I'll try this at my end as well.

[karthiknadig]

@DonJayamanne This might be a limitation of jedi-language-server at this point. LSP now has full support for notebooks but the jedi language server implementation still only looks at whole documents. It gets partial view, because it is seeing each cell as a whole document. If jedi-language-server uses the notebook features in LSP, it should be able to provide a richer experience here.

Created a feature request on jedi-language-server: pappasam/jedi-language-server#324

[DonJayamanne]

Weird, I thought this did work in the past with Jedi LSP.

[DonJayamanne]

I can confirm, this is a regression, I can verify that it worked with the older versions

non-Pylance intellisense only works as expected in the notebook for jupyter v2023.9.100, python v2024.2.1 or older. For newer

See here #23897

[DonJayamanne]

Is it possible some code was removed that was actually crucial for Jedi LSP, e.g. some of the middle ware or the like.

[tmchartrand]

Sounds like you made the connection, but "middleware addon component" was referring to the PR resolving the previous issue, #23898

[karthiknadig]

@DonJayamanne I tried versions all the way back to 2023.0.* (keeping jupyter version latest), I was not able to get it to work just moving python back. Moving Jupyter back to 2023.9.100 and python to 2024.2.1, it seems to look for Notebook Concatenator. Triggering goto definition on something in second cell causes this:

This is what I was trying to mention, previously we used to concatenate cells into a virtual document. I think this got removed because LSP now fully supports notebooks.

Was the concatenator in Jupyter or in Python extension. I can't seem to find it.

[DonJayamanne]

@karthiknadig Yes, I believe this was in Jupyter extension.
Let me see what we can do to address this. Lets leave this issue in this repo for now.
I think we might want to just create a Jedi LS extension for now, instead of moving stuff back into Jupyter, when this can be fixed by Jedi in the future (when the Notebook LSP protocol parts are implemented).

Will try to whip up an extension for this and see where we can go from there.