Missing package hints raised in error on requirements.txt

[philipnye]

Type: Bug

Behaviour

Open a project that does not have an associated virtual environment - i.e. my system-wide Python installation is the interpreter.

View requirements.txt, which contains the following:

numpy==1.26.0
pandas==2.1.1
pytest==7.4.4
rapidfuzz==3.5.2
streamlit==1.31.0

Missing package hints are raised for all packages, despite them being installed. Hints are e.g.

Package `streamlit` is not installed in the selected environment. Python-InstalledPackagesChecker(not-installed)

Expected

No missing package hints are raised.

Extension version: 2024.0.1
VS Code version: Code 1.86.0 (05047486b6df5eb8d44b2ecd70ea3bdf775fd937, 2024-01-31T10:28:19.990Z)
OS version: Windows_NT x64 10.0.19045
Modes:

System Info

Item	Value
CPUs	Intel(R) Core(TM) i5-8365U CPU @ 1.60GHz (8 x 1896)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: enabled_on direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok skia_graphite: disabled_off video_decode: enabled video_encode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: enabled
Load (avg)	undefined
Memory (System)	7.82GB (0.83GB free)
Process Argv	--folder-uri file:///c%3A/Users/nyep/OneDrive%20-%20Institute%20for%20Government/Documents/Development/st_fuzzy_match --crash-reporter-id 9caa3fea-c07d-4c78-9e6c-cc9cdf50faa9
Screen Reader	no
VM	0%

A/B Experiments

vsliv368:30146709
vspor879:30202332
vspor708:30202333
vspor363:30204092
vstes627:30244334
vscoreces:30445986
vscod805cf:30301675
binariesv615:30325510
vsaa593:30376534
py29gd2263:30899288
c4g48928:30535728
azure-dev_surveyone:30548225
vscrpc:30673769
a9j8j154:30646983
962ge761:30951796
pythongtdpath:30769146
welcomedialog:30910333
pythonidxpt:30866567
pythonnoceb:30805159
asynctok:30898717
pythontestfixt:30902429
pythonregdiag2:30936856
pyreplss1:30897532
pythonmypyd1:30879173
pythoncet0:30885854
pythontbext0:30879054
dsvsc016:30899300
dsvsc017:30899301
dsvsc018:30899302
dsvsc019bcf:30953938
7j2b6412:30949929
ccp2r6:30958158

[luabud]

Hi @philipnye, can you ensure you have the right selected interpreter in the workspace (the one that has the installed requirements)? You can check the status bar on the bottom right and click on the version information to find the right one:

[philipnye]

Hi @luabud, I can confirm that I do. I don't get any missing package messages on my scripts, and the scripts run fine using that packages - it's just requirements.txt where there seems to be an issue. Thanks

[karthiknadig]

@philipnye Can you do a python -m pip list? and also python -c "import sys;print(sys.executable)" from your terminal. I am specifically looking at output of python -m pip list, see here why https://snarky.ca/why-you-should-use-python-m-pip/

[philipnye]

@karthiknadig Thanks.

python -m pip list gives:

altair                    5.2.0
asttokens                 2.4.0
attrs                     23.1.0
backcall                  0.2.0
beautifulsoup4            4.12.2
blinker                   1.7.0
build                     1.0.3
cachetools                5.3.2
cairocffi                 1.6.1
CairoSVG                  2.7.1
certifi                   2023.7.22
cffi                      1.16.0
cfgv                      3.4.0
charset-normalizer        3.3.2
click                     7.1.2
click-plugins             1.1.1
cligj                     0.7.2
colorama                  0.4.6
comm                      0.1.4
contourpy                 1.2.0
cryptography              41.0.7
cssselect2                0.7.0
csv-to-sqlite             2.1.3
csvs-to-sqlite            1.3
cycler                    0.12.1
dateparser                1.2.0
debugpy                   1.8.0
decorator                 5.1.1
defusedxml                0.7.1
Deprecated                1.2.14
distlib                   0.3.8
drawsvg                   2.3.0
et-xmlfile                1.1.0
executing                 2.0.0
filelock                  3.13.1
fiona                     1.9.5
fonttools                 4.45.0
GDAL                      3.7.3
geopandas                 0.14.1
gitdb                     4.0.11
GitPython                 3.1.41
greenlet                  3.0.0
identify                  2.5.33
idna                      3.4
importlib-metadata        6.11.0
iniconfig                 2.0.0
ipykernel                 6.25.2
ipython                   8.16.1
jedi                      0.19.1
Jinja2                    3.1.2
joblib                    1.3.2
jsonschema                4.20.0
jsonschema-specifications 2023.12.1
jupyter_client            8.3.1
jupyter_core              5.3.2
kiwisolver                1.4.5
markdown-it-py            3.0.0
MarkupSafe                2.1.3
matplotlib                3.8.2
matplotlib-inline         0.1.6
mdurl                     0.1.2
nameparser                1.1.3
nest-asyncio              1.5.8
nodeenv                   1.8.0
numpy                     1.26.0
odfpy                     1.4.1
openpyxl                  3.1.2
packaging                 23.2
pandas                    2.1.1
parso                     0.8.3
pickleshare               0.7.5
Pillow                    10.1.0
pip                       24.0
platformdirs              3.11.0
pluggy                    1.3.0
pre-commit                3.6.0
prompt-toolkit            3.0.39
protobuf                  4.25.2
psutil                    5.9.5
pure-eval                 0.2.2
py_lru_cache              0.1.4
pyarrow                   14.0.2
pycparser                 2.21
pydeck                    0.8.1b0
PyGithub                  2.1.1
Pygments                  2.16.1
pyjson5                   1.6.4
PyJWT                     2.8.0
PyNaCl                    1.5.0
pyodbc                    5.0.1
pyparsing                 3.1.1
pyproj                    3.6.1
pyproject_hooks           1.0.0
pytest                    7.4.4
python-dateutil           2.8.2
pytz                      2023.3.post1
pywin32                   306
PyYAML                    6.0.1
pyzmq                     25.1.1
rapidfuzz                 3.5.2
referencing               0.32.1
regex                     2023.12.25
requests                  2.31.0
rich                      13.7.0
rpds-py                   0.16.2
scikit-learn              1.4.0
scipy                     1.12.0
setuptools                68.2.2
shapely                   2.0.2
six                       1.16.0
smmap                     5.0.1
soupsieve                 2.5
SQLAlchemy                2.0.21
stack-data                0.6.3
streamlit                 1.31.0
tenacity                  8.2.3
threadpoolctl             3.2.0
tinycss2                  1.2.1
toml                      0.10.2
toolz                     0.12.0
tornado                   6.3.3
traitlets                 5.11.2
ttf-opensans              2020.10.30
typing_extensions         4.8.0
tzdata                    2023.3
tzlocal                   5.2
urllib3                   2.0.7
validators                0.22.0
virtualenv                20.25.0
watchdog                  3.0.0
wcwidth                   0.2.8
webencodings              0.5.1
wrapt                     1.16.0
xlrd                      2.0.1
XlsxWriter                3.1.9
zipp                      3.17.0

python -c "import sys;print(sys.executable)" gives C:\Program Files\Python312\python.exe.

I'd been wondering whether this might be to do with the fact that python is installed in C:\Program Files\Python312 but packages are installed in C:\Users\<username> \AppData\Roaming\Python\Python312 as they're installed without administrator rights. Will the installed packages checker be checking that location?

[karthiknadig]

@philipnye You can see the script of what it does here:
https://github.com/microsoft/vscode-python/blob/main/pythonFiles/installed_check.py

It uses importlib_metadata to get the details, you should be able to run this script from the extension manually in your terminal and see the response. It should be able to find the packages the same way your code does. My suspicion is that it fails to match something like the version or some constraint.

The script should be located at C:\Users\<username>\.vscode\extensions\ms-python.python-<version>\pythonFiles\installed_check.py. Run this, to see it locally:

python <path to> \installed_check.py <path to>\requirement.txt

[philipnye]

@karthiknadig Thanks. Running that returns an empty list - which I think indicates all of the packages could be found?

[karthiknadig]

That is correct. So, in the output python logs do you see an entry for this installed_check.py. and see which exact python executable was used to launch that.

[philipnye]

@karthiknadig Sorry, where do I find the logs?

[karthiknadig]

It should be in the Output tab, you would need to select Python:

Please share the lines that run interpreter info and installed check. Also the line that says "Python Interpreter Path"

[philipnye]

@karthiknadig Gotcha, thanks (I'd been running installed_check.py via cmd).

These are my logs:

2024-02-09 06:36:38.772 [info] Experiment 'pythonRecommendTensorboardExt' is active
2024-02-09 06:36:38.773 [info] Experiment 'pythonREPLSmartSend' is active
2024-02-09 06:36:38.773 [info] Experiment 'pythonSurveyNotificationcf' is active
2024-02-09 06:36:38.773 [info] Experiment 'pythonTerminalEnvVarActivation' is active
2024-02-09 06:36:38.773 [info] Experiment 'pythonTestAdapter' is active
2024-02-09 06:36:38.773 [info] Default formatter is set to null for workspace c:\Users\<username>\OneDrive - Institute for Government\Documents\Development\st_fuzzy_match
2024-02-09 06:36:38.773 [info] Running installed packages checker:  C:\Program Files\Python312\python.exe c:\Users\<username>\.vscode\extensions\ms-python.python-2024.0.1\pythonFiles\installed_check.py c:\Users\<username>\OneDrive - Institute for Government\Documents\Development\st_fuzzy_match\requirements.txt
2024-02-09 06:36:38.773 [info] > "C:\Program Files\Python312\python.exe" ~\.vscode\extensions\ms-python.python-2024.0.1\pythonFiles\installed_check.py "~\OneDrive - Institute for Government\Documents\Development\st_fuzzy_match\requirements.txt"
2024-02-09 06:36:38.773 [info] Test server listening.
2024-02-09 06:36:38.773 [info] Python interpreter path: C:\Program Files\Python312\python.exe
2024-02-09 06:36:46.044 [info] > conda info --json
2024-02-09 06:37:00.597 [info] Starting Pylance language server.

[karthiknadig]

This is really strange, it seems to be using the exact same python with same conditions, and somehow not able to find streamlit. I will need to investigate further.

Different question, is there any reason for not using a virtual environment?

[philipnye]

This is really strange, it seems to be using the exact same python with same conditions, and somehow not able to find streamlit. I will need to investigate further.

Thanks (I think it's not able to find any of the packages - I get the hint on all five of them).

Different question, is there any reason for not using a virtual environment?

I've had problems using virtual environments on OneDrive due to the large number of files involved, so I don't use them in all projects.

[karthiknadig]

I am able to repro this issue. Looks like it only occurs if shell execute is turned off when running something from node. Basically, user site packages are not included. From the docs it looks like it is intentional, but I need to dig more and see why this happens.

[karthiknadig]

Figured out the issue, created a PR.

[eleanorjboyd]

Hi @philipnye are you able to verify if this fix works? You can do so by trying on the pre-release of the python extension on vscode insiders. Thanks!

[TylerLeonhardt]

@eleanorjboyd are there verification steps here if they don't get back to us in time?

[TylerLeonhardt]

ope, just realized @karthiknadig fixed this. Same question to you ^ 😄

[eleanorjboyd]

@TylerLeonhardt - Karthik is gone on vacation so he won't be able to provide steps. I understand most of it other than ll execute is turned off when running something from node. which I am not sure how to do this step

[TylerLeonhardt]

Removed the steps-needed so this issue still shows up in endgame queries.

[rzhao271]

Verified the original comment is now fixed.