Support symlink Python environments

[damienrj]

Hello, we have been using Hermit to install portable versions of python and this is working great except that with VS Code we can not select that python as an interpreter.

For example
/Users/damien/Development/example/bin/python3 because in reality it is:

python3 -> .python3@3.8.pkg
pip3 -> .python3@3.8.pkg
 .python3@3.8.pkg -> hermit

I think this is similar to Dart-Code/Dart-Code#3961

❯ code --version
1.67.1
da15b6fd3ef856477bf6f4fb29ba1b7af717770d
arm64

[karthiknadig]

Thanks for the feature request! We are going to give the community 60 days from when this issue was created to provide 7 👍 upvotes on the opening comment to gauge general interest in this idea. If there's enough upvotes then we will consider this feature request in our future planning. If there's unfortunately not enough upvotes then we will close this issue.

[karrtikr]

I was wondering why the extension is not able to select it. can you send the logs after attempting to select it?

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

XXX

[damienrj]

Thanks for the feature request! We are going to give the community 60 days from when this issue was created to provide 7 👍 upvotes on the opening comment to gauge general interest in this idea. If there's enough upvotes then we will consider this feature request in our future planning. If there's unfortunately not enough upvotes then we will close this issue.

Thanks, that is fair. @karrtikr Also looks like the extension logging has improved a lot since the last time I looked. (Sorry for the delay, some stuff came up but this is still high interest to us)

> ./bin/hermit ~/.vscode/extensions/ms-python.python-2022.8.1/pythonFiles/linter.py -m flake8 ./app.py
cwd: .
> ./bin/hermit ~/.vscode/extensions/ms-python.python-2022.8.1/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir . -s --cache-clear
cwd: .
##########Linting Output - flake8##########

Usage: hermit <command>

🐚 Hermit is a hermetic binary package manager.

Flags:
  -h, --help                 Show context-sensitive help.
      --version              Show version.
  -d, --debug                Enable debug logging.
  -t, --trace                Enable trace logging.
  -q, --quiet                Disable logging and progress UI, except fatal
                             errors ($HERMIT_QUIET).
      --level=auto           Set minimum log level
                             (auto,trace,debug,info,warn,error,fatal)
                             ($HERMIT_LOG).
      --lock-timeout=30s     Timeout for waiting on the lock
                             ($HERMIT_LOCK_TIMEOUT)
      --env=KEY=VALUE;...    Extra environment variables to apply to
                             environments.

Environment:
  Commands for creating and managing environments.

  init               Initialise an environment (idempotent).
  status             Show status of Hermit environment.
  install            Install packages.
  uninstall          Uninstall packages.
  upgrade            Upgrade packages
  list               List local packages.
  exec               Directly execute a binary in a package.
  env                Manage environment variables.
  validate script    Verify a shell script uses only builtin or Hermit commands.

Global:
  Commands for interacting with the shared global Hermit state.

  version                        Show version.
  manifest validate              Check a package manifest source for errors.
  manifest auto-version          Upgrade manifest versions automatically where
                                 possible.
  manifest create                Create a new manifest from an existing package
                                 artefact URL.
  info                           Show information on packages.
  shell-hooks (install-hooks)    Manage Hermit auto-activation hooks of a shell.
  update (sync)                  Update manifest sources.
  search                         Search for packages to install.
  gen-installer                  Generate Hermit installer script.
  validate source                Check a package manifest source for errors.
  validate env                   Verify an environment.
  clean                          Clean hermit cache.
  test                           Run package sanity tests.

Run "hermit <command> --help" for more information on a command.


[ERROR 2022-5-30 0:18:48.918]: Error discovering pytest tests:
 [n [Error]: hermit: error: unexpected argument /Users/damien/.vscode/extensions/ms-python.python-2022.8.1/pythonFiles/testing_tools/run_adapter.py

	at ChildProcess.<anonymous> (/Users/damien/.vscode/extensions/ms-python.python-2022.8.1/out/client/extension.js:2:232795)
	at Object.onceWrapper (node:events:510:26)
	at ChildProcess.emit (node:events:390:28)
	at maybeClose (node:internal/child_process:1064:16)
	at Process.ChildProcess._handle.onexit (node:internal/child_process:301:5)]
Starting Pylance language server.

[brettcannon]

Thank you to everyone who upvoted this issue! Since the community showed interest in this feature request we will leave this issue open as something to consider implementing at some point in the future.

We do encourage people to continue 👍 the first/opening comment as it helps us prioritize our work based on what the community seems to want the most.

[gandhis1]

Does someone mind enumerating what the issues currently are if you were to attempt to use a symlink environment? This feature is important enough to me that I'm probably going to spend some of my time to see if I can address any of these issues in a PR.

The first issue I notice is that conda run gets passed a name, when it probably needs to be passed a path.

[brettcannon]

Does someone mind enumerating what the issues currently are if you were to attempt to use a symlink environment?

I don't think anyone has a specific list. It's going to come down to testing what happens and seeing if there are some bad assumptions we make about everything we find not being a symlink.

But I think a bigger concern is being clear about what problem you're trying to solve? Is it just interpreters found on PATH? What about .venv being a symlink to a different directory? What a "symlinked environment" is has no concrete definition in terms of what people want to see supported.

The first issue I notice is that conda run gets passed a name, when it probably needs to be passed a path.

That should work regardless as we ask conda to tell us what environments it knows about. If conda doesn't support symlinked environments, then that would be an issue to open on https://github.com/conda/conda itself.

[gandhis1]

For starters, the equivalent functionality works in PyCharm, so I don't think it's a conda issue. But I need to take a closer look at the code that determines whether conda run is passed a name or passed a path. I can observe which it chooses by looking at the Python output window (this is for linting and testing tools that I observe this). In my current project, there is something making it such that when I use a symlink, it passes a name, but when I use a regular directory, it passes a path. From the perspective of me as a VS Code user, I am always passing a path to "Select Interpreter", but that path may or may not include a symlink. The same is also an issue in any feature that requires invoking conda activate, such as "Run Python File" - tries to activate a name, not a path.

In terms of what I view the issue as, I would like to put a path to a interpreter using a symlink-based path in "Select Interpreter" and all functionality (linting, formatting, code running, test running, etc) to work. I am personally testing this in the context of a conda env, so can't say what other things may come up with a venv.

Anyway, I understand you guys are busy, so I'm planning on taking a closer look to try to identify what the issue might be

Edit: I just looked a bit further. So, separate from what I mentioned above, it seems like everyone else in this issue is merely asking for auto-discovery of symlink-based virtual environments. For conda specifically - since it appears this extension parses the output of conda env list to view the list of environments, and that ostensibly does not include symlink environments, that may be a limitation of conda. But my point above still appears to be an issue.

Edit 2: Issue is probably somewhere in here, I'll need to step through this in the debugger to see exactly what assumptions are being made:

vscode-python/src/client/pythonEnvironments/discovery/locators/services/condaService.ts

Lines 237 to 316 in 3698950

	public async getCondaEnvironment(interpreterPath: string): Promise<{ name: string; path: string } | undefined> {
	const found = await this.pyenvs.getCondaEnvironment(interpreterPath);
	if (found !== undefined) {
	return found;
	}
	const isCondaEnv = await this.isCondaEnvironment(interpreterPath);
	if (!isCondaEnv) {
	return undefined;
	}
	let environments = await this.getCondaEnvironments(false);
	const dir = path.dirname(interpreterPath);
	// If interpreter is in bin or Scripts, then go up one level
	const subDirName = path.basename(dir);
	const goUpOnLevel = ['BIN', 'SCRIPTS'].indexOf(subDirName.toUpperCase()) !== -1;
	const interpreterPathToMatch = goUpOnLevel ? path.join(dir, '..') : dir;
	// From the list of conda environments find this dir.
	let matchingEnvs = Array.isArray(environments)
	? environments.filter((item) => this.fileSystem.arePathsSame(item.path, interpreterPathToMatch))
	: [];
	if (matchingEnvs.length === 0) {
	environments = await this.getCondaEnvironments(true);
	matchingEnvs = Array.isArray(environments)
	? environments.filter((item) => this.fileSystem.arePathsSame(item.path, interpreterPathToMatch))
	: [];
	}
	if (matchingEnvs.length > 0) {
	return { name: matchingEnvs[0].name, path: interpreterPathToMatch };
	}
	// If still not available, then the user created the env after starting vs code.
	// The only solution is to get the user to re-start vscode.
	return undefined;
	}
	/**
	* Return the list of conda envs (by name, interpreter filename).
	*/
	@traceDecorators.verbose('Get Conda environments')
	public async getCondaEnvironments(ignoreCache: boolean): Promise<CondaEnvironmentInfo[] | undefined> {
	// Global cache.
	const globalPersistence = this.persistentStateFactory.createGlobalPersistentState<{
	data: CondaEnvironmentInfo[] | undefined;
	}>('CONDA_ENVIRONMENTS', undefined);
	if (!ignoreCache && globalPersistence.value) {
	return globalPersistence.value.data;
	}
	try {
	const condaFile = await this.getCondaFile();
	const processService = await this.processServiceFactory.create();
	let envInfo = await processService.exec(condaFile, ['env', 'list']).then((output) => output.stdout);
	traceVerbose(`Conda Env List ${envInfo}}`);
	if (!envInfo) {
	traceVerbose('Conda env list failure, attempting path additions.');
	// Try adding different folders to the path. Miniconda fails to run
	// without them.
	const baseFolder = path.dirname(path.dirname(condaFile));
	const binFolder = path.join(baseFolder, 'bin');
	const condaBinFolder = path.join(baseFolder, 'condabin');
	const libaryBinFolder = path.join(baseFolder, 'library', 'bin');
	const newEnv = process.env;
	newEnv.PATH = `${binFolder};${condaBinFolder};${libaryBinFolder};${newEnv.PATH}`;
	traceVerbose(`Attempting new path for conda env list: ${newEnv.PATH}`);
	envInfo = await processService
	.exec(condaFile, ['env', 'list'], { env: newEnv })
	.then((output) => output.stdout);
	}
	const environments = parseCondaEnvFileContents(envInfo);
	await globalPersistence.updateValue({ data: environments });
	return environments;
	} catch (ex) {
	await globalPersistence.updateValue({ data: undefined });
	// Failed because either:
	// 1. conda is not installed.
	// 2. `conda env list has changed signature.
	traceError('Failed to get conda environment list from conda', ex);
	return undefined;