Components

What is a Component?

In the software world "component" describes many different (though mostly similar) concepts. In the context of the Python extension we're using a specific meaning, described here.

A software "component" is a discrete encapsulation of a set of related functionality. It's part of a larger "system" into which the component is integrated, usually along with other resources and components. The boundary between the component and that system is represented as (1) a set of functionality the component provides, AKA its API, and (2) its dependencies. An effective component keeps both as small and tight as possible. To avoid extra long-term cost, all components in the system should be on the same abstraction layer.

Typically a component represents either a feature or some cross-cutting machinery of the system. You can represent those graphically as "lanes", where features are vertical lanes and machinery is horizontal lanes. Each intersection between vertical and horizontal lanes in the system is a point at which integration between components happens. In fact you can think of the system as the set of those integration points, particularly in how they are composed.

Often components can be decomposed even more discretely at the next (lower) layer of abstraction, resulting in "sub-components". Such a component can be treated as a self-contained system. Its sub-components, both horizontal and vertical, should always be at the same layer of abstraction. This can recurse down many layers. Layering is more common with horizontal components.

Sometimes the components of a system are only an abstraction, but ideally each corresponds to single tree of source code. At the top of the tree we would find code that initializes the component and code that exposes the API. Usually the two are facilitated through a factory function (or class) that takes the dependencies and returns an object that provides the API. That would be the only instance of coupling with code outside the component. Sometimes adapter shims are necessary when using more general resources to satisfy the component's dependencies. Those adapters are never part of the component, but live near where the component is initialized.

VS Code: API Components

The Python extension for VS Code provides functionality to users through the VS Code UI. This involves heavy use of the VS Code API. That API can be divided into various components. Those components are:

• language services / Language Server
  • declarative:
    • syntax highlighting
    • snippet completion
    • bracket matching
    • bracket autoclosing
    • bracket autosurrounding
    • comment toggling
    • auto indentation
    • folding (by markers)
  • programatic (see also ...):
    • hover
    • auto completion
    • jump to definition
    • error checking
    • formatting
    • refactoring
    • folding
    • ...
• debugging
• terminal & code execution
• testing
• ipywidget support
• snippets
• ...

Python Extension: Functional Components

The functionality provided by the Python extension can be divided conceptually into various (vertical) components:

• language services
• debugging
• code execution (terminal)
• testing
• formatters
• linters
• datascience

Likewise for the cross-cutting machinery of the extension (horizontal components):

• interpreter environments
• ipywidget support
• process management
• filesystem
• networking

See below in "Details for Specific Components" for more info.

Each component has the following layers:

1. integration (providing / consuming)
   • with the rest of the extension
   • with other extensions
2. integration with VS Code (incl. API / UI)
3. component boundary
4. "global" capability & sub-component composition
5. sub-components
6. low-level implementations (e.g. npm modules)

Changing the Python Extension

To a certain degree the extension's code aligns with the above lists. However, there is a tremendous amount of coupling, in part due to our use of a dependency injection framework (injectify). This leads to significant extra complexity, leading to more bugs, higher maintenance costs, and slower feature development. We are planning on addressing at least a little at a time. The following describes the plan.

target project tree

<vscode-python>/
  src/
    client/
      extension.ts
      extensionInit.ts
      extensionActivation.ts
      ...  # other legacy code
    <component>/  # e.g. pythonEnvironments
      <sub-component>/  # e.g. discovery
        ...
      tests/
        ...  # new component-specific tests (jest)
    tests/
      ...  # new system-integration tests (jest), UI tests (puppeteer?)
    test/
      ...  # legacy tests (mocha)

steps

("EXT_ACT": per-component "activation" that happens via src/client/extensionInit.ts & extensionActivation.ts)

1. restructure extension activation around components (done)
2. separate each component
   1. create a new source tree for the component
      • including code added for activation (component-specific functions called in EXT_ACT)
   2. for the general parts and then for each sub-component:
      1. move all relevant code to the tree
         • some files can be moved as-is
         • some files will be partially copied
         • some methods will be factor out a functions in the component
      2. isolate the component (fix coupling)
         1. eliminate each use of @inject
            • this will involve switching to explicit dep. injection (pass as args) via EXT_ACT
            • where not too much trouble or too invasive, reduce the dependency as tightly as possible to what is specifically needed (further tightening will be done later)
            • this may involve adding (reductive / converting) adapters that live in EXT_ACT
         2. eliminate each use of @injectable
            • for now this will involve adding injectify-compatible adapters to wrap the component's API, either in some relevant serviceRegistry.ts or in EXT_ACT
         3. reduce / eliminate each implicit dependency (see imports)
            • as with @inject, switch to explicit injection via the component init
         4. ??? eliminate each export out of the component (see imports elsewhere)?
            • this is unnecessary in the long term but may be helpful in getting better tests in the short term
            • doing this fully may not be easy enough to be worth it
      3. reduce the component's dependencies and API
         • for each dependency, reduce it as tightly as possible to what is specifically needed
           • this will probably involve adding (reductive / converting) adapters that live in EXT_ACT
         • reduce the component API to as tight and minimal as possible
      4. extra credit
         • split component into 2 layers: VSCode-aware (incl. UI aspects) and (lower level) VSCode-agnostic
         • factor out separate npm module(s) for general low-level functionality
         • clean up: e.g. strip out any interfaces, methods, etc. we no longer need
3. fix known problems

about settings

Each component will have it's own config and the extension will manage exposing settings to components so they can convert that to config.

Python Extension: Details for Specific Components

• (V) language services
  • sub-components: language server (LSP), language config (e.g. indentation), syntax color, snippets, ...
• (V) debugging
• (V) code execution (terminal)
• (V) testing
  • sub-components: (H) discovery, (H) execution, (V) pytest support, (V) unittest support
• (V) formatters
  • sub-components: (V) black, ...
• (V) linters
  • sub-components: (V) pylint, (V) ...
• (V) datascience
• (H) interpreter environments
  • sub-components: discovery, "selected" / config, env activation, pkg installation, env mgmt
• (H) ipywidget support
• (H) process management
• (H) filesystem
• (H) networking

discovery

• includes caching
• sub-components: (V) global environments (system, pyenv, other)
• sub-components: (V) "virtual" environments (venv, virtualenv, conda, etc.)

Components as Separate Extensions

The Python extension's components may be thought of as mini extensions. Making some of them (or sub-components) into separate extensions may make sense, particularly with the extension as an extension pack (or if VS Code exposes the "builtin" extension capability they use internally). That does not necessarily mean we would publish them to the marketplace, but that might make sense too. Such sub-extensions may or may not have their own repositories and issue trackers.

Candidates:

• managing interpreter environments ("python-interpreter"?)
• each of the interpreter environments (conda, virtualenv, venv, pipenv, pyenv, system-installed)
• testing, including the test adapter script
• each of the testing frameworks (unittest, pytest, nose2)
• formatting
• each of the formatters (black, yapf, autopep8)
• linting
• each of the linters (pylint, etc.)
• language server
• terminal support (e.g. run-in-terminal)

Fundamental language config, etc. would stay in the main extension.

Note that we already have plans to factor out independent npm packages (possibly published publicly) for some of the low-level functionality of various components.