Pyspark: cannot access member "config" for type "classproperty"

[Daniel-Considine]

Environment data

• Language Server version: 2023.6.40
• OS and version: linux x64
• Python version (and distribution if applicable, e.g. Anaconda): 3.11
• python.analysis.indexing: true
• python.analysis.typeCheckingMode: basic

Code Snippet

import pyspark.sql
global spark
spark = (
        pyspark.sql.SparkSession.builder
        .config('spark.driver.memory', '100g')
        .appName('spark')
        .getOrCreate()
)
spark.version

Actual behavior

This code works fine when running scripts and Jupyter cells, but still the above message and failure to access functions etc.

I believe it's a stub issue, as pip install pyspark-stubs fixes the issue. However this library is meant to be integrated with the base pyspark distribution now, and is not compatible with more recent pyspark versions. I am unsure on whether the issue is with pylance accessing the stubs/typings that are now included in pyspark, or if the issue is on their end.

Logs

pylance.log

[rchiodo]

Without the stubs, pyspark has this for the 'builder' definition

    # TODO(SPARK-38912): Replace @classproperty with @classmethod + @property once support for
    # Python 3.8 is dropped.
    #
    # In Python 3.9, the @property decorator has been made compatible with the
    # @classmethod decorator (https://docs.python.org/3.9/library/functions.html#classmethod)
    #
    # @classmethod + @property is also affected by a bug in Python's docstring which was backported
    # to Python 3.9.6 (https://github.com/python/cpython/pull/28838)
    @classproperty
    def builder(cls) -> Builder:
        """Creates a :class:`Builder` for constructing a :class:`SparkSession`."""
        return cls.Builder()

Whereas the stubs have this (and force pyspark to also move backwards to 3.0.1):

    builder: SparkSession.Builder

The @classproperty is what is messing up Pylance. There's nothing that indicates @classproperty actually creates a property on a class.

If you switch pyspark to add what they suggested in the comment, it starts working.

For example this:

    if TYPE_CHECKING:
        @classmethod
        @property
        def builder(cls) -> Builder:
            """Creates a :class:`Builder` for constructing a :class:`SparkSession`."""
            return cls.Builder()
    else:
        @classproperty
        def builder(cls) -> Builder:
            """Creates a :class:`Builder` for constructing a :class:`SparkSession`."""
            return cls.Builder()

I think this would be a bug for pyspark to fix. By either providing the appropriate decorators, or perhaps fixing their classproperty decorator which is doing some weird stuff.

@erictraut do you have any thoughts?

[rchiodo]

A simpler repro is this:

class classproperty(property):
    def __get__(self, instance: Any, owner: Any = None) -> "TestSession.Builder":
        if self.fget:
            return classmethod(self.fget).__get__(None, owner)()  
        raise Exception()

class TestSession:
    class Builder:
        x: int
        def __init__(self):
            x = 4

    @classproperty
    def builder(cls) -> Builder:
        return TestSession.Builder()

b = TestSession.builder
reveal_type(b.x) # Error here.  Type of b.x is unknown

[Daniel-Considine]

@rchiodo Thanks for the prompt response, and apologies for the lack of succinct reproducible example.

This is a bit beyond my coding level. Should I just be adding that code to the class classproperty(property): decorator? Or do I need to remove parts too? I will raise an issue with the pyspark team once I've got it working.

[rchiodo]

@Daniel-Considine I don't think there's anything you can add to classproperty to get Pylance to understand it's adding a class level property, but I could be wrong. I'm not an expert at this stuff by any means.

I think the change I suggested for pyspark would be the way to "fix" it.

    if TYPE_CHECKING:
        @classmethod
        @property
        def builder(cls) -> Builder:
            """Creates a :class:`Builder` for constructing a :class:`SparkSession`."""
            return cls.Builder()
    else:
        @classproperty
        def builder(cls) -> Builder:
            """Creates a :class:`Builder` for constructing a :class:`SparkSession`."""
            return cls.Builder()

That change goes here:
https://github.com/apache/spark/blob/d12bec39cb141fbfab24d158022b327441d17945/python/pyspark/sql/session.py#L546

At least that would allow Pylance to understand that builder is a class level property. Pyspark might have other suggestions though.

[Daniel-Considine]

@rchiodo
Sorry I'm a certified idiot. I thought in your original comment you were suggesting adding that code snippet somewhere inside the already existing method. As you say - completely replacing it makes the code work and ensures pylance can understand what is going on.
Thanks for your help. I will submit an issue with the pyspark team so that they are aware their current solution to keep python<3.10 compatibility results in pylance errors.

[DavidArmendariz]

I still have this error, was it solved? I am using pyspark 3.5.0

[rchiodo]

I believe you'd have to ask Pyspark, It's a problem in their usage of classproperty which is not a known decorator.

The list of allowed/known decorators is specified here:
https://typing.readthedocs.io/en/latest/source/stubs.html#decorators

That's the python typing spec.

[nachobek]

It looks like the Pyspark 4.0 preview release takes care of this issue. I guess that's a good sign.

[eevmanu]

It looks like the Pyspark 4.0 preview release takes care of this issue. I guess that's a good sign.

In version 4.0.0.dev2 and 4.0.0.dev1 works great 👌🏼.

However, the problem persists in version 3.5.4.