Can not create Spark Lineage with Iceberg

[tranhan02]

I am currently experimenting with OpenMetadata's Spark lineage feature using Iceberg as the database. While I have successfully added Iceberg metadata, I am unable to generate any pipeline lineage with Spark. My code is similar to the following sample:

from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .config("spark.cores.max", "2") \
    .config("spark.executor.memory", "2g") \
    .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") \
    .config("spark.sql.catalog.warehouse", "org.apache.iceberg.spark.SparkCatalog") \
    .config("spark.sql.catalog.warehouse.type", "hive") \
    .config("spark.sql.catalog.warehouse.uri", "thrift://10.111.4.144:9083") \
    .config("spark.sql.catalog.warehouse.io-impl", "org.apache.iceberg.aws.s3.S3FileIO") \
    .config("spark.sql.catalog.warehouse.s3.endpoint", "http://10.111.4.144:9000") \
    .config("spark.sql.catalog.warehouse.s3.access-key-id", "testabc") \
    .config("spark.sql.catalog.warehouse.s3.secret-access-key", "testabc") \
    .config("spark.sql.defaultCatalog", "warehouse") \
    .config("spark.sql.catalog.warehouse.warehouse", "s3a://test") \
    .config("spark.extraListeners", "org.openmetadata.spark.agent.OpenMetadataSparkListener") \
    .config("spark.openmetadata.transport.hostPort", "http://10.111.4.144:8585") \
    .config("spark.openmetadata.transport.type", "openmetadata") \
    .config("spark.openmetadata.transport.jwtToken", "token_ingestion_bot") \
    .config("spark.openmetadata.transport.pipelineServiceName", "spark_test") \
    .config("spark.openmetadata.transport.pipelineName", "test") \
    .config("spark.openmetadata.transport.pipelineSourceUrl", "http://10.111.4.144:8585/service/pipelineServices/spark_test") \
    .config("spark.openmetadata.transport.pipelineDescription", "abc") \
    .config("spark.openmetadata.transport.timeout", "100") \
    .config("spark.openmetadata.transport.databaseServiceNames", "iceberg_test") \
    .getOrCreate()

table = 'warehouse.test.test1'
df = spark.read \
    .format("iceberg") \
    .load(table)

table = 'warehouse.test.test2'
df.write \
  .format("iceberg") \
  .mode("append") \
  .save(table) 

spark.stop()

Has anyone successfully set up pipeline lineage with Spark and Iceberg? If so, I would greatly appreciate it if you could share your experience or any reference materials.