[Bug]: The submission_environment_dependencies.txt file does not get staged when running with Flink runner on Dataproc

[liferoad]

What happened?

In some cases, "submission_environment_dependencies.txt" might not be staged.

#32752 added a workaround to ignore the error for a missing artifact, but we should rootcause why it didn't get staged.

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[tvalentyn]

Do we crash? Seems like we should just print sth:

beam/sdks/python/container/boot.go

Line 416 in 2049e6b

bufLogger.Printf(ctx, "couldn't fetch the submission environment dependencies: %v", err)

[liferoad]

Flink on Dataproc returns this:
failed to retrieve staged files: failed to retrieve /tmp/staged in 3 attempts: failed to retrieve chunk for /tmp/staged/submission_environment_dependencies.txt
And the job then failed.

[tvalentyn]

ok, then the problem concerns the area of materialization of staged artifacts - we have a file that is being added to a manifest, but then not available when we try to materialize it.

It should either not be staged (and not included in the manifest), or be available in the staging location.

[tvalentyn]

Workaround: supply --experiments=disable_logging_submission_environment

beam/sdks/python/apache_beam/runners/portability/stager.py

Line 378 in 4cf99ea

'disable_logging_submission_environment')):

[liferoad]

Thanks, @tvalentyn. Let me investigate this later when I have time.

[zendesk-kjaanson]

I also ran into this bug when trying to run Beam 2.59.0 using PortableRunner on Kubernetes with Apache Flink Operator. When looking into task manager pods then the submission_environment_dependencies.txt file at /tmp/staging seems to be there but completely empty. All task-managers spawned will simply sit idle after logging this error.

[liferoad]

You can disable this by using --experiments disable_logging_submission_environment

[zendesk-kjaanson]

I solved the issue for myself, not sure how relevant it is to the issue at hand here. In my case, when trying to use PortableRunner with flink using Apache Flink Operator, the staging volume was not accessible/same for job manager and task managers/workers. For some reason this causes empty files for submission_environment_dependencies.txt (and if you use save_main_session then also for pickled_main_session to appear in /tmp/staged which then results in the failed to retrieve staged files: failed to retrieve /tmp/staged in 3 attempts: failed to retrieve chunk for /tmp/staged/submission_environment_dependencies.txt error to appear when worker process tries to load these.

My issue was solved when I was able to create a working shared staging volume across pods.

Fun side note that might be helpful for someone: When you try to create host mounted PersistantVolume with ReadWriteMany access mode on Googles GKE and use it as a volume then it never actually tells you that you can't do it, but will simply mount random (different) volumes across all pods. Docs mention that it is not supposed to be supported :D. I went with FUSE CSI driver that solved the issue for GKE.

[liferoad]

Some notes:

beam/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/DockerEnvironmentFactory.java

Line 114 in 53080f1

String semiPersistDir = pipelineOptions.as(RemoteEnvironmentOptions.class).getSemiPersistDir();

: --semi_persist_dir - > https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/DockerCommand.java#L111

[liferoad]

https://github.com/liferoad/beam-ml-flink/blob/main/Makefile#L159 another workaroud for my case

[kennknowles]

I was just trying to understand this but I am really naive here - is this file not just transmitted over artifact API? we should really never rely on shared storage or sharing with docker containers, etc.

[kennknowles]

What is "semi persist dir" anyhow?