HDDS-11590. Unexpected character - in Docker Compose env_file variable names

[adoroszlai]

What changes were proposed in this pull request?

Merge configuration from docker-config into docker-compose.yaml. Benefits:

1. Fix unexpected character "-" problem that happens with some versions of Docker Compose.
2. Easier to use for getting started when trying Ozone with Docker, only single file needs to be downloaded.

https://issues.apache.org/jira/browse/HDDS-11590

How was this patch tested?

$ docker compose up -d --scale datanode=3
...

$ docker-compose exec -T scm ozone freon ockg -n1 -t1
...
INFO  BaseFreonGenerator:75 - Successful executions: 1

$ docker compose ps
NAME                      IMAGE                      COMMAND                  SERVICE    CREATED          STATUS          PORTS
docker-ozone-datanode-1   apache/ozone:1.4.1-rocky   "/usr/local/bin/dumb…"   datanode   34 seconds ago   Up 33 seconds   0.0.0.0:32892->9864/tcp, :::32892->9864/tcp
docker-ozone-datanode-2   apache/ozone:1.4.1-rocky   "/usr/local/bin/dumb…"   datanode   34 seconds ago   Up 33 seconds   0.0.0.0:32890->9864/tcp, :::32890->9864/tcp
docker-ozone-datanode-3   apache/ozone:1.4.1-rocky   "/usr/local/bin/dumb…"   datanode   34 seconds ago   Up 33 seconds   0.0.0.0:32891->9864/tcp, :::32891->9864/tcp
docker-ozone-om-1         apache/ozone:1.4.1-rocky   "/usr/local/bin/dumb…"   om         34 seconds ago   Up 33 seconds   0.0.0.0:9874->9874/tcp, :::9874->9874/tcp
docker-ozone-recon-1      apache/ozone:1.4.1-rocky   "/usr/local/bin/dumb…"   recon      34 seconds ago   Up 33 seconds   0.0.0.0:9888->9888/tcp, :::9888->9888/tcp
docker-ozone-s3g-1        apache/ozone:1.4.1-rocky   "/usr/local/bin/dumb…"   s3g        34 seconds ago   Up 33 seconds   0.0.0.0:9878->9878/tcp, :::9878->9878/tcp
docker-ozone-scm-1        apache/ozone:1.4.1-rocky   "/usr/local/bin/dumb…"   scm        34 seconds ago   Up 33 seconds   0.0.0.0:9876->9876/tcp, :::9876->9876/tcp

[errose28]

Overall LGTM. A few high level questions:

• Is envtoconf.py being invoked somewhere to act on the variables? Can we test that it is not actually using the ozone-site.xml that is already included in the image?
• Do we need a different set of configs for docker compose vs the ozone-site.xml already included in the image?

[errose28]

Is this config necessary? It looks like the intent is for all other metadata configs to fall back to ozone.metadata.dirs

[adoroszlai]

Probably not necessary, but falling back to ozone.metadata.dirs produces warning in the log.

And it was present in docker-config, I just moved and sorted the properties.

[adoroszlai]

• Do we need a different set of configs for docker compose vs the ozone-site.xml already included in the image?

ozone-site.xml in the image is a placeholder, which comes from the Ozone release tarball. The image used to overwrite the placeholder, but it was removed in b9fe9d2.

$ docker run -it --rm ghcr.io/apache/ozone:b38e5431f535d7a56752c0eb7e58031d2b6a4e4a bash -c 'cat $OZONE_CONF_DIR/ozone-site.xml'
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
   Licensed to the Apache Software Foundation (ASF) under one or more
   contributor license agreements.  See the NOTICE file distributed with
   this work for additional information regarding copyright ownership.
   The ASF licenses this file to You under the Apache License, Version 2.0
   (the "License"); you may not use this file except in compliance with
   the License.  You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>

</configuration>

• Is envtoconf.py being invoked somewhere to act on the variables? Can we test that it is not actually using the ozone-site.xml that is already included in the image?

envtoconf.py is invoked by /opt/hadoop/libexec/entrypoint.sh, the entrypoint of the (parent) image.

[errose28]

Ah I was looking at ozone-latest (which is an alias for 1.4.0), not latest so I still saw the old files. Side note, we need to move ozone-latest to point to ozone-1.4.1. That step may not be present in the release guide.

Can we simplify this config process? The whole env variable + python script thing always looked strange to me. IMO the most intuitive way to do this would be:

• Ozone tarball ships with an empty config like it has now.
• Docker image puts a sensible default config for docker deployments into the image like it was doing before HDDS-11809
  • Using the config baked into the image would still only require the docker-compose.yaml file to start the cluster.
• To override configs in the image, mount a config file as a volume on top of the existing config file in the image. This will effectively replace it.
  • This approach can also be used by all Ozone acceptance tests, so their config files can be actual ozone-site.xml files that are familiar to users and their compose files can handle the config mounting

To me this makes the quick start aspect much more intuitive, because new users won't think that the env var approach is some standard way of configuring Ozone and that the variables are handled by the Ozone process itself. The downside is that the compose file working without other files OOTB is coupled to the config shipped with the Ozone image it is using. Since they are managed in the same repo I feel that this is ok though.

If we do want specific service configurations baked into the docker-compose.yaml file (SCM/OM service IDs and membership, for example), passing -D<key>=<value> to the service's entrypoint is probably better than cluster wide variables anyways. This uses a standard Ozone feature to get the configs to the process while also making it clear which services actually need the specified config keys.

[adoroszlai]

Side note, we need to move ozone-latest to point to ozone-1.4.1. That step may not be present in the release guide.

https://issues.apache.org/jira/browse/HDDS-12046?focusedCommentId=17911354&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17911354

To override configs in the image, mount a config file as a volume on top of the existing config file in the image. This will effectively replace it.

Injecting config via environment variables makes it easy to add or replace some properties. XML files cannot be composed this way: replace ozone-site.xml as a whole. One more config file can be added with the -conf option. We may improve it by allowing multiple -conf options (or preferably --config-file...).

passing -D= to the service's entrypoint

The -D settings defined for entrypoints are not applied to any command run with docker compose exec in the same container.

The whole env variable + python script thing always looked strange to me.

Using it only for Ozone is probably overkill in a way. Marton designed it as a uniform way to define configs for images of various services in the ecosystem (Hadoop, Spark, Kafka, etc.).

I think fundamentally changing the way configuration is defined is out of scope in this PR.

[smengcl]

Thanks @adoroszlai for this.

I imagine we would want to do the same to compose/* under ozone repo later as well?

https://github.com/apache/ozone/tree/master/hadoop-ozone/dist/src/main/compose/ozone

[adoroszlai]

I imagine we would want to do the same to compose/* under ozone repo later as well?

Yes, eventually.

[errose28]

Thanks for the improvement @adoroszlai. This will be a good step forward to reducing config complexity. I agree that we can discuss further changes to how Ozone handles config input in a separate issue.