Difference container image output from Kaniko build

[worapojc]

Hello, I acknowledged that Kaniko already mentioned "Running kaniko in any Docker image other than the official kaniko image is not supported due to implementation details".

I'm trying to achieve this for GitLab Runner to build container images with Kaniko.

Here is the example Dockerfile, https://github.com/aws-samples/cdk-fargate-gitlab-runner/blob/main/docker_images/kaniko/Dockerfile.

I extracted a part from my Dockerfile

RUN apk upgrade --no-cache &&
apk add --no-cache openssh git git-lfs ca-certificates envsubst &&
ssh-keygen -A &&
sed -i -e 's/^#HostKey/HostKey/g' /etc/ssh/sshd_config &&
sed -i -e 's/^#Port 22/Port 22/g' /etc/ssh/sshd_config &&
sed -i -e 's/^#PubkeyAuthentication yes/PubkeyAuthentication yes/g' /etc/ssh/sshd_config &&
sed -i -e 's/^#PasswordAuthentication yes/PasswordAuthentication no/g' /etc/ssh/sshd_config &&
sed -i -e 's/^AuthorizedKeysFile\t.ssh/authorized_keys/AuthorizedKeysFile\t%h/.ssh/authorized_keys/g' /etc/ssh/sshd_config

Here is the build logs on my laptop.

INFO[2025-02-11T04:08:24Z] Taking snapshot of full filesystem...
INFO[2025-02-11T04:08:24Z] RUN apk upgrade --no-cache && apk add --no-cache tini openssh git git-lfs ca-certificates envsubst && ssh-keygen -A && sed -i -e 's/^#HostKey/HostKey/g' /etc/ssh/sshd_config && sed -i -e 's/^#Port 22/Port 22/g' /etc/ssh/sshd_config && sed -i -e 's/^#PubkeyAuthentication yes/PubkeyAuthentication yes/g' /etc/ssh/sshd_config && sed -i -e 's/^#PasswordAuthentication yes/PasswordAuthentication no/g' /etc/ssh/sshd_config && sed -i -e 's/^AuthorizedKeysFile\t.ssh/authorized_keys/AuthorizedKeysFile\t%h/.ssh/authorized_keys/g' /etc/ssh/sshd_config
INFO[2025-02-11T04:08:24Z] Cmd: /bin/sh
INFO[2025-02-11T04:08:24Z] Args: [-c apk upgrade --no-cache && apk add --no-cache tini openssh git git-lfs ca-certificates envsubst && ssh-keygen -A && sed -i -e 's/^#HostKey/HostKey/g' /etc/ssh/sshd_config && sed -i -e 's/^#Port 22/Port 22/g' /etc/ssh/sshd_config && sed -i -e 's/^#PubkeyAuthentication yes/PubkeyAuthentication yes/g' /etc/ssh/sshd_config && sed -i -e 's/^#PasswordAuthentication yes/PasswordAuthentication no/g' /etc/ssh/sshd_config && sed -i -e 's/^AuthorizedKeysFile\t.ssh/authorized_keys/AuthorizedKeysFile\t%h/.ssh/authorized_keys/g' /etc/ssh/sshd_config]
INFO[2025-02-11T04:08:24Z] Running: [/bin/sh -c apk upgrade --no-cache && apk add --no-cache tini openssh git git-lfs ca-certificates envsubst && ssh-keygen -A && sed -i -e 's/^#HostKey/HostKey/g' /etc/ssh/sshd_config && sed -i -e 's/^#Port 22/Port 22/g' /etc/ssh/sshd_config && sed -i -e 's/^#PubkeyAuthentication yes/PubkeyAuthentication yes/g' /etc/ssh/sshd_config && sed -i -e 's/^#PasswordAuthentication yes/PasswordAuthentication no/g' /etc/ssh/sshd_config && sed -i -e 's/^AuthorizedKeysFile\t.ssh/authorized_keys/AuthorizedKeysFile\t%h/.ssh/authorized_keys/g' /etc/ssh/sshd_config]
fetch https://dl-cdn.alpinelinux.org/alpine/v3.21/main/aarch64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.21/community/aarch64/APKINDEX.tar.gz
(1/5) Upgrading busybox (1.37.0-r9 -> 1.37.0-r12)
Executing busybox-1.37.0-r12.post-upgrade
(2/5) Upgrading busybox-binsh (1.37.0-r9 -> 1.37.0-r12)
(3/5) Upgrading libcrypto3 (3.3.2-r4 -> 3.3.2-r6)
(4/5) Upgrading libssl3 (3.3.2-r4 -> 3.3.2-r6)
(5/5) Upgrading ssl_client (1.37.0-r9 -> 1.37.0-r12)
Executing busybox-1.37.0-r12.trigger
OK: 7 MiB in 15 packages
fetch https://dl-cdn.alpinelinux.org/alpine/v3.21/main/aarch64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.21/community/aarch64/APKINDEX.tar.gz
(1/27) Installing ca-certificates (20241121-r1)
(2/27) Installing libintl (0.22.5-r0)
(3/27) Installing gettext-envsubst (0.22.5-r0)
(4/27) Installing brotli-libs (1.1.0-r2)
(5/27) Installing c-ares (1.34.3-r0)
(6/27) Installing libunistring (1.2-r0)
(7/27) Installing libidn2 (2.3.7-r0)
(8/27) Installing nghttp2-libs (1.64.0-r0)
(9/27) Installing libpsl (0.21.5-r3)
(10/27) Installing zstd-libs (1.5.6-r2)
(11/27) Installing libcurl (8.11.1-r1)
(12/27) Installing libexpat (2.6.4-r0)
(13/27) Installing pcre2 (10.43-r0)
(14/27) Installing git (2.47.2-r0)
(15/27) Installing git-init-template (2.47.2-r0)
(16/27) Installing git-lfs (3.6.0-r2)
Executing git-lfs-3.6.0-r2.post-install
Git LFS initialized.
(17/27) Installing openssh-keygen (9.9_p1-r2)
(18/27) Installing ncurses-terminfo-base (6.5_p20241006-r3)
(19/27) Installing libncursesw (6.5_p20241006-r3)
(20/27) Installing libedit (20240808.3.1-r0)
(21/27) Installing openssh-client-common (9.9_p1-r2)
(22/27) Installing openssh-client-default (9.9_p1-r2)
(23/27) Installing openssh-sftp-server (9.9_p1-r2)
(24/27) Installing openssh-server-common (9.9_p1-r2)
(25/27) Installing openssh-server (9.9_p1-r2)
(26/27) Installing openssh (9.9_p1-r2)
(27/27) Installing tini (0.19.0-r3)
Executing busybox-1.37.0-r12.trigger
Executing ca-certificates-20241121-r1.trigger
OK: 39 MiB in 42 packages

There is no issues for the container image which built on my laptop.

However, when my GitLab runner execute the same command on the same container image (GitLab Runner + Kaniko) on AWS ECS (Fargate Task). I noticed that there is no installation logs of the step.

INFO[2025-02-11T03:42:15Z] Taking snapshot of full filesystem...
INFO[2025-02-11T03:42:15Z] RUN apk upgrade --no-cache && apk add --no-cache openssh git git-lfs ca-certificates envsubst && ssh-keygen -A && sed -i -e 's/^#HostKey/HostKey/g' /etc/ssh/sshd_config && sed -i -e 's/^#Port 22/Port 22/g' /etc/ssh/sshd_config && sed -i -e 's/^#PubkeyAuthentication yes/PubkeyAuthentication yes/g' /etc/ssh/sshd_config && sed -i -e 's/^#PasswordAuthentication yes/PasswordAuthentication no/g' /etc/ssh/sshd_config && sed -i -e 's/^AuthorizedKeysFile\t.ssh/authorized_keys/AuthorizedKeysFile\t%h/.ssh/authorized_keys/g' /etc/ssh/sshd_config
INFO[2025-02-11T03:42:15Z] Cmd: /bin/sh
INFO[2025-02-11T03:42:15Z] Args: [-c apk upgrade --no-cache && apk add --no-cache openssh git git-lfs ca-certificates envsubst && ssh-keygen -A && sed -i -e 's/^#HostKey/HostKey/g' /etc/ssh/sshd_config && sed -i -e 's/^#Port 22/Port 22/g' /etc/ssh/sshd_config && sed -i -e 's/^#PubkeyAuthentication yes/PubkeyAuthentication yes/g' /etc/ssh/sshd_config && sed -i -e 's/^#PasswordAuthentication yes/PasswordAuthentication no/g' /etc/ssh/sshd_config && sed -i -e 's/^AuthorizedKeysFile\t.ssh/authorized_keys/AuthorizedKeysFile\t%h/.ssh/authorized_keys/g' /etc/ssh/sshd_config]
INFO[2025-02-11T03:42:15Z] Running: [/bin/sh -c apk upgrade --no-cache && apk add --no-cache tini openssh git git-lfs ca-certificates envsubst && ssh-keygen -A && sed -i -e 's/^#HostKey/HostKey/g' /etc/ssh/sshd_config && sed -i -e 's/^#Port 22/Port 22/g' /etc/ssh/sshd_config && sed -i -e 's/^#PubkeyAuthentication yes/PubkeyAuthentication yes/g' /etc/ssh/sshd_config && sed -i -e 's/^#PasswordAuthentication yes/PasswordAuthentication no/g' /etc/ssh/sshd_config && sed -i -e 's/^AuthorizedKeysFile\t.ssh/authorized_keys/AuthorizedKeysFile\t%h/.ssh/authorized_keys/g' /etc/ssh/sshd_config]
INFO[2025-02-11T03:42:17Z] Taking snapshot of full filesystem...

Other libraries look good except tini. Normally, it will installed in /sbin

/ # ls /sbin
acpid blkid fdisk getty ifdown inotifyd iplink iptunnel logread mkdosfs modprobe poweroff route swapon tunctl zcip
adjtimex blockdev findfs halt ifenslave insmod ipneigh klogd losetup mkfs.vfat nameif raidautorun setconsole switch_root udhcpc
apk depmod fsck hwclock ifup ip iproute ldconfig lsmod mkswap nologin reboot slattach sysctl vconfig
arp fbsplash fstrim ifconfig init ipaddr iprule loadkmap mdev modinfo pivot_root rmmod swapoff syslogd watchdog

/ # ls /bin
arch chattr dd egrep getopt iostat linux64 makemime mountpoint ping reformime setpriv su uname
ash chgrp df false grep ipcalc ln mkdir mpstat ping6 rev setserial sync usleep
base64 chmod dmesg fatattr gunzip kbd_mode login mknod mv pipe_progress rm sh tar watch
bbconfig chown dnsdomainname fdflush gzip kill ls mktemp netstat printenv rmdir sleep touch zcat
busybox cp dumpkmap fgrep hostname link lsattr more nice ps run-parts stat true
cat date echo fsync ionice linux32 lzop mount pidof pwd sed stty umount

/ # apk version tini
WARNING: opening from cache https://dl-cdn.alpinelinux.org/alpine/v3.21/main: No such file or directory
WARNING: opening from cache https://dl-cdn.alpinelinux.org/alpine/v3.21/community: No such file or directory
Installed: Available:
tini-0.19.0-r3 = 0.19.0-r3

apk is showing tini is installed but no executable file in /sbin path.

Could you please guide me what could be the cause of this issue?

My base image in Dockerfile is alpine:3.21, and the architecture is arm64.

[worapojc]

Update... the build output is not reliable. I found some executable files are missing for each builds.

[thetric]

Yeah, I have noticed a similar thing. Multiple invocations with different AWS credentials lead to broken images. The first build correctly installs the required Ubuntu packages but subsequent builds seem to exclude them