Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Memory issue with v0.19.0 #1173

Closed
tmaier opened this issue Mar 31, 2020 · 1 comment
Closed

Memory issue with v0.19.0 #1173

tmaier opened this issue Mar 31, 2020 · 1 comment

Comments

@tmaier
Copy link

tmaier commented Mar 31, 2020 •
edited
Loading

Actual behavior

I am using kaniko within GitLab CI.
I am using gcr.io/kaniko-project/executor:debug, which is v0.19.0 at the time of writing.

Recently, I started to get a "out of memory" error on each build.
Even on builds of commits, which used to work.

Expected behavior

The build should not fail.

I downgraded kaniko to gcr.io/kaniko-project/executor:debug-v0.18.0 and the builds worked again.

To Reproduce
Steps to reproduce the behavior:

  1. ...
  2. ...

Additional Information

# Kaniko image:

gcr.io/kaniko-project/executor:debug

# Stack trace:

fatal error: runtime: out of memory

runtime stack:
runtime.throw(0x1b4c7a1, 0x16)
	/usr/local/go/src/runtime/panic.go:1112 +0x72
runtime.sysMap(0xc0b0000000, 0x1c000000, 0x2c6f6f8)
	/usr/local/go/src/runtime/mem_linux.go:169 +0xc5
runtime.(*mheap).sysAlloc(0x2c5a440, 0x1ac00000, 0x2c5a448, 0xd59c)
	/usr/local/go/src/runtime/malloc.go:715 +0x1cd
runtime.(*mheap).grow(0x2c5a440, 0xd59c, 0x0)
	/usr/local/go/src/runtime/mheap.go:1286 +0x11c
runtime.(*mheap).allocSpan(0x2c5a440, 0xd59c, 0x7fff1f950100, 0x2c6f708, 0xfffffffffffffade)
	/usr/local/go/src/runtime/mheap.go:1124 +0x6a0
runtime.(*mheap).alloc.func1()
	/usr/local/go/src/runtime/mheap.go:871 +0x64
runtime.(*mheap).alloc(0x2c5a440, 0xd59c, 0x7fff1f950101, 0x43d715)
	/usr/local/go/src/runtime/mheap.go:865 +0x81
runtime.largeAlloc(0x1ab3789e, 0xc000050101, 0xc000000180)
	/usr/local/go/src/runtime/malloc.go:1152 +0x92
runtime.mallocgc.func1()
	/usr/local/go/src/runtime/malloc.go:1047 +0x46
runtime.systemstack(0x461904)
	/usr/local/go/src/runtime/asm_amd64.s:370 +0x66
runtime.mstart()
	/usr/local/go/src/runtime/proc.go:1041

goroutine 1 [running]:
runtime.systemstack_switch()
	/usr/local/go/src/runtime/asm_amd64.s:330 fp=0xc00054d1e8 sp=0xc00054d1e0 pc=0x461a00
runtime.mallocgc(0x1ab3789e, 0x1847ec0, 0x8001, 0x8000)
	/usr/local/go/src/runtime/malloc.go:1046 +0x895 fp=0xc00054d288 sp=0xc00054d1e8 pc=0x40c7c5
runtime.makeslice(0x1847ec0, 0x1ab3789e, 0x1ab3789e, 0xc00486f278)
	/usr/local/go/src/runtime/slice.go:49 +0x6c fp=0xc00054d2b8 sp=0xc00054d288 pc=0x44a55c
bytes.makeSlice(0x1ab3789e, 0x0, 0x0, 0x0)
	/usr/local/go/src/bytes/buffer.go:229 +0x73 fp=0xc00054d2f8 sp=0xc00054d2b8 pc=0x510683
bytes.(*Buffer).grow(0xc08af57d40, 0x8000, 0xc00054d3b8)
	/usr/local/go/src/bytes/buffer.go:142 +0x15b fp=0xc00054d348 sp=0xc00054d2f8 pc=0x50ffcb
bytes.(*Buffer).Write(0xc08af57d40, 0xc0acda8000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/bytes/buffer.go:172 +0xdc fp=0xc00054d378 sp=0xc00054d348 pc=0x5102cc
archive/tar.(*regFileWriter).Write(0xc0acd7e1a0, 0xc0acda8000, 0x8000, 0x8000, 0x8000, 0x0, 0x0)
	/usr/local/go/src/archive/tar/writer.go:497 +0xe5 fp=0xc00054d3c8 sp=0xc00054d378 pc=0x80b0b5
archive/tar.(*Writer).Write(0xc08b857500, 0xc0acda8000, 0x8000, 0x8000, 0x8000, 0x0, 0x0)
	/usr/local/go/src/archive/tar/writer.go:435 +0x6d fp=0xc00054d428 sp=0xc00054d3c8 pc=0x80ad4d
io.copyBuffer(0x1e85be0, 0xc08b857500, 0x1e85bc0, 0xc08b839d40, 0xc0acda8000, 0x8000, 0x8000, 0xc86, 0x0, 0x0)
	/usr/local/go/src/io/io.go:407 +0x1fb fp=0xc00054d4a0 sp=0xc00054d428 pc=0x471a2b
io.Copy(...)
	/usr/local/go/src/io/io.go:364
github.com/google/go-containerregistry/pkg/v1/mutate.layerTime(0x1edcb80, 0xc000592360, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/GoogleContainerTools/kaniko/vendor/github.com/google/go-containerregistry/pkg/v1/mutate/mutate.go:320 +0x3ee fp=0xc00054d5a8 sp=0xc00054d4a0 pc=0x146b4ce
github.com/google/go-containerregistry/pkg/v1/mutate.Time(0x1eea2a0, 0xc003d7a540, 0x0, 0x0, 0x0, 0x1eea2a0, 0xc003d7a540, 0x0, 0x0)
	/go/src/github.com/GoogleContainerTools/kaniko/vendor/github.com/google/go-containerregistry/pkg/v1/mutate/mutate.go:258 +0x1ac fp=0xc00054d6d0 sp=0xc00054d5a8 pc=0x146ab4c
github.com/google/go-containerregistry/pkg/v1/mutate.Canonical(0x1eea2a0, 0xc003d7a540, 0xbf990e94b107151e, 0x3eb217f0e4, 0x2c41480, 0x1eea2a0)
	/go/src/github.com/GoogleContainerTools/kaniko/vendor/github.com/google/go-containerregistry/pkg/v1/mutate/mutate.go:348 +0x4a fp=0xc00054d738 sp=0xc00054d6d0 pc=0x146ba9a
github.com/GoogleContainerTools/kaniko/pkg/executor.DoBuild(0x2c41d00, 0x1, 0x0, 0x0, 0x3)
	/go/src/github.com/GoogleContainerTools/kaniko/pkg/executor/build.go:609 +0x10f6 fp=0xc00054dc98 sp=0xc00054d738 pc=0x1724e26
github.com/GoogleContainerTools/kaniko/cmd/executor/cmd.glob..func2(0x2c304c0, 0xc00032b5f0, 0x0, 0xd)
	/go/src/github.com/GoogleContainerTools/kaniko/cmd/executor/cmd/root.go:105 +0x12b fp=0xc00054dda8 sp=0xc00054dc98 pc=0x173e84b
github.com/spf13/cobra.(*Command).execute(0x2c304c0, 0xc00003a0f0, 0xd, 0xd, 0x2c304c0, 0xc00003a0f0)
	/go/src/github.com/GoogleContainerTools/kaniko/vendor/github.com/spf13/cobra/command.go:830 +0x29d fp=0xc00054de80 sp=0xc00054dda8 pc=0x17319bd
github.com/spf13/cobra.(*Command).ExecuteC(0x2c304c0, 0x0, 0x181fa00, 0xc000048178)
	/go/src/github.com/GoogleContainerTools/kaniko/vendor/github.com/spf13/cobra/command.go:914 +0x2fb fp=0xc00054df58 sp=0xc00054de80 pc=0x17325bb
github.com/spf13/cobra.(*Command).Execute(...)
	/go/src/github.com/GoogleContainerTools/kaniko/vendor/github.com/spf13/cobra/command.go:864
main.main()
	/go/src/github.com/GoogleContainerTools/kaniko/cmd/executor/main.go:26 +0x2d fp=0xc00054df88 sp=0xc00054df58 pc=0x173ee2d
runtime.main()
	/usr/local/go/src/runtime/proc.go:203 +0x212 fp=0xc00054dfe0 sp=0xc00054df88 pc=0x435cf2
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc00054dfe8 sp=0xc00054dfe0 pc=0x463991

goroutine 7 [chan receive]:
github.com/golang/glog.(*loggingT).flushDaemon(0x2c417c0)
	/go/src/github.com/GoogleContainerTools/kaniko/vendor/github.com/golang/glog/glog.go:882 +0x8b
created by github.com/golang/glog.init.0
	/go/src/github.com/GoogleContainerTools/kaniko/vendor/github.com/golang/glog/glog.go:410 +0x26f

goroutine 15 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc0002ba740)
	/go/src/github.com/GoogleContainerTools/kaniko/vendor/go.opencensus.io/stats/view/worker.go:146 +0xdd
created by go.opencensus.io/stats/view.init.0
	/go/src/github.com/GoogleContainerTools/kaniko/vendor/go.opencensus.io/stats/view/worker.go:29 +0x57

goroutine 426 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7f43ad96af18, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc005e81298, 0x72, 0x1700, 0x17a8, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc005e81280, 0xc003ce2000, 0x17a8, 0x17a8, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc005e81280, 0xc003ce2000, 0x17a8, 0x17a8, 0x203000, 0x638ec0, 0xc002ca04b8)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0047e5c38, 0xc003ce2000, 0x17a8, 0x17a8, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x8e
crypto/tls.(*atLeastReader).Read(0xc003ca0bc0, 0xc003ce2000, 0x17a8, 0x17a8, 0x1a, 0x10a4, 0xc0005369c8)
	/usr/local/go/src/crypto/tls/conn.go:760 +0x60
bytes.(*Buffer).ReadFrom(0xc002ca05d8, 0x1e85ec0, 0xc003ca0bc0, 0x40a195, 0x190c800, 0x1acda20)
	/usr/local/go/src/bytes/buffer.go:204 +0xb1
crypto/tls.(*Conn).readFromUntil(0xc002ca0380, 0x1e89460, 0xc0047e5c38, 0x5, 0xc0047e5c38, 0x9)
	/usr/local/go/src/crypto/tls/conn.go:782 +0xec
crypto/tls.(*Conn).readRecordOrCCS(0xc002ca0380, 0x0, 0x0, 0xc000536d38)
	/usr/local/go/src/crypto/tls/conn.go:589 +0x115
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:557
crypto/tls.(*Conn).Read(0xc002ca0380, 0xc003cbf000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1233 +0x15b
bufio.(*Reader).Read(0xc003c91da0, 0xc003cc6038, 0x9, 0x9, 0xc000536d38, 0x1bf0000, 0x6d67f5)
	/usr/local/go/src/bufio/bufio.go:226 +0x24f
io.ReadAtLeast(0x1e85c80, 0xc003c91da0, 0xc003cc6038, 0x9, 0x9, 0x9, 0xc000074040, 0x0, 0x1e86140)
	/usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
	/usr/local/go/src/io/io.go:329
net/http.http2readFrameHeader(0xc003cc6038, 0x9, 0x9, 0x1e85c80, 0xc003c91da0, 0x0, 0x0, 0xc003c834a0, 0x0)
	/usr/local/go/src/net/http/h2_bundle.go:1479 +0x87
net/http.(*http2Framer).ReadFrame(0xc003cc6000, 0xc003c834a0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/h2_bundle.go:1737 +0xa1
net/http.(*http2clientConnReadLoop).run(0xc000536fa8, 0x1, 0x0)
	/usr/local/go/src/net/http/h2_bundle.go:8246 +0x8d
net/http.(*http2ClientConn).readLoop(0xc00179d200)
	/usr/local/go/src/net/http/h2_bundle.go:8174 +0x6f
created by net/http.(*http2Transport).newClientConn
	/usr/local/go/src/net/http/h2_bundle.go:7174 +0x64a

goroutine 45 [select, 5 minutes]:
net/http.(*persistConn).readLoop(0xc000503320)
	/usr/local/go/src/net/http/transport.go:2099 +0x99e
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1647 +0xc56

goroutine 46 [select, 5 minutes]:
net/http.(*persistConn).writeLoop(0xc000503320)
	/usr/local/go/src/net/http/transport.go:2277 +0x11c
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1648 +0xc7b

goroutine 69 [select, 5 minutes]:
net/http.(*persistConn).readLoop(0xc000503b00)
	/usr/local/go/src/net/http/transport.go:2099 +0x99e
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1647 +0xc56

goroutine 70 [select, 5 minutes]:
net/http.(*persistConn).writeLoop(0xc000503b00)
	/usr/local/go/src/net/http/transport.go:2277 +0x11c
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1648 +0xc7b

goroutine 439 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7f43ad96a118, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc005e81a18, 0x72, 0x1000, 0x109d, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc005e81a00, 0xc003d26000, 0x109d, 0x109d, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc005e81a00, 0xc003d26000, 0x109d, 0x109d, 0x203000, 0x598f0b, 0xc003d46191)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0047e5ca8, 0xc003d26000, 0x109d, 0x109d, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x8e
crypto/tls.(*atLeastReader).Read(0xc003d130c0, 0xc003d26000, 0x109d, 0x109d, 0x0, 0xc003cd1c80, 0xc000539918)
	/usr/local/go/src/crypto/tls/conn.go:760 +0x60
bytes.(*Buffer).ReadFrom(0xc002ca13d8, 0x1e85ec0, 0xc003d130c0, 0x40a195, 0x190c800, 0x1acda20)
	/usr/local/go/src/bytes/buffer.go:204 +0xb1
crypto/tls.(*Conn).readFromUntil(0xc002ca1180, 0x1e89460, 0xc0047e5ca8, 0x5, 0xc0047e5ca8, 0x6ecd00)
	/usr/local/go/src/crypto/tls/conn.go:782 +0xec
crypto/tls.(*Conn).readRecordOrCCS(0xc002ca1180, 0x0, 0x0, 0xc002c49c20)
	/usr/local/go/src/crypto/tls/conn.go:589 +0x115
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:557
crypto/tls.(*Conn).Read(0xc002ca1180, 0xc003d46000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1233 +0x15b
net/http.(*persistConn).Read(0xc001aac900, 0xc003d46000, 0x1000, 0x1000, 0xc000539eb0, 0x460db0, 0xc000539eb0)
	/usr/local/go/src/net/http/transport.go:1825 +0x75
bufio.(*Reader).fill(0xc003cd1c80)
	/usr/local/go/src/bufio/bufio.go:100 +0x103
bufio.(*Reader).Peek(0xc003cd1c80, 0x1, 0x2, 0x0, 0x0, 0xc002c49b00, 0x0)
	/usr/local/go/src/bufio/bufio.go:138 +0x4f
net/http.(*persistConn).readLoop(0xc001aac900)
	/usr/local/go/src/net/http/transport.go:1978 +0x1a8
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1647 +0xc56

goroutine 488 [select]:
net/http.(*persistConn).writeLoop(0xc0080718c0)
	/usr/local/go/src/net/http/transport.go:2277 +0x11c
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1648 +0xc7b

goroutine 487 [IO wait]:
internal/poll.runtime_pollWait(0x7f43ad96a578, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc005e81198, 0x72, 0x1000, 0x109d, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc005e81180, 0xc00055a000, 0x109d, 0x109d, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc005e81180, 0xc00055a000, 0x109d, 0x109d, 0x203002, 0x598f0b, 0xc00827c35c)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0047e40d0, 0xc00055a000, 0x109d, 0x109d, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x8e
crypto/tls.(*atLeastReader).Read(0xc009e11920, 0xc00055a000, 0x109d, 0x109d, 0x0, 0xc000362a20, 0xc000532918)
	/usr/local/go/src/crypto/tls/conn.go:760 +0x60
bytes.(*Buffer).ReadFrom(0xc0001a2958, 0x1e85ec0, 0xc009e11920, 0x40a195, 0x190c800, 0x1acda20)
	/usr/local/go/src/bytes/buffer.go:204 +0xb1
crypto/tls.(*Conn).readFromUntil(0xc0001a2700, 0x1e89460, 0xc0047e40d0, 0x5, 0xc0047e40d0, 0xc0047e40b0)
	/usr/local/go/src/crypto/tls/conn.go:782 +0xec
crypto/tls.(*Conn).readRecordOrCCS(0xc0001a2700, 0x0, 0x0, 0xc00281ad20)
	/usr/local/go/src/crypto/tls/conn.go:589 +0x115
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:557
crypto/tls.(*Conn).Read(0xc0001a2700, 0xc00827c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1233 +0x15b
net/http.(*persistConn).Read(0xc0080718c0, 0xc00827c000, 0x1000, 0x1000, 0xc000532eb0, 0x460db0, 0xc000532eb0)
	/usr/local/go/src/net/http/transport.go:1825 +0x75
bufio.(*Reader).fill(0xc000362a20)
	/usr/local/go/src/bufio/bufio.go:100 +0x103
bufio.(*Reader).Peek(0xc000362a20, 0x1, 0x2, 0x0, 0x0, 0xc00281ac00, 0x0)
	/usr/local/go/src/bufio/bufio.go:138 +0x4f
net/http.(*persistConn).readLoop(0xc0080718c0)
	/usr/local/go/src/net/http/transport.go:1978 +0x1a8
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1647 +0xc56

goroutine 442 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc001aac7e0)
	/usr/local/go/src/net/http/transport.go:2277 +0x11c
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1648 +0xc7b

goroutine 440 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc001aac900)
	/usr/local/go/src/net/http/transport.go:2277 +0x11c
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1648 +0xc7b

goroutine 498 [IO wait]:
internal/poll.runtime_pollWait(0x7f43ad96ab98, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc005e81a98, 0x72, 0x7e00, 0x7eea, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc005e81a80, 0xc00f8e8000, 0x7eea, 0x7eea, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc005e81a80, 0xc00f8e8000, 0x7eea, 0x7eea, 0x203011, 0x46937469c3ed9415, 0x100000072858261)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0047e4110, 0xc00f8e8000, 0x7eea, 0x7eea, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x8e
crypto/tls.(*atLeastReader).Read(0xc044a18140, 0xc00f8e8000, 0x7eea, 0x7eea, 0xc0005379b0, 0x495b72, 0xc0005379c8)
	/usr/local/go/src/crypto/tls/conn.go:760 +0x60
bytes.(*Buffer).ReadFrom(0xc000062cd8, 0x1e85ec0, 0xc044a18140, 0x40a195, 0x190c800, 0x1acda20)
	/usr/local/go/src/bytes/buffer.go:204 +0xb1
crypto/tls.(*Conn).readFromUntil(0xc000062a80, 0x1e89460, 0xc0047e4110, 0x5, 0xc0047e4110, 0x27)
	/usr/local/go/src/crypto/tls/conn.go:782 +0xec
crypto/tls.(*Conn).readRecordOrCCS(0xc000062a80, 0x0, 0x0, 0xc000062da0)
	/usr/local/go/src/crypto/tls/conn.go:589 +0x115
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:557
crypto/tls.(*Conn).Read(0xc000062a80, 0xc008296000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1233 +0x15b
bufio.(*Reader).Read(0xc00554b260, 0xc007eabed8, 0x9, 0x9, 0x0, 0xc000537d20, 0x708b84)
	/usr/local/go/src/bufio/bufio.go:226 +0x24f
io.ReadAtLeast(0x1e85c80, 0xc00554b260, 0xc007eabed8, 0x9, 0x9, 0x9, 0xc0082910e0, 0xc008285000, 0x11)
	/usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
	/usr/local/go/src/io/io.go:329
net/http.http2readFrameHeader(0xc007eabed8, 0x9, 0x9, 0x1e85c80, 0xc00554b260, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/h2_bundle.go:1479 +0x87
net/http.(*http2Framer).ReadFrame(0xc007eabea0, 0xc04495b5c0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/h2_bundle.go:1737 +0xa1
net/http.(*http2clientConnReadLoop).run(0xc000537fa8, 0x1, 0x0)
	/usr/local/go/src/net/http/h2_bundle.go:8246 +0x8d
net/http.(*http2ClientConn).readLoop(0xc003c9d980)
	/usr/local/go/src/net/http/h2_bundle.go:8174 +0x6f
created by net/http.(*http2Transport).newClientConn
	/usr/local/go/src/net/http/h2_bundle.go:7174 +0x64a

goroutine 441 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7f43ad96a038, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc005e81b18, 0x72, 0x1000, 0x109d, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc005e81b00, 0xc003d27300, 0x109d, 0x109d, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc005e81b00, 0xc003d27300, 0x109d, 0x109d, 0x203000, 0x598f0b, 0xc003d5018d)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0047e5ca0, 0xc003d27300, 0x109d, 0x109d, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x8e
crypto/tls.(*atLeastReader).Read(0xc003d12fc0, 0xc003d27300, 0x109d, 0x109d, 0x0, 0xc003cd1e00, 0xc000096918)
	/usr/local/go/src/crypto/tls/conn.go:760 +0x60
bytes.(*Buffer).ReadFrom(0xc002ca1058, 0x1e85ec0, 0xc003d12fc0, 0x40a195, 0x190c800, 0x1acda20)
	/usr/local/go/src/bytes/buffer.go:204 +0xb1
crypto/tls.(*Conn).readFromUntil(0xc002ca0e00, 0x1e89460, 0xc0047e5ca0, 0x5, 0xc0047e5ca0, 0xc0047e5d78)
	/usr/local/go/src/crypto/tls/conn.go:782 +0xec
crypto/tls.(*Conn).readRecordOrCCS(0xc002ca0e00, 0x0, 0x0, 0xc002c49740)
	/usr/local/go/src/crypto/tls/conn.go:589 +0x115
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:557
crypto/tls.(*Conn).Read(0xc002ca0e00, 0xc003d50000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1233 +0x15b
net/http.(*persistConn).Read(0xc001aac7e0, 0xc003d50000, 0x1000, 0x1000, 0xc000096eb0, 0x460db0, 0xc000096eb0)
	/usr/local/go/src/net/http/transport.go:1825 +0x75
bufio.(*Reader).fill(0xc003cd1e00)
	/usr/local/go/src/bufio/bufio.go:100 +0x103
bufio.(*Reader).Peek(0xc003cd1e00, 0x1, 0x2, 0x0, 0x0, 0xc002c49600, 0x0)
	/usr/local/go/src/bufio/bufio.go:138 +0x4f
net/http.(*persistConn).readLoop(0xc001aac7e0)
	/usr/local/go/src/net/http/transport.go:1978 +0x1a8
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1647 +0xc56

goroutine 460 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7f43ad96a1f8, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc0000d8698, 0x72, 0xa500, 0xa509, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0000d8680, 0xc0003c6000, 0xa509, 0xa509, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc0000d8680, 0xc0003c6000, 0xa509, 0xa509, 0x203002, 0x638ec0, 0xc0001a2138)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0047e5e38, 0xc0003c6000, 0xa509, 0xa509, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x8e
crypto/tls.(*atLeastReader).Read(0xc008060ca0, 0xc0003c6000, 0xa509, 0xa509, 0x19, 0xa2ca, 0xc0005359c8)
	/usr/local/go/src/crypto/tls/conn.go:760 +0x60
bytes.(*Buffer).ReadFrom(0xc0001a2258, 0x1e85ec0, 0xc008060ca0, 0x40a195, 0x190c800, 0x1acda20)
	/usr/local/go/src/bytes/buffer.go:204 +0xb1
crypto/tls.(*Conn).readFromUntil(0xc0001a2000, 0x1e89460, 0xc0047e5e38, 0x5, 0xc0047e5e38, 0x9)
	/usr/local/go/src/crypto/tls/conn.go:782 +0xec
crypto/tls.(*Conn).readRecordOrCCS(0xc0001a2000, 0x0, 0x0, 0xc000535d38)
	/usr/local/go/src/crypto/tls/conn.go:589 +0x115
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:557
crypto/tls.(*Conn).Read(0xc0001a2000, 0xc000390000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1233 +0x15b
bufio.(*Reader).Read(0xc003c5c2a0, 0xc00062a818, 0x9, 0x9, 0xc000535d38, 0x1bf0000, 0x6d67f5)
	/usr/local/go/src/bufio/bufio.go:226 +0x24f
io.ReadAtLeast(0x1e85c80, 0xc003c5c2a0, 0xc00062a818, 0x9, 0x9, 0x9, 0xc000074040, 0x0, 0x1e86140)
	/usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
	/usr/local/go/src/io/io.go:329
net/http.http2readFrameHeader(0xc00062a818, 0x9, 0x9, 0x1e85c80, 0xc003c5c2a0, 0x0, 0x0, 0xc00066a120, 0x0)
	/usr/local/go/src/net/http/h2_bundle.go:1479 +0x87
net/http.(*http2Framer).ReadFrame(0xc00062a7e0, 0xc00066a120, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/h2_bundle.go:1737 +0xa1
net/http.(*http2clientConnReadLoop).run(0xc000535fa8, 0xc000aa8c60, 0xc00005b750)
	/usr/local/go/src/net/http/h2_bundle.go:8246 +0x8d
net/http.(*http2ClientConn).readLoop(0xc000001980)
	/usr/local/go/src/net/http/h2_bundle.go:8174 +0x6f
created by net/http.(*http2Transport).newClientConn
	/usr/local/go/src/net/http/h2_bundle.go:7174 +0x64a
section_end:1585693463:build_script
�[0Ksection_start:1585693463:after_script
�[0K�[0K�[36;1mRunning after_script�[0;m
�[0;msection_end:1585693464:after_script
�[0Ksection_start:1585693464:upload_artifacts_on_failure
�[0K�[0K�[36;1mUploading artifacts for failed job�[0;m
�[0;msection_end:1585693466:upload_artifacts_on_failure
�[0K�[31;1mERROR: Job failed: exit code 2
�[0;m

# GitLab Ci job

build Docker image:
  image:
    name: gcr.io/kaniko-project/executor:debug-v0.18.0
    entrypoint: [""]
  variables:
    CONTAINER_TEST_IMAGE: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG:$CI_COMMIT_SHA
    CONTAINER_TEST_IMAGE_LATEST: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG:latest
    CONTAINER_RELEASE_IMAGE: $CI_REGISTRY_IMAGE:latest
    CONTAINER_DEVELOP_IMAGE: $CI_REGISTRY_IMAGE/develop:latest
  script:
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
    - /kaniko/executor
        --context $CI_PROJECT_DIR
        --dockerfile $CI_PROJECT_DIR/Dockerfile.dev
        --cache=true
        --cache-repo=$CI_REGISTRY_IMAGE/cache
        --reproducible=true
        --build-arg BUNDLE_GEMS=${BUNDLE_GEMS?"BUNDLE_GEMS must be set"}
        --destination $CONTAINER_TEST_IMAGE
        --destination $CONTAINER_TEST_IMAGE_LATEST
  stage: pre-test and build

Triage Notes for the Maintainers

Description Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile
@tmaier
Copy link
Author

tmaier commented Apr 22, 2020

The problem seems to be related to #862. I am closing this as a duplicate

@tmaier tmaier closed this as completed Apr 22, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@tmaier