Table 'http_events' not found on GKE 1.28.3

[maxlemieux]

Describe the bug
Scripts which use eBPF fail to run on GKE 1.28.3.

To Reproduce
Steps to reproduce the behavior:

1. Install Pixie on a new GKE Standard cluster (version 1.28.3-gke.1286000). I used machine type e2-standard-2, and installed through the New Relic chart bundle.
2. Go to the px/cluster script in Pixie.
3. See error:

Failed to execute script
Compiler error on line 33, column 22: Table 'http_events' not found..

Only the px/agent_status script seems to work.

Expected behavior
Expected eBPF scripts to run.

Screenshots

Logs
I couldn't use px logs to collect the logs since this is using a New Relic shadow account for Pixie. Here are the vizier-pem logs showing an error:

Click me

Defaulted container "pem" out of: pem, qb-wait (init)
I20240411 02:00:35.018565 11276 env.cc:47] Started: /app/src/vizier/services/agent/pem/pem
Started external stacktrace collection signal processor thread
I20240411 02:00:35.019026 11276 kernel_version.cc:199] Found Linux kernel version using .note section.
I20240411 02:00:35.019078 11276 pem_main.cc:68] Pixie PEM. Version: v0.14.8+Distribution.ce1e3f3.202312122235.1.RELEASE.jenkins, id: 23dcaa75-0167-47f1-a5bc-6a9a21038d05, kernel version: 6.1.58
I20240411 02:00:35.019224 11276 stirling.cc:958] Creating Stirling, registered sources: [process_stats, network_stats, jvm_stats, socket_tracer, perf_profiler, proc_exit_tracer, stirling_error]
I20240411 02:00:35.019318 11276 system_info.cc:42] Location of proc: /proc
I20240411 02:00:35.019340 11276 system_info.cc:43] Location of sysfs: /sys/fs
I20240411 02:00:35.019387 11276 system_info.cc:44] Number of CPUs: 2
I20240411 02:00:35.019507 11276 system_info.cc:35] /proc/version:
Linux version 6.1.58+ (builder@d367cb498058) (Chromium OS 16.0_pre484197_p20230405-r10 clang version 16.0.0 (/var/tmp/portage/sys-devel/llvm-16.0_pre484197_p20230405-r10/work/llvm-16.0_pre484197_p20230405/clang 2916b99182752b1aece8cc4479d8d6a20b5e02da), LLD 16.0.0) #1 SMP PREEMPT_DYNAMIC Sat Nov  4 14:14:29 UTC 2023
I20240411 02:00:35.019842 11276 system_info.cc:35] /host/etc/lsb-release:
CHROMEOS_AUSERVER=https://tools.google.com/service/update2
CHROMEOS_BOARD_APPID={76E245CF-C0D0-444D-BA50-36739C18EB00}
CHROMEOS_CANARY_APPID={90F229CE-83E2-4FAF-8479-E368A34938B1}
CHROMEOS_DEVSERVER=
CHROMEOS_RELEASE_APPID={76E245CF-C0D0-444D-BA50-36739C18EB00}
CHROMEOS_RELEASE_BOARD=lakitu
CHROMEOS_RELEASE_BRANCH_NUMBER=66
CHROMEOS_RELEASE_BUILD_NUMBER=17800
CHROMEOS_RELEASE_BUILD_TYPE=Official Build
CHROMEOS_RELEASE_CHROME_MILESTONE=109
CHROMEOS_RELEASE_DESCRIPTION=17800.66.19 (Official Build) stable-channel lakitu 
CHROMEOS_RELEASE_KEYSET=v10
CHROMEOS_RELEASE_NAME=Chrome OS
CHROMEOS_RELEASE_PATCH_NUMBER=19
CHROMEOS_RELEASE_TRACK=stable-channel
CHROMEOS_RELEASE_VERSION=17800.66.19
DEVICETYPE=OTHER
GOOGLE_RELEASE=17800.66.19
HWID_OVERRIDE=LAKITU DEFAULT
I20240411 02:00:35.019901 11276 system_info.cc:35] /host/etc/os-release:
NAME="Container-Optimized OS"
ID=cos
PRETTY_NAME="Container-Optimized OS from Google"
HOME_URL="https://cloud.google.com/container-optimized-os/docs"
BUG_REPORT_URL="https://cloud.google.com/container-optimized-os/docs/resources/support-policy#contact_us"
GOOGLE_METRICS_PRODUCT_ID=26
KERNEL_COMMIT_ID=77d913a39ed76e488be139379dffe44e172527f9
GOOGLE_CRASH_ID=Lakitu
VERSION=109
VERSION_ID=109
BUILD_ID=17800.66.19
I20240411 02:00:35.019943 11276 probe_cleaner.cc:102] Cleaning probes from /sys/kernel/debug/tracing/kprobe_events with the following marker: __pixie__
I20240411 02:00:35.020022 11276 probe_cleaner.cc:117] All Stirling probes removed (count=0)
I20240411 02:00:35.020035 11276 probe_cleaner.cc:102] Cleaning probes from /sys/kernel/debug/tracing/uprobe_events with the following marker: __pixie__
I20240411 02:00:35.020079 11276 probe_cleaner.cc:117] All Stirling probes removed (count=0)
I20240411 02:00:35.020092 11276 source_connector.cc:35] Initializing source connector: process_stats
I20240411 02:00:35.020104 11276 stirling.cc:438] Adding info class: [process_stats/process_stats]
I20240411 02:00:35.020120 11276 source_connector.cc:35] Initializing source connector: network_stats
I20240411 02:00:35.020133 11276 stirling.cc:438] Adding info class: [network_stats/network_stats]
I20240411 02:00:35.020159 11276 source_connector.cc:35] Initializing source connector: jvm_stats
I20240411 02:00:35.020172 11276 stirling.cc:438] Adding info class: [jvm_stats/jvm_stats]
I20240411 02:00:35.020443 11276 source_connector.cc:35] Initializing source connector: socket_tracer
I20240411 02:00:35.020495 11276 kernel_version.cc:82] Obtained Linux version string from `uname`: 6.1.58+
I20240411 02:00:35.020514 11276 linux_headers.cc:385] Detected kernel release (uname -r): 6.1.58+
I20240411 02:00:35.021205 11276 linux_headers.cc:400] Not Found : Could not find 'source' or 'build' under /lib/modules/6.1.58+.
I20240411 02:00:35.021323 11276 linux_headers.cc:248] Looking for host Linux headers at /host/lib/modules/6.1.58+/build.
I20240411 02:00:35.021364 11276 linux_headers.cc:403] Not Found : Did not find host headers at path: /host/lib/modules/6.1.58+/build.
I20240411 02:00:35.021394 11276 linux_headers.cc:346] Attempting to install packaged headers.
W20240411 02:00:35.021690 11276 linux_headers.cc:319] Ignoring /px/linux-headers-x86_64-4.14.304.tar.gz since it does not conform to the naming convention
W20240411 02:00:35.021770 11276 linux_headers.cc:319] Ignoring /px/linux-headers-x86_64-4.19.271.tar.gz since it does not conform to the naming convention
I20240411 02:00:35.021883 11276 linux_headers.cc:352] Using packaged header: /px/linux-headers-x86_64-6.1.8.tar.gz
I20240411 02:00:35.707615 11276 linux_headers.cc:56] Overriding linux version code to 393530
I20240411 02:00:35.708001 11276 kernel_version.cc:82] Obtained Linux version string from `uname`: 6.1.58+
I20240411 02:00:35.708133 11276 linux_headers.cc:98] Found kernel config at: /proc/config.gz.
I20240411 02:00:35.712800 11276 linux_headers.cc:377] Successfully installed packaged copy of headers at /lib/modules/6.1.58+/build
I20240411 02:00:35.712833 11276 bcc_wrapper.cc:94] Resolving task_struct offsets.
I20240411 02:00:35.801630 11290 bcc_wrapper.cc:166] Initializing BPF program ...
I20240411 02:00:36.804966 11290 scoped_timer.h:48] Timer(init_bpf_program) : 1.00 s
cannot create /var/tmp/bcc
WARNING: cannot get prog tag, ignore saving source with program tag
I20240411 02:00:36.817472 11276 bcc_wrapper.cc:166] Initializing BPF program ...
I20240411 02:00:37.834250 11276 scoped_timer.h:48] Timer(init_bpf_program) : 1.02 s
cannot create /var/tmp/bcc
WARNING: cannot get prog tag, ignore saving source with program tag
I20240411 02:00:37.849454 11276 bcc_wrapper.cc:97] Successfully resolved task_struct offsets: {real_start_time=1832, group_leader=1584, exit_code=1416}
I20240411 02:00:37.849560 11276 bcc_wrapper.cc:166] Initializing BPF program ...
src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1300:25: error: expected ')'
    if (({ typeof(void  btf_type_tag(user)*const) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_name); _val; }) != NULL) {
                        ^
src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1300:18: note: to match this '('
    if (({ typeof(void  btf_type_tag(user)*const) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_name); _val; }) != NULL) {
                 ^
src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1300:51: error: variable has incomplete type 'typeof(void)' (aka 'void')
    if (({ typeof(void  btf_type_tag(user)*const) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_name); _val; }) != NULL) {
                                                  ^
src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1303:43: error: expected ')'
      connect_args.addr = ({ typeof(void  btf_type_tag(user)*const) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_name); _val; });
                                          ^
src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1303:36: note: to match this '('
      connect_args.addr = ({ typeof(void  btf_type_tag(user)*const) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_name); _val; });
                                   ^
src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1303:69: error: variable has incomplete type 'typeof(void)' (aka 'void')
      connect_args.addr = ({ typeof(void  btf_type_tag(user)*const) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_name); _val; });
                                                                    ^
src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1311:46: error: expected ')'
    write_args.iov = ({ typeof(struct iovec  btf_type_tag(user)*const) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_iov); _val; });
                                             ^
src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1311:31: note: to match this '('
    write_args.iov = ({ typeof(struct iovec  btf_type_tag(user)*const) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_iov); _val; });
                              ^
src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1311:20: error: assigning to 'const struct iovec *' from incompatible type 'typeof(struct iovec)' (aka 'struct iovec')
    write_args.iov = ({ typeof(struct iovec  btf_type_tag(user)*const) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_iov); _val; });
                   ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1424:25: error: expected ')'
    if (({ typeof(void  btf_type_tag(user)*) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_name); _val; }) != NULL) {
                        ^
src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1424:18: note: to match this '('
    if (({ typeof(void  btf_type_tag(user)*) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_name); _val; }) != NULL) {
                 ^
src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1424:46: error: variable has incomplete type 'typeof(void)' (aka 'void')
    if (({ typeof(void  btf_type_tag(user)*) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_name); _val; }) != NULL) {
                                             ^
src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1427:43: error: expected ')'
      connect_args.addr = ({ typeof(void  btf_type_tag(user)*) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_name); _val; });
                                          ^
src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1427:36: note: to match this '('
      connect_args.addr = ({ typeof(void  btf_type_tag(user)*) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_name); _val; });
                                   ^
src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1427:64: error: variable has incomplete type 'typeof(void)' (aka 'void')
      connect_args.addr = ({ typeof(void  btf_type_tag(user)*) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_name); _val; });
                                                               ^
src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1435:45: error: expected ')'
    read_args.iov = ({ typeof(struct iovec  btf_type_tag(user)*) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_iov); _val; });
                                            ^
src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1435:30: note: to match this '('
    read_args.iov = ({ typeof(struct iovec  btf_type_tag(user)*) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_iov); _val; });
                             ^
src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1435:19: error: assigning to 'const struct iovec *' from incompatible type 'typeof(struct iovec)' (aka 'struct iovec')
    read_args.iov = ({ typeof(struct iovec  btf_type_tag(user)*) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_iov); _val; });
                  ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
12 errors generated.
I20240411 02:00:41.324113 11276 scoped_timer.h:48] Timer(init_bpf_program) : 3.47 s
W20240411 02:00:41.326061 11276 stirling.cc:416] Source Connector (registry name=socket_tracer) not instantiated, error: Internal : Unable to initialize BCC BPF program: Unable to initialize BPF program
I20240411 02:00:41.326159 11276 source_connector.cc:35] Initializing source connector: perf_profiler
I20240411 02:00:41.326224 11276 kernel_version.cc:82] Obtained Linux version string from `uname`: 6.1.58+
I20240411 02:00:41.326244 11276 linux_headers.cc:385] Detected kernel release (uname -r): 6.1.58+
I20240411 02:00:41.326279 11276 linux_headers.cc:206] Using Linux headers from: /lib/modules/6.1.58+/build.
I20240411 02:00:41.326292 11276 bcc_wrapper.cc:90] Returning the previously resolved TaskStructOffsets object
I20240411 02:00:41.326350 11276 bcc_wrapper.cc:166] Initializing BPF program ...
I20240411 02:00:43.133134 11276 scoped_timer.h:48] Timer(init_bpf_program) : 1.81 s
cannot create /var/tmp/bcc
WARNING: cannot get prog tag, ignore saving source with program tag
I20240411 02:00:43.134491 11276 perf_profile_connector.cc:153] PerfProfiler: Stack trace profiling sampling probe successfully deployed.
I20240411 02:00:43.134653 11276 perf_profile_connector.cc:169] PerfProfiler: Java symbolization enabled.
I20240411 02:00:43.134789 11276 java_symbolizer.cc:233] JavaSymbolizer found agent lib /px/libpx-java-agent.so.
I20240411 02:00:43.134922 11276 stirling.cc:438] Adding info class: [perf_profiler/stack_traces.beta]
I20240411 02:00:43.135090 11276 source_connector.cc:35] Initializing source connector: proc_exit_tracer
I20240411 02:00:43.135188 11276 kernel_version.cc:82] Obtained Linux version string from `uname`: 6.1.58+
I20240411 02:00:43.135290 11276 linux_headers.cc:385] Detected kernel release (uname -r): 6.1.58+
I20240411 02:00:43.135391 11276 linux_headers.cc:206] Using Linux headers from: /lib/modules/6.1.58+/build.
I20240411 02:00:43.135548 11276 bcc_wrapper.cc:90] Returning the previously resolved TaskStructOffsets object
I20240411 02:00:43.135702 11276 bcc_wrapper.cc:166] Initializing BPF program ...
I20240411 02:00:43.976743 11276 scoped_timer.h:48] Timer(init_bpf_program) : 840.96 ms
I20240411 02:00:43.976873 11276 proc_exit_connector.cc:86] Writing exit_code's offset to BPF array: offset=1416 bpf_array=control_values index=0
cannot create /var/tmp/bcc
WARNING: cannot get prog tag, ignore saving source with program tag
I20240411 02:00:43.982354 11276 stirling.cc:438] Adding info class: [proc_exit_tracer/proc_exit_events]
I20240411 02:00:43.982455 11276 source_connector.cc:35] Initializing source connector: stirling_error
I20240411 02:00:43.982564 11276 stirling.cc:438] Adding info class: [stirling_error/stirling_error]
I20240411 02:00:43.982631 11276 stirling.cc:438] Adding info class: [stirling_error/probe_status]
I20240411 02:00:43.982671 11276 stirling.cc:419] Stirling successfully initialized.
E0411 02:00:43.983477093   11276 dns_resolver_ares.cc:456]             no server name supplied in dns URI
E0411 02:00:43.983625510   11276 channel.cc:120]                       channel stack builder failed: UNKNOWN: the target uri is not valid: dns:///
I20240411 02:00:43.994021 11276 manager.cc:154] Hostname: gke-mlemieux-gke-pool-1-e24203b4-vksd
I20240411 02:01:07.767810 11276 cgroup_path_resolver.cc:141] Auto-discovered CGroup base path: /sys/fs/cgroup
I20240411 02:01:07.773299 11276 cgroup_path_resolver.cc:144] Auto-discovered example path: /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod581052f4_3e86_404a_87c6_267ac4c2f39a.slice/cri-containerd-e59fc320752e647cd963a1561602e3b0f744f209801def33a1259175ef182714.scope/cgroup.procs
I20240411 02:01:07.773522 11276 cgroup_path_resolver.cc:148] Auto-discovered template: /sys/fs/cgroup/kubepods.slice/kubepods-$2.slice/kubepods-$2-pod$0.slice/cri-containerd-$1.scope/cgroup.procs
I20240411 02:01:07.773583 11276 cgroup_metadata_reader.cc:41] Using path_resolver with configuration: template=/sys/fs/cgroup/kubepods.slice/kubepods-$2.slice/kubepods-$2-pod$0.slice/cri-containerd-$1.scope/cgroup.procs pod_id_separators=_ qos_spearator=-
I20240411 02:01:07.774121 11656 stirling.cc:792] Stirling is running.
I20240411 02:01:07.775919 11656 perf_profile_connector.cc:428] PerfProfileConnector statistics: kBPFMapSwitchoverEvent=1 kCumulativeSumOfAllStackTraces=4480 kLossHistoEvent=0 
I20240411 02:01:07.775952 11656 perf_profile_connector.cc:442] PerfProfileConnector u_symbolizer num_symbols_cached=0 hits=0 accesses=0 hit_rate=0
I20240411 02:01:07.775966 11656 perf_profile_connector.cc:445] PerfProfileConnector k_symbolizer num_symbols_cached=0 hits=0 accesses=0 hit_rate=0
I20240411 02:01:07.850221 11276 heartbeat.cc:143] Heartbeat ACK latency moving average: 3 ms
I20240411 02:01:07.875310 11656 run_core_stats.cc:111] |main_loop_iters,no_work_iters,useful_iters,push+transfer,transfer,push,min_push+transfer,max_push+transfer,total_0.00_ms,total_0.01_ms,total_0.02_ms,total_0.03_ms,total_0.06_ms,total_0.10_ms,total_0.18_ms,total_0.32_ms,total_0.56_ms,total_1.00_ms,total_1.78_ms,total_3.16_ms,total_5.62_ms,total_10.00_ms,total_17.78_ms,total_31.62_ms,total_56.23_ms,total_100.00_ms,total_177.83_ms,total_316.23_ms,total_562.34_ms,total_1000.00_ms,total_10000000.00_ms,no_work_0.00_ms,no_work_0.01_ms,no_work_0.02_ms,no_work_0.03_ms,no_work_0.06_ms,no_work_0.10_ms,no_work_0.18_ms,no_work_0.32_ms,no_work_0.56_ms,no_work_1.00_ms,no_work_1.78_ms,no_work_3.16_ms,no_work_5.62_ms,no_work_10.00_ms,no_work_17.78_ms,no_work_31.62_ms,no_work_56.23_ms,no_work_100.00_ms,no_work_177.83_ms,no_work_316.23_ms,no_work_562.34_ms,no_work_1000.00_ms,no_work_10000000.00_ms
I20240411 02:01:08.927611 11276 exec.cc:97] Queries in flight: 0
I20240411 02:01:08.928225 11663 exec.cc:51] Executing query: id=39cc10f9-ac60-4869-a9bb-73e15cb7b03a
I20240411 02:01:08.945636 11663 exec.cc:63] Completed query: id=39cc10f9-ac60-4869-a9bb-73e15cb7b03a
W20240411 02:01:12.769654 11276 state_manager.cc:276] Failed to read PID info for pod=882acabe-2a97-44cd-a81d-af3ead9e3915, cid=ed9e71ab6216ed2de124fa02e755c7910220ad446b6aaeed546669d5d3136c30 [msg=Failed to open file /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod882acabe_2a97_44cd_a81d_af3ead9e3915.slice/cri-containerd-ed9e71ab6216ed2de124fa02e755c7910220ad446b6aaeed546669d5d3136c30.scope/cgroup.procs]
W20240411 02:01:12.770524 11276 state_manager.cc:276] Failed to read PID info for pod=9cc0bf12-a9d1-4fe7-98b6-4b0342d970f3, cid=59f0291a730159a7c735e1a0577f2927f8aac63263d9cb02523bdab585782ee6 [msg=Failed to open file /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cc0bf12_a9d1_4fe7_98b6_4b0342d970f3.slice/cri-containerd-59f0291a730159a7c735e1a0577f2927f8aac63263d9cb02523bdab585782ee6.scope/cgroup.procs]
W20240411 02:01:12.770881 11276 state_manager.cc:276] Failed to read PID info for pod=581052f4-3e86-404a-87c6-267ac4c2f39a, cid= [msg=Failed to open file /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod581052f4_3e86_404a_87c6_267ac4c2f39a.slice/cri-containerd-.scope/cgroup.procs]

App information (please complete the following information):

• Pixie version: 0.14.8
• K8s cluster version: 1.28.3
• Node Kernel version: Kernel 6.1.58+ (Container-Optimized OS with containerd (cos_containerd))
• Browser version: latest Chrome

Additional context
Please let me know if I can provide any more information about this issue. It's also affecting 0.14.6, but did not reportedly happen in the same environment with 0.12.x.

[ddelnano]

@maxlemieux this will be fixed in the next Pixie release v0.14.10 and should be available sometime early this week. This issue was addressed in #1909.

[ddelnano]

v0.14.10 is released.