Extracting vmlinux and binary info from Android target

[imgod2u]

Hi all,

I've been following the instructions here: https://android.googlesource.com/platform/system/extras/+/master/simpleperf/doc/collect_etm_data_for_autofdo.md

For an Android system. After much kernel and bootloader hacking, I've finally gotten CoreSight up and running and was able to collect a cs-etm trace via simpleperf. What I'm stuck on now is this step:

# pull ETM data to host.
host $ adb pull /data/local/tmp/branch_list.data
# download vmlinux and kernel modules to <binary_dir>
# host simpleperf is in <aosp-top>/system/extras/simpleperf/scripts/bin/linux/x86_64/simpleperf,
# or you can build simpleperf by `mmma system/extras/simpleperf`.
host $ simpleperf inject --symdir <binary_dir> -i branch_list.data

I can pull the branch_list.data just fine but I'm not sure where to get vmlinux and kernel modules. This is an Android 12 target with Linux kernel 4.19.157.

Also, if I were to try to inject the binary info offline (not on target), what files would I need to download? The guide here:

https://github.com/Linaro/OpenCSD/blob/master/HOWTO.md

Indicates that there's a ~/.debug folder for Linaro linux. An equivalent doesn't seem to exist on Android...

I want to also use OpenCSD to decode the instruction stream...

[enh-google]

@yabinc

[yabinc]

I'm not sure where to get vmlinux and kernel modules

Not stripped vmlinux and kernel modules are not available on device. They are generated when you build the kernel. For example, if you use hikey960 board as in https://source.android.com/docs/setup/build/devices#960kernel. You can build the kernel locally, flash it on device, collect ETM data, pull branch_list.data on host. Then you can use vmlinux and kernel modules generated when building the kernel to produce AutoFDO profiles.

if I were to try to inject the binary info offline (not on target), what files would I need to download?

You can just pull perf.data (generated by record -e cs-etm) on host. And run simpleperf inject on host.
For branch-list output, simpleperf inject only needs perf.data.
For autofdo output, simpleperf inject needs the corresponding binary. either stripped or non-stripped. provided via --symdir option.
For create_llvm_prof (in autofdo) to work, it needs non-stripped binary.

I want to also use OpenCSD to decode the instruction stream...

OpenCSD is a library. I don't know how you can use it directly. Internally, simpleperf inject uses OpenCSD to decode the instruction stream. Simpleperf is open source, so you can modify simpleperf inject locally to do whatever you want. Or you can try linux tools perf.
Simpleperf stores ETM data in perf.data in the same format as linux tools perf. So it's likely to work out of box.

By the way, we have a patch updating collect_etm_data_for_autofdo.md, in https://android-review.googlesource.com/c/platform/system/extras/+/2200048. Not sure if that's helpful to you.

[imgod2u]

Thanks for the reply. I think I'm confused about this part:

For autofdo output, simpleperf inject needs the corresponding binary. either stripped or non-stripped. provided via --symdir option.

Where would I point the --symdir directory on the host? I should mention this is a system-wide trace I collected (simpleperf record -a) so it's not app specific. For sampled data, Android NDK provides a binary_cache_builder.py script that downloads all of the required binaries. Is there something similar that can be done for cs-etm data?

OpenCSD is a library. I don't know how you can use it directly. Internally, simpleperf inject uses OpenCSD to decode the instruction stream.

I see. I guess my issue is that I want to decode both userspace and kernel space instruction streams all in one go. And have the output format as a "list-of-instructions". What would you recommend as the most direct way to do that?

[yabinc]

Where would I point the --symdir directory on the host?

You need to build a directory manually. Put binaries (better not stripped binaries) you want to decode in the directory. And pass the directory path to simpleperf inject via --symdir option. There isn't something similar to binary_cache_builder.py. If needed, you can use adb pull to pull binaries on host.

I guess my issue is that I want to decode both userspace and kernel space instruction streams all in one go. And have the output format as a "list-of-instructions". What would you recommend as the most direct way to do that?

ETM data is a bunch of branch choice information. You need binaries (both userspace and vmlinux) to convert the branch choices into instruction streams. The AutoFDO format of simpleperf generates a list of executed instruction ranges. I guess you can modify simpleperf inject code to let it output executed instruction streams.

[imgod2u]

Thanks! I realize that if I do simpleperf inject on-device, it'll just detect the userspace binaries. Then all I need is to grab the vmlinux and push it onto the device and point simpleperf to it. So now I can have both user and kernel space injection in one go.

One follow-on question is about AutoFDO format. You said that it generates a "list of executed instruction ranges". Is there documentation somewhere (I googled but couldn't find it) of how exactly this looks like? Maybe this is all I need and I can write a custom program to decode it. Messing with simpleperf code would be a bit more out of my wheelhouse.

[yabinc]

It's actually an interface of using OpenCSD library. Document is https://github.com/Linaro/OpenCSD/blob/master/decoder/docs/prog_guide/prog_guide_generic_pkts.md#ocsd_gen_trc_elem_instr_range. But maybe the code is easier to understand.

A clear view of code is from https://android.googlesource.com/platform/system/extras/+/master/simpleperf/ETMDecoder.h#72. ETMDecoder is a wrapper of using OpenCSD. One function of ETMDecoder is

using InstrRangeCallbackFn = std::function<void(const ETMInstrRange&)>;
virtual void RegisterCallback(const InstrRangeCallbackFn& callback) = 0;

The callback is called whenever OpenCSD figures out a range of executed instructions. ETMInstrRange represents the range:

struct ETMInstrRange {
  // the binary containing the instruction range
  Dso* dso = nullptr;
  // the address of the first instruction in the binary
  uint64_t start_addr = 0;
  // the address of the last instruction in the binary
  uint64_t end_addr = 0;
  // If the last instruction is a branch instruction, and it branches
  // to a fixed location in the same binary, then branch_to_addr points
  // to the branched to instruction.
  uint64_t branch_to_addr = 0;
  // times the branch is taken
  uint64_t branch_taken_count = 0;
  // times the branch isn't taken
  uint64_t branch_not_taken_count = 0;
  };

ETMInstrRange mixes instruction ranges from different cpus. To distinguish them, you can add trace_id or cpu field.
To get executed instructions in time order, you need to convert from perf.data to autofdo directly. Because branch-list output will combine branch lists.