jq syntax errors cause the configuration to fail

[hishamhm]

Originally reported by @teb510

i can grab the response from headers from a callnode - but the filter seems to bomb with headers that are multiple words connected by a "-" - downcasing/snakecasing/etc doesn't do anything

no issue with single word response headers (vary/nel/etc)

- name: CAT_FACT
            type: call
            url:  https://catfact.ninja/fact

          - name: CHUCK_NORRIS_FACT
            type: call
            url:  https://api.chucknorris.io/jokes/random

          - inputs:
            - cat: CAT_FACT.body
            - chuck: CHUCK_NORRIS_FACT.body
            - chuck_headers: CHUCK_NORRIS_FACT.headers
            name: JOIN
            jq: |
              {
                 "cat_fact": $cat.fact,
                 "chuck_norris_fact": $chuck.value,
                 "cf-ray-chuck": $chuck_headers.cf-ray || $chuck_headers.cf_ray , // either one will error - i don't try to use || in the config :)
              }
            type: jq

          - name: EXIT
            type: exit
            inputs:
            - body: JOIN
            status: 200

^^ will cause an error that looks like:

Caused by:
kong-db-less  |     wasm trap: wasm `unreachable` instruction executed <module: "datakit", vm: "main", runtime: "wasmtime">
kong-db-less  | 2024/11/04 08:34:55 [error] 2529#0: *6453 [proxy-wasm]["datakit" #3] filter compile error: undefined filter while closing request, client: 192.168.65.1, server: 0.0.0.0:8000
kong-db-less  | 2024/11/04 08:34:55 [crit] 2529#0: *6453 [proxy-wasm]["datakit" #3] panicked at src/nodes/jq.rs:237:51:
kong-db-less  | called `Result::unwrap()` on an `Err` value: "filter compilation failed" while closing request, client: 192.168.65.1, server: 0.0.0.0:8000
kong-db-less  | 2024/11/04 08:34:55 [error] 2529#0: *6453 [wasm] error while executing at wasm backtrace:
kong-db-less  |     0: 0x39ec0e - datakit.wasm!__rust_start_panic
kong-db-less  |     1: 0x39ead3 - datakit.wasm!rust_panic
kong-db-less  |     2: 0x39e889 - datakit.wasm!std::panicking::rust_panic_with_hook::h50e657195af0239c
kong-db-less  |     3: 0x39db4b - datakit.wasm!std::panicking::begin_panic_handler::{{closure}}::h0187e6969a85aab0
kong-db-less  |     4: 0x39dab2 - datakit.wasm!std::sys_common::backtrace::__rust_end_short_backtrace::h575fb82445d56667
kong-db-less  |     5: 0x39e421 - datakit.wasm!rust_begin_unwind
kong-db-less  |     6: 0x3a48d6 - datakit.wasm!core::panicking::panic_fmt::ha6764f2272b7fb95
kong-db-less  |     7: 0x3abfc2 - datakit.wasm!core::result::unwrap_failed::had3752ac9f022e32
kong-db-less  |     8: 0x81a5c - datakit.wasm!<datakit::nodes::jq::JqFactory as datakit::nodes::NodeFactory>::new_node::h5b626fe6983cb32b
kong-db-less  |     9: 0x463b2 - datakit.wasm!datakit::nodes::new_node::h47b9a8934de025ad
kong-db-less  |    10: 0x4de45 - datakit.wasm!datakit::config::Config::build_nodes::hf2fdf362a5a9b71c
kong-db-less  |    11: 0x7ae23 - datakit.wasm!<datakit::DataKitFilterRootContext as proxy_wasm::traits::RootContext>::create_http_context::haaec3287cebb434d
kong-db-less  |    12: 0x3898ec - datakit.wasm!proxy_wasm::dispatcher::Dispatcher::create_http_context::h4bac58e69dab0f8b
kong-db-less  |    13: 0x38a587 - datakit.wasm!proxy_wasm::dispatcher::Dispatcher::on_create_context::h03839dfd114fc01c
kong-db-less  |    14: 0x39514e - datakit.wasm!proxy_on_context_create
kong-db-less  | 
kong-db-less  | Caused by:

[hishamhm]

@teb510 Root cause here is that the jq syntax is invalid. The giveaway is the message "filter compilation failed", which is internal to the jq node. For our first alpha MVP that was fine, but yes, we should try to provide better diagnostics for this. From a quick look at the jq library we're using, the good news is that I think we can! https://docs.rs/jaq-interpret/latest/jaq_interpret/struct.ParseCtx.html#structfield.errs

@flrgh My first approach at thinking about this is that we could, instead of failing at building the node, succeed and store the errors, and then on run(), we'd make it return Fail to produce an HTTP 500, and then include the error diagnostics in the Fail payload, so that they appear as part of the debug trace. How does that sound?

[teb510]

hey @hishamhm i'm still seeing errors here with a variety of strategies and seeing success for single word headers.

maybe the right question is how do i refer to the value of a header, and in particular, one with a two word name within a jq node? i'm seeing single word headers work and multi word headers fail when using the $ not using the $ will fail for even single word headers..

for example: the below works -

 - name: JOIN
            inputs:
            - cat: CAT_FACT.body
            - chuck: CHUCK_NORRIS_FACT.body
            - chuck_headers: CHUCK_NORRIS_FACT.headers
            jq: |
              {
                 "cat_fact": $cat.fact,
                 "chuck_norris_fact": $chuck.value,
                 "cf-ray-chuck": $chuck_headers.vary,
              }
            type: jq

the following would fail

 - name: JOIN
            inputs:
            - cat: CAT_FACT.body
            - chuck: CHUCK_NORRIS_FACT.body
            - chuck_headers: CHUCK_NORRIS_FACT.headers
            jq: |
              {
                 "cat_fact": $cat.fact,
                 "chuck_norris_fact": $chuck.value,
                 "cf-ray-chuck": chuck_headers.vary,
              }
            type: jq

and then any use of cf-ray cf_ray with chuck_headers or $chuck_headers will fail. when i return the headers direclty to the body - i see that Cf-Ray is normalized to cf-ray so i thought that would work but it doesn't.

[teb510]

headers with multiple words seems like an edge case except that proxies add headers with multiple words - and you may even want to grab the kong headers and do something with them