RHEL 8 - Error: COMMAND_FAILED: 'python-nftables' failed

[nasa-dan]

Affected Puppet, Ruby, OS and module versions/distributions

Puppet: PE 2021.4
RHEL 8.5

How to reproduce (e.g Puppet code you use)

This was for an initial Code Manager deployment, using code that ran OK on RHEL/CentOS 7

What are you seeing

In the puppet log, when it tried to reload firewalld, it spewed this:

Error: COMMAND_FAILED: 'python-nftables' failed: internal:0:0-0: Error: Could not process rule: No such file or directory

internal:0:0-0: Error: Could not process rule: No such file or directory

internal:0:0-0: Error: Could not process rule: No such file or directory

internal:0:0-0: Error: Could not process rule: No such file or directory

internal:0:0-0: Error: Could not process rule: No such file or directory

internal:0:0-0: Error: Could not process rule: No such file or directory

internal:0:0-0: Error: Could not process rule: No such file or directory

internal:0:0-0: Error: Could not process rule: No such file or directory

JSON blob:
{"nftables": [{"metainfo": {"json_schema_version": 1}}, {"insert": {"rule": {"family": "inet", "table": "firewalld", "chain": "filter_INPUT_ZONES", "expr": [{"match": {"left": {"meta": {"key": "iifname"}}, "op": "==", "right": "ens192"}}, {"goto": {"target": "filter_IN_drop"}}]}}}, {"insert": {"rule": {"family": "inet", "table": "firewalld", "chain": "filter_FORWARD_OUT_ZONES", "expr": [{"match": {"left": {"meta": {"key": "oifname"}}, "op": "==", "right": "ens192"}}, {"goto": {"target": "filter_FWDO_drop"}}]}}}, {"insert": {"rule": {"family": "ip", "table": "firewalld", "chain": "nat_POSTROUTING_ZONES", "expr": [{"match": {"left": {"meta": {"key": "oifname"}}, "op": "==", "right": "ens192"}}, {"goto": {"target": "nat_POST_drop"}}]}}}, {"insert": {"rule": {"family": "ip6", "table": "firewalld", "chain": "nat_POSTROUTING_ZONES", "expr": [{"match": {"left": {"meta": {"key": "oifname"}}, "op": "==", "right": "ens192"}}, {"goto": {"target": "nat_POST_drop"}}]}}}, {"insert": {"rule": {"family": "inet", "table": "firewalld", "chain": "filter_FORWARD_IN_ZONES", "expr": [{"match": {"left": {"meta": {"key": "iifname"}}, "op": "==", "right": "ens192"}}, {"goto": {"target": "filter_FWDI_drop"}}]}}}, {"insert": {"rule": {"family": "ip", "table": "firewalld", "chain": "nat_PREROUTING_ZONES", "expr": [{"match": {"left": {"meta": {"key": "iifname"}}, "op": "==", "right": "ens192"}}, {"goto": {"target": "nat_PRE_drop"}}]}}}, {"insert": {"rule": {"family": "ip6", "table": "firewalld", "chain": "nat_PREROUTING_ZONES", "expr": [{"match": {"left": {"meta": {"key": "iifname"}}, "op": "==", "right": "ens192"}}, {"goto": {"target": "nat_PRE_drop"}}]}}}, {"insert": {"rule": {"family": "inet", "table": "firewalld", "chain": "mangle_PREROUTING_ZONES", "expr": [{"match": {"left": {"meta": {"key": "iifname"}}, "op": "==", "right": "ens192"}}, {"goto": {"target": "mangle_PRE_drop"}}]}}}]}

Firewalld is now broken.
I put in a help ticket to Red Hat, but I wanted to see if you folks have seen this mess before.

[nasa-dan]

Red Hat Support found the problem:

firewalld::custom_services:
  'puppet-agent':
    short: 'puppet-agent'
    description: 'Puppet Client access Puppet Server'
    port:
      - port: 8140
        protocol: 'tcp'
      - port: 8142
        protocol: 'tcp'
    module:
      - 'nf_conntrack_netbios_ns'   <--------------------- THIS IS THE TROUBLE-MAKER !!

That "module" parameter did not cause problems in RHEL 7, but RHEL 8.5 really does not like it.

[nasa-dan]

Help me out here, please.
I think I just copied from the Forge examples.
What does this parameter do to the service definition, please ?
What will removing it do to the service definition, please ?

[nasa-dan]

More details from Red Hat:

That parameter shows in /etc/firewalld/services/.xml as
<module name="nf_conntrack_netbios_ns">

Red Hat says changing it to
<helper name="netbios-ns"/>
works.

And it does work, but how do I get it that way using this module ?

[oberon227]

From where $module is used in the code

puppet-firewalld/lib/puppet/provider/firewalld_custom_service/firewall_cmd.rb

Line 167 in 3aaf4f4

execute_firewall_cmd(['--service', @resource[:name], '--add-module', entry], nil)

it looks like it's trying to add the module to the firewalld custom service.

I'd chalk this one up to changes between RHEL 7 and RHEL 8, and the documentation not perfectly reflecting that.

Have you tried setting the module parameter to netbios-ns? If that's what RedHat is telling you to set it to, it seems like the module might support that, without further modification.

Side note: firewalld::custom_service is deprecated in favour of firewalld_custom_service. Looks like firewalld::custom_service now just does a light transform on the input and sends it to firewalld_custom_service anyway.

[nasa-dan]

I can try putting <module name="netbios-ns"/> in the /etc/firewalld/services/service.xml file to see what it does, but Red Hat said to set the HELPER parameter rather than the MODULE.

Does anyone have references for netfilter kernel helper modules ?

About firewalld_custom_service : Can I still define it in Hiera ? There is no documentation to that effect.

[nasa-dan]

With either <helper name ="netbios-ns"/> or <module name ="netbios-ns"/> in the XML file, the firewall reloads wwithout problem and this command
nft -a list chain inet firewalld filter_IN_drop_allow
shows a line of
udp dport 137 ct helper set "helper-netbios-ns-udp" # handle 98
either way.

To paraphrase Professor HIggins, "By George1 I think we've got it !"

[nasa-dan]

Now if I only knew what it does and why I would want to include it …

[oberon227]

About firewalld_custom_service : Can I still define it in Hiera ? There is no documentation to that effect.

Hmmmm, yeah. Despite the deprecation notice, there isn't a parameter set up to take a hiera hash of firewalld_custom_services.

What I do in situations like this is something like
hiera:

profile::firewalld::services:
  'service1':
    port: 1234
    etc

profile/firewalld.pp

profile::firewalld(
  $services = {},
  ...
  ensure_resources(firewalld_custom_service, $services)

As I was typing this out, I see you posted another message with success. Huzzah!
But if you don't know why you need it, maybe take it out and see what breaks? ;-)
I think from context, that it's adding the netbios name server helper to that connection. Like, if you're using FTP and you have separate data and control ports, you'd need an FTP helper on that service. If you don't know you need the helper, you might not need it! It was possibly just there as an example in the documentation.

[nasa-dan]

I have to admit, I copied it from the example given in the documentation.
I have no clue what it does.

About Hiera, could we get the needed parameter before firewalld::custom_service is removed ?
Pretty please ?

[oberon227]

I have to admit, I copied it from the example given in the documentation.
I have no clue what it does.

I think in your stated case (allowing connections into the Puppet server), you don't need the netbios-ns helper. Puppet's not using netbios, so you probably don't need it.

About Hiera, could we get the needed parameter before firewalld::custom_service is removed ?
Pretty please ?

That part I can't speak to. I'm just a Puppet user who happens to work heavily with firewalls at work. :-)
But it looks like, when the time comes, they can just update init.pp here

puppet-firewalld/manifests/init.pp

Line 122 in 3aaf4f4

firewalld::custom_service { $key:

to say instead firewalld_custom_service { $key:

[adam-kosseck]

Just noting that we've experienced outages due to this issue twice in the last 3 months, against both RHEL 8.9 and 9.3, but we do not have any modules declared against our custom services (and can not yet reliably reproduce the problem).

I've also raised a case with Red Hat to try and determine root cause.

[adam-kosseck]

Red Hat analysis of the SOS Report seems to indicate that the Puppet Firewalld module may have a bug in the order that it processes the various components/rules. Here is a verbatim extract of just the summary of their case findings:

In summary, my feeling is that the puppet-firewalld module tried to do the things in the wrong order:

• It created the rich rule the first of all, making reference to a service:rubrik object and an ipset:rubrik object that didn't exist yet. IMHO the rule creation should be the last step of all, precisely to avoid these kind of reference errors.
• Then it creates service:rubrik and reloads firewalld. It would make sense to trigger a firewalld reload automatically right after a new service has been added, but it should make sure that no rule has been added yet.
• All the work with the ipsets seems to be done as the last step. IMHO the ipsets should be handled after doing the work with the services, and definitely before any rule is added.

Looking in more detail at the open issues against this module, I found two which seem to concur:

• firewalld_rich_rule needs to autorequire firewalld_custom_service on service #321 (rich rule / service / custom service dependencies)
• firewalld_zone doesn't autorequire consumed firewalld_ipset elements #362 (zone / ipset dependencies)

This seems to be evidence that dependencies need to be investigated at a wider level.

Firewalld logs on affected hosts show 3 error lines which repeat multiple times:

ERROR: 'python-nftables' failed: internal:0:0-0: Error: Could not process rule: No such file or directory
JSON blob:
{"nftables": [{"metainfo": {"json_schema_version": 1}}, {"insert": {"rule": {"family": "inet", "table": "firewalld", "chain": "filter_INPUT_ZONES", "expr": [{"match": {"left": {"meta": {"key": "iifname"}}, "op": "==", "right": "eth0"}}, {"goto": {"target": "filter_IN_internal"}}]}}}, {"insert": {"rule": {"family": "inet", "table": "firewalld", "chain": "nat_POSTROUTING_ZONES", "expr": [{"match": {"left": {"meta": {"key": "oifname"}}, "op": "==", "right": "eth0"}}, {"goto": {"target": "nat_POST_internal"}}]}}}, {"insert": {"rule": {"family": "inet", "table": "firewalld", "chain": "filter_FORWARD_ZONES", "expr": [{"match": {"left": {"meta": {"key": "iifname"}}, "op": "==", "right": "eth0"}}, {"goto": {"target": "filter_FWD_internal"}}]}}}, {"insert": {"rule": {"family": "inet", "table": "firewalld", "chain": "nat_PREROUTING_ZONES", "expr": [{"match": {"left": {"meta": {"key": "iifname"}}, "op": "==", "right": "eth0"}}, {"goto": {"target": "nat_PRE_internal"}}]}}}, {"insert": {"rule": {"family": "inet", "table": "firewalld", "chain": "mangle_PREROUTING_ZONES", "expr": [{"match": {"left": {"meta": {"key": "iifname"}}, "op": "==", "right": "eth0"}}, {"goto": {"target": "mangle_PRE_internal"}}]}}}, {"add": {"rule": {"family": "inet", "table": "firewalld", "chain": "filter_FWD_internal_allow", "expr": [{"match": {"left": {"meta": {"key": "oifname"}}, "op": "==", "right": "eth0"}}, {"accept": null}]}}}]}
ERROR: INVALID_IPSET: ipset type '' not usable

Our declaration of the firewalld class in is as follows:

  class { 'firewalld':
    default_zone         => 'internal',
    default_service_zone => 'internal',
    default_port_zone    => 'internal',
    firewall_backend     => 'nftables',   
    install_gui          => false,
    log_denied           => 'unicast',  
    package_ensure       => 'installed',  
    package              => 'firewalld',  
    service_ensure       => 'running',    
    service_enable       => true,
    zone_drifting        => 'no',         
  }

All of the firewall rules are defined in hiera, with the base ruleset in "common.yaml", and where needed additional host rules at the hiera node level.
The rule change that triggered the issue for us was to add a new ipset and custom service to hiera common.yaml along with a rich rule which references them.

See the below hiera extract (with other rules excluded for clarity)

firewalld::zones:
  internal:
    ensure: present
    target: '%%REJECT%%'
    sources: [ ]
    purge_rich_rules: true
    purge_services: true
    purge_ports: true
    icmp_block_inversion: false
    icmp_blocks: [ ]

firewalld::custom_services:
  rubrik:
    short: '12800, 12801'
    ports:
      - port: 12800
        protocol: tcp
      - port: 12801
        protocol: tcp

firewalld::ipsets:
  rubrik:
    type: 'hash:net'
    entries:
      - x.x.x.x/24

firewalld::rich_rules:
  '050 Accept Rubrik Agent Traffic':
    zone: internal
    source:
      ipset: rubrik
    service: rubrik
    action: accept

Happy to provide further information and run tests as needed to get to the bottom of this.
I've taken a VM snapshot (with memory) of a machine we have in the "broken" state (which is 'resolved' by executing firewall-cmd --reload or restarting firewalld). This may help with any testing.