Combination of memo stats and substing linker command line fails

[mwichmann]

In a somewhat obscure combination, if memo stats printing is enabled (--debug=memoizer) and a shared library is created, scons will fail with an exception subst'ing _LIBFLAGS. The variable being expanded that causes the problem is set in the link tool:

env['_LIBFLAGS'] = '${_stripixes(LIBLINKPREFIX, LIBS, LIBLINKSUFFIX, LIBPREFIXES, LIBSUFFIXES, __env__, LIBLITERALPREFIX)}'

Suspicious the LIBLITERALPREFIX construction variable is involved, as that's a relatively new setting appearing in this line, but it might also be due to the shared-library-version symlink handling - that has also received work in recent memory.

A rather obscure error message is emitted, without a backtrace, so you need to add stacktrace to the debug options - and make sure #4677 is applied, as otherwise the memoizer debug won't actually be enabled.

In the failure, _stripixes will eventually create a PathList object; PathList is actually a method of the intentionally obscured _PathListCache, and as a caching dictionary, it is decorated with @SCons.Memoize.CountDictCall. The two memoizing counters are conditional decorators, they are do-nothing unless --debug=memoizer is used to enable them. Somehow the dictionary decorator on PathList changes the call method so in this one particular case too many arguments are passed to the function generating the cache key, leading to this failure:

scons: *** [prog] TypeError `PathListCache._PathList_key() takes 2 positional arguments but 3 were given' trying to evaluate `${_stripixes(LIBLINKPREFIX, LIBS, LIBLINKSUFFIX, LIBPREFIXES, LIBSUFFIXES, __env__, LIBLITERALPREFIX)}'

---

A reproducer is to "preserve" the output of this existing test:

$ PRESERVE=1 ./runtest.py test/Libs/LIBLITERALPREFIX.py

In the preserved directory, build as:

$ scons --debug=presub,memoizer,stacktrace

This is the slightly trimmed output (I additionally added presub to the debug options to get a bit more verbosity):

$ scons --debug=memoizer,stacktrace
,,,
scons: Building targets ...
Building foo.o with action:
  $CC -o $TARGET -c $CFLAGS $CCFLAGS $_CCCOMCOM $SOURCES
gcc -o foo.o -c foo.c
Building libfoo.a with action:
  $AR $ARFLAGS $TARGET $SOURCES
ar rc libfoo.a foo.o
Building libfoo.a with action:
  $RANLIB $RANLIBFLAGS $TARGET
ranlib libfoo.a
Building shfoo.os with action:
  $SHCC -o $TARGET -c $SHCFLAGS $SHCCFLAGS $_CCCOMCOM $SOURCES
gcc -o shfoo.os -c -fPIC shfoo.c
Building libfoo.so with action:
  SharedFlagChecker(target, source, env)
Building libfoo.so with action:
  $SHLINK -o $TARGET $SHLINKFLAGS $__SHLIBVERSIONFLAGS $__RPATH $SOURCES $_LIBDIRFLAGS $_LIBFLAGS
gcc -o libfoo.so -shared shfoo.os -L.
Building libfoo.so with action:
  LibSymlinksActionFunction(target, source, env)
scons: *** [prog] TypeError `PathListCache._PathList_key() takes 2 positional arguments but 3 were given' trying to evaluate `${_stripixes(LIBLINKPREFIX, LIBS, LIBLINKSUFFIX, LIBPREFIXES, LIBSUFFIXES, __env__, LIBLITERALPREFIX)}'
scons: internal stack trace:
  File "/home/mats/github/scons/SCons/Taskmaster/Job.py", line 670, in _work
    task.prepare()
  File "/home/mats/github/scons/SCons/Script/Main.py", line 209, in prepare
    return SCons.Taskmaster.OutOfDateTask.prepare(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mats/github/scons/SCons/Taskmaster/__init__.py", line 176, in prepare
    self.exception_raise()
  File "/home/mats/github/scons/SCons/Taskmaster/__init__.py", line 554, in _exception_raise
    raise exc_value.with_traceback(exc_traceback)
  File "/home/mats/github/scons/SCons/Taskmaster/__init__.py", line 1035, in next_task
    task.make_ready()
  File "/home/mats/github/scons/SCons/Script/Main.py", line 356, in make_ready
    SCons.Taskmaster.OutOfDateTask.make_ready(self)
  File "/home/mats/github/scons/SCons/Taskmaster/__init__.py", line 404, in make_ready_current
    t.disambiguate().make_ready()
  File "/home/mats/github/scons/SCons/Node/FS.py", line 3133, in make_ready
    self.get_binfo()
  File "/home/mats/github/scons/SCons/Node/__init__.py", line 1189, in get_binfo
    binfo.bactsig = hash_signature(executor.get_contents())
                                   ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mats/github/scons/SCons/Memoize.py", line 207, in wrapper
    return fn(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mats/github/scons/SCons/Executor.py", line 461, in get_contents
    result = bytearray("",'utf-8').join([action.get_contents(all_targets,
                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mats/github/scons/SCons/Action.py", line 558, in get_contents
    result = self.get_presig(target, source, env)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mats/github/scons/SCons/Action.py", line 1346, in get_presig
    return c.get_presig(self, target, source, env)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mats/github/scons/SCons/Action.py", line 1117, in get_presig
    return env.subst_target_source(cmd, SUBST_SIG, target, source)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mats/github/scons/SCons/Environment.py", line 720, in subst
    return SCons.Subst.scons_subst(string, self, raw, target, source, gvars, lvars, conv, overrides=overrides)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mats/github/scons/SCons/Subst.py", line 854, in scons_subst
    result = ss.substitute(strSubst, lvars)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mats/github/scons/SCons/Subst.py", line 459, in substitute
    result = _dollar_exps.sub(sub_match, args)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mats/github/scons/SCons/Subst.py", line 454, in sub_match
    return self.conv(self.expand(match.group(1), lvars))
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mats/github/scons/SCons/Subst.py", line 415, in expand
    return self.substitute(s, lv)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mats/github/scons/SCons/Subst.py", line 459, in substitute
    result = _dollar_exps.sub(sub_match, args)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mats/github/scons/SCons/Subst.py", line 454, in sub_match
    return self.conv(self.expand(match.group(1), lvars))
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mats/github/scons/SCons/Subst.py", line 415, in expand
    return self.substitute(s, lv)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mats/github/scons/SCons/Subst.py", line 459, in substitute
    result = _dollar_exps.sub(sub_match, args)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mats/github/scons/SCons/Subst.py", line 454, in sub_match
    return self.conv(self.expand(match.group(1), lvars))
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mats/github/scons/SCons/Subst.py", line 393, in expand
    raise_exception(e, lvars['TARGETS'], old_s)
  File "/home/mats/github/scons/SCons/Subst.py", line 54, in raise_exception
    raise SCons.Errors.BuildError(target[0], msg)
scons: building terminated because of errors.
Memoizer (memory cache) hits and misses:
...