Uploaded image for project: 'CFEngine Community'
  1. CFEngine Community
  2. CFE-445

cf-promises and cf-agent run shell commands excessively in returnszero() execresult()

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Priority: Low
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Based on this thread:
      https://groups.google.com/forum/#!topic/help-cfengine/2KclZOxonKc

      The bug report at https://cfengine.com/dev/issues/1506 can probably be merged into this one.

      Excessive evaluations of returnszero() and execresult() can cause poor performance and scalability.
      The problem may be that all classes and vars are evaluated excessively - I'm not sure as I have not dug into the code.

      Take this example file test-returnszero.cf:

      body common control {
      bundlesequence =>

      { "test_config" };
      }
      bundle common test_config { classes: "test" expression => returnszero("/bin/echo setting test class >> /tmp/cfagent-class-count", "useshell"); }

      Running it:

      user@cerberus:~/.cfagent/masterfiles$ rm /tmp/cfagent-class-count
      user@cerberus:~/.cfagent/masterfiles$ cf-agent -I -f ./test-returnszero.cf
      user@cerberus:~/.cfagent/masterfiles$ cat /tmp/cfagent-class-count
      setting test class
      setting test class
      setting test class
      setting test class
      setting test class
      setting test class
      setting test class
      setting test class
      setting test class
      setting test class
      setting test class
      setting test class
      setting test class
      setting test class
      user@cerberus:~/.cfagent/masterfiles$ wc -l /tmp/cfagent-class-count
      14 /tmp/cfagent-class-count


      14 times seems a little excessive. In a production environment, promise caching reduces the number a little, but it still seems a bit ridiculous. I don't see why it is run more than twice (once in cf-promises and once in cf-agent). And ideally, it would only be run once (though that might require neutering or replacing cf-promises functionality).


      Naturally this is also an issue with execresult, test-execresult.cf:

      body common control {
      bundlesequence => { "test_config" }

      ;
      }
      bundle common test_config

      { vars: "blah" string => execresult("/bin/echo setting test var >> /tmp/cfagent-vars-count", "useshell"); }

      user@cerberus:~/.cfagent/masterfiles$ cf-agent -I -f ./test-execresult.cf
      user@cerberus:~/.cfagent/masterfiles$ cat /tmp/cfagent-vars-count
      setting test var
      setting test var
      setting test var
      setting test var
      setting test var
      setting test var
      setting test var
      user@cerberus:~/.cfagent/masterfiles$ wc -l /tmp/cfagent-vars-count
      7 /tmp/cfagent-vars-count

      Of course, with verbosity turned up, you get a better idea of where these are coming from, test-returnszero2.cf:

      body common control {
      bundlesequence =>

      { "test_config" }

      ;
      }
      bundle common test_config

      { classes: "test" expression => returnszero("/bin/echo setting test class", "useshell"); }

      <pre>
      user@cerberus:~/.cfagent/masterfiles$ cf-agent -Iv -f ./test-returnszero.cf |grep -B3 'setting test class'
      cf3> -> Input file is changed since last validation, validating it
      cf3> -> Verifying the syntax of the inputs...
      cf3> Checking policy with command ...
      setting test class
      setting test class
      setting test class
      setting test class

      cf3> > Parsing file ./test-returnszero.cf
      cf3> Initiate variable convergence...
      cf3> -> Checking common class promises...
      setting test class

      cf3> ?> defining explicit global class test
      cf3> Initiate variable convergence...
      cf3> -> Checking common class promises...
      setting test class

      cf3> -> Hard classes =

      { redacted }

      cf3> -> Additional classes =

      { test }

      cf3> -> Negated Classes = { }
      setting test class

      cf3> Initiate variable convergence...
      cf3> -> Checking common class promises...
      setting test class

      cf3> classes in bundle test_config (1)
      cf3> =========================================================
      cf3>
      setting test class

      cf3> Initiate variable convergence...
      cf3> -> Checking common class promises...
      setting test class

      cf3> classes in bundle test_config (2)
      cf3> =========================================================
      cf3>
      setting test class

      cf3> Initiate variable convergence...
      cf3> -> Checking common class promises...
      setting test class

      cf3> classes in bundle test_config (3)
      cf3> =========================================================
      cf3>
      setting test class

      cf3> Initiate variable convergence...
      cf3> -> Checking common class promises...
      setting test class
      </pre>

      You can see how heavy hitting shell commands could easily eat into performance. It seems like cfagent should store classes, rather than re-evaulate every time it is accessed. Are all classes evaluated this often, or only shell commands? If this is done for convergence reasons, couldn't convergence be achieved after multiple cf-agent runs, instead of inside the same run?

      According to Aleksey Tsalolikhin, there should be 3 lines in that logfile instead of 14. But supposedly Mark Burgess said that more than 3 runs are required. In any case, I think the number of runs should be minimized (and though admittedly ignorant about the details of convergence cycles, I still think 1 is optimal).

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                a10025 Volker Hilsheimer (Inactive)
                Reporter:
                chrishiestand Chris Hiestand
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel