Uploaded image for project: 'CFEngine Community'
  1. CFEngine Community
  2. CFE-254

iteration over large lists is prohibitively slow

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Priority: Medium
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.6.0
    • Component/s: Evaluation
    • Labels:
      None

      Description

      This has been discussed a few times on the mailing list. I run into this
      problem more and more. It affects Cfengine's ability to work in large
      environments.

      Consider these examples.

      A large template file
      Suppose there is a large template file for ifcfg-eth0. It contains blocks and classes for 10,000 hosts.

      <pre>
      [%CFEngine myhost10000:: %]

      DEVICE=eth0
      IPADDR=172.26.2.10000
      HWADDR=00:18:71:70:8F:AA
      NETMASK=255.255.0.0
      NETWORK=172.26.0.0
      BROADCAST=172.26.255.255
      ONBOOT=yes
      </pre>

      The above is a block for one host. For 10,000 hosts the file size is 100,000
      lines. The agent takes over 4 minutes to process this template.

      One option is to break this template into multiple smaller ones. I consider
      this a poor work around. With policies of this scale, breaking one file into ten
      reduces the simplicity of management.

      Large lists
      There have been multiple discussions about list iteration and how the time to
      do so increases dramatically beyond a few hundred. Such lists become more
      common in large environments.

      In the past flex or bison has been blamed for this 'limitation'. I believe it
      is time to move beyond the limitation. Cfengine is touted as scaling better
      than the competition. Indeed it does but, we now know that Cfengine has its own
      scale limits. It's time we move beyond them.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                a10025 Volker Hilsheimer (Inactive)
                Reporter:
                neilhwatson Neil Watson
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel