Uploaded image for project: 'CFEngine Community'
  1. CFEngine Community
  2. CFE-2166

cf-serverd 3.7.1 showing high system CPU usage

    XMLWordPrintable

    Details

    • Type: Task
    • Status: Open
    • Priority: High
    • Resolution: Unresolved
    • Affects Version/s: 3.7.1
    • Fix Version/s: None
    • Component/s: cf-serverd
    • Labels:
      None

      Description

      Opening a bug to help with profiling information on a busy cf-serverd per Dimitrios (see "relevant mailing list thread":https://groups.google.com/forum/#!topic/help-cfengine/gn3e8pw_nU8).

      • This system is 8 CPUs / 8GB RAM / 4GB swap (0 swap currently used) / VMWare virtualized.
      • about 5500 clients (22000 divided by 4 because of load-balancer)
      • 5 minute schedule, 3 minute splaytime, custom failsafe (see comments for details)
      • Huge directory tree with ~5 files per host, recursively copied by all clients

      Summary of the profiling data from the comments:

      • @vmstat@ shows high @sys@ CPU usage, no swapping, low I/O
      • @strace@ shows most CPU time is spent in @futex()@ system call; this indicates high lock contention
      • @gdb@ backtraces show most of the threads in cf-serverd waiting on lastseen db LMDB mutex for I/O
        • This was remedied by touching @am_policy_hub@ file in @state/@ directory - see redmine #7640

        Attachments

        1. 2015_10_29.logs.tar.gz
          670 kB
        2. 2015_11_30.logs.tar.gz
          386 kB
        3. gdblogs.tar.gz
          189 kB
        4. gdblogs.tar.gz
          197 kB

          Issue Links

            Activity

              People

              • Assignee:
                a10038 jimis (Dimitrios Apostolou)
                Reporter:
                dannysauer Danny Sauer
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Summary Panel