Uploaded image for project: 'CFEngine Community'
  1. CFEngine Community
  2. CFE-1872

cf-agent processes piling up waiting for LMDB

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.7.0
    • Component/s: Internal Databases
    • Labels:
      None

      Description

      I'm seeing a build up of cf-agent procs on a RHEL6 host running CFEngine 3.6.3. Here's an lsof and gdb backtrace of one of the early processes. The host is a VM and may be having storage issues, will get more details on that.

      <pre>

      1. lsof -p 32741
        COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
        cf-agent 32741 root cwd DIR 253,0 4096 352257 /root
        cf-agent 32741 root rtd DIR 253,0 4096 2 /
        cf-agent 32741 root txt REG 253,1 414225 2395737 /var/cfengine/bin/cf-agent
        cf-agent 32741 root mem REG 253,1 57344 2392212 /var/cfengine/state/cf_lock.lmdb
        cf-agent 32741 root mem REG 253,0 383504 466954 /lib64/libfreebl3.so
        cf-agent 32741 root mem REG 253,0 65928 467323 /lib64/libnss_files-2.12.so
        cf-agent 32741 root mem REG 253,0 113432 467319 /lib64/libnsl-2.12.so
        cf-agent 32741 root mem REG 253,0 40400 466973 /lib64/libcrypt-2.12.so
        cf-agent 32741 root mem REG 253,0 113096 467125 /lib64/libaudit.so.1.0.0
        cf-agent 32741 root mem REG 253,0 14584 467093 /lib64/libutil-2.12.so
        cf-agent 32741 root mem REG 253,1 95063 2416753 /var/cfengine/lib/libz.so.1.2.8
        cf-agent 32741 root mem REG 253,1 57431 2416715 /var/cfengine/lib/libattr.so.1.1.0
        cf-agent 32741 root mem REG 253,0 1916528 466965 /lib64/libc-2.12.so
        cf-agent 32741 root mem REG 253,0 142464 467076 /lib64/libpthread-2.12.so
        cf-agent 32741 root mem REG 253,0 595800 467317 /lib64/libm-2.12.so
        cf-agent 32741 root mem REG 253,0 43832 467330 /lib64/librt-2.12.so
        cf-agent 32741 root mem REG 253,0 19536 467316 /lib64/libdl-2.12.so
        cf-agent 32741 root mem REG 253,0 52560 467068 /lib64/libnss_nis-2.12.so
        cf-agent 32741 root mem REG 253,0 55848 467236 /lib64/libpam.so.0.82.2
        cf-agent 32741 root mem REG 253,1 4268745 2416751 /var/cfengine/lib/libxml2.so.2.9.1
        cf-agent 32741 root mem REG 253,1 157841 2416744 /var/cfengine/lib/libpq.so.5.3
        cf-agent 32741 root mem REG 253,1 7230594 2416749 /var/cfengine/lib/libvirt.so.0.1001.3
        cf-agent 32741 root mem REG 253,1 753528 2416740 /var/cfengine/lib/libpcre.so.1.2.1
        cf-agent 32741 root mem REG 253,1 1662897 2416722 /var/cfengine/lib/libcrypto.so.0.9.8
        cf-agent 32741 root mem REG 253,1 365357 2416747 /var/cfengine/lib/libssl.so.0.9.8
        cf-agent 32741 root mem REG 253,1 166096 2416713 /var/cfengine/lib/libacl.so.1.1.0
        cf-agent 32741 root mem REG 253,1 2955175 2416746 /var/cfengine/lib/libpromises.so.3.0.6
        cf-agent 32741 root mem REG 253,0 154464 469389 /lib64/ld-2.12.so
        cf-agent 32741 root mem-r REG 253,1 320768 2392211 /var/cfengine/state/cf_lock.lmdb-lock
        cf-agent 32741 root mem REG 253,1 217016 1589255 /var/db/nscd/passwd
        cf-agent 32741 root mem REG 253,1 217016 1589257 /var/db/nscd/services
        cf-agent 32741 root mem REG 253,1 209313 2416724 /var/cfengine/lib/liblmdb.so
        cf-agent 32741 root 0r FIFO 0,8 0t0 676210295 pipe
        cf-agent 32741 root 1w FIFO 0,8 0t0 676210296 pipe
        cf-agent 32741 root 2w FIFO 0,8 0t0 676210296 pipe
        cf-agent 32741 root 4ur REG 253,1 320768 2392211 /var/cfengine/state/cf_lock.lmdb-lock
        cf-agent 32741 root 5u REG 253,1 57344 2392212 /var/cfengine/state/cf_lock.lmdb
        cf-agent 32741 root 6u REG 253,1 57344 2392212 /var/cfengine/state/cf_lock.lmdb

      gdb backtrace:
      Reading symbols from /usr/lib64/libfreebl3.so...(no debugging symbols found)...done.
      Loaded symbols for /usr/lib64/libfreebl3.so
      0x00007f1fa0266054 in __lll_lock_wait () from /lib64/libpthread.so.0
      Missing separate debuginfos, use: debuginfo-install audit-libs-2.2-2.el6.x86_64 glibc-2.12-1.80.el6_3.6.x86_64 nss-softokn-freebl-3.12.9-11.el6.x86_64 pam-1.1.1-10.el6_2.1.x86_64
      (gdb) bt
      #0 0x00007f1fa0266054 in __lll_lock_wait () from /lib64/libpthread.so.0
      #1 0x00007f1fa0261388 in _L_lock_854 () from /lib64/libpthread.so.0
      #2 0x00007f1fa0261257 in pthread_mutex_lock () from /lib64/libpthread.so.0
      #3 0x00007f1fa1ecb252 in mdb_txn_renew0 (txn=0xc23670) at mdb.c:2202
      #4 0x00007f1fa1ecb905 in mdb_txn_begin (env=0xb574c0, parent=0x0, flags=<value optimized out>, ret=0xa50c80) at mdb.c:2366
      #5 0x00007f1fa1bf6aa1 in GetWriteTransaction (db=0x126dc40, txn=0x7fff92a3e908) at dbm_lmdb.c:122
      #6 0x00007f1fa1bf732b in DBPrivDelete (db=0x7f1fa1e08040, key=0x7fff92a3e970, key_size=12728144) at dbm_lmdb.c:425
      #7 0x00007f1fa1bf5ebc in DeleteDB (handle=0x7f1fa1db27e0, key=0x7fff92a3e970 "ce6d011fd796cb8826ce2db62a681240") at dbm_api.c:327
      #8 0x00007f1fa1c1780a in RemoveLock (
      name=0x8a6a90 "lock.fs_inputs.processes.comment.handle.signals.-atlgit01.proc_cf_monitord__norestart_6749_MD5=f554349cdb95bbf1ceddb052b989f0c0") at locks.c:262
      #9 0x00007f1fa1c178ad in YieldCurrentLock (lock=...) at locks.c:780
      #10 0x000000000041f170 in VerifyProcessesPromise ()
      #11 0x000000000040ce9d in KeepAgentPromise ()
      #12 0x00007f1fa1c0b565 in ExpandPromiseAndDo (ctx=0x8a66e0, pp=<value optimized out>, ActOnPromise=0x40ccb0 <KeepAgentPromise>, param=0x0)
      at expand.c:207
      #13 ExpandPromise (ctx=0x8a66e0, pp=<value optimized out>, ActOnPromise=0x40ccb0 <KeepAgentPromise>, param=0x0) at expand.c:161
      #14 0x000000000040d36a in ScheduleAgentOperations ()
      #15 0x000000000041c1ca in VerifyMethod ()
      #16 0x000000000041c6f1 in VerifyMethodsPromise ()
      #17 0x000000000040cfd1 in KeepAgentPromise ()
      #18 0x00007f1fa1c0b565 in ExpandPromiseAndDo (ctx=0x8a66e0, pp=<value optimized out>, ActOnPromise=0x40ccb0 <KeepAgentPromise>, param=0x0)
      at expand.c:207
      #19 ExpandPromise (ctx=0x8a66e0, pp=<value optimized out>, ActOnPromise=0x40ccb0 <KeepAgentPromise>, param=0x0) at expand.c:161
      #20 0x000000000040d36a in ScheduleAgentOperations ()
      #21 0x000000000040db67 in KeepPromises ()
      #22 0x000000000040ecf8 in main ()
      </pre>

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              a10003 Eystein Maloy Stenberg
              Reporter:
              neilhwatson Neil Watson
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - 1 hour
                  1h
                  Remaining:
                  Not Specified
                  Logged:
                  Time Not Required
                  Not Specified