Uploaded image for project: 'Mender'
  1. Mender
  2. MEN-2497

lzma compression is slow

    XMLWordPrintable

    Details

    • Type: Task
    • Status: Open
    • Priority: (None)
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None

      Description

      This is based on information in this commit where lzma is a factor 10x slower time wise.

      The "11 minutes to compress" is a surprising to me,

      running the `xz` command it is around 1 minute to compress a artifact of the same size as mentioned in above commit.

      e.g

      $ time xz -f -k -c -3 -T 0 --check=crc32 2018-11-13-raspbian-stretch-lite.ext4 > 2018-11-13-raspbian-stretch-lite.ext4.xz
      
      real	1m17,426s
      user	8m24,607s
      sys	0m4,020s
      mirzak@mk2000 ~/src/mender-convert/output ((HEAD detached at 1.1.0b1))
      $ ls -alh 2018-11-13-raspbian-stretch-lite.ext4
      -rw-r--r-- 1 root root 3,9G 30 apr 18.02 2018-11-13-raspbian-stretch-lite.ext4
      mirzak@mk2000 ~/src/mender-convert/output ((HEAD detached at 1.1.0b1))
      $ ls -alh 2018-11-13-raspbian-stretch-lite.ext4.xz 
      -rw-r--r-- 1 mirzak mirzak 269M  2 maj 13.19 2018-11-13-raspbian-stretch-lite.ext4.xz
      

      Would be interesting to investigate why the there is a factor 10x when using mender-artifact. I suspect that we are not utilizing parallelism which the `-T 0` flag indicates in my command which means utilize all available CPU cores. I have also tried setting the highest compression preset (`-9`) but it still takes around 3 minutes.

      Note that I copied the command arguments from Yocto, and how it is producing `xz` compressed artifacts.

      Would be interesting to investigate the following on our end:

      • which compression preset are we using (this can heavily impact memory usage and can be a problem on constraint devices if we are using a "high" preset)
      • are we using parallelism? If not, is it possible?

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              mirzak Mirza Krak
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:

                Summary Panel