This is just a quick walk through of a small test I just did with the ZPAQ archiver. The test was done on Windows 64bit (Win7Pro) environment but this app (ZPAQ) is available for a variety of platforms, ie, Win32 and Linux also.
For more info please refer to the homepage of ZPaq.
Disclaimer. I have nothing to do with this software, and I never heard of it before today. I just was curious to find a tool that did what I needed, and stumbled across this. Wanted to test if it worked as expected, and since I prepped the small test doc for myself, put it here for reference.
I was looking for an incremental archival compression too that behaved in a sane way with a use case scenario:
Anyhow. Lots of fun.
Kudos to the author of ZPAQ for making such a lovely tool / and making it open source for all to benefit from.
- Tim Chipman Jan.2016
STARTING POINT: WE HAVE ZPAQ BINARY READY TO USE. WE HAVE A TESTDIR WITH A SMALL TEXT FILE IN IT.
Microsoft Windows [Version 6.1.7601] Copyright (c) 2009 Microsoft Corporation. All rights reserved. C:\Windows\system32>cd \ C:\>dir Volume in drive C has no label. Volume Serial Number is 2E9C-DB05 Directory of C:\ 12/31/2015 11:47 AM <DIR> Brother 04/28/2014 03:13 PM <DIR> Intel 12/31/2015 11:17 AM <DIR> maintenance 07/13/2009 11:20 PM <DIR> PerfLogs 01/08/2016 01:09 PM <DIR> Program Files 01/08/2016 01:14 PM <DIR> Program Files (x86) 04/28/2014 03:06 PM 211 setup.log 12/31/2015 11:50 AM <DIR> temp 01/16/2016 10:31 AM <DIR> testdir 12/22/2015 04:40 AM <DIR> Users 01/13/2016 06:21 PM <DIR> Windows 01/16/2016 10:29 AM 657,920 zpaq64.exe 2 File(s) 658,131 bytes 10 Dir(s) 259,510,542,336 bytes free C:\>zpaq64.exe add part??? testdir zpaq v7.05 journaling archiver, compiled Apr 17 2015 Creating part001.zpaq dated 2016-01-16 14:32:56 assuming 0 prior bytes Adding 0.000014 MB in 1 files -method 14 -threads 4 at 2016-01-16 14:32:56. + testdir/test document.txt 14 [1..1] 26 -method 14,14,0 + testdir/ 2 +added, 0 -removed. 0.000000 + (0.000014 -> 0.000014 -> 0.001137) = 0.001137 MB 0.047 seconds (all OK)
NOW TAKE A LOOK AT CHANGES:
C:\>dir Volume in drive C has no label. Volume Serial Number is 2E9C-DB05 Directory of C:\ 12/31/2015 11:47 AM <DIR> Brother 04/28/2014 03:13 PM <DIR> Intel 12/31/2015 11:17 AM <DIR> maintenance 01/16/2016 10:32 AM 705 part000.zpaq 01/16/2016 10:32 AM 1,137 part001.zpaq 07/13/2009 11:20 PM <DIR> PerfLogs 01/08/2016 01:09 PM <DIR> Program Files 01/08/2016 01:14 PM <DIR> Program Files (x86) 04/28/2014 03:06 PM 211 setup.log 12/31/2015 11:50 AM <DIR> temp 01/16/2016 10:31 AM <DIR> testdir 12/22/2015 04:40 AM <DIR> Users 01/13/2016 06:21 PM <DIR> Windows 01/16/2016 10:29 AM 657,920 zpaq64.exe 4 File(s) 659,973 bytes 10 Dir(s) 259,510,534,144 bytes free
NOTE ZPAQ CREATED AN INDEX FILE (part000) and ALSO a DATA File (part001)
NOW BEHIND THE SCENES DROP ANOTHER TEXT FILE INTO TESTDIR, NOTHING BIG YET. RE-RUN ZPAQ IN INCREMENTAL MODE
C:\>zpaq64.exe add part??? testdir zpaq v7.05 journaling archiver, compiled Apr 17 2015 part???.zpaq: 1 versions, 2 files, 1 fragments, 0.001137 MB part000.zpaq: 1 versions, 2 files, 1 fragments, 0.000705 MB Adding 0.002332 MB in 1 files -method 14 -threads 4 at 2016-01-16 14:33:31. + testdir/another test.txt 2332 [2..2] 2344 -method 14,169,1 # testdir/ 2 +added, 0 -removed. 0.001137 + (0.002332 -> 0.002332 -> 0.002976) = 0.004113 MB 0.093 seconds (all OK)
TAKE A LOOK AT THE OUTPUT ON DISK: NOTE WE NOW HAVE PART002 ADDED / EXISTS. SMALL DELTA IN SIZE TO PART000, NO DELTA TO PART001
C:\>dir Volume in drive C has no label. Volume Serial Number is 2E9C-DB05 Directory of C:\ 12/31/2015 11:47 AM <DIR> Brother 04/28/2014 03:13 PM <DIR> Intel 12/31/2015 11:17 AM <DIR> maintenance 01/16/2016 10:33 AM 1,409 part000.zpaq 01/16/2016 10:32 AM 1,137 part001.zpaq 01/16/2016 10:33 AM 2,976 part002.zpaq 07/13/2009 11:20 PM <DIR> PerfLogs 01/08/2016 01:09 PM <DIR> Program Files 01/08/2016 01:14 PM <DIR> Program Files (x86) 04/28/2014 03:06 PM 211 setup.log 12/31/2015 11:50 AM <DIR> temp 01/16/2016 10:33 AM <DIR> testdir 12/22/2015 04:40 AM <DIR> Users 01/13/2016 06:21 PM <DIR> Windows 01/16/2016 10:29 AM 657,920 zpaq64.exe 5 File(s) 663,653 bytes 10 Dir(s) 259,510,525,952 bytes free
BEHIND THE SCENES ADD ANOTHER FILE INTO TEST DIR. IT IS A COPY OF THE FILE ADDED IN LAST PASS. IDENTICAL CONTENT JUST SLIGHTLY DIFFERENT NAME (append copy to end of name)
C:\>zpaq64.exe add part??? testdir zpaq v7.05 journaling archiver, compiled Apr 17 2015 part???.zpaq: 2 versions, 3 files, 2 fragments, 0.004113 MB part000.zpaq: 2 versions, 3 files, 2 fragments, 0.001409 MB Adding 0.002332 MB in 1 files -method 14 -threads 4 at 2016-01-16 14:33:42. + testdir/another test - Copy.txt 2332 -> 0 # testdir/ 2 +added, 0 -removed. 0.004113 + (0.002332 -> 0.000000 -> 0.000586) = 0.004699 MB 0.047 seconds (all OK)
LOOK AT WHERE WE END UP - PART000 gets index addition, part001 AND part002 are unchanged; part003 is created and is quite tiny - very little data.
C:\>dir Volume in drive C has no label. Volume Serial Number is 2E9C-DB05 Directory of C:\ 12/31/2015 11:47 AM <DIR> Brother 04/28/2014 03:13 PM <DIR> Intel 12/31/2015 11:17 AM <DIR> maintenance 01/16/2016 10:33 AM 1,995 part000.zpaq 01/16/2016 10:32 AM 1,137 part001.zpaq 01/16/2016 10:33 AM 2,976 part002.zpaq 01/16/2016 10:33 AM 586 part003.zpaq 07/13/2009 11:20 PM <DIR> PerfLogs 01/08/2016 01:09 PM <DIR> Program Files 01/08/2016 01:14 PM <DIR> Program Files (x86) 04/28/2014 03:06 PM 211 setup.log 12/31/2015 11:50 AM <DIR> temp 01/16/2016 10:33 AM <DIR> testdir 12/22/2015 04:40 AM <DIR> Users 01/13/2016 06:21 PM <DIR> Windows 01/16/2016 10:29 AM 657,920 zpaq64.exe 6 File(s) 664,825 bytes 10 Dir(s) 259,510,517,760 bytes free
NOW BEHIND THE SCENES DROP A BINARY OF NON_TRIVIAL SIZE INTO THE TESTDIR: Installer EXE for MS_security essentials, approx 14 megs.
C:\>zpaq64.exe add part??? testdir zpaq v7.05 journaling archiver, compiled Apr 17 2015 part???.zpaq: 3 versions, 4 files, 2 fragments, 0.004699 MB part000.zpaq: 3 versions, 4 files, 2 fragments, 0.001995 MB Adding 14.243034 MB in 2 files -method 14 -threads 4 at 2016-01-16 14:34:47. 100.00% 0:00:00 + testdir/mseinstall.exe 14243008 100.00% 0:00:00 + testdir/mseinstall.exe:Zone.Identifier:$DATA 26 100.00% 0:00:00 [3..199] 14243830 -method 14,4,0 # testdir/ 3 +added, 0 -removed. 0.004699 + (14.243034 -> 14.243034 -> 14.251026) = 14.255725 MB 1.466 seconds (all OK)
SEE WHAT WE HAVE GOT - Update to Part000 Index, modest increase but much less than data delta AND part001,2,3 are unchanged AND part004 is created, which has majority of data just added - 14megs of dense data.
C:\>dir Volume in drive C has no label. Volume Serial Number is 2E9C-DB05 Directory of C:\ 12/31/2015 11:47 AM <DIR> Brother 04/28/2014 03:13 PM <DIR> Intel 12/31/2015 11:17 AM <DIR> maintenance 01/16/2016 10:34 AM 8,213 part000.zpaq 01/16/2016 10:32 AM 1,137 part001.zpaq 01/16/2016 10:33 AM 2,976 part002.zpaq 01/16/2016 10:33 AM 586 part003.zpaq 01/16/2016 10:34 AM 14,251,026 part004.zpaq 07/13/2009 11:20 PM <DIR> PerfLogs 01/08/2016 01:09 PM <DIR> Program Files 01/08/2016 01:14 PM <DIR> Program Files (x86) 04/28/2014 03:06 PM 211 setup.log 12/31/2015 11:50 AM <DIR> temp 01/16/2016 10:34 AM <DIR> testdir 12/22/2015 04:40 AM <DIR> Users 01/13/2016 06:21 PM <DIR> Windows 01/16/2016 10:29 AM 657,920 zpaq64.exe 7 File(s) 14,922,069 bytes 10 Dir(s) 259,481,960,448 bytes free
FINALLY: MAKE A COPY OF THE 14meg MSE INSTALLER, AND RE RUN THE ZPAQ:
C:\>zpaq64.exe add part??? testdir zpaq v7.05 journaling archiver, compiled Apr 17 2015 part???.zpaq: 4 versions, 6 files, 199 fragments, 14.255725 MB part000.zpaq: 4 versions, 6 files, 199 fragments, 0.008213 MB Adding 14.243034 MB in 2 files -method 14 -threads 4 at 2016-01-16 14:34:57. 100.00% 0:00:00 + testdir/mseinstall - Copy.exe 14243008 -> 0 100.00% 0:00:00 + testdir/mseinstall - Copy.exe:Zone.Identifier:$DATA 26 -> 0 # testdir/ 3 +added, 0 -removed. 14.255725 + (14.243034 -> 0.000000 -> 0.001394) = 14.257119 MB 0.202 seconds (all OK)
SEE WHAT WE HAVE: THIS TIME
- another modest increment to index file part000 - Files 1,2,3,4 are unchanged - File 005 is created and is VERY SMALL (compared to part004) since the binary data is virtually identical. (just filename is different)
C:\>dir Volume in drive C has no label. Volume Serial Number is 2E9C-DB05 Directory of C:\ 12/31/2015 11:47 AM <DIR> Brother 04/28/2014 03:13 PM <DIR> Intel 12/31/2015 11:17 AM <DIR> maintenance 01/16/2016 10:34 AM 9,607 part000.zpaq 01/16/2016 10:32 AM 1,137 part001.zpaq 01/16/2016 10:33 AM 2,976 part002.zpaq 01/16/2016 10:33 AM 586 part003.zpaq 01/16/2016 10:34 AM 14,251,026 part004.zpaq 01/16/2016 10:34 AM 1,394 part005.zpaq 07/13/2009 11:20 PM <DIR> PerfLogs 01/08/2016 01:09 PM <DIR> Program Files 01/08/2016 01:14 PM <DIR> Program Files (x86) 04/28/2014 03:06 PM 211 setup.log 12/31/2015 11:50 AM <DIR> temp 01/16/2016 10:34 AM <DIR> testdir 12/22/2015 04:40 AM <DIR> Users 01/13/2016 06:21 PM <DIR> Windows 01/16/2016 10:29 AM 657,920 zpaq64.exe 8 File(s) 14,924,857 bytes 10 Dir(s) 259,467,665,408 bytes free C:\>