* fix: resolve clippy warning in nightly
* wip: major rework of cde location
* wip: rework CDE lookup
* refactor: magic finder, eocd lookup retry
* wip: handle empty zips
* fix: satisfy tests, add documentation
* chore: remove unused dependencies
* feat: support both zip32 and zip64 comments
* feat: add zip64 comment functions to ZipWriter
* fix: first pass on maintainer comments
* fix: continue searching for EOCD when the central directory is invalid
* chore: satisfy clippy lints
* chore: satisfy style_and_docs
* feat: support both directions in MagicFinder, correctly find first CDFH
* fix: more checks to EOCD parsing, move comment size error from parse to write
* fix: use saturating add when checking eocd64 record_size upper bound
* fix: correctly handle mid window offsets in forward mode
* fix: compare maximum possible comment length against file size, not search region end
* feat: handle zip64 detection as a hint
* fix: detect oversized central directories when locating EOCD64
* fix: oopsie
---------
Signed-off-by: Chris Hennick <4961925+Pr0methean@users.noreply.github.com>
Co-authored-by: Chris Hennick <4961925+Pr0methean@users.noreply.github.com>
* Use the tempfile crate instead of the tempdir crate (which is deprecated)
https://github.com/rust-lang-deprecated/tempdir?tab=readme-ov-file#deprecation-note
* perf: Add benchmark that measures the rejection speed of a large non-zip file
* perf: Speed up non-zip rejection by increasing END_WINDOW_SIZE
I tested several END_WINDOW_SIZEs across 2 machines:
Machine 1: macOS 15.0.1, aarch64 (apfs /tmp)
512: test parse_large_non_zip ... bench: 30,450,608 ns/iter (+/- 673,910)
4096: test parse_large_non_zip ... bench: 7,741,366 ns/iter (+/- 521,101)
8192: test parse_large_non_zip ... bench: 5,807,443 ns/iter (+/- 546,227)
16384: test parse_large_non_zip ... bench: 4,794,314 ns/iter (+/- 419,114)
32768: test parse_large_non_zip ... bench: 4,262,897 ns/iter (+/- 397,582)
65536: test parse_large_non_zip ... bench: 4,060,847 ns/iter (+/- 280,964)
Machine 2: Debian testing, x86_64 (tmpfs /tmp)
512: test parse_large_non_zip ... bench: 65,132,581 ns/iter (+/- 7,429,976)
4096: test parse_large_non_zip ... bench: 14,109,503 ns/iter (+/- 2,892,086)
8192: test parse_large_non_zip ... bench: 9,942,500 ns/iter (+/- 1,886,063)
16384: test parse_large_non_zip ... bench: 8,205,851 ns/iter (+/- 2,902,041)
32768: test parse_large_non_zip ... bench: 7,012,011 ns/iter (+/- 2,222,879)
65536: test parse_large_non_zip ... bench: 6,577,275 ns/iter (+/- 881,546)
In both cases END_WINDOW_SIZE=8192 performed about 6x better than 512 and >8192
didn't make much of a difference on top of that.
* perf: Speed up non-zip rejection by limiting search for EOCDR.
I benchmarked several search sizes across 2 machines
(these benches are using an 8192 END_WINDOW_SIZE):
Machine 1: macOS 15.0.1, aarch64 (apfs /tmp)
whole file: test parse_large_non_zip ... bench: 5,773,801 ns/iter (+/- 411,277)
last 128k: test parse_large_non_zip ... bench: 54,402 ns/iter (+/- 4,126)
last 66,000: test parse_large_non_zip ... bench: 36,152 ns/iter (+/- 4,293)
Machine 2: Debian testing, x86_64 (tmpfs /tmp)
whole file: test parse_large_non_zip ... bench: 9,942,306 ns/iter (+/- 1,963,522)
last 128k: test parse_large_non_zip ... bench: 73,604 ns/iter (+/- 16,662)
last 66,000: test parse_large_non_zip ... bench: 41,349 ns/iter (+/- 16,812)
As you might expect these significantly increase the rejection speed for
large non-zip files.
66,000 was the number previously used by zip-rs. It was changed to zero in
7a55945743.
128K is what Info-Zip uses[1]. This seems like a reasonable (non-zero)
choice for compatibility reasons.
[1] Info-zip is extremely old and doesn't not have an official git repo to
link to. However, an unofficial fork can be found here:
bb0c4755d4/zipfile.c (L4073)
---------
Co-authored-by: Chris Hennick <4961925+Pr0methean@users.noreply.github.com>
* fix: Rare combination of settings could lead to writing a corrupt archive with overlength extra data
* fix: Previous fix was breaking alignment
* style: cargo fmt --all
* fix: ZIP64 header was being written twice
* style: cargo fmt --all
* ci(fuzz): Add check that file-creation options are individually valid
* fix: Need to update extra_data_start in deep_copy_file
* style: cargo fmt --all
* test(fuzz): fix bug in Arbitrary impl
* fix: Cursor-position bugs when merging archives or opening for append
* fix: unintended feature dependency
* style: cargo fmt --all
* fix: merge_contents was miscalculating new start positions for absorbed archive's files
* fix: shallow_copy_file needs to reset CDE location since the CDE is copied
* fix: ZIP64 header was being written after AES header location was already calculated
* fix: ZIP64 header was being counted twice when writing extra-field length
* fix: deep_copy_file was positioning cursor incorrectly
* test(fuzz): Reimplement Debug so that it prints the method calls actually made
* test(fuzz): Fix issues with `Option<&mut Formatter>`
* chore: Partial debug
* chore: Revert: `merge_contents` already adjusts header_start and data_start
* chore: Revert unused `mut`
* style: cargo fmt --all
* refactor: eliminate a magic number for CDE block size
* chore: WIP: fix bugs
* refactor: Minor refactors
* refactor: eliminate a magic number for CDE block size
* refactor: Minor refactors
* refactor: Can use cde_start_pos to locate ZIP64 end locator
* chore: Fix import that can no longer be feature-gated
* chore: Fix import that can no longer be feature-gated
* refactor: Confusing variable name
* style: cargo fmt --all and fix Clippy warnings
* style: fix another Clippy warning
---------
Signed-off-by: Chris Hennick <4961925+Pr0methean@users.noreply.github.com>
* refactor: eliminate a magic number for CDE block size
* refactor: Minor refactors
* refactor: Can use cde_start_pos to locate ZIP64 end locator
* chore: Fix import that can no longer be feature-gated
* chore: Fix import that can no longer be feature-gated
