Package::LoadDir("") returns nothing on large package

[SpiderPig]

I've compressed 5,259 4096x4096 r16 heightmap files into a zip file.  It's about 7GB of data.  Using LoadDir("") to list the files at the root returns a size of 0 ONLY for this zip file.

I used the code below on a zip file of 5,259 1024x1024 r16 heightmaps (1.2GB) and it worked fine.  Pretty sure all the files have the exact same names in both zip files.  It's just the map resolution that is different.

I thought maybe LoadDir() or LoadPackage() doesn't like the large zip file?

 

This is the snippet of code that handles the loading.

if (extension == "zip") {
	auto package = LoadPackage(filepath);

	if (package != nullptr) {
		auto dir = package->LoadDir("");

 

[Josh]

I think 4 GB is the max size for a zip file:
https://superuser.com/questions/1305867/what-is-the-maximum-size-of-a-zip-file-on-windows-10-pro-64-bit#:~:text=1 Answer&text=4 GB size is a,about 16 exabytes size limitation).

[SpiderPig]

So 4GB was a limit in the past?  So it should work now...?

[Josh]

What happens when you open this zip file in another program?

[Josh]

https://comp.lang.idl-pvwave.narkive.com/L2ukAnN1/zlib-uncompress-limited-to-4gb

[SpiderPig]

13 minutes ago, Josh said:

https://comp.lang.idl-pvwave.narkive.com/L2ukAnN1/zlib-uncompress-limited-to-4gb

I compressed it with 7zip, it's interesting that last comment saying it's not a zip?  Even though it is... and I've compressed large files with window's default zip too and it worked above 4GB.

Anyway, I can open the 7GB in windows explorer to search the files and I can extract from it.  Don't know any other software besides 7zip to try it with...

[SpiderPig]

I've just confirmed ReadFile() returns nullptr too.  The package seems to load okay - well there's no errors and it's not nullptr.

[SpiderPig]

If this is a limitation if zlib then this might be an opportunity for a 7zip plugin. 

[IceBurger]

24 minutes ago, SpiderPig said:

I compressed it with 7zip, it's interesting that last comment saying it's not a zip?

I was confused with that as well, knowing the 7z does do zip. I decided that he was referring to his own data, not what 7z does.

[SpiderPig]

Yeah probably.  It's defiantly a zip.

[Josh]

Zip64 sounds like one of those things like JPEG2000 where it just gets no adoption.

It appears ziplib does not support zip64 files:
https://www.mjtnet.com/forum/viewtopic.php?t=7120

Libzip probably does not either:
https://github.com/nih-at/libzip/issues/239

[reepblue]

5 hours ago, SpiderPig said:

If this is a limitation if zlib then this might be an opportunity for a 7zip plugin. 

The library that 7z uses should be apart of ziplib unless Josh excluded it.

[SpiderPig]

3 hours ago, Josh said:

Libzip probably does not either:
https://github.com/nih-at/libzip/issues/239

In this link the guy reckons it does actually work on 64 bit machines... just not on 32bit.  To me that makes sense that a 32bit machine can't load something bigger than 4GB but I'm no expert.

Quote

Note that the same code (same libzip version, same ZIP file) works correctly on 64-bit devices. I checked on both AMD-64 and ARM-64.

 

26 minutes ago, reepblue said:

The library that 7z uses should be apart of ziplib unless Josh excluded it.

I did try loading a 7GB ".7z" file just to be sure but it didn't load.  I don't know much about 7z, it uses LZMA or something?  Maybe this SDK is what I need to make a 7z plugin.

https://www.7-zip.org/sdk.html

But I don't know, maybe libzip has a bug with loading over 4GB files...

[reepblue]

LZMA is in the ziplib repo and should be easily compilable. Josh just has to include it.

I actually rearranged ziplib so it all compiled in one fat static lib and it worked fine.

[SpiderPig]

I wonder if that will solve this issue or of there's something else at play...

[Alienhead]

RaR handles up to 8,589,934,591 GB but still limited by fat32. Besides, rar isn't open source is it?

EDIT: Nvm, just found thgis https://github.com/peazip/PeaZip

 

[Canardia]

7z has 7 different algorithms, hence the name 7z.
The most powerful compression is LZMA2 Ultra.

[Josh]

On 1/8/2024 at 5:41 PM, reepblue said:

LZMA is in the ziplib repo and should be easily compilable. Josh just has to include it.

I actually rearranged ziplib so it all compiled in one fat static lib and it worked fine.

So you are saying ziplib can be compiled in a way that supports more than 4 GB zip files? Have you done this?

[Josh]

@reepblue I found this in ZipMethodResolver.h but it appears that it already includes everything by default:

#define ZIP_METHOD_TABLE          \
  ZIP_METHOD_ADD(StoreMethod);    \
  ZIP_METHOD_ADD(DeflateMethod);  \
  ZIP_METHOD_ADD(Bzip2Method);    \
  ZIP_METHOD_ADD(LzmaMethod);

#define ZIP_METHOD_ADD(method_class)                                                            \
  if (compressionMethod == method_class::GetZipMethodDescriptorStatic().GetCompressionMethod()) \
    return method_class::Create()

 

[Josh]

I think LMZA is already in there.

[Josh]

Apparently this version of minizip would maybe work, as long as you have less than 65536 files:
https://github.com/zlib-ng/minizip-ng

However, adoption of Zip64 appears to be almost non-existent in the real world. Writing a new archive format isn't exactly difficult, and may be less work than implementing another library, then dealing with all the bad design decisions a programmer who isn't me will invariably make.

If I were to make an encryption format, I would include:

• unsigned 64-bit integers for everything so we don't have this stupid size limitation bullshit
• file table would be at the end of the file
• file removal without shifting all other files

Maybe for really big archives we should be looking at something like that. The advantage would be that you can upload a new version of the archive on Steam and it would only upload the sections that changed, because each build would not shift the entire contents around. I think Valve did something like this with their packages files, which is why they have a defragment feature.

In the immediate term, I recommend splitting your data up into multiple zip archives.

[SpiderPig]

1 hour ago, Josh said:

In the immediate term, I recommend splitting your data up into multiple zip archives.

Yeah I'll do this for now.  Thankyou.