Unable to uncompress the compressed core file if the file size is so big

[sehdshskj]

Hello, when i use the tool minicoredumper on my pc, i have encountered a problem. If the core file size is so big(14 GB), it is unable to uncompress the compressed file. I think the reason is that the file size is more than the maximum size a single tar file supports. Are there some alternate solutions to be sought for this?

[jogness]

Thank you for reporting this.

I can easily confirm the problem. After looking deeper I notice that the used tar format (ustar) does not support files above 8GB. In order to support larger core files a more modern tar format will be needed. As a first step I would like to release a version of minicoredumper that detects the problem and either does a partial capture or none at all.

[jogness]

One possible alternative is to disable "in_tar" in the recept. However, by >8GB core files, this is probably horribly inefficient. To make things worse, the dump_zero() function in the minicoredumper is also horribly inefficient.
I did some testing of my own, and with "in_tar" set to false and a modified dump_zero() to use 1KB buffers, my test program with 16GB allocated takes about 1.5 minutes to core dump. (With tar it would only takes a few seconds, but the minicoredumper tar will be broken.)
I suppose for the next step (a 2.0.3 release) I will add detection if the core file is >8GB and in that case fallback to direct dumping to gzip. Horribly slow and the compressed core file is much larger than with the tar, but at least a core file would exist.

[jogness]

In the master branch I have pushed a workaround that will be released as 2.0.3 soon. See the commit message for details. It refers to the mailing list thread where various alternatives, workarounds and application implementations were discussed.

I am leaving this issue open because the proper solution is to implement a more modern tar format to support large files.

[sehdshskj]

Hello,
Thank you very much for your reply! I have great admiration for your rigor and pragmatism, and it is a great honor to continue to discuss this issue with you in depth.
I have read the commit message on github and the content of the mailing list thread. Same as I analyzed the other day, the best alternative solution for this is to use modern tar format which support the sparse file with unlimited size. But it takes plenty of time to fix it,
I have another easier but less elegant workaround here. If the core file size is more than 8GB, we can truncate it and split it into multiple segments, and a single tar header with a single segment which size is smaller than 8GB. So after core data dumped, there are multiple file in core.tar.gz. After uncompressing core.tar.gz, we use cat command to merge the files. In this way we can avoid size limitations on a single files. I have used this way to fix on my own computer without much coding and impact on performance I know this is just a temporary replacement. All in all, I'm very much looking forward to the new release.

[dharmendra-nutanix]

Hello John,
Thanks for this awesome tool! Has the work to move to a modern tar format started? If not, we were planning to add pax support and create a pull request for it.

[jogness]

Unfortunately I will not get to updating the tar format anytime soon. Patches to support additional formats to fix this issue are very welcome!

[dharmendra-nutanix]

Hey @jogness, Prakhar from Nutanix has created a pull request at #10 for adding the PAX support. Can you please take a look? Thanks

[jogness]

Yes, I have seen it. I am taking a look at it. Thanks for the ping.

[ShaharHD]

Any updates on getting #10 merged?

Got the same issue with a small tester applicaiton on an Aarch64 linux (Yocto)

[FengFei18]

Are there any plans to further improve and eventually merge this patch? #10
The basic functionality has been validated and seems to work reliably.

[jogness]

My problem with the patch (aside from it doing a lot of changes and refactoring in a single patch) is that it is touching lots of things that are not strictly necessary to support PAX. I need to make some time to do a proper review. I do not have a problem supporting PAX per se.

[haferburg]

We ran into the problem described in this issue. I've prepared an alternative patch.

After reverting the workaround 499f8c1, it uses GNU tar base-256 encoding: set bit 7 of the first byte and store the value big-endian in the remaining bytes.

2026-07-14_minicoredumper_issue_6.patch

My initial tests were successful. I could generate a core.tar.gz that extracts to 10 GB, and load it in gdb.

Would you be open to a PR?

Full disclosure: I've generated this with Github Copilot.

[jogness]

To look at your proposal I did a bit of reading up on the tar format. To my surprise, minicoredumper is already using the GNU tar magic+version. So this change is even an official part of the format.

Would you be open to a PR?

Yes, but please cleanup the code so that it is easier to read/understand by a human. The Copilot version is ridiculously complex. The decision to use octal or base-256 should be a simple value comparison. And naming the function put_gnu_number() is more appropriate.