1
The backup process creates a tar file and then applies gzip to it. There are two types of things in the tar file: an already gzipped sql dump and the contents of uploads (if requested.) In my case every upload file is already compressed: gz, gzip, gif, jpeg, png, zip. So the final gzipping gains only 1% of size.
I believe it would be better to demand less free space.
A previous topic from 2016 mentions disabling backup compression, but it looks like the sql dump was at that time not compressed, which shifted the tradeoffs.
11 Likes
gerhard (Gerhard Schlager) 2
I’m already working on a new backup format that removes the double compression. My hope is that it will be ready within a month or two.
13 Likes
Ed_S (Ed S) 3
2 Likes
Ed_S (Ed S) 5
Not to bug you too much, but how is this progressing?
gerhard (Gerhard Schlager) 6
Development of that feature is currently paused and it isn’t on our current roadmap. I hope we will get to it in 2024.
4 Likes
Isambard (Isambard) 7
If I wrote a patch to accept a 0 in the compression rate to disable gzip, would that be something that you would accept?
1 Like
Ed_S (Ed S) 8
(I’m guessing that you’d save CPU time that way, but not space, because the gzipped tar file would still be created.)
Isambard (Isambard) 9
I’m aiming to save cpu time. Actually, I was thinking of using the 0 as a flag that would change the code path so that it doesn’t gzip (sadly, zero is not a valid compression level supported across all gzip versions, afaik).
Ed_S (Ed S) 10
Hmm that wouldn’t help me at all! (Likewise others who’ve had the same problem with limited disk space.)
If tar were being used, it could be used with z or j options. If a subshell were being used, the output of tar could be piped into gzip. But I think in fact some higher level ruby functions may be in use.
1 Like
Ed_S (Ed S) 12
Maybe it shouldn’t be too difficult… I appreciate that making changes to backup and restore must be made with great care, but I think just inlining the compression would save a lot of space requirement without any compatibility question.
From tar --help
-a, --auto-compress use archive suffix to determine the compression
-z, --gzip, --gunzip, --ungzip filter the archive through gzip
1 Like
Isambard (Isambard) 13
Does -z actually do an in-place compression? I always assumed that it just ran gzip after the tar file is completed.
Ed_S (Ed S) 14
Unwisely, in this case! The bytes which represent the uncompressed tar file never hit the disk.
2 Likes
MentalNomad (Mental Nomad) 15
Are you saying we can simply add
"--gzip",
And it will stop requiring fully double the actually space used used the data?
1 Like
Ed_S (Ed S) 17
Looks like an even better choice is --zstd but then we’d also need ‘zstd’ package installed in the docker image.
2 Likes
Ed_S (Ed S) 18
Possibly a better approach which might simplify things - being able to deal with existing backups which might be *.gz or *.zst - is to use tar’s automatic detection:
tar --auto-compress -c -f ../file.tar.gz .
tar --auto-compress -c -f ../file.tar.zst .
More important for the unpacking, of course, where we might not know what we’re going to see.
Presently the ruby code seems to do lots of things which tar itself can do. Hopefully this can be simplified, rather than becoming more complex.
2 Likes
Ed_S (Ed S) 19
zstd is also a great deal quicker - which makes it less problematic that we spend time compressing the near-incompressible.
(If zstd were also used for the sql dump, in my case that comes out 10% smaller.)
1 Like
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。