Haven't found any archiving functionality in Calibre by myself, so assuming it doesn't exist. I am right? How can I shrink the library size using Calibre without changing a books format?
Calibre does not provide any compression tools, but indeed one could use third party tools to compress files in the directory structure. The predominant files to target would be DJVU and PDF files. Using ImageMagick's convert utility, it is not difficult to do programmatic compression (write a script that recursively goes through the library directory structure and compresses all PDF files). Testing is important to ensure no loss of visual quality.
Note that the same thing can be done for video files using the ffmpeg utility. Works a dream.