A download checker for Humble Bundle book bundles.
Since there seems to be no answer from this community regarding an application meeting the request in my question, I decided to make one myself.
When you buy a bundle from Humble Bundle, whether books or games, they send you a web link to a HTML page (which I call here the download page).. You can then click on various buttons on the page to download the files composing your bundle. This page can also display on request the MD5 checksums (or signatures) for each of the files, which may be very large for comics and audio-books (and for games).
I wish to check mechanically with a single command that all files have been downloaded, and that the MD5 signature of downloaded files is identical to the MD5 signature specified in the download page provided by Humble Bundle.
The next section describes the rather simple script wrote to achieve my purpose. Given the doubts expressed in a comment to my initial question, it is worth noting that, while applying this script to my own purchases from Humble Bundle, both to check my downloads and to debug the script, I discovered a variety of problems, some related to mixed up MD5 specifications
(not really critical) and others related to various problems that must have occurred during download, that actually required action on my part. So the work was worth my effort.
As usual, this program is doing thing that were not initially intended by its author.
The download checker
The obvious first step is to have a command that will extract from the HTML source of the download page the names of the files together with the corresponding MD5. Here is such a command which is called humblemd5. It takes a single argument which is a file containing the HTML source code of the download page. This HTML source code can be downloaded with your browser.
The output is the list of names of the files to be dowloaded, each preceded by its MD5 signature. This is the standard format of the Linux command md5sum for computing the MD5 of a bunch of files, probably chosen for readability reasons.
Here is such a command written in Perl, developed on Linux, but hopefully working on other platforms. I have also a version written in Linux command line language BASH, since some users seem to have a liking for the command line, which I can add if requested.
#!/usr/bin/perl
# USAGE: humblemd5 <local download page>
# Attribution : This code was written by Babou and is licensed as
# Creative Commons By-SA or GNU GPL. Live long and prosper.
# This script "humblemd5" is intended to extract MD5 signatures of
# Humble-Bundle files books to be downloaded, from the HTML download page
# that provides the hyperlinks to the files.
# The only argument <local download page> is the path to a local copy
# of the download page.
# The standard output contains one line for each file, containing
# first the MD5 signature, followed by 2 spaces, followed by the file name.
# This is the format used by the command "md5sum" in Linux (probably
# also in other Unices).
use strict;
use warnings;
my $file= $ARGV[0];
my $pattmd5='^ *(</div>)?<div data-order-gamekey=".*"" data-md5="(.{32})">$';
my $pattname='^ <a class="a" (download="" )?href="https://dl.humble.com/(.*)[?]gamekey=.*" data-web="https://dl.humble.com/\2[?]gamekey=.*$';
my $md5sig;
open my $fh, '<', $file or die "unable to open file '$file' for reading : $!";
while(my $line=<$fh>) {
if ($line =~ s/$pattmd5/${2}/) {
$md5sig = $line;
chomp $md5sig
} elsif ($line =~ s/$pattname/$2/) {
print $md5sig.' '.$line
}
}
close($fh);
The next step is to compute the MD5 signature of all downloaded files and check that they correspond to the list of names and md5 computed by this humblemd5 command from the download page.
The proper Unix/Linux way would be to have a standard command that checks whether a given bunch of files meet a specification defined by a list of file names preceded by MD5 signatures. I do not know whether such a command exists. Combined with the previous command, it would solve my question.
Instead, since the local topic is not programming but ebooks, I designed a command humblecheck that can be used directly to check downloads. This command is written in the BASH command language of Linux, which I think runs only on Unix systems. The reason is that my knowledge of Perl is getting very rusty, and that I am not sure I could write properly in a system independent way, given that I have no Windows or Mac for testing.
Hopefully the use of this command humblecheck is self explanatory. Its first argument is the download page, as for the previous command, and the second argument is the name of a directory supposed to contain the downloaded files.
#!/bin/bash
# USAGE: humblecheck <local download page> <books directory>
# Attribution : This code was written by Babou and is licensed as
# Creative Commons By-SA or GNU GPL. Live long and prosper.
# This script "humblecheck" is intended to check MD5 signatures for
# book files downloaded from humblebundle.com. It probably works as
# well for other files, such as games, but I have not tried it.
# The first argument is the path to a downloaded local copy of HTML
# page used to download the book files, for example (this is a fake):
# https://www.humblebundle.com/downloads?key=kabjr3jkj5&guard=JGUDEOP7
# The second argument is the path to a directory that contains the
# downloaded book files.
downpage="$1"
downdir="$2"
currentdir=`pwd`
tmpdir=/tmp/hb-md5check
rm -rf $tmpdir
mkdir $tmpdir
[ -s "$downpage" ] || { cat << END-text
First argument file $downpage does not exist or is empty. ABORT.
END-text
exit; }
[ -d "$downdir" ] || { cat << END-text
Second argument $downdir is not a directory. ABORT.
END-text
exit; }
[ "$(ls -A "$downdir"|head -1)" ] || { cat << END-text
Directory $downdir is empty. ABORT.
END-text
exit; }
cat << END-text
The files to be downloaded are accessed from a download page on the
Humble Bundle web site. This html page is copied locally in the file:
$downpage
It contains the names of the files to be downloaded and the corresponding
MD5 signatures.
The files are supposed to have been downloaded in the download directory:
$downdir
This script will now check that all files have been downloaded, and
that they have the correct MD5 signature.
It will also list files found in the download directory that are not
specified in the download page.
--- --- ---
END-text
(cd "$downdir"; md5sum *) > $tmpdir/copysig
humblemd5 "$downpage" > $tmpdir/srcsig
cd $tmpdir
# We sort the lines of the two files in order to compare them.
# However, it seem more logical to sort with respect to file names in
# order to have a readable result, since it is the usual order of
# files. Hence, we have to exchange the order of MD5 and name on each
# line, before we compare the two files with "diff".
# However, we will change back the lines of the "diff" result, if any,
# to display the MD5 before the file name.
sed -e 's|^\([0-9,a-f]\{32\}\) \(.*\)$|\2 \1|g' srcsig > srcsig-r
sort srcsig-r > srcsig-rs
sed -e 's|^\([0-9,a-f]\{32\}\) \(.*\)$|\2 \1|g' copysig > copysig-r
sort copysig-r > copysig-rs
diff srcsig-rs copysig-rs > diff-r
sed -e 's|^\([<,>] \)\(.*\) \([0-9,a-f]\{32\}\)$|\1\3 \2|g' diff-r > diffmd5
grep '^< \([0-9,a-f]\{32\}\) ' diffmd5 > missing
grep '^> \([0-9,a-f]\{32\}\) ' diffmd5 > extra
sed -e 's|^< \([0-9,a-f]\{32\} .*\)$|\1|g' missing > MISSING
sed -e 's|^> \([0-9,a-f]\{32\} .*\)$|\1|g' extra > EXTRA
if [ -s MISSING ]
then
cat << END-text
The following files, specified in the download page, are missing in
the download directory, or contains errors (wrong MD5 signature).
`cat MISSING`
END-text
else
cat << END-text
All files specified in the download page have been correctly downloaded
in the download directory.
END-text
fi
if [ -s EXTRA ]
then
cat << END-text
The following files, not specified in the download page, or downloaded with
an error (wrong MD5), are found in the download directory:
`cat EXTRA`
END-text
else
cat << END-text
All files found in the download directory are correct dowload of files
specified in the download page.
END-text
fi
( [ -s MISSING ] || [ -s EXTRA ] ) && cat << END-text
All file names listed are preceded by their MD5 signature.
It is the MD5 of the intended file in the download page, or of the actual
file found in the download directory.
END-text
