3

Humble-Bundle has a book section that sells weekly ebooks bundles. They are very professional and provide a MD5 signature for each file, i.e., for each format of each book title. This makes it easy to check ebook files integrity, i.e., that the ebook files have been dowloaded on the computer without error. One only has to recompute locally the MD5 signatures, and compare them with those provided by Humble-Bundle in the HTML dowload page, from which ebooks are dowloaded.

My problem is that when you download 3 files corresponding to 3 formats for each of 40 books in a bundle, that makes 120 files, and the checking is a tedious and time consuming task, that should nevertheless be done carefully (I think). Actually, I feel that this remains true even when you download only one format for each title.

I have started looking for a way of checking automatically the signatures for all the downloaded files in a bundle. Of course, my first question is whether existing tools are available for that.

I am specificaly looking for a tool running on Linux, but I guess answers for other systems may be relevant answers, though probably useless for me.

The Humble-Bundle site does not seem to suggest anything of the kind. All I found is the basic explanation for checking the MD5 signature of a file.

  • Dow you have some reasons to believe that file integrity is compromised so often to justify checking the MD5 sum every time? I never acquired ebooks through Humble-Bundle, but from the other stores that I use I have never had any problem. If I were in your shoes, I would simply load the ebooks into my ereader, and if they load and open correctly, I won't assume that they are corrupted. In the remote case that you experience some problems, you can check the MD5 sum for that particular file only. – Sekhemty May 25 '17 at 18:15
  • Checksums is a standard way of preventing loss of integrity, even though less critical for ebooks than for code. I did run into integrity problems, though that is rare. It is more likely for e-comics because they may exceed 1 GB, close to DVD size, and checksums are standard for downloading DVD images. Checking on my ereader is more work than checking MD5, and is not mechanizable. Postponing till I read the book may wait for a time when downloading a better version is no longer an option. The whole point of the question is to check a whole bundle in one click, and feel safe at no cost. – babou May 25 '17 at 22:09
  • I understand. If these ebook files can be downloaded just for a limited time, then checking them could be important. Unfortunately I don't have other suggestions for you. Maybe other users could be more useful. – Sekhemty May 25 '17 at 22:23
  • They do not actually set a time limit. But everything is mortal in this universe (afaik). I learned the hard way to be paranoid, and I am very surprised that so many people trust the e-world to be stable and reliable. And we seldom get advance warning. Thanks for the comment. – babou May 25 '17 at 22:39
  • 1
    @Sekhemty Not getting an adequate answer to my question, I wrote a script to do the job, which I tested on several of my own purchases from Humble Bundle (see my answer below). I did get some MD5 mismatch, probably due to some file switching problems at Humble Bundle (I had also found one by hand a year ago). I also discovered various other problems regarding unreadable files, or files I had forgotten to download. I guess that should answer your doubts about the usefulness of automated checking of download. I hope you appreciate that it is a BASH script :). – babou Jun 7 '17 at 15:12
1

A download checker for Humble Bundle book bundles.

Since there seems to be no answer from this community regarding an application meeting the request in my question, I decided to make one myself.

When you buy a bundle from Humble Bundle, whether books or games, they send you a web link to a HTML page (which I call here the download page).. You can then click on various buttons on the page to download the files composing your bundle. This page can also display on request the MD5 checksums (or signatures) for each of the files, which may be very large for comics and audio-books (and for games).

I wish to check mechanically with a single command that all files have been downloaded, and that the MD5 signature of downloaded files is identical to the MD5 signature specified in the download page provided by Humble Bundle.

The next section describes the rather simple script wrote to achieve my purpose. Given the doubts expressed in a comment to my initial question, it is worth noting that, while applying this script to my own purchases from Humble Bundle, both to check my downloads and to debug the script, I discovered a variety of problems, some related to mixed up MD5 specifications (not really critical) and others related to various problems that must have occurred during download, that actually required action on my part. So the work was worth my effort.

As usual, this program is doing thing that were not initially intended by its author.

The download checker

The obvious first step is to have a command that will extract from the HTML source of the download page the names of the files together with the corresponding MD5. Here is such a command which is called humblemd5. It takes a single argument which is a file containing the HTML source code of the download page. This HTML source code can be downloaded with your browser. The output is the list of names of the files to be dowloaded, each preceded by its MD5 signature. This is the standard format of the Linux command md5sum for computing the MD5 of a bunch of files, probably chosen for readability reasons.

Here is such a command written in Perl, developed on Linux, but hopefully working on other platforms. I have also a version written in Linux command line language BASH, since some users seem to have a liking for the command line, which I can add if requested.

#!/usr/bin/perl
# USAGE:  humblemd5 <local download page>

# Attribution : This code was written by Babou and is licensed as
#               Creative Commons By-SA or GNU GPL. Live long and prosper.

# This script "humblemd5" is intended to extract MD5 signatures of
# Humble-Bundle files books to be downloaded, from the HTML download page
# that provides the hyperlinks to the files.

# The only argument <local download page> is the path to a local copy
# of the download page.
# The standard output contains one line for each file, containing
# first the MD5 signature, followed by 2 spaces, followed by the file name.

# This is the format used by the command "md5sum" in Linux (probably
# also in other Unices).

use strict;
use warnings;

my $file= $ARGV[0];

my $pattmd5='^ *(</div>)?<div data-order-gamekey=".*"" data-md5="(.{32})">$';

my $pattname='^      <a class="a" (download="" )?href="https://dl.humble.com/(.*)[?]gamekey=.*" data-web="https://dl.humble.com/\2[?]gamekey=.*$';

my $md5sig;

open my $fh, '<', $file or die "unable to open file '$file' for reading : $!";
while(my $line=<$fh>) {
    if ($line =~ s/$pattmd5/${2}/) {
      $md5sig = $line;
      chomp $md5sig
    } elsif ($line =~ s/$pattname/$2/) {
      print  $md5sig.'  '.$line
    }
}
close($fh);

The next step is to compute the MD5 signature of all downloaded files and check that they correspond to the list of names and md5 computed by this humblemd5 command from the download page.

The proper Unix/Linux way would be to have a standard command that checks whether a given bunch of files meet a specification defined by a list of file names preceded by MD5 signatures. I do not know whether such a command exists. Combined with the previous command, it would solve my question.

Instead, since the local topic is not programming but ebooks, I designed a command humblecheck that can be used directly to check downloads. This command is written in the BASH command language of Linux, which I think runs only on Unix systems. The reason is that my knowledge of Perl is getting very rusty, and that I am not sure I could write properly in a system independent way, given that I have no Windows or Mac for testing.

Hopefully the use of this command humblecheck is self explanatory. Its first argument is the download page, as for the previous command, and the second argument is the name of a directory supposed to contain the downloaded files.

#!/bin/bash
# USAGE: humblecheck <local download page> <books directory>

# Attribution : This code was written by Babou and is licensed as
#               Creative Commons By-SA or GNU GPL. Live long and prosper.

# This script "humblecheck" is intended to check MD5 signatures for
# book files downloaded from humblebundle.com. It probably works as
# well for other files, such as games, but I have not tried it.

# The first argument is the path to a downloaded local copy of HTML
# page used to download the book files, for example (this is a fake):
#   https://www.humblebundle.com/downloads?key=kabjr3jkj5&guard=JGUDEOP7

# The second argument is the path to a directory that contains the
# downloaded book files.

downpage="$1"
downdir="$2"
currentdir=`pwd`
tmpdir=/tmp/hb-md5check
rm -rf $tmpdir
mkdir $tmpdir

[ -s "$downpage" ] || { cat << END-text
First argument file $downpage does not exist or is empty. ABORT.
END-text
                     exit; }

[ -d "$downdir" ] || { cat << END-text
Second argument $downdir is not a directory. ABORT.
END-text
                     exit; }

[ "$(ls -A "$downdir"|head -1)" ] || { cat << END-text
Directory $downdir is empty. ABORT.
END-text
                     exit; }

cat << END-text
The files to be downloaded are accessed from a download page on the
Humble Bundle web site. This html page is copied locally in the file:
   $downpage
It contains the names of the files to be downloaded and the corresponding
MD5 signatures.

The files are supposed to have been downloaded in the download directory:
   $downdir

This script will now check that all files have been downloaded, and
that they have the correct MD5 signature.

It will also list files found in the download directory that are not
specified in the download page.

---  ---  ---

END-text

(cd "$downdir"; md5sum *) > $tmpdir/copysig
humblemd5 "$downpage" > $tmpdir/srcsig
cd $tmpdir


# We sort the lines of the two files in order to compare them.
# However, it seem more logical to sort with respect to file names in
# order to have a readable result, since it is the usual order of
# files. Hence, we have to exchange the order of MD5 and name on each
# line, before we compare the two files with "diff".

# However, we will change back the lines of the "diff" result, if any,
# to display the MD5 before the file name.

sed -e 's|^\([0-9,a-f]\{32\}\)  \(.*\)$|\2  \1|g' srcsig > srcsig-r
sort srcsig-r > srcsig-rs

sed -e 's|^\([0-9,a-f]\{32\}\)  \(.*\)$|\2  \1|g' copysig > copysig-r
sort copysig-r > copysig-rs

diff srcsig-rs copysig-rs > diff-r

sed -e 's|^\([<,>] \)\(.*\)  \([0-9,a-f]\{32\}\)$|\1\3  \2|g' diff-r > diffmd5

grep '^< \([0-9,a-f]\{32\}\)  ' diffmd5 > missing
grep '^> \([0-9,a-f]\{32\}\)  ' diffmd5 > extra

sed -e 's|^< \([0-9,a-f]\{32\}  .*\)$|\1|g' missing >  MISSING
sed -e 's|^> \([0-9,a-f]\{32\}  .*\)$|\1|g' extra >  EXTRA

if [ -s MISSING ]
then
   cat << END-text
The following files, specified in the download page, are missing in
the download directory, or contains errors (wrong MD5 signature).

`cat MISSING`

END-text
else
   cat << END-text
All files specified in the download page have been correctly downloaded
in the download directory.

END-text
fi  

if [ -s EXTRA ]
then
   cat << END-text
The following files, not specified in the download page, or downloaded with
an error (wrong MD5), are found in the download directory:

`cat EXTRA`

END-text
else
   cat << END-text
All files found in the download directory are correct dowload of files
specified in the download page.

END-text
fi

 ( [ -s MISSING ] || [ -s EXTRA ] ) &&   cat << END-text
All file names listed are preceded by their MD5 signature.
It is the MD5 of the intended file in the download page, or of the actual
file found in the download directory.

END-text
0

Humble Bundle is great. Great service with fun offerings.

I typically don't worry about md5 checksums, but this web page lists some tools to verify file integrity. https://www.lifewire.com/validate-md5-checksum-file-4037391

I think the search term you want would be "batch md5 checksum tool". Here's one app that seems to do what you want. http://getmd5checker.com/features.html (PS, I haven't tried it).

By the way, the last time I downloaded a Humble ebook bundle, the files were PDF and were huge!

Batching tools are great. I used to unzip dozens of zip and rar files manually until I asked myself, "Surely someone has created a free tool to unzip a batch of zipped files!" As it turns out, someone has (it's called "ExtractNow")

  • Thanks for the links. The more interesting is the second one, unfortunately available only on Windows. I am not sure I understand it all: it seem to be a kind of adjunct to the underlying OS to preserve and check MD5 for files when they are manipulated, with a dedicated interface for that. Actually, my problem is Humble-Bundle specific, because I also need to extract the MD5 information from the HTML code of the download page, and put it in the right format for a checking tool, or build an ad-hoc checking tool myself. The aim to to check a whole downloaded bundle with a single command. – babou May 27 '17 at 12:33
  • 1
    "Batching tools are great. I used to unzip dozens of zip and rar files manually until I asked myself, "Surely someone has created a free tool to unzip a batch of zipped files!" As it turns out, someone has (it's called "ExtractNow")" And what is even greater, is the Linux command line ;) – Sekhemty Jun 1 '17 at 17:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.