I am looking for a software or a way to list, extract, count, in short, analyze a book which is in epub or pdf format

Question

As someone who is interested in learning languages, I want to find a way to know the each word and their repetitions inside a book that I am going to read.

Answer 1

Here is one way to find the number of words in an epub file sorted by their frequency, with the words used the most at the top of the list.

This is done on a Mac laptop and will also work on Unix hosts.

The overview of the process:

1. Install Calibre
2. Use the ./ebook-convert command in Calibre to convert the epub file to text
3. Transform the entire text file to lowercase (so "Word" and "word" match)
4. Convert punctuation to whitespace (so "period." and "period " match)
5. Convert all whitespace to a new line. This puts each word on its own line.
6. Exclude any blank lines from the list
7. Sort the list of words alphabetically
8. Pipe (send) that list of words through uniq -c You now have a count of how often each word appears.
9. Sort the result in numerical order. If you use the sort command with the -r argument, the most frequent words are at the top.

Here's an example of steps (2) through (9). The head command lists the top ten words in the final output.

$ ./ebook-convert ./book.epub ./book.txt
$ cat   ./book.txt | tr '[:upper:]' '[:lower:]' | tr "“" " " | tr "”" " " | tr "," " " | tr "." " " | tr " " "\n" | grep -v ^$ |  sort | uniq -c | sort -gr  | head
 5303 the
 1960 and
 1934 of
 1910 to
 1874 a
 1168 i
 1067 you
  844 in
  812 that
  703 it
 $

The result is pretty boring. The word 'the" appears 5303 times, while the word 'it' appears 703 times.

I suspect in most books the most common words are the tiny conjunctions, articles, prepositions and pronouns. Perhaps on something that is not a novel this might be more interesting.

Good luck!

Answer 2

We can achieve this using Windows PowerShell (with an example epub from IDPF):

$book = [System.IO.File]::ReadAllBytes("$env:TEMP\cc-shared-culture.epub")
[System.Reflection.Assembly]::LoadWithPartialName('System.IO.Compression')
# The epub format is essentially a structured wrapper for zip archival
$zipStream = New-Object System.IO.Memorystream # Rather than expanding the archive to disk, let's do the vocab work in memory
$zipStream.Write($book, 0, $book.Count) | Out-Null
$zipFile = [System.IO.Compression.ZipArchive]::new($zipStream)
# The chapter files in this epub are like 
$chapterFiles = $zipFile.Entries | Where-Object -Property FullName -Match "p\d+\.x?html"

# Strip remaining xml markup
function Remove-XmlMarkup {
    param (
        [System.__ComObject]$xml
    )
    # Check if $HTMLNode is HTMLFile
    if (-not (Test-IsHTMLFile $xml)) {
        Write-Verbose "Node '$xml' is not HTMLFile type; cannot continue"
        #throw exception wrong type
        throw "Get-InnerText: '$xml' is not an HTMLFile"
    }
    if($null -ne $xml -and $null -ne $xml.innerText) {
        return [string](($xml.innerText) -replace "<[^>]+>(?:[^<]+</[^>]+>)?",'')
    } else {
        return [string]::Empty
    }
}

# Create a function to recursively replace each node with its inner text
function Get-InnerText {
    [CmdletBinding()]
    param (
        [Parameter(Mandatory)]
        [System.__ComObject] $HTMLNode        
    )
    
    begin {
        # Check if $HTMLNode is HTMLFile
        if (-not (Test-IsHTMLFile $HTMLNode)) {
            Write-Verbose "Node '$HTMLNode' is not HTMLFile type; cannot continue"
            #throw exception wrong type
            throw "Get-InnerText: '$HTMLNode' is not an HTMLFile"
        }
        # Check recursion depthl

        if (($null -eq $recursionDepth) -or -not (Test-Path variable:\recursionDepth)) {
            New-Variable -Name recursionDepth -Value 0 -Description "Recursion depth" -Option AllScope -Visibility Public -Scope Local
        } else {
            Set-Variable -Name recursionDepth -Value $((Get-Variable -Name recursionDepth -ValueOnly) + 1) -Scope Local
        }
        Write-Verbose "Recursion depth is $recursionDepth"
    }
    
    process {
        #check if $HTMLNode has children
        if ($HTMLNode.hasChildNodes()) {
            Write-Verbose "Node has children to process"
            #Call this function recursively on each child
            foreach ($child in $HTMLNode.childNodes) {
                #try catch block to check for Type mismatch exception
                try {
                    #Call this function recursively on each child
                    Get-InnerText $child
                } catch {
                    Write-Verbose -Message "$($child.innerHTML) could not be splatted"
                }
            }
        } else {
            # If no children, get the inner text
            return (Remove-XmlMarkup $HTMLNode) 
        }
    }
    end {
        #Reset recursion depth
        Set-Variable -Name recursionDepth -Value $((Get-Variable -Name recursionDepth -ValueOnly) -1) -Scope Local
    }
}
foreach ($chapterFile in $chapterFiles) {
    $chapter = $zipFile.GetEntry($chapterFile)
    $chapter.Open().Read(($chContentBytes = [byte[]]::new($chapter.Length)), 0, $($chapter.Length)) | Out-Null
    $chContentStr = [System.Text.Encoding]::Default.GetString($chContentBytes)
    $HTML = New-Object -Com "HTMLFile"
    $HTML.write([ref]$chContentStr) | Out-Null
    [System.Collections.Generic.Dictionary[string, int]] $wordCount = @{}
    [char[]] $delims = @(' ', "`r", "`n", '"', "'", ".", ',', "’")
    # Create an array of common English words to ignore
    $ignoreWords = @("i", "me", "my", "myself", "we", "our", "ours", "ourselves", "you", "your", "yours", "yourself", "yourselves", "he", "him", "his", "himself", "she", "her", "hers", "herself", "it", "its", "itself", "they", "them", "their", "theirs", "themselves", "what", "which", "who", "whom", "this", "that", "these", "those", "am", "is", "are", "was", "were", "be", "been", "being", "have", "has", "had", "having", "do", "does", "did", "doing", "a", "an", "the", "and", "but", "if", "or", "because", "as", "until", "while", "of", "at", "by", "for", "with", "about", "against", "between", "into", "through", "during", "before", "after", "above", "below", "to", "from", "up", "down", "in", "out", "on", "off", "over", "under", "again", "further", "then", "once", "here", "there", "when", "where", "why", "how", "all", "any", "both", "each", "few", "more", "most", "other", "some", "such", "no", "nor", "not", "only", "own", "same", "so", "than", "too", "very", "s", "t", "can", "will", "just", "don", "should", "now")
    Get-InnerText $HTML | ForEach-Object { 
        $words = $_.Split($delims , @([System.StringSplitOptions]::RemoveEmptyEntries, [System.StringSplitOptions]::TrimEntries )) | Where-Object -FilterScript { ($_.Length -gt 1) -and ($_.ToLowerInvariant() -notmatch "(?:$($ignoreWords -join '|'))")}
        $words.ForEach( {
                $count = 0
                if ($wordCount.TryGetValue($_, [ref]$count)) { 
                    $wordCount[$_] = ++$count 
                }
                else { 
                    $wordCount.Add($_, 0)
                } 
            }) 
    }
}

The result is in $wordCount.

Answer 3

As English learner maybe you will be also interested in https://github.com/artemtomchuck/MyKnownWordsDictionary project. This extracts unique words from your book (straight txt should be an input, but it is not a problem to transform your pdf or epub into txt). Then it filters out the word you already know (based on your own dictionary). Then it prints new words from book ordered by frequency (the most frequent on the top) with their translation to your native language (given that book is written in English or any other language which you are learning).

Notes:

1. in output csv file frequency of word is not printed, but if you understand the source code then you can include this frequency column into your output (currently it is just hidden).
2. this mechanism is very simplistic and does not recognize plural/singular forms, names or words with "-" symbol (e.g. single-minded is treated like two separate words: single and minded)

Full disclosure: this is my personal project and it may not fully suit your purposes or may be complex for using/setup by non-technical people (non-software engineers). But I tried to document setup as best as I can in github. The reason why I am posting the answer here: initially I looked some solutions from answers in this question. I found the solution in https://ebooks.stackexchange.com/a/6557 , but it was quite a routine for regular usage and that's why I automated process a bit.