As someone who is interested in learning languages, I want to find a way to know the each word and their repetitions inside a book that I am going to read.

  • Welcome to the site! Can you clarify a little bit? Are you looking for software that you could run to generate a list of all the words in a book and how many times they are used? Also, are you looking to do this before you download or buy a book, or after?
    – elixenide
    Mar 28 '15 at 17:49
  • Hi Ed, As you mentioned, I want it to list the words and show how many times they are used. It is like the books which are printed for English Language learners and indicate the words that the reader will run across in the book. For now, I am interested in doing this for the books that are licensed as Creative Commons like in Gutenberg Project. Mar 28 '15 at 20:10
  • Interesting question -- I don't have an answer, but the clarification may help other users.
    – elixenide
    Mar 29 '15 at 5:08
  • @Anthon: Please send me an EPUB that you would like to count the words in, I'll give it a shot. A shorter EPUB is better for testing purposes. I can't PM you so you'll have to PM me to ask for my email address. Thanks.
    – Bulrush
    Apr 21 '16 at 13:12
  • @Bulrush I think you are confusing me with the OP.
    – Anthon
    Apr 21 '16 at 13:52

Here is one way to find the number of words in an epub file sorted by their frequency, with the words used the most at the top of the list.

This is done on a Mac laptop and will also work on Unix hosts.

The overview of the process:

  1. Install Calibre
  2. Use the ./ebook-convert command in Calibre to convert the epub file to text
  3. Transform the entire text file to lowercase (so "Word" and "word" match)
  4. Convert punctuation to whitespace (so "period." and "period " match)
  5. Convert all whitespace to a new line. This puts each word on its own line.
  6. Exclude any blank lines from the list
  7. Sort the list of words alphabetically
  8. Pipe (send) that list of words through uniq -c You now have a count of how often each word appears.
  9. Sort the result in numerical order. If you use the sort command with the -r argument, the most frequent words are at the top.

Here's an example of steps (2) through (9). The head command lists the top ten words in the final output.

$ ./ebook-convert ./book.epub ./book.txt
$ cat   ./book.txt | tr '[:upper:]' '[:lower:]' | tr "“" " " | tr "”" " " | tr "," " " | tr "." " " | tr " " "\n" | grep -v ^$ |  sort | uniq -c | sort -gr  | head
 5303 the
 1960 and
 1934 of
 1910 to
 1874 a
 1168 i
 1067 you
  844 in
  812 that
  703 it

The result is pretty boring. The word 'the" appears 5303 times, while the word 'it' appears 703 times.

I suspect in most books the most common words are the tiny conjunctions, articles, prepositions and pronouns. Perhaps on something that is not a novel this might be more interesting.

Good luck!

  • Thank you for the detailed answer. As a windows user, I would like to know if all the steps you described are done in Calibre. Mar 8 '17 at 20:37
  • Step 1 is done at the OS level. Step 2 is done in Calibre from the command line. The remaining steps are done at the Mac OS command line using the program named Terminal. I know the Unix tools much better than I know the MS Windows tools. I wish you good luck! Mar 8 '17 at 23:24

We can achieve this using Windows PowerShell (with an example epub from IDPF):

$book = [System.IO.File]::ReadAllBytes("$env:TEMP\cc-shared-culture.epub")
# The epub format is essentially a structured wrapper for zip archival
$zipStream = New-Object System.IO.Memorystream # Rather than expanding the archive to disk, let's do the vocab work in memory
$zipStream.Write($book, 0, $book.Count) | Out-Null
$zipFile = [System.IO.Compression.ZipArchive]::new($zipStream)
# The chapter files in this epub are like 
$chapterFiles = $zipFile.Entries | Where-Object -Property FullName -Match "p\d+\.x?html"

# Strip remaining xml markup
function Remove-XmlMarkup {
    param (
    # Check if $HTMLNode is HTMLFile
    if (-not (Test-IsHTMLFile $xml)) {
        Write-Verbose "Node '$xml' is not HTMLFile type; cannot continue"
        #throw exception wrong type
        throw "Get-InnerText: '$xml' is not an HTMLFile"
    if($null -ne $xml -and $null -ne $xml.innerText) {
        return [string](($xml.innerText) -replace "<[^>]+>(?:[^<]+</[^>]+>)?",'')
    } else {
        return [string]::Empty

# Create a function to recursively replace each node with its inner text
function Get-InnerText {
    param (
        [System.__ComObject] $HTMLNode        
    begin {
        # Check if $HTMLNode is HTMLFile
        if (-not (Test-IsHTMLFile $HTMLNode)) {
            Write-Verbose "Node '$HTMLNode' is not HTMLFile type; cannot continue"
            #throw exception wrong type
            throw "Get-InnerText: '$HTMLNode' is not an HTMLFile"
        # Check recursion depthl

        if (($null -eq $recursionDepth) -or -not (Test-Path variable:\recursionDepth)) {
            New-Variable -Name recursionDepth -Value 0 -Description "Recursion depth" -Option AllScope -Visibility Public -Scope Local
        } else {
            Set-Variable -Name recursionDepth -Value $((Get-Variable -Name recursionDepth -ValueOnly) + 1) -Scope Local
        Write-Verbose "Recursion depth is $recursionDepth"
    process {
        #check if $HTMLNode has children
        if ($HTMLNode.hasChildNodes()) {
            Write-Verbose "Node has children to process"
            #Call this function recursively on each child
            foreach ($child in $HTMLNode.childNodes) {
                #try catch block to check for Type mismatch exception
                try {
                    #Call this function recursively on each child
                    Get-InnerText $child
                } catch {
                    Write-Verbose -Message "$($child.innerHTML) could not be splatted"
        } else {
            # If no children, get the inner text
            return (Remove-XmlMarkup $HTMLNode) 
    end {
        #Reset recursion depth
        Set-Variable -Name recursionDepth -Value $((Get-Variable -Name recursionDepth -ValueOnly) -1) -Scope Local
foreach ($chapterFile in $chapterFiles) {
    $chapter = $zipFile.GetEntry($chapterFile)
    $chapter.Open().Read(($chContentBytes = [byte[]]::new($chapter.Length)), 0, $($chapter.Length)) | Out-Null
    $chContentStr = [System.Text.Encoding]::Default.GetString($chContentBytes)
    $HTML = New-Object -Com "HTMLFile"
    $HTML.write([ref]$chContentStr) | Out-Null
    [System.Collections.Generic.Dictionary[string, int]] $wordCount = @{}
    [char[]] $delims = @(' ', "`r", "`n", '"', "'", ".", ',', "’")
    # Create an array of common English words to ignore
    $ignoreWords = @("i", "me", "my", "myself", "we", "our", "ours", "ourselves", "you", "your", "yours", "yourself", "yourselves", "he", "him", "his", "himself", "she", "her", "hers", "herself", "it", "its", "itself", "they", "them", "their", "theirs", "themselves", "what", "which", "who", "whom", "this", "that", "these", "those", "am", "is", "are", "was", "were", "be", "been", "being", "have", "has", "had", "having", "do", "does", "did", "doing", "a", "an", "the", "and", "but", "if", "or", "because", "as", "until", "while", "of", "at", "by", "for", "with", "about", "against", "between", "into", "through", "during", "before", "after", "above", "below", "to", "from", "up", "down", "in", "out", "on", "off", "over", "under", "again", "further", "then", "once", "here", "there", "when", "where", "why", "how", "all", "any", "both", "each", "few", "more", "most", "other", "some", "such", "no", "nor", "not", "only", "own", "same", "so", "than", "too", "very", "s", "t", "can", "will", "just", "don", "should", "now")
    Get-InnerText $HTML | ForEach-Object { 
        $words = $_.Split($delims , @([System.StringSplitOptions]::RemoveEmptyEntries, [System.StringSplitOptions]::TrimEntries )) | Where-Object -FilterScript { ($_.Length -gt 1) -and ($_.ToLowerInvariant() -notmatch "(?:$($ignoreWords -join '|'))")}
        $words.ForEach( {
                $count = 0
                if ($wordCount.TryGetValue($_, [ref]$count)) { 
                    $wordCount[$_] = ++$count 
                else { 
                    $wordCount.Add($_, 0)

The result is in $wordCount.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.