As someone who is interested in learning languages, I want to find a way to know the each word and their repetitions inside a book that I am going to read.

  • Welcome to the site! Can you clarify a little bit? Are you looking for software that you could run to generate a list of all the words in a book and how many times they are used? Also, are you looking to do this before you download or buy a book, or after?
    – elixenide
    Mar 28, 2015 at 17:49
  • Hi Ed, As you mentioned, I want it to list the words and show how many times they are used. It is like the books which are printed for English Language learners and indicate the words that the reader will run across in the book. For now, I am interested in doing this for the books that are licensed as Creative Commons like in Gutenberg Project. Mar 28, 2015 at 20:10
  • Interesting question -- I don't have an answer, but the clarification may help other users.
    – elixenide
    Mar 29, 2015 at 5:08
  • @Anthon: Please send me an EPUB that you would like to count the words in, I'll give it a shot. A shorter EPUB is better for testing purposes. I can't PM you so you'll have to PM me to ask for my email address. Thanks.
    – Bulrush
    Apr 21, 2016 at 13:12
  • @Bulrush I think you are confusing me with the OP.
    – Anthon
    Apr 21, 2016 at 13:52

3 Answers 3


Here is one way to find the number of words in an epub file sorted by their frequency, with the words used the most at the top of the list.

This is done on a Mac laptop and will also work on Unix hosts.

The overview of the process:

  1. Install Calibre
  2. Use the ./ebook-convert command in Calibre to convert the epub file to text
  3. Transform the entire text file to lowercase (so "Word" and "word" match)
  4. Convert punctuation to whitespace (so "period." and "period " match)
  5. Convert all whitespace to a new line. This puts each word on its own line.
  6. Exclude any blank lines from the list
  7. Sort the list of words alphabetically
  8. Pipe (send) that list of words through uniq -c You now have a count of how often each word appears.
  9. Sort the result in numerical order. If you use the sort command with the -r argument, the most frequent words are at the top.

Here's an example of steps (2) through (9). The head command lists the top ten words in the final output.

$ ./ebook-convert ./book.epub ./book.txt
$ cat   ./book.txt | tr '[:upper:]' '[:lower:]' | tr "“" " " | tr "”" " " | tr "," " " | tr "." " " | tr " " "\n" | grep -v ^$ |  sort | uniq -c | sort -gr  | head
 5303 the
 1960 and
 1934 of
 1910 to
 1874 a
 1168 i
 1067 you
  844 in
  812 that
  703 it

The result is pretty boring. The word 'the" appears 5303 times, while the word 'it' appears 703 times.

I suspect in most books the most common words are the tiny conjunctions, articles, prepositions and pronouns. Perhaps on something that is not a novel this might be more interesting.

Good luck!

  • Thank you for the detailed answer. As a windows user, I would like to know if all the steps you described are done in Calibre. Mar 8, 2017 at 20:37
  • Step 1 is done at the OS level. Step 2 is done in Calibre from the command line. The remaining steps are done at the Mac OS command line using the program named Terminal. I know the Unix tools much better than I know the MS Windows tools. I wish you good luck! Mar 8, 2017 at 23:24

We can achieve this using Windows PowerShell (with an example epub from IDPF):

$book = [System.IO.File]::ReadAllBytes("$env:TEMP\cc-shared-culture.epub")
# The epub format is essentially a structured wrapper for zip archival
$zipStream = New-Object System.IO.Memorystream # Rather than expanding the archive to disk, let's do the vocab work in memory
$zipStream.Write($book, 0, $book.Count) | Out-Null
$zipFile = [System.IO.Compression.ZipArchive]::new($zipStream)
# The chapter files in this epub are like 
$chapterFiles = $zipFile.Entries | Where-Object -Property FullName -Match "p\d+\.x?html"

# Strip remaining xml markup
function Remove-XmlMarkup {
    param (
    # Check if $HTMLNode is HTMLFile
    if (-not (Test-IsHTMLFile $xml)) {
        Write-Verbose "Node '$xml' is not HTMLFile type; cannot continue"
        #throw exception wrong type
        throw "Get-InnerText: '$xml' is not an HTMLFile"
    if($null -ne $xml -and $null -ne $xml.innerText) {
        return [string](($xml.innerText) -replace "<[^>]+>(?:[^<]+</[^>]+>)?",'')
    } else {
        return [string]::Empty

# Create a function to recursively replace each node with its inner text
function Get-InnerText {
    param (
        [System.__ComObject] $HTMLNode        
    begin {
        # Check if $HTMLNode is HTMLFile
        if (-not (Test-IsHTMLFile $HTMLNode)) {
            Write-Verbose "Node '$HTMLNode' is not HTMLFile type; cannot continue"
            #throw exception wrong type
            throw "Get-InnerText: '$HTMLNode' is not an HTMLFile"
        # Check recursion depthl

        if (($null -eq $recursionDepth) -or -not (Test-Path variable:\recursionDepth)) {
            New-Variable -Name recursionDepth -Value 0 -Description "Recursion depth" -Option AllScope -Visibility Public -Scope Local
        } else {
            Set-Variable -Name recursionDepth -Value $((Get-Variable -Name recursionDepth -ValueOnly) + 1) -Scope Local
        Write-Verbose "Recursion depth is $recursionDepth"
    process {
        #check if $HTMLNode has children
        if ($HTMLNode.hasChildNodes()) {
            Write-Verbose "Node has children to process"
            #Call this function recursively on each child
            foreach ($child in $HTMLNode.childNodes) {
                #try catch block to check for Type mismatch exception
                try {
                    #Call this function recursively on each child
                    Get-InnerText $child
                } catch {
                    Write-Verbose -Message "$($child.innerHTML) could not be splatted"
        } else {
            # If no children, get the inner text
            return (Remove-XmlMarkup $HTMLNode) 
    end {
        #Reset recursion depth
        Set-Variable -Name recursionDepth -Value $((Get-Variable -Name recursionDepth -ValueOnly) -1) -Scope Local
foreach ($chapterFile in $chapterFiles) {
    $chapter = $zipFile.GetEntry($chapterFile)
    $chapter.Open().Read(($chContentBytes = [byte[]]::new($chapter.Length)), 0, $($chapter.Length)) | Out-Null
    $chContentStr = [System.Text.Encoding]::Default.GetString($chContentBytes)
    $HTML = New-Object -Com "HTMLFile"
    $HTML.write([ref]$chContentStr) | Out-Null
    [System.Collections.Generic.Dictionary[string, int]] $wordCount = @{}
    [char[]] $delims = @(' ', "`r", "`n", '"', "'", ".", ',', "’")
    # Create an array of common English words to ignore
    $ignoreWords = @("i", "me", "my", "myself", "we", "our", "ours", "ourselves", "you", "your", "yours", "yourself", "yourselves", "he", "him", "his", "himself", "she", "her", "hers", "herself", "it", "its", "itself", "they", "them", "their", "theirs", "themselves", "what", "which", "who", "whom", "this", "that", "these", "those", "am", "is", "are", "was", "were", "be", "been", "being", "have", "has", "had", "having", "do", "does", "did", "doing", "a", "an", "the", "and", "but", "if", "or", "because", "as", "until", "while", "of", "at", "by", "for", "with", "about", "against", "between", "into", "through", "during", "before", "after", "above", "below", "to", "from", "up", "down", "in", "out", "on", "off", "over", "under", "again", "further", "then", "once", "here", "there", "when", "where", "why", "how", "all", "any", "both", "each", "few", "more", "most", "other", "some", "such", "no", "nor", "not", "only", "own", "same", "so", "than", "too", "very", "s", "t", "can", "will", "just", "don", "should", "now")
    Get-InnerText $HTML | ForEach-Object { 
        $words = $_.Split($delims , @([System.StringSplitOptions]::RemoveEmptyEntries, [System.StringSplitOptions]::TrimEntries )) | Where-Object -FilterScript { ($_.Length -gt 1) -and ($_.ToLowerInvariant() -notmatch "(?:$($ignoreWords -join '|'))")}
        $words.ForEach( {
                $count = 0
                if ($wordCount.TryGetValue($_, [ref]$count)) { 
                    $wordCount[$_] = ++$count 
                else { 
                    $wordCount.Add($_, 0)

The result is in $wordCount.


As English learner maybe you will be also interested in https://github.com/artemtomchuck/MyKnownWordsDictionary project. This extracts unique words from your book (straight txt should be an input, but it is not a problem to transform your pdf or epub into txt). Then it filters out the word you already know (based on your own dictionary). Then it prints new words from book ordered by frequency (the most frequent on the top) with their translation to your native language (given that book is written in English or any other language which you are learning).


  1. in output csv file frequency of word is not printed, but if you understand the source code then you can include this frequency column into your output (currently it is just hidden).
  2. this mechanism is very simplistic and does not recognize plural/singular forms, names or words with "-" symbol (e.g. single-minded is treated like two separate words: single and minded)

Full disclosure: this is my personal project and it may not fully suit your purposes or may be complex for using/setup by non-technical people (non-software engineers). But I tried to document setup as best as I can in github. The reason why I am posting the answer here: initially I looked some solutions from answers in this question. I found the solution in https://ebooks.stackexchange.com/a/6557 , but it was quite a routine for regular usage and that's why I automated process a bit.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.