5

As someone who is interested in learning languages, I want to find a way to know the each word and their repetitions inside a book that I am going to read.

  • Welcome to the site! Can you clarify a little bit? Are you looking for software that you could run to generate a list of all the words in a book and how many times they are used? Also, are you looking to do this before you download or buy a book, or after? – Ed Cottrell Mar 28 '15 at 17:49
  • Hi Ed, As you mentioned, I want it to list the words and show how many times they are used. It is like the books which are printed for English Language learners and indicate the words that the reader will run across in the book. For now, I am interested in doing this for the books that are licensed as Creative Commons like in Gutenberg Project. – Karel Capek Mar 28 '15 at 20:10
  • Interesting question -- I don't have an answer, but the clarification may help other users. – Ed Cottrell Mar 29 '15 at 5:08
  • @Anthon: Please send me an EPUB that you would like to count the words in, I'll give it a shot. A shorter EPUB is better for testing purposes. I can't PM you so you'll have to PM me to ask for my email address. Thanks. – Bulrush Apr 21 '16 at 13:12
  • @Bulrush I think you are confusing me with the OP. – Anthon Apr 21 '16 at 13:52
3

Here is one way to find the number of words in an epub file sorted by their frequency, with the words used the most at the top of the list.

This is done on a Mac laptop and will also work on Unix hosts.

The overview of the process:

  1. Install Calibre
  2. Use the ./ebook-convert command in Calibre to convert the epub file to text
  3. Transform the entire text file to lowercase (so "Word" and "word" match)
  4. Convert punctuation to whitespace (so "period." and "period " match)
  5. Convert all whitespace to a new line. This puts each word on its own line.
  6. Exclude any blank lines from the list
  7. Sort the list of words alphabetically
  8. Pipe (send) that list of words through uniq -c You now have a count of how often each word appears.
  9. Sort the result in numerical order. If you use the sort command with the -r argument, the most frequent words are at the top.

Here's an example of steps (2) through (9). The head command lists the top ten words in the final output.

$ ./ebook-convert ./book.epub ./book.txt
$ cat   ./book.txt | tr '[:upper:]' '[:lower:]' | tr "“" " " | tr "”" " " | tr "," " " | tr "." " " | tr " " "\n" | grep -v ^$ |  sort | uniq -c | sort -gr  | head
 5303 the
 1960 and
 1934 of
 1910 to
 1874 a
 1168 i
 1067 you
  844 in
  812 that
  703 it
 $

The result is pretty boring. The word 'the" appears 5303 times, while the word 'it' appears 703 times.

I suspect in most books the most common words are the tiny conjunctions, articles, prepositions and pronouns. Perhaps on something that is not a novel this might be more interesting.

Good luck!

  • Thank you for the detailed answer. As a windows user, I would like to know if all the steps you described are done in Calibre. – Karel Capek Mar 8 '17 at 20:37
  • Step 1 is done at the OS level. Step 2 is done in Calibre from the command line. The remaining steps are done at the Mac OS command line using the program named Terminal. I know the Unix tools much better than I know the MS Windows tools. I wish you good luck! – StandardEyre Mar 8 '17 at 23:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.