The book starts very well with a great first chapter on the ancient history of numbers - the Egyptian and Babylonian systems, and the effect numbers have on spoken languages. It continues with another two good chapters: one on geometric interpretation of numbers, and the other on number sequences, where a particularly great section on guessing the next number in a sequence is a true pleasure to read.
Unfortunately, it goes downhill from there. Starting with chapter 4 the book becomes more and more of an overly-packed whirlwind of algebra and number theory. A new topic is presented on almost every page, with very little introduction and explanation, and is immediately dumped for another topic. Indeed, the lack of depth is perhaps the most disappointing. The authors spend very little time explaining, results are almost never proved and the topics change too quickly.
This is not to say that there aren't gems hidden here and there, but overall I was quite disappointed. After the first 3 chapters the book wasn't fun to read.
It's interesting to note that lots of the material in the book will be familiar to fans of Project Euler. The correlation is so high that I'm quite sure that the problem authors of PE used "The book of numbers" as an inspiration for at least a few problems posed on the website.
Eli Bendersky's website