Refactoring is very healthy for the code and can also be fun when it goes well and you feel the code becomes more modular.

Some say it's a daunting task, and it can be, if the code is large. There are special tools that help with refactoring, but for me Find In Files and grep work pretty well anyway. True, sometimes you have to go through tons of code, but it's very educational and makes the code quality much better. Not once while manually going through the files changing a direct member access to an encapsulated function call I noticed other things to refactor.

What refactoring also gives is the freedom to write prototypes and experiment freely. Once the prototype is done, it can be refactored into a final, clean form.

One big problem with refactoring is that it's unsafe to do without a good test suite. And in some cases, like heavily GUI-oriented apps, the test bases are non-existant or negligible, which makes the process dangerous. But it's still worth it if you know what you're doing, IMHO.