There is a performance rule that 80 percent of a program's execution runs in 20 percent of the code, therefore you should only have to optimize that 20 percent. In my opinion (he wrote, reaching for his flak vest), time invested in refactoring should work the same way. What is the chance the code you are refactoring will be modified by some future programmer? Does the 80-20 rule apply here, too?