Agile development says when something is done, it stays done. Don't touch it again! On the other hand refactoring is an extremely important part of agility. How do you combine these, possibly conflicting requirements?
Refactoring is a planned action. When you develop something, it is done as well as it can be at that specific moment. It is scheduled, planned, designed, implemented, tested, documented and released as well as needs to be at the specific moment. When it's done, it's really done.
Developing a new feature, additional requirements or fixing bugs might bring in new information, which makes it possible to improve something already done. Make it more generic, robust or faster. Search for reusing something old.
Refactoring is a fact of life. Experiment, seek alternatives, gain more experience, receive more detailed information, learn from own mistakes or from experts who have made more mistakes than you. Use this for your advantage. Refactoring is not experimenting, but cleaning up afterwards to the most suitable solution.
Don't refactor for the sake of refactoring. If something works, don't touch it without really good reasons. Developing something twice is quite ok, requires less time than refactoring. More safe. Refactoring has to be an improvement.
When you find need for a third similar object/module, then take a moment to consider advantages and dangers of refactoring. Will the be more similar cases in future, how critical areas are you dealing with, how stable have they been bug and feature wise. Would refactoring be worth the risk and investment of time and effort?
Don't refactor for fun. Only when needed, understanding risks.