If it distracts from your plot and from your theme, then yes - sooner or later, that dialogue should be rewritten or cut entirely.
Real conversations get sidetracked; they go off on tangents. If you do a lot of your writing by figuring out "what would they say next," you're likely to follow that natural tendency. The problem is, if it's really nothing but a random offshoot of conversation, then it probably won't interest the reader. It may confuse him, or he may find it boring.
Not every digression from the initial subject is a distraction. Some digressions have thematic significance; some digressions serve important roles like exposition, developing relationships, etc.; some digressions switch from one aspect of the plot to another one. But if you don't know why that bit of dialogue is there, if you don't feel that it adds to your story, than you do the same as with any bit of writing that isn't contributing: you change it, or you cut it out.
You will probably be able to tell the difference simply by assessing your own intentions while writing. If you feel that you're grasping for something, anything, for the characters to say, or that you're writing the dialogue based on loose association rather than sticking to some central point, then the dialogue is probably unnecessary. On the other hand, if you're trying to accomplish something (anything!) specific by changing topics, then you're probably fine - you can judge whether the topic change accomplishes your goal or not.