"Add ever-branching alternate realities to your baseline perception of physics, and it resolves itself." You can only travel back along one line; you can only travel forward along one line; but when you do travel back, you can choose to go up a different line than you did before. Simple.