Why can't I make it work - in other words, how many first drafts do I need?

I've been trying to write Marked for over a year now. It's gone through many incarnations. It was Only A Name for a while. Then I put out a call on Twitter for a new title. I was given several suggestions. Marked is the one that stuck. I completely revamped it from the earlier version of the story and thought that was it. I'd finish the first draft and then move on to another novel, saving the first draft for future revisions.

I'm on version number three now, and I don't mean through edits. I'm on the third complete rewrite of Marked. In the two previous versions there was so much I didn't like that editing wouldn't have done me any good. I have both versions saved so I can refer back to them for the things I liked, and I've taken notes on that to help with the third version.

The problem is I'm not sure if the third version is going to work either. I've already taken a large chunk of it and put it into a side file for future reference because I didn't like how the story was going at the point it was at. I like it so far but what happens if I find I don't like how it's going again? I'll probably cut the offending text and paste it into another document to save for dissection.

Why do I do this? Well, the main problem is I have a hard time settling on one story. I'm also forever tweaking the world so even though I've done extensive world building things are perpetually in flux. I try to settle on one thing and a new idea springs up and I have to examine it. If I like it, it stays. If I don't like it, it doesn't get included and I move on. I've finally settled on the look of the world and the way politics are set up in one of the countries. But there are still two more countries and a lot of races to deal with that are only vaguely formed.

Now, you might ask how that works out for me. I'm a pantser not a plotter. I sketch out my characters and get an idea of their world and then let them go from there. I let them tell their story as I write it. For a few of the shorts I've written, though they do need to be edited, it worked out great. With Marked, it's not. But if I try to write from an outline the story falters and dies and I can't get anything done.

So right now I'm just allowing the story to flow forward as it wants to, stopping every now and then to check details and make sure I'm not changing things halfway through. I'm keeping lists of things so I don't forget them and making notes as I go. Those will help with my synopsis I'm sure when I'm ready to write it. But first I have to settle on one version of the story.

So tell me, how do you do it? What do you do that makes you love or hate the first draft of your story, and what do you do about it? How many versions of your first draft do you go through before you settle on one and go from there? Am I the only one with this problem?