Personally, I prefer to start with the pattern, that way I know exactly how much fabric I need (although I always buy extra just in case).
If I absolutely fall in love with a fabric I'll usually buy the whole bolt, but that's usually when I'm shopping for clothing fabrics!