I do suggest starting small - such as a table runner, place mat, or lap quilt. Something useful to you.
(I have trouble making pot holders - I think it's the extra thickness - so I would avoid them for a bit)
The reasons for starting small - in case you tried it and didn't like it (and strange as it may be, there may be a few that this has happened to) - you don't have a lot of money invested.
It's easier to finish something smaller and it's a lot easier to handle.
Is it "hard"? As others have said, patterns come in easy, medium, hard, difficult, and extremely challenging.
Straight lines and right angled pieces (squares and rectangles) are easier for most beginners than triangles and curves.
One can make a respectable top with only fabric, thread, needle, templates (pattern pieces), pencil and scissors.
There are some people that are "big shots" in the quilting world that hand piece. Jinny Beyer is one of them.