It is common to compare the banjo to the guitar which is popular and which multitudes seem to learn how to play. The banjo has 5 strings instead of 6, making it that much simpler -- and the fifth string is special and is rarely if ever involved in chords. I almost all cases, a beginner will be playing the banjo tuned to G, and will be able to play tunes without forming chords or doing much with the left hand.
If you are content to strum the banjo, things can indeed be fairly simple. When you introduce picking and the use of two fingers and the thumb on the right hand, the game changes, but plenty of people master it.
The bottom line is that it is going to require work and lots of practice time to learn to play any instrument. I think the reputation of the banjo as being especially hard, has served to discourage students in the early stages of learning.
Also worth considering is that there are other sorts of banjos besides the 5 string, and other styles other than bluegrass (such as clawhammer). Learning to play clawhammer could well be less demanding than bluegrass, at least at the early stages. Learning to play any style really well will be demanding.
Also it is one thing to learn to play an instrument, and another to learn to play it well. This is true of any instrument. It has been suggested that 2000 hours of practice on the banjo would take you to a level where you could play well enough to participate in a bluegrass band. Give an hour a day of practice, this is 5 or 6 years. This may well be an accurate estimate, but even so the claim is made that given 6 months of diligent practice you should be playing and enjoying yourself.
Finally, there are these comments by Stuart Rosemberg
Technically a banjo can best be understood as a cross between a musical instrument and a hardware store.The banjo is an incredibly loud instrument, resulting in ordinances in most municipalities restricting performance to houses of ill reput and government offices.
The combination of physical complexity, unbearable loudness, and confusing string layout make the instrument virtually impossible to learn to play.
To the consternation and disappointment of multitudes of innocent bystanders, every year thousands of people selfishly acquire banjos and make a futile attempt to learn how to play them.