The exact definition for "casual gamers" depends on the person being asked. It is difficult to define a casual gamer exactly, but some things can be observed relatively often.
There are people who only play on the train, in the waiting room or on the toilet. Mostly they play mobile games with a low level of difficulty, simple rules and simple controls. These people are casual gamers, for example.
On the other hand, there are people who play a lot, but change games frequently, don't feel tied to a game, and basically don't want to be good at anything. Such people would also be called casual gamers.
Just like people who only play solitaire, occasionally play a game of Fifa with their friends or catch Pokemon while riding shotgun.
What they all have in common is that they play "when it suits" and regard gaming as a casual pastime rather than a core hobby. Casual gamers don't usually follow the latest developments in the e-sports scene, stream on Twitch, or play 10 hours of MMORPGs a day. So basically, a majority of gamers can probably be described as casual gamers.