My son is 5 now. He started riding with me when he was 4. Naturally his mother freaks out. But we never go far or fast, and we never leave our neighborhood streets. The police told me that if he's old enough to reach the pegs, he can ride anywhere anytime. But that doesn't mean I'll take him on a country highway at midnight.
My son and I have a set of matching gear: white helmets, red/black gloves, red/black jackets, black jeans. Our boots don't match yet.
We both have HJC Symax flip-up helmets. I wanted the flip-up to make it easy to communicate quickly if we have a problem. Our next helmets won't be flip-up, since he's patient enough to wait to communicate. Fortunately for me, his head is too big for an extra-small adult helmet, so he wears a small adult helmet. Those were easy to find.
His jacket is an adult extra-small Joe Rocket Phoenix 2.0. I tailored the body and sleeves to be shorter and thinner. It's still too big for him, but it's better than it was.
His gloves are adult size extra-small.
I bought some kevlar material to sew into our jeans (knees, hips, and seat), but I haven't done the work yet.
To answer your question, I think you're dead on. If the kid can hold on securely, and he has the desire and the ability to follow instructions, then he can ride. With age and experience (you need both), the kid can ride farther and faster.
Curt