AITA for making my son cry in front of his kid?

My son(M28) is mentally stunted (I'm not sure what the correct word in english is, basically he is an adult but mentally and emotionally he is no older than a teenager).

When he was 18 he had unprotected sex which resulted in a girl. The woman she slept with didn't want to be stuck with him for the rest of her life so she just left the kid with us.

My son and his child still live with us because well he can't look after a kid or hold a full time job. We take care of our granddaughter.

A few days ago, we had to be somewhere and I asked my son if he can look after his child for a few hours. She is old enough that she doesn't need a babysitter so all he had to do was stay with her and give her lunch. He said he can do this.

We went out and a few hours later he called us panicking and said he took his daughter to the hospital. My granddaughter is severely allergic to cow milk. HE KNOWS THIS. Yet he cooked a food with milk. Thank god he brought her to the hospital quickly and she was fine. I didn't say anything while we were in the hospital but as soon as we got home I started yelling at him. I told him that he is a useless piece of sh*t who can't even look after his own child for an hour. I said a bunch of other things that I can't remember right now.

He was crying the entire time and my granddaughter was crying as well and yelling at me to stop being mean to him.

Well he won't come out of his room now. My granddaughter is staying with him and only comes out to bring him food and when I tried to talk to her she started yelling at me telling me I'm an asshole and she hates me.

I was just worried about her.