Am I the AH for getting upset because my husband wants a hall pass

To start off I’m 8 weeks pregnant and the doctor advised we not have intercourse until after 13 weeks. Prior to this I had two miscarriages last year and we finally have our rainbow baby. As excited as we both are my husband decided to ask me for a hall pass to have intercourse with another woman since we cannot. He believes it is just sex nothing more (me: rolls eyes). He said he loves me and only me and that he wants to spend his life with me and our children but he “has needs”. I of course got extremely upset and refused. I told him we could get a divorce and he could be with as many women as he wants. In case anyone wants to know, no he’s not cheating. We are together literally all the time. So please tell me AITA? Am I in the right here?