Do have any rights as an immigrant mom?

No, you cant take the kids out of the country without the father's signed consent. You must have something up your sleeve. If it is so important that you go, then go! You didnt say anything about him stopping YOU from going.