Critics warned that the optics from the meeting would be damaging for the US president