A new poll finds a small but fast-growing group of people in the U.S. have soured somewhat on President Trump this year.