The Federal Trade Communication got a skeptical reception from a judge over its targeting of a nonprofit in an investigation into alleged advertising boycotts of conservative websites.