What does Holden mean when he says that his brother D.B. is out in Hollywood being a prostitute?

It means that his brother is out there being a sellout.
According to Holden, the entire world, especially people who love appareances, looks,
and the superficial (namely, the Hollywood society) are phony, fake, and Holden hates
them. At first he was a huge fan of his brother for his writing ability. Holden held
that quite dear. When the brother got a chance to make movies Holden felt that he was
just being a sell out and a fake like the rest.