What are the townspeople’s reactions to Chillingworth’s lodging in the same house as Dimmesdale in The Scarlet Letter?

According to the story, the townsmen accepted and
respected Chillingworth because he was educated, and because he showed an accentuated
interest in the man whom the town loved and admired the way they would venerate God
himself.  Yet once Chillingworth' true intentions began to take place (which was to find
Dimmesdale out as his wife's lover), his action spoke louder than
words.

At first, his
expression had been calm, meditative, scholar-like. Now, there was something ugly and
evil in his face, which they had not previously noticed . . . it grew to be a wisely
diffused opinion, that the Reverend . . . was haunted either by Satan himself, or
Satan's emissary, in the guise of old Roger Chillingworth.
(1224)

Therefore,
the townspeople felt that Chillingworth was haunting Dimmesdale, but what they do not
know is that the reason why Dimmesdale is allowing himself to be haunted is because he
is guilty, and because Chillingworths other mission was basically to drive Dimmesdale to
insanity so he would confess who he really was.