Evaluation of On-the-Job Behavior
Change Results
of Interviewing With Style by Tim Gard
Presented May 20, 1999
Income Maintenance workers who attended the ‘Interviewing with Style’ training were asked to use a Training Impact Evaluation for the course. To enable data gathering for this
Evaluation form, workers were given the assignment of applying any techniques presented in the course in three separate interviews.
The goal of this evaluation is to assess how likely it is that the Interviewing with Style training results in behavior change.
On the whole, worker responses indicate that the training had a very positive impact --some behaviors were changed, new behaviors were learned and applied. The Council Bluffs Region is
fortunate to have many years of experience represented in its IM staff. Some of these experienced workers state in the evaluations that they already knew the techniques presented in the
course or noted on the form that applying the technique was not a change in their behavior. For others, the techniques were very new and comments on the forms were enthusiastic. Because
anecdotal evidence helps make reports more real, some of these comments are reproduced at the end of this report.
Learned behaviors most often tried and reported were
¨
Greeting the client with a handshake and introduction—workers observed that
greeting the client and engaging in some small talk helped put the client at ease and helped establish rapport.
¨
the power of silence—some workers reported realizing that they had previously
been “filling in the blanks” themselves. This technique helps clients realize the worker is going to wait for an answer that they are responsible for.
¨
open ended questions—this technique was reported as leading to more accurate
information
¨
Explaining what is needed and why—workers noted positive differences in client
cooperation and in interview progress when the reasons for the interview and its questions, and how these related to client eligibility were explained first.
¨
gaining understanding of the client’s intentions, beliefs, motivations—workers
wrote that this helped them understand the client’s responses and actions, and better work with the client
¨
having eye contact, showing an interest—workers reported that the client was
more willing and the interviews went better when these were used
¨
investigative techniques –when appropriate, these techniques were found to
work, improving accuracy of information (though not always client satisfaction) and
¨
closing techniques—various ways to cue the client that the interview is ending
were tried and valued
45 workers returned Evaluation forms for three interviews, 4 workers returned forms for only two interviews and 5 workers returned forms for only one interview.
This means that results were reported for 148 interviews. There were multiple response areas on the form and workers often listed more than one response for an item on the form.
The Training Impact Evaluation form lists these under ‘Topic Trained’ down the left hand column: Contact and control (instant rapport, getting to yes, power of silence); Framework
(boundaries, frame of reference, truth and consequences); Environment (credibility); Understanding of client (intentions, beliefs, motivation); Detective work (questions, investigation);
Counseling, coach; and Closing the Interview. The other columns are ‘How used in Interview?’; ‘Change from previous interview behavior?’; and ‘Impact: positive or negative?’
The form turns out to have been too complex for use ‘on the job’, well after the training session. Workers did very well at reporting techniques applied, fairly well at naming the
techniques but not as well at reporting those techniques consistently in the seven categories. If workers have learned and applied interviewing techniques presented in a short training
session, whether or not the techniques are understood or reported in a certain category is not important to evaluating results. The responses then, are reported and subtitled here by
technique applied but the report does not divide techniques by the seven categories.
|
Number with this response |
Subtitle/
How used in interview
|
Was this a change for worker? |
Was the impact positive or negative? |
|
|
Introduce self, offer handshake
|
|
|
|
16 |
Offer handshake, introduce self, make small talk on way to office, etc |
Yes |
Positive |
|
4 |
Offer handshake, introduce self, make small talk on way to office, etc |
No |
Positive |
|
|
Explain what’s needed and why
|
|
|
|
25 |
Explained what and why |
Yes |
Positive |
|
4 |
Explained what and why |
No or blank |
Negative / NA/blank |
|
11 |
Explained what and why |
No |
Positive |
|
5 |
Explained what and why
|
Blank |
Positive |
|
1 |
Discussed needs |
Unknown |
Positive |
|
|
Gain an understanding of client
|
|
|
|
6 |
Gain understanding of client intentions, beliefs, motivation |
No |
Blank or NA |
|
1 |
Gain understanding of client intentions, beliefs, motivations |
Yes |
Negative |
|
44 |
Gain understanding of client
Intentions, beliefs, motivations |
Yes |
Positive |
|
7 |
Gain understanding of client intentions, beliefs, motivations
|
No |
Positive |
|
9 |
Gain understanding of client intentions beliefs, motivations |
Blank |
Blank |
|
3 |
Gain understanding of client intentions beliefs, motivations |
No |
Negative |
|
1 |
Gave referral info pertinent to client situation
|
Blank |
Blank |
|
3 |
Gave referral info pertinent to client situation |
Yes |
Positive |
|
2 |
Gain understanding of client intentions beliefs, motivations |
Blank |
Positive |
|
|
Starting w easy or yes answer questions |
|
|
|
5 |
“Columbo”-start with easy questions |
Yes |
Positive |
|
1 |
Start with the yes answer questions |
Yes |
Positive |
|
1 |
Start with the yes answer questions |
Blank |
Positive |
|
|
Eye contact
|
|
|
|
24 |
Eye contact, show interest in client and what they say |
Yes |
Positive |
|
1 |
Eye contact, show interest in client and what they say |
Yes |
Blank |
|
5 |
Eye contact, show interest in client and what they say |
No |
Positive |
|
2 |
Eye contact, show interest in client and what they say |
No |
Blank |
|
Speech (of worker) |
|
|
|
4 |
Slow and or soft speech, calm tone, smile as appropriate |
Yes |
Positive |
|
1 |
Slowed when going over what is for worker repetitious information, checked for client understanding
|
Yes |
Positive |
|
1 |
Slow and or soft speech, calm tone, smile as appropriate |
No |
Positive |
|
|
Listening
|
|
|
|
1 |
Mental movie of client situation |
Yes |
Positive |
|
1 |
Repeated what client said |
Blank |
Blank |
|
2 |
Leaning forward to listen |
Yes |
Positive |
|
1 |
Asked question and then sat quietly to let client answer |
Yes |
Positive |
|
6 |
Listening intently, not finishing sentences or filling in the blanks for the client |
Yes |
Positive |
|
|
‘Instant rapport’
|
|
|
|
5
|
‘Instant rapport’ |
Yes |
Positive |
|
5 |
‘Instant rapport’
|
No |
Positive |
|
1 |
‘Instant rapport’ |
No |
Negative |
|
|
Power of silence
|
|
|
|
11 |
Power of silence |
Yes
|
Positive |
|
4 |
Power of silence |
No |
Positive |
|
|
‘Getting to yes’
|
|
|
|
1 |
‘Getting to yes’ |
Yes |
Positive |
|
1
|
‘Getting to yes’
|
No |
Positive |
|
Focus
|
|
|
|
3 |
Letting client ramble, repeating what is said, confronting discrepancies |
Yes |
Positive |
|
1 |
Asked Has anything changed since this application was filled out? |
Positive |
|
|
1 |
Focus on the business at hand
|
No |
Positive |
|
|
Establish control
|
|
|
|
6 |
Established control with boundaries |
Yes |
Positive |
|
1 |
Established control with boundaries |
Yes |
Negative, couldn’t
|
|
2 |
Didn’t get off the subject |
Unknown |
Positive |
|
1 |
More control |
Unknown |
Positive |
|
3 |
Pacing, leading |
Yes |
Positive |
|
2
|
Established control with boundaries
|
No |
Positive |
|
|
Open ended questions
|
|
|
|
19 |
Open ended questions |
Yes
|
Positive |
|
2 |
Open ended questions
|
Yes |
Blank |
|
1 |
Open ended questions |
Yes |
Negative |
|
|
Investigative
|
|
|
|
3 |
Cover, move on, go back and review what was covered earlier |
Yes |
Positive |
|
1 |
Tried to make the story complete w more questions |
Yes |
Negative-
Couldn’t
|
|
2 |
Tried to make the story complete w more questions |
Yes |
Both positive and negative
|
|
4 |
Tactfully asked in various ways til got answer |
Yes
|
Positive |
|
3 |
Asked what has happened since last we visited |
Yes |
Positive |
|
5 |
Continued to ask for answers to unanswered questions |
Yes |
Positive |
|
2 |
Continued to ask for answers to unanswered questions |
No |
Positive |
|
1 |
Continued to ask for answers to unanswered questions |
Blank |
Blank |
|
12 |
More direct in my questions |
Yes |
Positive |
|
1 |
Asked what changes do you foresee? |
Yes |
Positive |
|
1 |
Asked follow-up question to see if another change had happened |
Yes |
Positive |
|
1 |
Called client’s bluff/verified info
|
Yes |
Positive
re accuracy |
|
|
Frame of reference
|
|
|
|
2
|
Frame of reference |
Yes |
Positive |
|
1
|
Frame of reference |
No |
Positive |
|
|
Truth and Consequences
|
|
|
|
1 |
Truth and consequences |
Yes |
Unknown |
|
5 |
Truth and consequences |
No |
Positive |
|
3 |
Truth and consequences |
No |
Unknown |
|
5 |
Truth and consequences |
Yes |
Positive |
|
4 |
Truth and consequences |
Blank |
Unknown |
|
|
Attention to client non verbal
Communication |
|
|
|
1 |
Attention to client’s body language |
No |
Positive |
|
1 |
Attention to client’s body language |
Yes |
Positive |
|
|
Environment
|
|
|
|
1 |
Tried to create environment of trust and respect |
Yes |
Positive |
|
2 |
Tried to create trustworthy, credible
Environment |
Blank |
Blank |
|
2 |
Tried to create more professional environment |
Blank |
Blank |
|
3 |
Set a comfortable, relaxed environment |
No |
Positive |
|
5 |
Set a comfortable, relaxed environment |
Yes |
Positive |
|
3 |
Cleared desk of other work |
Yes |
Positive |
|
1 |
Friendly |
Blank |
Blank |
|
1 |
Hung a poster in office about reporting
|
Yes |
Positive |
|
Closing
|
|
|
|
11 |
Asked “anything else you need/want to tell me?” |
Yes |
Positive |
|
2 |
Asked “anything else you need/want to tell me?” |
Yes |
Unknown |
|
1 |
Summarized |
Yes |
Blank |
|
3 |
Gave all written info, stressed reporting
|
Yes |
Positive |
|
1 |
Used redirecting and focusing |
Yes |
Positive |
|
1 |
Emphasized importance of PJ |
Yes |
Unknown |
|
6 |
Stood to signal end of interview |
Yes |
Positive |
|
2 |
Stood to signal end of interview |
No |
Positive |
|
2 |
Stood, talked about next steps
|
Yes |
Positive |
|
1 |
Stood, talked about next steps |
No |
Blank |
|
1 |
Reviewed, talked about next steps
|
Yes |
Negative |
|
2 |
Reviewed, talked about next steps |
Yes |
Positive |
|
6 |
Stressed reporting, contacting us |
Yes |
Positive |
|
1 |
Stressed need for communication
|
Blank |
Positive |
|
1 |
Reviewed options, encouraged to stay in touch with us |
Yes |
Positive |
|
3 |
Thanked for coming |
Yes |
Positive |
|
. |
Miscellaneous
|
|
|
|
2 |
Maintain positive attitude |
Blank |
Positive |
|
2
|
More prep time for interview
|
Yes |
Positive |
|
7 |
Encouraged client re positive actions
|
Yes |
Positive |
|
1 |
Gave advice on our services |
Yes |
Positive |
|
2 |
Sympathized with client’s situation |
Yes |
Positive |
|
1 |
Gave client an ‘out’/ “take app home and discuss this” |
Blank |
Positive |
|
3 |
Workers expressed concerns if DHS management will back workers up esp re investigative interviewing techniques |
-- |
-- |
|
3 |
Workers expressed concerns re time allotted for interviews |
-- |
-- |
|
1 |
Referred client to another agency
|
No |
Positive |
|
|
Response either lacked usable information or stated no changes made
|
|
|
|
10 |
No changes made or impact found to date |
Blank |
Blank |
|
19 |
Response that could not decipher OR Response reported on client behavior without explaining techniques applied
|
Na |
Na |
|
6 |
Form response area was all blank or noted ‘none’ |
Blank or ‘none’ |
Blank or ‘none’ |
|
Sub Totals |
All |
Was this a change? |
the impact? |
|
275 |
|
Yes |
Positive |
|
56 |
|
No |
Positive |
|
5 |
|
Yes |
Negative |
|
2 |
|
No |
Negative |
|
6 |
|
Yes |
Blank |
|
12 |
|
No |
Blank |
|
52 |
|
Blank or unknown |
|
ODS summary
Of 408 responses, 275 indicate that a technique that represents a change in worker interviewing behavior, was applied with a positive impact. This is a very high proportion.
56 responses indicate that the technique applied with a positive impact was not a change for the worker. It is unknown whether workers interpreted this as a change from usual practice
OR or a change from things the worker had learned earlier. Careful reading of each form suggests that it probably is a combination. Either way, positive impact is reported for the
technique.
While labeling of an impact as positive or negative is subjective, independent measurement of interviewing impacts would likely also be subjective.
The instances of negative impacts reported is very low for an evaluation based on three interviews per worker where multiple, often new-to- the- worker, techniques were applied.
Evaluation strategies did not ensure that the various techniques from the training were equally applied. The choice of techniques to apply was the worker’s and since real clients and
real work was involved, that seems appropriate. It may also be that the composite view of workers’ choices reflected here is also a view of the most generally useful techniques.
Organization Development and Support welcomed this opportunity to evaluate impact. In the field of Evaluation of training, this is rarely accomplished. ODS is new at the task but
whatever expectations we might made have had about impact, would not have been for it to have been this significant. We believe this report can be very useful to IM Training Redesign,
especially in design of Advanced courses. This report is also being shared with Tim Gard per his request.
Finally, we must acknowledge that this report is very, very overdue. We thank Council Bluffs Region for demonstrating considerable patience and we humbly apologize for the delays.
Some worker comments on positive results
“I have a problem of not guessing what the client is going to say and I have helped them finish a sentence.”
[The] “client threatened to walk out and go home to call Des Moines. Proceeded to explain to him that he would be required to attend another interview
……..so to save his time, we could continue.”
“I like these changes but have to reprogram myself to change the routine.”
“I have really changed the way I interview since this training.”
[The client] “revealed resources not on the application”
[I] “was able to grasp an understanding of what client needed . Really.”
“I feel this class should be given to all new employees before they start interviewing. Some times it’s hard to change our ways after 20 years. “
“I have changed the way I sit-instead of completely talking and writing, I now look directly at them and listen. [Have] more eye contact. Let them talk and not talk for them.”
[before] “usually rambled fast when explaining the program requirements as it was so repetitious to me”
[while waiting, client was mad/upset and ] “had upset the clerical a lot with their comments. I asked if there was a problem.” {There was an unrelated problem and] “I listened to her
briefly, then explained I would probably be asking her some more questions that might upset her. Gave her option of rescheduling or going on……She even apologized to clerical when
leaving.”
“Felt the [training] day was very well spent”
[Before]” I only assumed I knew [the client’s] intentions were kept within program of application.” [Now] “I can/do understand their needs better by my opening to the interview.”
|
