Response Rate Calculation Formula


OVERVIEW

MRIA ADOPTS RESPONSE RATE CALCULATION FORMULA

The Marketing Research and Intelligence Association has adopted the Data Collection Response Rate Calculation recommended by Statistics Canada in its Standards and Guidelines for Reporting of Non-Response Rates as the industry standard for market, survey and public opinion research in Canada.

This response rate calculation formula which MRIA refers to as the Empirical Method, is defined below.

You can also view several examples by clicking on the following links:

ILLUSTRATIVE EXAMPLES OF EMPIRICAL METHOD

MRIA has also adopted a secondary method of response rate calculation, the Estimation Method, is defined below.

You can also view several examples by clicking on the following links:

ILLUSTRATIVE EXAMPLES OF ESTIMATION METHOD


Empirical Method of Response Rate Calculation

MRIA has determined that the following Empirical Method of Response Rate Calculation is not only the most appropriate method, but also the most clearly defined method for the monitoring of survey operations and for the evaluation of data collection methodology and frame coverage.

Empirical Calculation for Data Collection

 

Total Numbers Attempted  
Invalid
NIS, fax/modem, business/non-res.
 

Unresolved (U)

Busy, no answer, answering machine

 

In-scope - non-responding (IS)

Language problem
Illness, incapable
Selected respondent not available

Household refusal
Respondent refusal
Qualified respondent break-off

 

In-scope - Responding units (R)

Language disqualify
No one 18+
Other disqualify
Completed interviews

 
Response Rate = R/(U+IS+R)  

 


Estimation Method of Response Rate Calculation

MRIA has also adopted a modified version of the Data Collection Response Rate Calculation as a secondary method, provided that it is reported in addition to the data collection response rate. This secondary method, which MRIA refers to as the Estimation Method, closely parallels the method used by the Association de l'industrie de la recherche marketing et sociale [AIRMS Québec]. Statistics Canada accepts this alternative response rate calculation when response rate is being used to assess the "fitness for use" of survey data or what StatsCan refers to as the "estimation phase".

The primary difference between the Estimation Method below and the Empirical Method used at the data collection stage is that "there should be an attempt to estimate the proportion of the unresolved units that are in-scope and hence should be treated like non-respondents as opposed to the proportion of units that are out-of-scope".

Estimation Calculation

 

Total Numbers Attempted  
Invalid
NIS, fax/modem, business/non-res.
 

Unknown (U)*

Busy, no answer, answering machine
Language problem
Illness, Incapable
Selected respondent not available
Household refusal
Respondent refusal

 

Ineligible (I)

Language disqualify
No one 18+
Other disqualify

 

Eligible (E)

Language problem
Illness, incapable
Selected respondent not available
Household refusal
Respondent refusal
Qualified respondent break-off

 
Completed interviews (C)  
Eligibility Rate: ER = (C+E)/(C+E+I)  
Response Rate = C/[(ER*U) + C + E]  

 


Scenario A

Every household qualifies (eg: "main grocery shopper" or "M or F head of household")

EMPIRICAL METHOD

A

 

Total Numbers Attempted

4000

 
Invalid
NIS, fax/modem, business/non-res.

1000
1000

 

Unresolved (U)

Busy, no answer, answering machine

900

900

 

In-scope - non-responding (IS)

Language problem
Illness, incapable
Selected respondent not available

Household refusal
Respondent refusal
Qualified respondent break-off

1050

100
50
100

500
250
50

 

In-scope - Responding units (R)

Language disqualify
No one 18+
Other disqualify
Completed interviews

1050

--
--
--
1050

 
Response Rate = R/(U+IS+R)

35%

 

 


Scenario A

Every household qualifies (eg: "main grocery shopper" or "M or F head of household")

ESTIMATION METHOD

A

 

Total Numbers Attempted

4000

 
Invalid
NIS, fax/modem, business/non-res.

1000
1000

 

Unknown (U)*

Busy, no answer, answering machine
Language problem
Illness, Incapable
Selected respondent not available
Household refusal
Respondent refusal

900

900

 

Ineligible (I)

Language disqualify
No one 18+
Other disqualify

--

 

Eligible (E)

Language problem
Illness, incapable
Selected respondent not available

Household refusal
Respondent refusal
Qualified respondent break-off

1050

100
50
100

500
250
50

 
Completed interviews (C)

1050

 
Eligibility Rate: ER = (C+E)/(C+E+I)

1.0

 
Response Rate = C/[(ER*U) + C + E]

35%

 

 

** "Unknown" = "Numbers in sample for which eligibility could not be established"

 

 


Scenario B

All individuals 18+ qualify
Illness, selected respondent NA and respondent refusal are still deemed to be eligible units since person initially answering phone will have confirmed selected respondent is 18+. Household refusal and language are now unknown as we don't know whether there is anyone 18+ in household

EMPIRICAL METHOD

B

 

Total Numbers Attempted

4000

 
Invalid
NIS, fax/modem, business/non-res.

1000
1000

 

1 Unresolved (U)

Busy, no answer, answering machine

900

900

 

2 In-scope - non-responding (IS)

Language problem
Illness, incapable
Selected respondent not available

Household refusal
Respondent refusal
Qualified respondent break-off

1050

100
50
100

500
250
50

 

3 In-scope - Responding units (R)

Language disqualify
No one 18+
Other disqualify
Completed interviews

1050

--
50
--
1000

 
Response Rate = R/(U+IS+R)

35.0%

 

 


Scenario B

All individuals 18+ qualify
Illness, selected respondent NA and respondent refusal are still deemed to be eligible units since person initially answering phone will have confirmed selected respondent is 18+. Household refusal and language are now unknown as we don't know whether there is anyone 18+ in household

ESTIMATION METHOD

B

 

Total Numbers Attempted

4000

 
Invalid
NIS, fax/modem, business/non-res.

1000
1000

 

Unknown (U)**

Busy, no answer, answering machine
Language problem
Illness, Incapable
Selected respondent not available
Household refusal
Respondent refusal

1500

900
100
--
--
500
--

 

Ineligible (I)

Language disqualify
No one 18+
Other disqualify

50

--
50
--

 

Eligible (E)

Language problem
Illness, incapable
Selected respondent not available

Household refusal
Respondent refusal
Qualified respondent break-off

450

--
50
100

--
250
50

 
Completed interviews (C)

1000

 
Eligibility Rate: ER = (C+E)/(C+E+I)

0.967

 
Response Rate = C/[(ER*U) + C + E]

34.5%

 

 

** "Unknown" = "Numbers in sample for which eligibility could not be established"

 

 


Scenario C

40% of individuals 18+ qualify.
Must speak directly to selected respondent to determine whether he/she qualifies.
Qualified break-off set at 5% of completes as in scenarios A & B

EMPIRICAL METHOD

C

 

Total Numbers Attempted

3970

 
Invalid
NIS, fax/modem, business/non-res.

1000
1000

 

Unresolved (U)

Busy, no answer, answering machine

900

900

 

In-scope - non-responding (IS)

Language problem
Illness, incapable
Selected respondent not available

Household refusal
Respondent refusal
Qualified respondent break-off

1020

100
50
100

500
250
20

 

In-scope - Responding units (R)

Language disqualify
No one 18+
Other disqualify
Completed interviews

1050

--
50
600
400

 
Response Rate = R/(U+IS+R)

35.4%

 

 


Scenario C

40% of individuals 18+ qualify.
Must speak directly to selected respondent to determine whether he/she qualifies.
Qualified break-off set at 5% of completes as in scenarios A & B

ESTIMATION METHOD

C

 

Total Numbers Attempted

3970

 
Invalid
NIS, fax/modem, business/non-res.

1000
1000

 

Unknown (U)**

Busy, no answer, answering machine
Language problem
Illness, Incapable
Selected respondent not available
Household refusal
Respondent refusal

1900

900
100
50
100
500
250

 

Ineligible (I)

Language disqualify
No one 18+
Other disqualify

650

--
50
600

 

Eligible (E)

Language problem
Illness, incapable
Selected respondent not available

Household refusal
Respondent refusal
Qualified respondent break-off

20

--
--
--

--
--
20

 
Completed interviews (C)

400

 
Eligibility Rate: ER = (C+E)/(C+E+I)

0.393

 
Response Rate = C/[(ER*U) + C + E]

34.3%

 

 

** "Unknown" = "Numbers in sample for which eligibility could not be established"

 

 


Scenario D

40% of individuals 18+ qualify.
Can determine from anyone in household whether selected respondent qualifies (eg: asking for a male aged 25 to 49)
Selected respondent NA, respondent refusal and qualified break-off reduced in proportion to reduction in number of completes

3.1 EMPIRICAL METHOD 

D

 

Total Numbers Attempted

3760

 
Invalid
NIS, fax/modem, business/non-res.

1000
1000

 

4 Unresolved (U)

Busy, no answer, answering machine

900

900

 

5 In-scope - non-responding (IS)

Language problem
Illness, incapable
Selected respondent not available

Household refusal
Respondent refusal
Qualified respondent break-off

810

100
50
40

500
100
20

 

6 In-scope - Responding units (R)

Language disqualify
No one 18+
Other disqualify
Completed interviews

1050

--
50
600
400

 
Response Rate = R/(U+IS+R)

38.0%

 

 


Scenario D

40% of individuals 18+ qualify.
Can determine from anyone in household whether selected respondent qualifies (eg: asking for a male aged 25 to 49)
Selected respondent NA, respondent refusal and qualified break-off reduced in proportion to reduction in number of completes

ESTIMATION METHOD 

D

 

Total Numbers Attempted

3760

 
Invalid
NIS, fax/modem, business/non-res.

1000
1000

 

Unknown (U)**

Busy, no answer, answering machine
Language problem
Illness, Incapable
Selected respondent not available
Household refusal
Respondent refusal

1550

900
100*
50*
--
500
--

 

Ineligible (I)

Language disqualify
No one 18+
Other disqualify

650

--
50
600

 

Eligible (E)

Language problem
Illness, incapable
Selected respondent not available

Household refusal
Respondent refusal
Qualified respondent break-off

160

--
--
40

--
100
20

 
Completed interviews (C)

400

 
Eligibility Rate: ER = (C+E)/(C+E+I)

0.462

 
Response Rate = C/[(ER*U) + C + E]

31.3%

 

 

*In fact, some of these could have been determined to be eligible
** "Unknown" = "Numbers in sample for which eligibility could not be established

 

 


Scenario E.1

20% of individuals 18+ qualify. Readership survey in Montreal - must speak English most often at home. Assumed any household member can confirm whether selected respondent qualifies as English speaker.
Selected respondent NA, respondent refusal and qualified break-off reduced in proportion to reduction in number of completes

EMPIRICAL METHOD 

E

Total Numbers Attempted

3680

Invalid
NIS, fax/modem, business/non-res.

1000
1000

Unresolved (U)

Busy, no answer, answering machine

900

900

In-scope - non-responding (IS)

Language problem
Illness, incapable
Selected respondent not available

Household refusal
Respondent refusal
Qualified respondent break-off

730

100
50
20

500
50
10

In-scope - Responding units (R)

Language disqualify
No one 18+
Other disqualify
Completed interviews

1050

800
50
--
200

Response Rate = R/(U+IS+R)

39.2%

 


Scenario E.1

20% of individuals 18+ qualify. Readership survey in Montreal - must speak English most often at home. Assumed any household member can confirm whether selected respondent qualifies as English speaker.
Selected respondent NA, respondent refusal and qualified break-off reduced in proportion to reduction in number of completes

ESTIMATION METHOD 

E

Total Numbers Attempted

3680

Invalid
NIS, fax/modem, business/non-res.

1000
1000

Unknown (U)

Busy, no answer, answering machine
Language problem
Illness, Incapable
Selected respondent not available
Household refusal
Respondent refusal

1550

900
100*
50*
--
500
--

Ineligible (I)

Language disqualify
No one 18+
Other disqualify

850

800
50
--

Eligible (E)

Language problem
Illness, incapable
Selected respondent not available

Household refusal
Respondent refusal
Qualified respondent break-off

80

--
--
20

--
50
10

Completed interviews (C)

200

Eligibility Rate: ER = (C+E)/(C+E+I)

0.248

Response Rate = C/[(ER*U) + C + E]

30.1%

 

*In fact, some of these could have been determined to be eligible
** "Unknown" = "Numbers in sample for which eligibility could not be established

 

 


Scenario E.2

The same as scenario E, except that household refusals are reduced reflecting what will actually happen in a live survey. The relative ease in determining a language disqualify will reduce the number of household refusals encountered. The net effect is to increase response rate under both methods (slightly more so with the Empirical method).

EMPIRICAL METHOD 

E

E-2

Total Numbers Attempted

3680

3480

Invalid
NIS, fax/modem, business/non-res.

1000
1000

1000
1000

Unresolved (U)

Busy, no answer, answering machine

900

900

900

900

In-scope - non-responding (IS)

Language problem
Illness, incapable
Selected respondent not available

Household refusal
Respondent refusal
Qualified respondent break-off

730

100
50
20

500
50
10

530

100
50
20

300
50
10

In-scope - Responding units (R)

Language disqualify
No one 18+
Other disqualify
Completed interviews

1050

800
50
--
200

1050

800
50
--
200

Response Rate = R/(U+IS+R)

39.2%

42.3%

 


Scenario E.2

ESTIMATION METHOD 

E

E-2

Total Numbers Attempted

3680

3480

Invalid
NIS, fax/modem, business/non-res.

1000
1000

1000
1000

Unknown (U)

Busy, no answer, answering machine
Language problem
Illness, Incapable
Selected respondent not available
Household refusal
Respondent refusal

1550

900
100*
50*
--
500
--

1350

900
100*
50*
--
300
--

Ineligible (I)

Language disqualify
No one 18+
Other disqualify

850

800
50
--

850

800
50
--

Eligible (E)

Language problem
Illness, incapable
Selected respondent not available

Household refusal
Respondent refusal
Qualified respondent break-off

80

--
--
20

--
50
10

80

--
--
20

--
50
10

Completed interviews (C)

200

200

Eligibility Rate: ER = (C+E)/(C+E+I)

0.248

0.248

Response Rate = C/[(ER*U) + C + E]

30.1%

32.5%

 

*In fact, some of these could have been determined to be eligible
** "Unknown" = "Numbers in sample for which eligibility could not be established"