Part I - Protocol Review and Feedback

Format and Presentation:

In general, there is commendable effort as per the flow of the protocol as seen in the table of contents. However, there are some gaps which i will outline as follows:

Background section- Information contained in this section of the protocol is fine, and it flows directly into Objectives of the survey. However, 'Target population' looks somewhat displaced from the background. The Immunization schedule breaks the flow of information herein and it may aid proper flow if it follows after presentation of all information on coverage and data disparities between the different surveys which have been done in Country A.
There should be a very clear section titled ‘Justification and rationale’ which follows after presentation of the situational analysis and gaps; this is not evident.
Objectives: This section flows from survey rationale even though the latter is tucked inside the background and thus somehow not prominent.
Materials and methods: The smooth flow of this section is somewhat evident; it does well to provide subtitles for specific survey methods' sections under its broad heading.
The next section after methods is 'implications of survey results on disease control ' which i think is not an appropriate heading. I propose this should be changed to ‘implications of survey to the national immunization program’.
Ethical considerations, Budgets, funding sources and dissemination of findings are all in the right place. The Timeline of activities is very important and should come immediately after Budget.

Protocol development:

Country A protocol is a product of WHO and staff of the Ministry of Public health. There is evidence of appropriate expertise and experience based on information available about conduct of other surveys such as MICS, DHS and Risk and Vulnerability Assessment surveys. However, we are not provided with information about availability of statistical expertise to specifically guide the team on issues of sampling, selection of clusters/enumeration areas and households, weighting and data analysis. The timing is valid, at 3-4 years after a prior vaccine coverage survey (CES, 2013).

Proposal or suggestions: Articulate clearly the contributions of the National bureau of statistics department in the survey. Outline clearly the specific stakeholders and their roles in the survey and protocol development. This will ensure the funders buy in.
I suggest Country A to be explicit on technical assistance for its statistical needs or requirements.
Define the survey steering committee and their roles in the survey within this protocol.

Background (Context) and survey Objectives

There is good information on demographic indicators, target population, geographical barriers and status of immunization services, challenges to routine immunization e.g. geographical factors leading to poor access, security issues, paucity of immunizing facilities, population displacements and utilization problems and mention of outbreaks of VPDs. There is a lot of information on CES 2013 and comparison of administrative data with WHO/UNICEF estimates. It is commendable that the protocol provides trend analysis graph for penta 3 coverage to make a case on data disparities at national level. However, there is need to discuss exhaustively for purposes of justifying this survey, the probable causes of the serious data variations out of the 2 credible sources of data, which come from very respectable surveys. There is need to quantify the 'high' rates of VPDs since these are a major contributors to deaths in children since Country A has it as one of the goals in the National health Policy.

Background information needs to outline specific areas of Country A which have challenges e.g. show these insecure areas, areas prone to population displacement and distribution of facilities in a map format for easier appreciation. In addition, the regional data variations need to come out to support a rationale for conducting a survey at this level if resources are available.

Proposals to make section more informative:

1. Highlight specific variations in coverage in Country A, in ways for example- where the unvaccinated children are; quantify measles drop outs; explain more about VPDs, are they concentrated in specific regions, rural/urbans coverages etc. in essence, are there areas where ‘over sampling’ should take place for?
2. Give more information about the areas which have challenges with security issues: comment on whether they have been included in past surveys etc.
3. Provide in map format, distribution of facilities, showing clearly areas which are disadvantaged.
4. Give limitations and lessons learnt in the previous surveys and why the data variations are evident in CES and WHO/Unicef estimates.

Some notable information which is missing from the protocol's background section relates to how this survey links with Global vaccine action plan; how it relates to measles elimination, Polio end game strategy etc. However, the writers do a commendable job of linking it to the national health policy, specifically the objectives which relate to the national immunization program as documented in the National health policy.

Proposal: provide link of this survey to Global targets and objectives such as Polio end game strategy and GVAP.

In the background section, the Target population is not correctly presented.

Proposal: I suggest that the team states clearly the eligible age groups early in the background/objectives; since issues which are discussed in the background revolve around this group.

Survey Objectives: One is likely to miss out the primary Objective and only take note of the secondary objectives.

Proposal: Clearly state: Primary Objective and Secondary Objectives. This allows the reader to focus on the important issues here.
I propose that the team looks at the secondary objectives again with the aim of making them focused and non-repetitive. They seem to repeat those related to data (baselines for cMYP, data accuracy and reliability).
It would be important to state the eligible age groups which are being targeted at this point.

Survey rationale:

Justification for the survey though sufficient it is not exhaustive. I propose addition of regional/provincial data to buttress their data disparity argument. This assist the reader or audience of this protocol to appreciate the need at the local level, adding to their justification. It is important to note that we are not seeing any hypothesis about some issues to be addressed by the objectives, e.g. data recording and HBR availability etc. This need to come out if they are to form a basis for conducting a survey.

Context Analysis:

In general, the drafters of the protocol have thought through the whole process. A review of the background tries to bring out the ‘story’ of this Country, including immunization issues at play. It is evident from the background that they have a valid reason for asking for a coverage survey, however we are left asking ourselves whether they have fully analyzed the situation and why we have disparities with credible survey results in the past. This is a fundamental issue which should be resolved as they plan to do this survey. Is it a question of methodology or estimates, Biases, etc; in essence if it is not resolved or adequately explained, then the results may add to more confusion especially if they appear different from the other two.

There is need to do a serious analysis in this survey of the issues at play in the other regions which are left out in immunization e.g. insecure regions, given the fact that only about 50% of the Country is covered with services; is it appropriate to stratify the Country into ‘secure versus insecure’ so that we also get a proper analysis of estimates from here?

The intervention is relevant for routine immunization and addresses the objectives. The inferential goal is relevant. However, is this goal applicable to insecure areas too? There is no relevant data on areas with equity challenges such as the insecure areas and this protocol does not present a proper sitautional analysis of insecure areas.

Overall, the flow of the protocol is wanting and there is need to follow a clear and consistent format which flags the main titles and subtitles for easier review. Some sections are inappropriately placed e.g. Timelines, Implications of the survey, survey report-which should not be there at all. Others are deficient incontent e.g. survey design. 

Survey Design: The survey design in this protocol is incomplete. We are not informed about the sampling method to be used, whether it will be a probability sampling design, what sampling frame will be used and if there will be any stratification. There is a section under sampling methodology which attempts to describe a 2-stage sampling methodology which is wrongly explained.

Proposal: Write out a survey design which is a probability based Cluster survey which incorporates stratification, e.g. rural/urban, areas which are secure versus insecure and one which addresses the age groups selected. This design should also state the inferential goal of estimation adequately.
There is need to correct the sampling stages (2-stage sampling methodology) since it is wrongly understood or represented. Stage 1 is selection of PSU (EAs or Clusters) and stage 2 is selection of SSU (Households).
Kindly interrogate the Sampling Frame which is available for this survey against the known desirable characteristics of a sampling frame; what is documented here are the characteristics of a good frame but we are not told about whether the one chosen fulfills these attributes. .

Sampling and Procedures:

The section on sampling is a mixture of standard operating procedures (SOPs) important in the field and sampling methodology. I propose dividing this section into: sampling methods to be used- stratified, systematic; procedures for selection of Clusters, Households and eligibility of respondents (explanation should also be provided on how the team will identify mothers for TT coverage estimates) as this is missed out in the whole protocol). Provide details of Cluster/HH replacement procedures or criterion in case it becomes inaccessible and Procedures to control for Bias. This section should also touch on ascertainment of vaccination in Children explicitly- this is important to control misclassification which can occur.

Finally, it would be important to attach a separate sheet in the annex on standard operating procedures for Field staff.

Data Management and analysis:

This section is well done. It includes operating procedures for data analysis methods such as double entry of data , pilot testing of tools in the field, code books, analysis indicators however, please include a section on weighting and the specific aspects of the survey which will be amenable to weighting and weighting procedures to ensure survey representativeness. The protocol data management and analysis section will be complete when we show how we are also going to analyze ‘quantitative’ data coming out of the survey respondents since these are part of the objectives.

Implications of the Survey:

This section is not appropriately head or titled; I propose a change to read ‘implications to the NIV program’, or alternatively, implications of survey; this should be followed by outlining implications to the program at various level in-country (Immunization program itself, provincial level program and staff, districts, community level) and a global audience. We need to align these with objectives set out at the beginning- ascertaining data estimates and how this will impact on programming, data disparities betwee surveys, VPD control, baseline for cMYP and what it means for programming and in addition whether the results shall contribute to evidence for global learning. Any plans for publications need to be stated here. Also, provide how the results will assist the Country to fulfill the requirements of various donors such as GAVI, USAID etc.

This section should come at the end of the protocol; at least after Budget and Ethical considerations.

Budget- should be presented in a Table format with details of the cost of activities and events.

Ethical Considerations – Mention the Ethical Review Boards whose consent shall be sought to give the survey credibility and ensure its utility in international/global audiences. There is need to mention how confidentiality shall be ensured.

Dissemination of results: This is part of the protocol and should also appear as a budget item.

Part II - Sampling Methods and Procedures for Country A

Country A vaccine coverage survey has the following objectives:

Primary Objective: To estimate the level of immunization coverage at national and sub-national level.

Secondary Objectives:

To establish baseline information to enable the monitoring of the cMYP progress,
To obtain vaccination coverage estimates that can be compared with administrative

Coverage data

To identify reasons for not immunizing children and women,
To obtain an estimate of ever receiving a home-based record (HBR) and HBR availability, as well as reasons for not having a card. In addition, qualitative information on recording practices (HBRs and registries) can be obtained, and
To make evidence-based recommendations for strategies and interventions that will enhance the achievement and sustainability of EPI planned activities.

Goal: Our inferential goal is estimation of point coverage of specific vaccines in the immunization schedule of Country A. The committee’s decision to use an anticipated Penta 3 coverage of greater than 70%, at 95% confidence interval with a precision of +/-5% is fine.

Study Design: This is a probability cluster survey using 2 strata namely: National and Provincial level. It is important to emphasize that we chose 2 strata because it is easier to conduct a survey at provincial strata, 34 than surveys in each of the 416 districts which can lead to a huge sample size and a logistical challenge, eventually compromising the quality of our estimates. The target group is 12-23 months and survey will involve a 2-stage stratified sampling of clusters and households to look for eligible children. All children in selected households will be eligible to be part of the survey. There will be stratification based on rural or urban populations as well as secure and not secure places. It is prudent to consider any displaced populations and ensure these are sampled. All methodologies will be as per the 2015 WHO vaccination coverage cluster survey reference manual.

Study Population:

The reference population for this survey is all children aged 12-23 months in selected households. This age group will be used to calculate the effective sample size (ESS) for the survey. Because of availability of various recent surveys, I propose use of a sampling frame used in either the CES 2013 survey, MICS survey or the last DHS from Country A national bureau of statistics.

Sample size determination

In order to manage costs of the survey, the steering committee will need to consider whether their primary objectives will be met with a survey at the level of the province or districts. This means that a decision has to be made based on the strata which is chosen and which is feasible within the context of Country A.

The estimation of the primary outcome of the survey will be based on sample size calculations for parameters shown below:

A – Strata: Provincial (34)

B – Effective Sample size (ESS): For Routine Immunization coverage, at an expected coverage of 70% and a precision for 95% confidence interval of +/- 5% confidence bound; the ESS

C – Design Effect (DEFF):

m=5; If target number of respondents per cluster, m=5 and an ICC taken as 1/3, a conservative choice for RI then DEFF will be 2.33.
m=10; if target number of respondents per cluster, m=10 and an ICC taken as 1/3, a conservative value then the DEFF will be 4.

D – Average number of Household to visit to find an eligible child aged between 12-23 months.

Despite having conducted recent surveys, we are not given information about this parameter; It would be important to get information about Birth rate, Infant mortality and average Household size to calculate this.

E – Inflation factor to account for non-response.

Selection of Clusters

The sampling frame will be from the national bureau of statistics. From this frame, a list of clusters or Enumeration areas, will form our Primary sampling units. A list of Provinces is available for the period 2015-2016. Each province shall be assigned a unique code or number. From each province, we shall compile the cumulative total number of districts, Enumeration areas and households within those districts. Refer to figure 1:

Prov Code	districts #	EA/Clusters #	Rural/Urban	Cumulative # , Cluster or EA

From the list of Enumeration areas within the province, we shall stratify the EAs based on rural/urban stratification and then populate a column for cumulative number of EAs. Based on the calculated number of clusters, we shall divide this by the total number of EAs and get the sampling interval to enable selection of the clusters. Those clusters which are more than double the sampling interval shall be segmented to get a manageable population while the small ones shall be combined.

Stage 1 sampling: The cluster selecting team shall comprise the survey coordinator and national bureau/ department of statistics staff. From the cumulative list of EAs, a random number between 1 and the sampling interval figure shall be selected. One can use the excel function RANDBETWEEN (1, X), with X being the sampling interval calculated above.

The specific EA selected will start from this random number and subsequent ones will be selected based on addition of the sampling interval X to this random number, and the corresponding EA selected.

With this process, the total required number of clusters or EA shall be reached.

Selection of Households

Stage 2 selection: From the List of EAs selected, a list of Households shall be requested from the census officials of the National statistics bureau for each of these EAs. Step 2 involves selecting the HH for the survey using systematic sampling based on an interval coming from dividing the total number of HH in the selected EAs with number of HH to be visited as calculated from the sample size calculation. Figure 2.

Selected EA Code	Total #HH in EA	Urban/Rural	Cumulative #HH	Selected HH based on calculated interval

From a random number between 1, and the interval Y, which can be got from the excel function RANDBETWEEN (1, Y), we can then select households. This selection of HH will be done by the same team.

These calculated clusters and households shall be used to plan for field work, budget for field activities including data collection in order to complete the interviews needed to make estimates provincial and National level.

This selection process has used implicit stratification through stratification of rural and urban thus taking care of respondents in households in the two regions.

Eligible children at Household level: This coverage survey is a probability two stage cluster coverage survey and will enroll all eligible children in a selected household. In order to be eligible, the child must be within the age group 12-23 months and be a resident in the household or shall have stayed in the household last night. In case a household with an eligible child is missed, a revisit of upto 2 times will be planned; for household where the respondents are missing, then clear documentation of this event shall be made. This will assist in weighting during analysis.

Women of children Aged 0-6 months: For selected households, all women with children aged 0-11 months shall be interviewed.

Definition of terms: Household- These is defined as a group of people who live under one roof and have got the same eating arrangement.

Survey Pilot:

Its imperative that this survey should have about 3-4 days of Pilot testing in clusters and Households not selected as part of the survey. It will ensure proper testing of operations of the survey tools.