6. Recruitment

6.1 Recruitment categories
6.2 Recruitment results
6.3 Comparison of study participants and non-participants.
6.4 Investigation of possible participation bias
6.5 Discussion
6.6 Technical Supplement – Details of the imputation procedure for non-participation

This chapter describes the results of the recruitment effort for this study, including a comparison of participants and non-participants. Recruitment for this study commenced in July 2000 with weekly mail-outs of invitation packages to Navy subjects. These were followed by mail-outs to Air Force subjects commencing in February 2001 and mail-outs to Army subjects commencing in May 2001. Sampling for each service type was conducted within the 8 weeks prior to each wave of mail-outs. Recruitment was closed in April 2002. A complete guide to the subject sampling methods, study inclusion and exclusion criteria, sample sizes and contact and recruitment methods, including methods to maximise recruitment success and minimise participation bias, can be found at chapter 5.

6.1 Recruitment categories

The original study sample comprised 1873 Gulf War veterans and 3192 comparison group subjects. Throughout the contact and recruitment period, they were classified into the following categories.

6.1.1 Ineligible subjects

It was assumed, upon commencement of the contact and recruitment effort, that some sampled subjects would prove to be ineligible for participation according to the study inclusion criteria. The original comparison group was specifically over-sampled on the assumption that some Navy subjects would prove to have been not serving at the time of the Gulf War and therefore ineligible for inclusion in the study. It was not known whether any of the Gulf War veteran group, identified from the Nominal Roll, would prove to be ineligible for participation. Ineligible subjects were usually identified by information that they provided to the contact and recruitment team.

6.1.2 Eligible subjects

Throughout the contact and recruitment period, and upon cessation of the contact and recruitment effort, remaining eligible subjects were classified as belonging to one of the following recruitment categories:

6.1.2.1 Not recruitable categories

Eligible subjects were classified as not recruitable if they were:

Reportedly deceased: These persons had either already been identified as deceased according to DVA records, or were reported as being deceased during the study contact and recruitment period.
Reportedly overseas long-term: These persons were found to be overseas during the study contact and recruitment period with no expectation of returning to Australia during the data collection period.

6.1.2.2 Recruitable categories

Remaining eligible subjects were considered recruitable. These subjects were classified into the following categories:

Participants

Full participant: These persons completed the study postal questionnaire and attended a medical assessment with HSA.
Postal questionnaire-only participant: These persons completed the study postal questionnaire but did not attend a medical assessment with HSA.

Non-participants

Telephone questionnaire-only: These persons completed the telephone-administered heath questionnaire only, and declined all other participation.
Declined all participation: These persons declined participation in all aspects of data collection including the medical assessment, postal questionnaire and telephone questionnaire.
Not Contactable: It was assumed that the study invitation packages were not received by these subjects. There was evidence to suggest that the held contact details were incorrect, and no alternative contact details could be found.
Non-responder: In these cases there was evidence that the Contact and Recruitment team had the correct contact details and that the subject had received the invitation package. Despite multiple contacts, or multiple contact attempts, these subjects never finally indicated whether they wished to participate or refuse.

6.2 Recruitment results

6.2.1 Final eligible sample sizes

Of the original 1873 Gulf War veterans, one proved to have been a serving member of the British armed forces during the Gulf War, and not the ADF, and two reported to have not deployed to the Gulf War. All three subjects were therefore categorised as ineligible for participation in the Gulf War veteran group and were removed from the eligible sample. One comparison group subject reported to have deployed to the Gulf War. This report was confirmed by the ADF and this comparison group subject was reclassified as a Gulf War veteran for the purpose of the study. The final Gulf War veteran sample totalled 1871 veterans, which included 38 females.

Throughout the recruitment period 267 comparison group subjects, of the originally sampled 3192, were found to be ineligible for inclusion in the study as they had not been serving members of the ADF in August 1990. These subjects were therefore removed from the comparison group sample along with the one subject who reported deploying to the Gulf War. The resulting eligible comparison group sample totalled 2924 including 74 females.

6.2.2 Recruitment outcomes for Gulf War veterans and the comparison group

6.2.2.1 Total eligible sample

The recruitment results for the total eligible sample, and for the total recruitable sample, are shown in Table 6.1 for Gulf War and comparison group subjects. Table 6.1 also shows the recruitment results for both study groups according to their ADF employment status; that is, whether they were classified as still serving members of the ADF, or not serving, at the time of sampling. Sampling dates were approximately June 2000 for the Navy subjects, January 2001 for the Air Force and April 2001 for the Army.

Of 1871 Gulf War veterans, removal of subjects who were reportedly deceased and reportedly overseas long-term resulted in a total recruitable sample of 1808 Gulf War veterans. Among these there was an overall participation rate of 80.5% (1456/1808), with 1414 Gulf War veterans (78.2%) completing both the medical assessment and postal questionnaire, and a further 42 (2.3%) completing the postal questionnaire alone. These included 32 female Gulf War veteran participants, 30 of whom completed both the medical assessment and the postal questionnaire.

The total recruitable sample of comparison group subjects was 2796 after removal of those reportedly deceased and reportedly overseas long-term. The overall participation rate in the comparison group was 56.8% (1588/2796) with 1588 participants of whom 40 were females. Compared with the Gulf War veteran group participants, the comparison group participants were less likely to participate in full (50.5% of the recruitable sample) and more likely to opt for postal questionnaire-only participation (6.3%). Overall the comparison group subjects were also more likely than the Gulf War veterans to participate in the telephone questionnaire, more likely to decline all participation and more likely to be classified as non-responders or not contactable.

Within both study groups, subjects classified as serving members of the ADF at the time of sampling were more likely to participate in full, or else were more likely to participate as postal questionnaire-only participants when compared with those no longer serving. Serving members were also less likely to be not contactable compared with not serving subjects.

Telephone questionnaire data was collected on a total of 411 subjects, representing more than 21% of all Gulf War veteran non-participants (77/352) and more than 27% of all comparison group non-participants (334/1208).

It should be noted that the tabulation of three Gulf War veterans, and 17 comparison group subjects, who were ‘reportedly deceased’ and yet also categorised as still serving with the ADF at the time of sampling, is likely to be the result of some misclassification of serving status, rather than the incidence of 20 new deaths in the period between sampling and recruitment closure.

6.2.2.2 Eligible sample by service type

The recruitment results for eligible Navy subjects, Army subjects and Air Force subjects are shown in Table 6.2, Table 6.3 and Table 6.4 respectively, for Gulf War veterans and comparison group subjects, and for those considered serving or not serving at the time of sampling.

Overall, Navy subjects represented 86% of all Gulf War veteran participants (1249/1456) and 72% of all comparison group participants (1139/1588). The participation rates within the service types were highest for the Navy Gulf War veteran group with 81.6% of this group participating either in full or via postal questionnaire alone (1249/1530), compared with 78.5% of Army Gulf War veterans (95/121), and 71.3% of Air Force Gulf War veterans (112/157). Participation rates, in the comparison group, were fairly consistent across the three service types and lower than those for Gulf War veterans.

Among Gulf War veterans, those who served in the Air Force were the least likely to participate in full and the most likely to decline all participation in the study, when compared with Gulf War veterans of the Navy and Army services.

Table 6.1 Recruitment results for all Gulf War veterans and comparison group subjects

go to table 6.1

Table 6.2 NAVY: Recruitment results for Navy Gulf War veterans and comparison group subjects

go to table 6.2

Table 6.3 ARMY: Recruitment results for Army Gulf War veterans and comparison group subjects

go to table 6.3

Table 6.4 AIR FORCE: Recruitment results for Air Force Gulf War veterans and comparison group subjects

go to table 6.4

Back to top

6.3 Comparison of study participants and non-participants.

A comparison of known information for study participants and non-participants was conducted to assess the level to which participants were representative of the recruitable samples from which they were drawn. For these investigations the group referred to as ‘participants’ include all subjects belonging to the two recruitable categories ‘Full participants’ and ‘Postal questionnaire-only participants’ as defined in section 6.1.2.2.

The investigation of differences between participants and non-participants was conducted in two ways. Firstly, all participants were compared with all non-participants, in other words those remaining recruitable subjects who either, only completed the telephone questionnaire, declined all participation, were non-responders, or were not-contactable. Secondly, all participants were compared with the ‘telephone questionnaire-only’ subjects.

6.3.1 Participants compared with all non-participants.

The information available for all sampled subjects, and therefore available for the comparison of participants with all non-participants, were:

Age at August 1990 (commencement of the Gulf War),
Sex
Service type (Navy, Army or Air Force) at 2 August 1990
Service rank at August 1990
ADF employment status (serving versus not serving) at the time of sampling.

The mean age and participation rates within subcategories of each variable are shown in Table 6.5 for Gulf War veterans and the comparison group.

Generally, the characteristics of participants across both study groups were similar. Subjects who were the youngest and the lowest in rank were least likely to participate in both groups. This difference, across age and rank subcategories, was more pronounced in the comparison group where subjects aged less than 20 years, at August 1990, were 30% less likely to participate in the study compared with subjects aged 35 years or older, and non-supervisory ranks were 21% less likely to participate than officer ranks. The source of rank category, at August 1990 for the comparison of participants and non-participants, was DVA-held archival records. Some inaccuracies are thought to have existed within that data source. These inaccuracies are expected to have occurred in both study groups and are not expected to have notably altered the true trend in participation rates across rank category.

As previously noted, Air Force Gulf War veterans were less likely to participate than Gulf War veterans of the Navy and Army. In the comparison group, however, participation was highest in the Air Force group.

Serving subjects were more likely to participate than non-serving subjects, in both groups.

Table 6.5 Mean age and participation rates across age category, sex, service type, rank and ADF employment status in recruitable Gulf War veterans and comparison group subjects: study participants versus all non-participants.

go to table 6.5

6.3.2 Comparison of study participants with telephone questionnaire-only subjects

The telephone questionnaire was specifically offered to subjects who declined to participate in the study’s full medical assessment or postal questionnaire, for the purpose of investigating any differences in the demographic profile and general health profile of study participants and non-participants. All questions included in the telephone questionnaire were drawn directly from the postal questionnaire, enabling responses from participants and telephone-only subjects to be directly compared. The information collected included:

Country of birth
Level of highest education achieved
Occupational status
Smoking history
Physical Component Summary (PCS) and Mental Component Summary (MCS) measures from the SF-12 Short Form Health Survey.

The comparison of study participants and telephone questionnaire-only participants is shown in Table 6.6. There were only minor differences in country of birth, occupational status and smoking history. Fewer of the telephone questionnaire-only participants had post-secondary education, possibly because officers were more likely to participate in the study (see Table 6.5) and officer training includes the attainment of a tertiary degree. Telephone questionnaire-only subjects received higher MCS scores (self-reported evidence of healthier mental status) than full participants in both the Gulf War veteran group and the comparison group. PCS scores were also a little higher (self-reported evidence of greater physical functioning) for telephone questionnaire-only subjects.

Telephone questionnaire-only subjects comprised only one quarter of all non-participants. Before any true interpretation can be made of differences between study participants and non-participants it is important to consider how representative the telephone questionnaire-only subjects are of the larger non-participant group. Therefore a comparison was made between telephone questionnaire-only subjects and the remainder of the non-participants on age, service type, service rank and ADF employment status (data not shown).

In the comparison group, where non-participation was highest, it was found that the mean age for telephone questionnaire-only subjects was 26.5 years; very similar to the mean age of 26.2 years for remaining comparison group non-participants. When analysed by age category, the telephone questionnaire-only subjects were slightly under-represented in the <20-year age group (12.9% versus 17.4%). Therefore, comparison group non-participants who did not complete the telephone questionnaire were a little younger than the non-participants who did complete the telephone questionnaire. The comparison group telephone questionnaire-only subjects were more likely to be serving than the other non-participants (45.8% versus 28.9%). Comparison group telephone questionnaire-only subjects were also more likely to have served with a non-supervisory rank (26.3% versus 11.4%).

Patterns between Gulf War veteran telephone questionnaire-only subjects and other non-participants were similar to those in the comparison group.

Table 6.6 Comparison of study participants with telephone questionnaire-only subjects.
	Gulf War veterans				Comparison group
	Participants (N=1456)		Telephone questionnaire (N=77)		Participants (N=1588)		Telephone questionnaire (N=334)
	n	(%)	N	(%)	n	(%)	n	(%)
Country of birth
Australia	1221	(83.9)	63	(81.8)	1324	(83.4)	280	(83.8)
Other	231	(15.9)	14	(18.2)	263	(16.6)	54	(16.2)
Education level
Up to year 12	542	(37.2)	38	(49.4)	518	(32.6)	166	(49.7)
Certif/Dipl/Tertiary	910	(62.5)	39	(50.6)	1066	(67.1)	168	(50.3)
Occupational status
Paid employment	1335	(91.7)	70	(90.9)	1468	(92.4)	307	(91.9)
Other	119	(8.2)	5	(6.5)	116	(7.3)	24	(7.2)
Smoking status
Current	375	(25.8)	18	(23.4)	366	(23.0)	93	(27.8)
Former	444	(30.5)	21	(27.3)	508	(32.0)	102	(30.5)
Never	634	(43.5)	38	(49.4)	710	(44.7)	138	(41.3)
SF-12	Median (range)		Median (range)		Median (range)		Median (range)
Physical Component Summary	52.2 (12.6 – 65.1)		53.8 (22.5 – 59.4)		53.1 (15.6 – 66.8)		54.2 (18.4 – 65.5)
Mental Component Summary	50.9 (10.1 – 66.1)		56.8 (24.0 – 67.3)		54.0 (16.9 – 69.5)		55.9 (15.9 – 65.7)

6.4 Investigation of possible participation bias

Having discovered that participation in the study was lower in the comparison group than in the Gulf War veteran group, and lower also among younger persons and lower ranks, we investigated whether this non-response might bias the results of various health outcomes. Bias might arise if the health status of non-participants differed markedly, on average, from the health status of participants. In particular, we investigated the likely direction and possible magnitude of any such bias.

A complete examination of possible participation bias would require the collection of comprehensive information on the health status of all non-participating Gulf War veteran and comparison group subjects. As such data were not available for non-participants, two different methods were adopted to assess possible participation bias.

The first method was a simple, general methodology in which the overall health status of Gulf War and comparison group non-participants was hypothesised, and an age-adjusted estimate, of the odds ratio that would have been observed in the complete sample, was computed. A graphical display is presented to assess the sensitivity of conclusions to the hypothesised values of the non-participants’ health status (for an example see Figure 6.1).

The second method used a series of probability models to impute (ie, predict) the health status of individual non-participants and to compute the odds ratio using the resultant “complete” data. As a stochastic mechanism (ie, random number generation) was used to perform the imputations, the imputation process was repeated multiple times. The average of the resulting odds ratios was taken as the estimate of the true odds ratio relating the health of Gulf War veterans and comparison group subjects. This procedure is an approximation of the statistical technique known as multiple imputation.^[332] Details of both methods are presented in the sections that follow.

The interpretation of the results of the first method above, and the core of the modelling process for the second method above, rely on the utilisation of the physical (PCS) and mental (MSC) component summary scores of the SF-12 questionnaire as proxy measures of health status, available for the non-participants who completed the telephone questionnaire. Using these data, and under assumptions about the representativeness of the non-participants who completed the telephone questionnaire (see Section 6.3.2), it is possible to get an indication of the possible health status of all non-participants. This is achieved, in a broad sense, by examining differences in SF-12 scores across participants and telephone questionnaire subjects, and across study groups and broad age category (<25 versus >=25 years in August, 1990). Note that age was chosen as a sub-grouping variable as it was clearly related to participation and is also associated with many health outcomes. Choosing a single cut-point to divide age into two categories makes presentation simpler and yet allows the control of this important confounding variable in assessments of differences in health outcome between Gulf War and comparison group subjects. The choice of the cut-point of 25 years for categorisation of age was made because 25 years was approximately the median age among telephone respondents. This cut-point therefore provided reasonable sample sizes in each category to facilitate numerical stability, in addition to separating low from high participation (see Table 6.5).

Table 6.7 Median SF-12 PCS and MCS scores for Gulf War veteran and comparison group participants and telephone questionnaire subjects across age categories
		Gulf War veterans			Comparison group
Age		N (% of non-participants)*	Median PCS	Median MCS	N (% of non-participants)*	Median PCS	Median MCS
< 25	Participants	576	53.1	50.5	532	54.2	54.5
	Telephone	35 (19%)	53.6	56.8	145 (25%)	54.3	55.9
>= 25	Participants	829	51.7	51.3	1021	52.4	53.9
	Telephone	40 (23%)	53.8	56.9	186 (30%)	53.8	55.9

* For telephone questionnaire-only subjects

The sample sizes for the participants and telephone questionnaire subjects, the percentage of total non-participants represented by the telephone questionnaire subjects, and the median SF-12 PCS and MCS scores are shown in Table 6.7 for the Gulf War veteran group and comparison group for each age category. Medians are presented as the PCS and MCS distributions were found to be left skewed. The means displayed similar relationships but were uniformly lower in magnitude. For interpretation of the SF-12 scores, recall that higher SF-12 scores represent better self-reported health.

Previously it was shown that telephone questionnaire-only subjects recorded higher MCS scores than full participants in both study groups, but particularly so in the Gulf War veteran group (see Table 6.6). Table 6.7 demonstrates, in addition, that the difference in MCS scores between participants and telephone questionnaire subjects is reasonably consistent across age categories within the Gulf War and comparison groups. In the case of the PCS scores, the differences are uniformly less pronounced.

The health differential between telephone participants and full participants therefore appears to be unidirectional within each age stratum and within each study group, with telephone participants recording better mental heath than full participants in each age stratum and in both study groups. If the SF-12 measure is a valid proxy of health in participants and non-participants, then the health differential implies that the Gulf War veterans and comparison group non-participants will be of better health status than the participants in both of these groups. As a result, the overall prevalence of illness, combining all participants and non-participants, will be lower than that found for the participants alone. However, odds ratios, which represent the extent to which the prevalence of ill-health differs between the two study groups, may be over or underestimated according to the relative differences in health between the participants and non-participants in these groups. The examination of the effect of participation bias on odds ratios, using two methods, is presented in the following sections 6.4.1 and 6.4.2.

6.4.1 Method 1: Grouped data assessment of participation bias

The essence of the first method is to hypothesise the extent (via the prevalence) of ill-health among non-participants within each age category of the Gulf War and comparison groups. These hypothesised prevalences are then applied in order to obtain an overall age-adjusted odds ratio as if there had been complete participation from the outset. Note that no individual data are imputed in this method.

The methodology proceeds using concepts of Mantel-Haenszel estimation of a common odds ratio from a series of 2x2 tables.^[333] Specifically, the hypothesised prevalences of ill-health among non-participants within each age category of the Gulf War and comparison groups are combined with the observed prevalences among the full participants to yield a “complete sample” prevalence within each age category. These are then combined by the Mantel-Hanszel method to estimate the overall “complete sample” odds ratio. The difference between this predicted “complete sample” odds ratio and the observed odds ratio among participants reflects the degree of participation bias in the observed odds ratio.

It is convenient to consider the prevalence of ill-health among non-participants via a ratio measure, proportionate to the participants, which we have called “proportionate bias”. For example a proportionate bias value of 0.5 indicates that the prevalence of ill-health among non-participants of a certain study group age category ( eg, Gulf War veterans aged <25 years), is half the prevalence of ill-health among the participants in the same study group and age stratum. Similarly a proportionate bias value of 2 indicates that the prevalence of ill-health among non-participants is twice that of participants in the same stratum.

For graphical presentation of results, it simplifies matters to constrain the two proportionate biases within the Gulf War group to be equal (ie, same proportionate bias for those aged <25 as for age >25), and to similarly constrain the two proportionate biases in the comparison group to be equal. This reduces the presentation load from four participation bias parameters to two.

The following figure presents an example of predicted odds ratios for the complete sample according to the proportionate bias in the comparison group, at each of four fixed levels (0.5, 1, 1.5, 2) of proportionate bias in the Gulf War group. The data are the post-Gulf War CIDI defined anxiety disorder results from the Psychological Health chapter (see Chapter 11). No substantive interpretation of these results will be provided here as the data are for illustration only.

Figure 6.1 Predicted odds ratios for compete sample, at varying levels of proportionate bias in non-participants, where observed age-adjusted odds ratio is 2.6.

The prevalence of post-gulf anxiety in Gulf War veteran participants, aged <25, was 9.5% whereas in the comparison group it was 2.8%. Among those aged >25, the prevalence was 6.1% in the Gulf War group and 3.0% in the comparison group. This produces an age-adjusted (Mantel-Haenszel) odds ratio of 2.6, represented by the horizontal line in the Figure, meaning that the odds of the anxiety is reported to be 2.6 times higher in Gulf War participants than in comparison group participants within the same age stratum. The curved lines represent the fixed proportionate bias values (0.5, 1, 1.5 and 2) in Gulf War veteran non-participants.

Figure 6.1 demonstrates that the predicted odds ratio in the complete sample may vary between 1.7 and 4.2, albeit in rather unlikely scenarios. The value of 1.7 arises when the Gulf War non-participants have half the prevalence of anxiety as the Gulf War participants, and the comparison group non-participants have twice the prevalence of anxiety as the comparison group participants. In this case the observed odds ratio among participants is an overestimate of the odds ratio in the complete sample. Similarly the odds ratio of 4.2 arises at the opposite extreme when the non-participants in the Gulf War group have twice the prevalence of anxiety as the participants, with the opposite being true for the comparison group.

Based on the demonstration that SF-12 MCS results displayed higher scores (ie, better health) for the telephone participants compared with full participants, and under the assumption that the telephone participants are representative of the health of the non-participants, the health of non-participants would be expected to also be better than that of the participants. It then follows that the proportionate bias within both groups would be expected to have values less than one. In this instance, the relevant section of Figure 6.1 to be examined would be the area between the curves labelled 0.5 and 1, and to the left of 1 on the horizontal axis. However, recalling that the difference between the SF-12 MCS scores of telephone participants and full participants was greater for Gulf War veterans than for comparison groups subjects, the proportionate bias for Gulf War veterans would, accordingly, be expected to be greater than that for comparison group subjects. This suggests that the relevant region on the horizontal axis (ie, comparison group proportionate bias axis) could be further constrained to be between approximately 0.7 and 1. This area is shaded in Figure 6.1 for convenient reference. The estimated complete sample odds ratios within this range vary from approximately 2.4 to 3.2, and these do not vary to a large degree from the observed odds ratio among full participants of 2.6. Observe also that it is possible for substantial proportionate bias to exist within each group, and yet the complete sample odds ratio need not be biased. For example, consider a proportionate bias of 0.50 in Gulf War veterans, and 0.80 in comparison group subjects; Figure 6.1 indicates that the predicted complete sample odds ratio is almost identical to the odds ratio among participants.

It should be noted, however, that the shaded region is only an approximation as the precise relationship between SF-12 scores and each health outcome among non-participants is not known and is postulated only generally here.

Other similar examples can be constructed. For example, an illness with a prevalence of approximately 7% among comparison group subjects and an odds ratio of 1.2 among participants would be expected to yield complete sample odds ratios between approximately 1.1 and 1.4 for proportionate biases in the range described in the previous example (ie, Gulf War veterans 0.50-1.0, comparison group 0.70-1.0). This range of proportionate biases does not indicate substantial participation bias in the odds ratio. Other examples demonstrate that the higher the prevalence for a given odds ratio among participants, the wider the variation in complete sample odds ratios.

6.4.2 Method 2: Individual data imputation-based assessment of participation bias

In this section we use the SF-12 data more directly than was the case with the grouped data method. In brief, the SF-12 data for full and telephone-questionnaire participants are used to generate predicted SF-12 responses for remaining non-participants. These predicted or imputed values are then used further to predict the broader health status of non-participants. The assumptions behind the procedure include being able to predict the SF-12 mental and physical health of non-participants using the relationship between SF-12 scores and deployment status, age, rank, service type and serving status that was observed among telephone-questionnaire participants. Furthermore, the health outcome of non-participants is assumed to be able to be predicted using the relationship between the health outcome and the above factors and SF-12 scores observed in full participants. Collectively, the missing data among non-participants are assumed to be “missing at random” given knowledge of all the observed data.^[332] Technical details of the imputation methodology are contained in a Technical Supplement at the conclusion of this chapter (see Section 6.6)

Using the two-step modelling mechanism of first imputing SF-12 scores, followed by imputing the health status of non-participants, a “complete” dataset was formed. This “complete” dataset was then used to compute an odds ratio via logistic regression relating deployment status to the health outcome after adjusting for age, rank and service. The entire imputation procedure was then replicated 100 times. The average imputed odds ratio from the 100 replications represents the best estimate, based on the observed data and the imputation model for the health status of non-participants, of the true odds ratio underlying the relative health of Gulf War veterans and non-deployed subjects. The difference between the average imputed odds ratio and the actual observed odds ratio among participants reflects the degree of participation bias, and is the focus of this assessment.

As an example of the methodology, consider the post-Gulf War CIDI anxiety disorder data and the results from 100 replications of the imputation process:


Participants			Imputed results
GWV prevalence	Comp group prevalence	Odds Ratio	Average GWV prevalence	Average Comp group prevalence	Average Odds Ratio	Range
8.3%	3.1%	2.77	7.5%	2.9%	2.68	1.95-3.71

Under the assumptions of the imputation model, the estimated true odds ratio is 2.68, which differs only slightly from the observed odds ratio of 2.77 among participants, indicating minimal participation bias. Note also that, as anticipated from the previous section, the imputed prevalence of post-gulf anxiety in either deployment group is lower than the corresponding prevalence among participants due to the more favourable SF-12 scores observed among telephone participants than full participants. More detailed discussion of this and other related health outcomes is deferred to the Psychological Health chapter (see chapter 11).

The imputation procedure described above relies upon the assumptions that the missing data are missing at random, and that the imputation models are correctly specified. The procedure is one, simplified application of a more general multiple imputation strategy. More comprehensive modelling and inference can be obtained using a variety of imputation models and more formal Bayesian inferential methods.^[334] An advantage of the grouped data method for assessing participation bias described in the previous section over that of the individual data imputation method is that the former can be applied to a range of hypothesised values of the proportionate bias in each group and need not rely on the specification of statistical models. The fact that the odds ratios are only adjusted by a binary age variable, as opposed to a fuller set of adjustment factors, does not severely detract from the method’s applicability. In fact, for most analyses described in this report, adjustment for further potential confounding factors had little impact on the results.

6.5 Discussion

Overall, more than 80% of Australia’s 1808 recruitable Gulf War veterans, and more than 56% of the 2796 recruitable comparison group subjects participated either in full or by postal questionnaire in this study. Full participation included completion of both the postal questionnaire and medical assessment, and often participants undertook lengthy travel, sometimes requiring overnight accommodation, to attend a HSA clinic. The recruitment results compare very favourably with international survey-based studies where response rates have been from as low as 31% in a study of more than 16,000 active duty and reserve personnel,^[335] to as high as 95% in a study of 630 Gulf War veterans directly recruited at training sessions.^[179] Table 6.8 contains a brief review of participation rates in international epidemiological studies. These studies have been primarily questionnaire-based surveys, with only one of these studies^[162] including a medical examination as a component of the data collection.

The inability to locate subjects, particularly those who have left the armed services since the Gulf War, has proven to be a major factor affecting participation rates in many studies. Kang et al reported 90% participation by all located subjects but a participation rate of 70% among all eligible subjects,^[20] and the Iowa Persian Gulf Study Group reported a 91% response rate amongst located subjects compared with their overall participation rate of 76%.^[16] Despite multiple strategies employed to locate current contact details for all recruitable subjects in this study, more than 300 such subjects (7%) remained non-contactable upon closure of the recruitment period.

The reduced response rate amongst the comparison group, compared with the Gulf War veteran group, is consistent with other major epidemiological studies, which utilised a non-Gulf War comparison group. For example, Unwin et al reported response rates of 70.4% in their Gulf War cohort compared with 61.9% in their Bosnia cohort and 62.9% in their non-deployed control population.^[21] Similarly, Ishoy et al reported participation rates of 83.6% in the Gulf War veteran group and 57.7% in the control population.^[162]

Despite the findings of reduced participation rates amongst non-Gulf War comparison groups, few studies have formally evaluated participation bias in published papers. We investigated the issue of participation bias in several ways. Firstly, we were able to make some comparisons between all participants and all non-participants on a few parameters thought to influence health status. This analysis showed that younger people tended to be under-represented in the participating groups, as were those of lowest rank category. The difference in rank distribution is likely to be related to the aforementioned age differential. Secondly, the inclusion of the SF-12 Health Survey in the telephone questionnaire, administered to approximately one quarter of the non-participants, enabled us to compare participants with this subset of non-participants, on this health measure as well as other demographic and lifestyle factors. This investigation showed the most apparent difference between groups was in the Mental Component Summary score derived from the SF-12, indicating that participants had slightly poorer self reported mental health than non-participants.

Table 6.8: Participation rates in international epidemiological studies.
Study reference	Gulf War veterans % Participation (total sample size)	Non-Gulf War comparison groups % Participation (total sample size)	Study characteristics
Perconte et al 1993^[179]	95% (620)	NA	Participants recruited during weekend training sessions. Data collected via questionnaire.
Cherry et al 2001^[157]	85.1% (9505)	82.9% (4749)	Recruitment via mail, personal visit to Units, telephone contact and home visits. Data collected via questionnaire.
Ishoy et al 1999^[162]	83.6% (821)	57.7% (400)	Postal questionnaire followed by health examination.
Wolfe et al 1998^[159]	78.4% (2949)	NA	Participants recruited within five days of return to US. Data collected via questionnaire.
Iowa Persian Gulf Study Group 1997^[16]	78.3% (2421)	73% (2465)	Telephone survey.
Kang et al 2000^[20]	75% (15,000)	64% (15,000)	Data collected via postal questionnaire followed by telephone administered interview with non-respondents.
Southwick et al 1993, 1995^{[336, 337]}	74.4% (160) at 1 mo 52.5% at 6 mo 38.8% at 2 yrs	NA	Participants recruited at training sessions at 1 mo, 6 mo and 2 yrs following return from Gulf. Data collected via questionnaire.
Goss Gilroy Inc., 1998^[22]	73% (4262)	60.3% (5699)	Data collected via postal questionnaire.
Unwin et al 1999^[21]	70.4% (4246)	Bosnia cohort: 61.9% (4250) Not deployed: 62.9% (4248)	Data collected via postal questionnaire.
Holmes et al 1998^[338]	57.3% (517)	42.2% (497)	Data collected via postal questionnaire.
Haley et al 1997^[158]	41% (606)	NA	Recruitment via mail and telephone. Data collected via self-administered questionnaire in supervised groups.
Stretch et al 1995^[335]	31% (16,167)	NA	Recruitment and data collection via distribution of questionnaires through Units.

We explored the impact of varying magnitudes and directions of hypothesised non-participation bias upon the prevalence of ill-health and odds ratios likely to be found in this study. These computations, based on the telephone questionnaire results, showed that non-participation is likely to produce only a small bias in the observed odds ratios among full participants. Therefore, participation bias is unlikely to explain large differences in measures of health, between the participating Gulf War veterans and comparison group subjects, in our study.

There are several limitations, however, to the use of telephone questionnaire results to predict health in non-participants, and to compare participants with non-participants. Firstly, the telephone questionnaire results may not be generalisable to the larger non-participant population. Telephone questionnaire subjects made up approximately one quarter of all comparison group non-participants and one fifth of all Gulf War veteran non-participants. It is not possible to determine whether the SF-12 scores of the telephone participants are similar, on average, to those of the remaining non-participants. Responses to the SF-12, and the relationship between those and variables such as study group, age and rank may not be predictive of accurate SF-12 scores in non-participants. Also, the relationship between participants’ SF-12 scores and other health outcomes, such as psychological health outcomes, may not be predictive of the health outcomes of non-participants.

The direct comparison of telephone questionnaire derived mental health measures, with those derived from the self-administered postal questionnaire, should also be undertaken with some caution. There is some evidence to expect telephone questionnaire respondents to report improved mental health when compared with those completing self-administered questionnaires.^[339-341] Therefore, improved SF-12 MCS scores among telephone questionnaire subjects in this study, when compared with participants who completed the SF-12 via the self-administered questionnaire, could partly be an artefact of the method of administration of the instrument.

In summary, participation rates in this study compared favourably with international studies, particularly considering the lengthy questionnaire, comprehensive medical assessment and, in many cases, lengthy travel undertaken by participants. Non-participation was highest in the comparison group, in the youngest subjects and in the lowest ranks. Despite some limitations, however, the inclusion of the telephone questionnaire proved a valuable tool for collecting brief, yet useful demographic, lifestyle and health information on a subset of non-participants and this data was used for an investigation of possible participation bias. It was concluded that whilst some health-related participation bias may exist, the magnitude and likely effects of this appear small, based both on the information drawn from the telephone questionnaire responses, and on predictions of health outcomes in non-participants derived from patterns observed in participants. Participation bias for some health outcome measures in this study remains, nonetheless, a possibility.

6.6 Technical Supplement – Details of the imputation procedure for non-participation

This Supplement describes the more technical details of the imputation procedure involved in the assessment of non-participation bias. The procedure consisted of two stages, reflecting the “monotone” missing data pattern, as described in Rubin (1987).^[332] Data concerning deployment group, age, rank, service and serving status were available for all sampled persons; SF-12 mental and physical scores for full telephone participants; and comprehensive health outcome data for all full participants. The imputation procedure operated by first fitting a joint multiple linear regression model predicting SF-12 physical and mental health scores from deployment group, age, rank, service and serving status. All available data were used, together with a binary variable indicating full or telephone participant status, and a term for the interaction of the binary status indicator variable with deployment group. To reflect uncertainty in the model parameters, a random draw was made of these parameters from their posterior distribution given the observed data, and assuming a non-informative prior distribution. (In brief, and more simply put, a prior distribution summarises the information available about a parameter prior to observing a new set of data; the posterior distribution summarises the updated information about the parameter after observing the data.)^[342] Predicted SF-12 scores were generated for non-participants using these sampled parameters and their deployment group, age, rank, service and serving status. A randomly generated normally distributed residual (with variance equal to a random draw from the posterior distribution of the residual variance from the linear regression) was then added to each predicted value to provide comparable variability of imputed SF-12 scores to that of the observed SF-12 scores.

The second step involved fitting a logistic regression model to the full participant data to predict the health outcome of individuals using their deployment group, age, rank, service, serving status and SF-12 scores as predictors. After drawing a value of the model parameters from their approximate posterior distribution (obtained from large sample maximum likelihood theory), predicted probabilities of the health outcome were generated for each non-participant based on their observed values and their imputed SF-12 scores (or observed SF-12 scores for telephone participants). Finally, the actual health status of each individual was imputed by generating a uniform random number and comparing it to the predicted probability for each individual. In this manner a “complete” data set was formed.

Note that the imputation procedure was implemented for binary health outcomes only. In addition, adjustment in the final logistic regression model was made for age, rank and service type only, as these were the key confounding variables that were available for all participants. Adjustment for other variables (eg, education level) would have required additional imputation, which was considered undesirable and therefore was not implemented.