Citation: Mohd Suan MA, et al. The Development and Validation of the Nurses’ Attitude towards Conducting Research Questionnaire (NA2CRESQ). Ann Med Health Sci Res. 2017; 7: 377-382

This open-access article is distributed under the terms of the Creative Commons Attribution Non-Commercial License (CC BY-NC) (http://creativecommons.org/licenses/by-nc/4.0/), which permits reuse, distribution and reproduction of the article, provided that the original work is properly cited and the reuse is restricted to noncommercial purposes. For commercial reuse, contact reprints@pulsus.com

Abstract

Background: To identify nurses’ needs and problems in relation to research, and thereby enhance nursing research output, a valid measure of nurses’ attitudes toward research is necessary. Aim: This study aimed to develop, validate and conduct reliability analysis of a newly developed questionnaire assessing nurses’ attitudes towards conducting research. Subject and methods: We evaluated the content validity of the new questionnaire using an expert panel, while nurses with similar characteristics as the target group assessed for face validity. The content and face validity were determined by six experts and 11 nurses, respectively. The construct validity of the draft questionnaire was assessed using exploratory and confirmatory factor analysis from pilot-study data. The reliability (internal consistency and composite) of the final version of the questionnaire was then assessed. Statistical analysis was carried out using SPSS version 20 (IBM Corp., Armonk, NY) and SPSS AMOS version 18.0 for confirmatory factor analysis. Results: Two factors were extracted using an exploratory factor analysis. The confirmatory factor analysis confirmed that the two-factor structure, which contained 10 items in total, was the best fit to the data. The Cronbach’s alpha was good (0.81 and 0.77 for Factors 1 and 2, respectively). The composite reliability and discriminant validity of the final model were confirmed. Conclusion: We confirmed the validity and reliability of this questionnaire, making it a useful measure of attitudes towards conducting research among nurses.

Keywords

Attitude; Questionnaire; Nurse; Reliability and validity; Research; Malaysia

Introduction

Research is vital to nursing practice and nurses’ career development. [1,2] The application of research and scientific evidence will ensure that nursing care is appropriate, safe, and of high quality. In the current era of evidence-based practice, it is imperative that nurses conduct credible research that aims to ultimately improve nursing practice. In a review of 13 studies, it has been shown that organization that incorporated research function into their approach had better health care performance. [3]

Nurses’ desire to engage in research is widely influenced by their attitude towards research. [4] Indeed, there are a considerable number of studies examining the attitudes of nurses and what views serve as barriers toward the utilization of research in their practice. [5-7] For instance, in a study involving more than a thousand Swedish nurses, a positive relationship was proven between positive attitudes towards research and research use. [7] There was also a definite link to the nurses’ perceived competence and ability to interpret research results. Despite a huge involvement of nurses in research globally, the research culture, however, remains new to the nursing profession in Malaysia, [8] and research activities have not been given sufficient emphasis during nursing training. Furthermore, only nurses who go on to obtain a degree or doctorate receive the opportunity to become involved in a research project.

As the local nurses are still at a comparative stage of infancy with regard to nursing research, it is timely now to assess their attitude towards conducting research. The result may serve as a basis for future improvement to engage more nurses in research.

However, many studies on the nurses’ attitude towards research were using the instrument that was focusing on measuring and assessing research utilization in nursing practice. [9,10] The published instruments to measure nurses’ attitude towards conducting research are scant. Thus, this study aimed to develop a Nurses’ Attitude towards Conducting Research Questionnaire (NA2CRESQ), and to evaluate its validity and reliability.

Subjects and Methods

Development of the questionnaire

Item development

A draft of the NA2CRESQ was developed in English through a review of the available literature. We primarily employed information from literature reviews to generate items. Overall, the first draft comprised of 37 items assessing attitudes; each item was rated on a 5-point Likert scale, ranging from 1 (strongly disagree) to 5 (strongly agree). This included seven items with negative phrasing, which were reverse scored before the data analysis. The total score is the sum of all item scores, with higher scores indicating more positive attitudes towards conducting research.

Content and face validity

To assess the content validity, we gave the first draft of the questionnaire to a panel of experts. The expert panel comprised the head of the nursing division, three nurse supervisors, and two experts in questionnaire design. The feedback and opinions of experts have been shown to be useful in measuring and improving the content validity of instruments. [11] All six experts were asked to rate the 37 items of the draft questionnaire concerning their relevance on a 4-point scale (1=test not being relevant; 2=somewhat relevant; 3=quite relevant and; 4=highly relevant). From here, the content validity index (CVI) of each item was calculated by dividing the number of experts that gave the item a 3 (quite relevant) or a 4 (highly relevant) by the total number of experts. [12,13] The cut-off for an acceptable item-level CVI was >0.78. [13] The face validity was then tested by administering the draft questionnaire to 11 nurses. The face validity is used to determine respondents’ perception of the appropriateness of each item. The 11 nurses were asked to assess the clarity and appropriateness of the phrasing of items, as well as the response format. Modifications were performed in line with the feedback of both the experts and the face validity sample, which generated the second draft of the questionnaire. This question was then piloted to the targeted group to collect the data for exploratory and confirmatory factor analyses to determine the underlying construct of the NA2CRESQ.

Pilot study and subject recruitment

The sample size of this pilot study was estimated based on our use of the exploratory factor analysis (EFA) as the main statistical test to elucidate the construct of the NA2CRESQ. Employing the calculation proposed by Bujang et al. [14] a minimum sample size of 3 respondents per item is necessary to produce a valid construct with relatively high loadings and communalities for a questionnaire with 29 items that utilize a 5-point Likert scale. Hence, the sample size required was 87. We added 20% to the current sample to account for possible missing values or nonresponses; therefore, the required sample size was 109.

A convenient sampling was chosen to achieve sample size whereby the second draft of the questionnaire was distributed to nurses that attending a state nursing program. An information sheet was attached to each questionnaire with a short description of the study aims and instructions on how to complete the questionnaire. Of the 238 nurses that attended this program, the questionnaire was answered voluntarily by 174 (response rate 73.1%). In addition to items assessing attitudes toward conducting research, the questionnaire also contained items for obtaining demographic information such as age, ethnicity, and level of education, working experience, and previous exposure to research.

Exploratory factor analysis

The data from this pilot study were subjected to an EFA using SPSS Statistics 20 (IBM Corp., Armonk, NY). To assess the sampling adequacy of these pilot data, we used the Kaiser- Meyer-Olkin (KMO) test and Bartlett’s test of sphericity. A KMO value of more than 0.7 indicates sampling adequacy and a significant Bartlett’s test of sphericity (p< .05) indicates that there are worthwhile correlations among the items according to the correlation matrix. We used a screen test and the Kaiser criterion (i.e., eigenvalues should be greater than 1) to determine the number of factors to retain. Furthermore, with the assumption that all items were allowed to correlate with each other, an oblique (promax) rotation was used to optimize the loading factors of each item in the extracted factors. We retained only items with factor loadings of ≥.5, as recommended by Hair et al. [15] These remaining items were subsequently subjected to the confirmatory factor analysis (CFA).

Confirmatory factor analysis

The CFA was used to determine the construct validity of the factor structure derived from the EFA using SPSS AMOS 18.0. Specifically, we employed structural equation modeling to confirm the factor structure. All of the tested models employed maximum likelihood estimation. The fit of the model was evaluated using eight indices: the chi-square (χ²) and degree of freedom (df), Akaike information criterion (AIC), expected cross-validation index (ECVI), Tucker-Lewis index (TLI), comparative fit index (CFI), goodness of fit index (GFI), root mean square error of approximation (RMSEA), and standardized root mean square residual (SRMR). A statistically significant chi-square test (< .05) indicates that the model has a poor fit to the data. For the AIC and ECVI, smaller values suggest a better fit. [16] For the TLI, CFI, and GFI, values .95 or greater are considered evidence of good model fit. [17] The RMSEA ranges from 0 to 1, with a smaller value indicating a better model fit. An acceptable fit was an RMSEA value of 0.06 or less. [18] The SRMR values also range from 0 to 1, with the best-fitting models obtaining values of less than 0.08. [18] To achieve adequate model fit, we determined whether items satisfied the cutoffs for several parameters. Specifically, we eliminated those items with low factor loadings < 0.5, modification indices (MI) of 10 or higher, and a standardized residual covariance of greater than 2.58. The models were revised until a good fitting model (i.e., the final model) was obtained.

Reliability

Finally, to measure the reliability of the instrument, we evaluated the internal consistency of the items. [19] Specifically, we determined the Cronbach’s alpha coefficient for each internal item in the final model. A coefficient of 0.7 or higher was indicative of an internally consistent questionnaire. [20] We also assessed the correlations between the factors to confirm whether they might have overlapped in some way. A correlation of ≥.85 is suggestive of overlapping factors. [21] We also calculated the composite reliability (CR), which is used to evaluate the overall reliability of a collection of relatively heterogeneous items. To complete this assessment, we used the average variance extracted (AVE) and shared variance (SV) to measure the discriminant validity of the model. [22] The AVE is the average degree of variation that a latent factor is able to explain the observed items to which it is theoretically related. The SV refers to the amount of variance that one variable (i.e., factor) can explain in another variable (factor). It is calculated by squaring the correlation between the two factors. The CR should be equal to or greater than 0.7 and the AVE should be greater than 0.5, as recommended by Hair et al. [15] If the AVE for each factor is greater than its SV with another factor, then the discriminant validity is supported.

Ethical considerations

This study was approved by the Medical Research Ethics Committee of Malaysia (NMRR-15-529-25670). Prior to the pilot test, the authors explained the purpose of the study to the potential participants (i.e., nurses). The participants were also informed that any information obtained for this study would be handled in a confidential manner. Informed consent was obtained after the participant was satisfied with these explanations.

Results

Content and face validity

The expert panel dropped eight items from the initial 37 items as those items had a CVI of less than 0.78. The remaining 29 items were subsequently subjected to the face validation process. This led to minor changes to the wording of these remaining items. For instance, two nurses suggested changing the term “research specialist” in items C11 and A10, as this term was not commonly used in the local setting. The term was replaced with the simpler term of “research mentor” for item C11 and “supervisor” for item A10, as this made the statements somewhat more comprehensible and relevant.

Pilot-study respondent characteristics

None of the completed questionnaires had missing values. Table 1 show the socio-demographic characteristics of the nurses who participated in this pilot study. Most of the respondents were staff nurses (71.8%) and were around 31 years old or above (71.3%). Only one nurse had a master’s degree in nursing, while the remaining 17.8% and 73% had obtained a bachelor’s degree and a diploma in nursing, respectively. Almost 44% of the nurses had attended a research-related course or workshop, while 59.2% claimed that they had never performed any research before.

Table 1: Socio-demographic characteristics of participants in the pilot study.

Exploratory factor analysis

The KMO value was 0.779, while Bartlett’s test of sphericity was significant (χ²=1821.09, p< .001). Fourteen items were loaded onto each factor, while one item did not load onto either factor. Factor 1 had relatively low factor loading (less than 0.5) for seven items, while Factor 2 had six such items. These items— both those with low factor loadings and those that did not load onto either factor—were removed from the model, after which we retested the model for EFA. This retest extracted the same number of factors, although another two items were removed from Factor 2 because of their low factor loadings. Finally, we retested the model with the remaining 13 items; seven items loaded to Factor 1 and six items to Factor 2 [Table 2]. At this point, we assigned names to each factor: Factor 1 was “support and opportunity,” while Factor 2 was “research values.” These two factors accounted for a total of 43.1% of the variance.

Table 2: Items with their factor loadings and extraction communalities in the exploratory factor analysis (N = 174).

Confirmatory factor analysis

Model 1: The results of the initial model, wherein all 13 of the remaining items from the EFA were included, did not have a satisfactory fit, as follows: χ²=135.09 (df=64; p< .05), AIC=189.09, ECVI=1.09, TLI=0.88, CFI=0.90, GFI=0.90, RMSEA=0.08, SRMR=0.08). All of the items in this model had factor loadings of greater than 0.5, and none had an MI of more than 10. However, item A11 had a high standardized residual covariance value of 2.75 with item C7. Thus, item A11 deleted and the model retested.

Model 2: This model was the initial model with item A11 removed. Although this led to a clear rise in the model fit from the initial model, the fit remained poor overall (χ²=103.14, df=53, p< .05; AIC=153.4; ECVI=0.89; TLI =0.90; CFI=0.91; GFI=0.92; RMSEA=0.07; SRMR=0.07). In Model 2, item C11 had a standardized residual covariance value of 2.74 with item C7. This indicated that item C11 had to be removed to improve the model fit.

Model 3: Model 3 was formed by removing item C11 from the initial model. The model fit was improved somewhat compared to Model 2. Additionally, based on the MI, the covariance between e9 and e11 was relatively high. Thus, the error covariance was added between items A15 and C14 (which represented by the measurement errors of e9 and e11, respectively) to form Model 4.

Model 4: The model fit for Model 4 was much improved when compared with Model 3. However, by adding the error covariance between A15 and C14, item C14 was found to have a low factor loading in Model 4. Thus, it was removed to form Model 5. We also dropped the error covariance between items A15 and C14 because of the exclusion of item C14.

Model 5: Model 5 contained ten items. In this model, none of the items had low factor loadings, an MI of more than 10, or a standardized residual covariance of more than 2.58. Furthermore, this model had acceptable fit according to the adopted criteria, and thus was accepted as the best model (��2=41.49, df=34, p=.18; AIC=83.49; ECVI=0.48; TLI=0.98; CFI=0.98; GFI=0.96; RMSEA=0.04; SRMR=0.05). The results of the goodness-of-fit indices for all models above are shown in Table 3.

Table 3: Goodness-of-fit indices and their comparisons for the confirmatory factor analysis.

Reliability analysis

Table 4 shows the internal consistency results of the final two factor, 10-item questionnaire. All of the correlations between individual items and the total scale ranged from 0.34 to 0.56, indicating that the items in the final model had the ability to differentiate respondents with good attitudes from those without positive attitudes. The Cronbach’s alpha coefficients (0.80 for the “support and opportunity” factor [Factor 1] and 0.77 for the “research values” [Factor 2]) also denoted good internal consistency. Deleting any one of the items did not result in a higher alpha value.

Table 4: Internal consistency analysis of the final two-factor, 10-item model.

Final model description

The final model of the NA2CRESQ is shown in Figure 1. The final two-factor model contains ten items, with factor loadings ranging from 0.53 to 0.77. All of the loadings were significant at p< .001, thus indicating that all of the items were useful for explaining their corresponding factor. The low correlation between the two factors of the final model (r=0.36) indicated that these factors did not overlap. The CR of both factors was well above the acceptable limit of 0.7 (0.81 and 0.77 for Factors 1 and 2, respectively). Additionally, the AVE for the “support and opportunity” factor was 0.50, while that for the “research values” factor was 0.40, making it slightly lower than the minimal threshold. Nevertheless, the SV of each factor was found to be less than the AVE, indicating that the model had good discriminant validity [Table 4]. The final version of the questionnaire is available as in the supplementary file.

Figure 1: Confirmatory factor analysis of the final two-factor, 10-item model of NA2CRESQ. Rectangles represent items (observed variables); large ovals represent factors (latent variables); and small ovals represent the measurement error. Arrows between each factor and items represent a regression path and the number represents the standardized regression weight of that path (used to determine the factor loading of the respective item). The arrow between the measurement error and the item represents the measurement error term. The bidirectional arrow and its number represent the correlations between the two factors of the model.

Discussion

Our results suggest that the NA2CRESQ would be a valid and reliable assessment tool for assessing nurses’ attitude towards conducting research. Specifically, we assessed the content validity using an expert panel; the face validity by administering the questionnaire to a sample of nurses with similar characteristics as the target group; the construct validity, including the convergent and divergent validity, by EFA and CFA; and the reliability by measures of internal consistency and composite reliability.

Content validation is an essential prerequisite for the development of an assessment instrument. It is used to assess the relevance and appropriateness of the construct measured by the instrument to the study objectives. [23] The content validity of the initial draft of the questionnaire was obtained via the consensus of six members of an expert panel. This coincides with the recommendations of De Von et al., [24] who reported that at least three experts are needed for adequate assessment of the content validity. Furthermore, for the determination of the face validity, the comprehensibility of the items according to the target group is important, especially for instruments that are developed for a particular population. Using nurses who have similar characteristics with the target group, we altered the items slightly, as mentioned in the Results section. These modifications helped make the new assessment tool more representatives of nurses before we used it in the pilot test.

The validation and reliability process yielded a final model (Model 5) with two factors and ten items, and that displayed the best fit for our sample. Based on this final model, nurses’ attitudes towards conducting research can be assessed by summing the item scores. The possible minimum total score is 10, while the possible maximum total score is 50. Higher total scores reflect a more positive attitude towards conducting research. The NA2CRESQ contains two negatively phrased statements that are reversed score. These negative items assessed respondents’ interest in conducting research, and the importance of research for their career.

This study has a notable limitation. Specifically, the development of this questionnaire was based on data gathered from a small population of nurses from a single state. Thus, this questionnaire may have limited generalizability to other nurse populations. Nevertheless, because of our step-by-step assessment of the validity and reliability, we believe that this limitation is minimal. We nevertheless encourage future studies to test the fitness of this two-domain model in various nurse populations. If a similar factor structure is obtained, confidence in the factor structure revealed in this study and the generalizability of the results will increase.

Conclusion

In conclusion, the results of this study supported the utility of the NA2CRESQ as a brief and reliable scale for measuring attitudes towards conducting research among nurses in a Malaysian population. The extracted model could provide a baseline for devising strategies that aim to increase nurses’ participation in research. We recommend that other research groups utilize this questionnaire to confirm the model in using their datasets.

Conflict of Interest

All authors disclose that there was no conflict of interest.

REFERENCES

1. Fawcett TJN, McCulloch C. Pursuing a career in nursing research. Nurs Stand 2014; 28: 54-58.
2. Cole A. Why nursing research matters. Nurs Stand 2016; 30: 16-18.
3. Hanney S, Boaz A, Jones T, Soper B. Engagement in research: an innovative three-stage review of the benefits for health-care performance. Health Serv Deliv Res 2013; 1.8
4. Akerjordet K, Lode K, Severinsson E. Clinical nurses' attitudes towards research, management and organisational resources in a university hospital: part 1. J Nurs Manag 2012; 20: 814-823.
5. Gerçek E, Okursoy A, Dal NA. Awareness and attitudes of Turkish nursing students towards research and development in nursing. Nurse Educ Today 2016; 46: 50-56.
6. Haslinger-Baumann E, Lang G, Müller G. Influence and correlation of attitude, availability and institutional support to research implementation in nursing practice – Results from an exploratory, cross-sectional quantitative study. Pflege 2015; 28: 145-155.
7. Nilsson Kajermo K, Alinaghizadeh H, Falk U, Wändell P, Törnkvist L. Psychometric evaluation of a questionnaire and primary healthcare nurses’ attitudes towards research and use of research findings. Scand J Caring Sci 2014; 28: 173-185.
8. Birks M, Francis K, Chapman Y, Mills J JP. Supporting the evolution of a research culture among nurses in Malaysia. Aust J Adv Nurs 2009; 27: 89-93.
9. Frasure J. Analysis of instruments measuring nurses’ attitudes towards research utilization: a systematic review. J Adv Nurs 2008; 61: 5-18.
10. Squires JE, Estabrooks CA, O’Rourke HM, Gustavsson P, Newburn-Cook CV, Wallin L. A systematic review of the psychometric properties of self-report research utilization measures used in healthcare. Implement Sci IS. 2011; 6: 83.
11. Jensen MP. Questionnaire validation: A brief guide for readers of the research literature. Clin J Pain. 2003; 19: 345-352.
12. Lynn MR. Determination and quantification of content validity. Nurs Res. 1986; 35: 382-385.
13. Polit DF, Beck CT, Owen SV. Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Res Nurs Health. 2007; 30: 459-467.
14. Bujang MA, Ghani P, Soelar SA. Sample size guideline for exploratory factor analysis when using small sample: taking into considerations of different measurement scales.  International Conference on Statistics in Science, Business and Engineering (ISCCBE); Langkawi 2012.
15. Hair JF, Black B, Babin B, Anderson RE. Multivariate data analysis. 7th edition. Upper Saddle River, NJ: Pearson Education; 2016. 900 p.
16. Akaike H. A new look at the statistical model identification. IEEE Trans Automat Contr. 1974; 19: 716-723.
17. Hooper D, Coughlan J, Mullen MR. Structural equation modelling: Guidelines for determining model fit. Electronic Journal of Business Research Methods 2008; 6: 53-60.
18. Brown TA. Confirmatory factor analysis for applied research. 2nd edition. New York: The Guilford Press; 2015; pp: 35-130.
19. Windsor R. Evaluation of health promotion and disease prevention programs: Improving population health through evidence-based practice. 5th edition. New York: Oxford University Press; 2015. 312 p.
20. Tavakol M, Dennick R. Making sense of Cronbach’s alpha. Int J Med Educ 2011; 2: 53-55.
21. Tabachnick BG, Fidell LS. Using multivariate statistics. 6th ed. Upper Saddle River, NJ: Pearson Education; 2013. 983 p.
22. Carter SR. Using confirmatory factor analysis to manage discriminant validity issues in social pharmacy research. Int J Clin Pharm 2016; 38: 731-737.
23. Polit DF. Assessing measurement in health: Beyond reliability and validity. Int J Nurs Stud 2015; 52: 1746-1753.
24. DeVon HA, Block ME, Moyle-Wright P, Ernst DM, Hayden SJ, Lazzara DJ, et al. A psychometric toolbox for testing validity and reliability. J Nurs Scholarsh. 2007; 39: 155-164.