1 / 33

COST WATCH 2006 O verview of national transport survey Pargue 11-13 October

COST WATCH 2006 O verview of national transport survey Pargue 11-13 October. The influence of verbal and non verbal language manipulations on data quality. Future work. Conclusion. The test for the data. collection, results and discussion. The sample.

lerato
Download Presentation

COST WATCH 2006 O verview of national transport survey Pargue 11-13 October

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. COST WATCH 2006Overview of national transport surveyPargue 11-13 October The influence of verbal and non verbal language manipulations on data quality Politecnico di Torino - Maria Lapietra & Cristina Pronello

  2. Future work Conclusion The test for the data collection, results and discussion The sample Analysis of the questionnaire instrument design Objective Structure of research

  3. Objective Structure of research Objective of this paper To define some guidelines for questionnaire design Objective of the research To define some guidelines to obtain high quality data from transport surveys

  4. Preliminary planning Choice of survey type Sampling scheme Choice of survey method(s) Survey instrument design Pilot survey SURVEY Data recording Codify of open-ended questions Data revision Data analysis Presentation of outcome Objective • “Core” questions • Standardisation of categories • Standard wordings • Start and end times for travel day • Multi-tasking of activities • In-home activities • Question ordering • Instrument design • Mailing materials

  5. Analysis of the questionnaire instrument design Objective Structure of research

  6. The respondents to self-administered questionnairs, above all trough the web, are often subjected to question where thay have to choice one or more answers among those given. From previous studies emerged that the forced choices increase the percentage of affirmative answer while it has not been explained yet how the order of the replies can influence the respondents. The scalar question are frequently used in the transport survey to determine the attitudes, the opinions, and the behaviours of the user. It is very Important to standardize the way in which the answers, either numeric or verbal are depicted, to have high quality and comparable data In the questionnaire design, a great care has to be taken of the position of the instruction because it is the first and fundamental step for the right drawing up of the self-admiministered questionnaires. Analysis of the questionnaire instrument design • Scalar questions • Skip instructions • Check-all-that-apply format

  7. The sample Analysis of the questionnaire instrument design Objective Structure of research

  8. The sample Sample: The university staff working in the department of transport and students attending transport courses. 52 units • The innovation in the design of the tests is that: • to each unit all the version of the test have been given, ad hoc distributed along the questionnaire, so that a comparison among the answers of each person has been possible; • the test has been done in the same time to investigate if the changes of the version could be influence the respondents in a short time range.

  9. On the basis of the studies carried out from other researches, a survey has been designed, to test different form of languages using a self-administered questionnaires I was present during the writing out of the questionnaire, to garantee the right drawing up and to avoid that people go up and down trough the questionnaire and correct the answers The test for the data collection, results and discussion The sample Analysis of the questionnaire instrument design Objective Structure of research

  10. Question (1= never, 6 = always) 1 2 3 4 Question 5 Never 6 Rarely Sometimes Often Nearly always Always The test for the data collection SCALAR QUESTIONS TEST N.1 Numeric Fully labelled In the test number one the same question was presented, in different point of the questionnarie, in two different version:

  11. In this experiment, as seen in other researches, the respondents give marks significantly more poositive in the format using labels expressed by words Results and discussion SCALAR QUESTIONS TEST N.1 Numeric Fully labelled Mean = 3.60 t = -4.435 p = 0.000 Mean = 4.22 The respondents tend more to choice the extreme answers when these are emphasized by words. 22% 8% The people who changed their mark are the 58%

  12. Question Never Rarely Sometimes Often Nearly always Always The test for the data collection SCALAR QUESTIONS TEST N.2 Linear horizontal Linear vertical Question (1= never, 6 = always) Always Nearly always Never Rarely Sometimes Often In the test number 2 the effect of the linear format with non linear format has been compared Non Linear horizontal Non Linear vertical Question Question Nearly always Never Rarely Sometimes Never Sometimes Nearly always Always Often Always Rarely Often

  13. This fact because the respondents were used from the previous questions to read vertically the answers; a non linear structure, horizzontally organaized, has made a muddle Confirming the results coming from other researchers who observed that people have difficulty in reading scales organized in non linear way Results and discussion SCALAR QUESTIONS TEST N.2 t = 1.386 p = 0.172 Linear horizontal Linear vertical Mean = 4.191 Mean = 4.383 t = -0.443 p = 0.660 t = -2.934 p = 0.005 t = -0.948 p = 0.348 t = -1.430 p = 0.160 t = 1.855 p = 0.070 Non Linear horizontal Non Linear vertical Mean = 4.553 Mean = 4.319 The only significant difference among the average marks occur between the linear version and the non linear horizontal version 27.7% of the respondents has given equal answers in the linear ver sion and different answers in non linear version

  14. The test for the data collection SCALAR QUESTIONS TEST N.3 Number scale points Question (date un voto da 1 a 7) Question (date un voto da 1 a 5) 1 2 3 4 1 2 3 4 5 5 6 7 Question (date un voto da 1 a 9) Question (date un voto da 1 a 11) 2 3 4 5 6 7 9 8 1 3 2 4 10 5 6 7 8 9 11 1 In the test number 3 the respondents have been subjected to four questions asking the same information using four type of scale having different lenght, ranging from 5, 7, 9 to 11 points.

  15. Significant differences occur between the scales having 5 and 7 points and the scales having 11 points Result and discussion SCALAR QUESTIONS: TEST N.3 Number scale points t = 1.950 p = 0.057 5 points 7 points Mean = 7.04 Mean = 6.48 t = -3.548 p = 0.001 t = -0.293 p = 0.771 t = -2.947 p = 0.005 t = -1.814 p = 0.076 9 points 11 points t = 2.561 p = 0.014 Mean = 7.15 Mean = 8.42 The marks have been all conducted to a 11 points scale Longer is the lenght of the scales bigger is the mark the respondents give

  16. Question Never Rarely Sometimes Often Nearly always Always The test for the data collection SCALAR QUESTIONS TEST N.4 Polar point Fully labelled Question (1= never, 6 = always) Never 1 2 3 4 5 Always 6 In the test number 4 two different sets are made up. One format with labels expressed by words and the other one with only extreme points expressed using words

  17. Result and discussion SCALAR QUESTIONS TEST N.4 Polar point Fully labelled Mean = 3.224 Mean = 3.041 t = -1.206 p = 0.310 The respondents tend more to choice the extreme answers when these are emphasized by words. 22.5% 2% The people who changed their mark are the 69.4%

  18. The test for the data collection SCALAR QUESTIONS TEST N.5 Balanced Unbalanced Question Question Very satisfied Excellent Very good Somewhat satisfied Good Sufficiently satisfied Sufficiently dissatisfied Fair Somewhat dissatisfied Poor Very dissatisfied Terrible During this test two ways of answer have been given for the same question. The first using a balanced scale between the positive and negative categories, the second using 4/6 positive answers.

  19. Results and discussion SCALAR QUESTIONS TEST N.5 Balanced Unbalanced Mean = 3.320 Mean = 3.040 t = -2.087 p = 0.042 More positive answers 54.9% have answered negatively 30% have answered negatively It is not positive to observe that the mark is in the average the same. This means that the respondents do not have recorded that change of the category of answers.

  20. Question Never Rarely Sometimes Often Nearly always Always The test for the data collection SCALAR QUESTIONS TEST N.6 Answer box Fully labelled Question (1= never, 6 = always) In the test number 6 the couple of format compared the format having the labels of the categories expressed by words with the format with free answers

  21. Result and discussion SCALAR QUESTIONS TEST N.6 Fully labelled Answer box Mean = 3.620 Mean = 4.180 t = -3.964 p = 0.001 The respondents tend more to choice the extreme answers when these are emphasized by words. 24% 6% The people who changed their mark are the 60% When the labels are totally described using words the people decisions are more directed towards a certain category while, in the other cases, they are distributed in an uniform way

  22. The test for the data collection SCALAR QUESTIONS TEST N.7 Midpoint No midpoint Question Question Never Rarely Never Rarely Sometimes Sometimes Neither…nor… Often Often Nearly always Nearly always Always Always With this experiment we want to test is adding a neutral point in the middle of a scale the respondents change reply

  23. Results and discussion SCALAR QUESTIONS TEST N.7 Midpoint No midpoint Mean = 4.195 Mean = 2.439 t = -4.174 p = 0.000 The 38.09% of the interwiewed have chosen the neutral point in the format where the central point occurred. Among these respondents (the 38.09%) the 37.5% has chosen the extreme values of the scales in the format without the neutral point.

  24. The test for the data collection SKIP INSTRUCTIONTEST N.8 In the questionnaire 8 types of logical skip have been proposed: 1. Question 1. Question 1. Question 1. Question yes yes yes Skip to 3 yes Skip to 3 no no no no 2. Question 2. Question 2. Question 2. Question yes yes yes yes no no no no 3. Question 3. Question 3. Question 3. Question yes yes yes yes no no no no 1. Question 1. Question Attention: Check for a skip after your answer Attention: Check for a skip after your answer yes If YES skip to 3 no yes If YES skip to 3 1. Question no 1. Question yes Skip to 3 2. Question 2. Question yes no yes yes no 2. Question no no 2. Question yes yes no no

  25. 1. Question Attention: Check for a skip after your answer yes no If YES skip to 3 1. Question yes Skip to 3 2. Question no yes 2. Question no 17.3% 21.2% yes no Results and discussion SKIP INSTRUCTIONTEST N.8 1. Question 1. Question 1. Question 1. Question yes yes yes Skip to 3 yes Skip to 3 no no no no 2. Question 2. Question 2. Question 2. Question 76.9% 73.1% 40.4% 28.8% yes yes yes yes no no no no 3. Question 3. Question 3. Question 3. Question The writing out of the questionnaire has been evaluated analysing the commission errors, calculating how many times the interviewed have answered to questions that they should have had to skip. It can be observed than the most effective method to make understandable the instructions is to put them after the question. yes yes yes yes no no no no 1. Question Attention: Check for a skip after your answer If YES skip to 3 yes 32.7% 28.8% no 1. Question 2. Question yes yes no no 2. Question yes no

  26. The test for the data collection Check-all-that-apply format TEST N.9 Reversing the order Question Question Answer 1 Answer 13 Answer 2 Answer 12 In this test has been asked to choice at most 3 alternative among 13 possible alternatives, then the same question has been asked reversing the order of the alternatives Answer 3 Answer 11 Answer 4 Answer 10 Answer 5 Answer 9 Answer 6 Answer 8 Answer 7 Answer 7 Answer 8 Answer 6 Answer 9 Answer 5 Answer 10 Answer 4 Answer 11 Answer 3 Answer 12 Answer 2 Answer 13 Answer 1

  27. The test for the data collection TEST N.10 Check-all-that-apply format YES/NO format Question Question In this test the respondents had to choice among 8 alternatives; then the same questions has been put asking to mark a yes or no for each alternative YES NO Answer 1 Answer 1 Answer 2 Answer 2 Answer 3 Answer 3 Answer 4 Answer 4 Answer 5 Answer 5 Answer 6 Answer 6 Answer 7 Answer 7 Answer 8 Answer 8

  28. Results and discussion Check-all-that-apply format TEST N.9 Reversing the order 26.9% of the respondents has changed the number of the choices 50% of the interviewed has changed the choice done some pages before. Theversion containing the forced answers has recorded a higher average of affirmative answers TEST N.10 Check-all-that-apply format YES/NO format Mean = 1.423 Mean = 3.135 t = -5.988 p = 0.000

  29. Conclusion The test for the data collection, results and discussion The sample Analysis of the questionnaire instrument design Objective Structure of research The analysys of the results leads to affirm that the manipulations of the visual languages introduce changes on how the respondente process and answers to different types of self-administered questions

  30. Scalar questions: - FULLY LABELLED - BALANCED - LINEAR - 5-7 POINTS - NO NEUTRAL POINT Question Never Rarely Sometimes Often 1. Question Question Nearly always yes no Always YES NO Answer 1 If YES skip to 3 • Skip instructions: Answer 2 AFTER THE QUESTION AND BEFORE THE RESPONDENT FACE THE SKIP OPTION 2. Question Answer 3 yes no Answer 4 Answer 5 Answer 6 • Check-all-that-apply format: YES/NO FORMAT YES/NO FORMAT Answer 7 Answer 8 Conclusion

  31. Future work Conclusion The test for the data collection, results and discussion The sample Analysis of the questionnaire instrument design Objective Structure of research

  32. Future work Non-standard surveys and emerging Techniques: • Stated Preference (SP) surveys • Internet surveys

  33. THANKS FOR YOUR ATTENTION! maria.lapietra@polito.it Politecnico di Torino - Maria Lapietra & Cristina Pronello

More Related