LAPS Policy Delphi social research with IdSurvey

The Laboratory for Political and Social Analyses (LAPS, Laboratorio Analisi Politiche e Sociali – website: www.laps.unisi.it) is a centre of excellence for public opinion and social research. It provides the equipment and expertise to undertake different research methods (face-to-face, telephone, online and mixed-mode surveys).

LAPS offers its expertise in questionnaire design and evaluation, data collection, and content analysis. The research team provided scientific advice to several international projects, such as the Transatlantic Trend Survey, the Transatlantic Trends Survey on Immigration and the European Elite Survey.
The team is composed by social scientists Prof. Pierangelo Isernia, Director and senior advisor, Dr. Francesco Olmastroni, Dr. Rossella Borri and Mrs. Manuela Maltese.

The range of research for social science conducted at LAPS goes from local issues to (inter)national politics and society.

LAPS uses IdSurvey software for its social research activities.

IdSurvey provides extreme flexibility in research design conducted with CATI, CAWI, CAPI or mixed modes. In particular, we find the following functionalities very useful:

• The organisation of random and stratified sampling frames.
• General or specific filters for interviewers, contacts, questions, and answer options.
• The wide range of research questions that can be created.
• Randomize all elements of the questionnaire: single pages or groups/blocks of pages, questions, answer options, etc.
• Display answers or information about the contacts during the questionnaire.
• Export the gathered data in different formats such as SPSS, CSV, Access and Excel.

Policy Delphi social research

One of the most challenging research conducted is the Policy Delphi. It was carried out as part of the TRANSWORLD project (Project no. 290454), funded by the EU under the 7th Framework Programme. It focused on the analysis of transatlantic relations and the future of global governance.

Within this framework, the Delphi was designed to primarily explore the following aspects:

• Whether and to what extent a set of global challenges are likely to emerge in the four areas addressed by the project (i.e., international security, global economy, global environment and climate change, human rights and democracy promotion). And what their impact would be, should they occur.
• What policy responses are desirable and/or feasible to such challenges.
• What is the state of governance in each of the four areas and how it is changing.

The final results of the Delphi provided a solid ground for the development of policy recommendations to decision-makers, in order to foster the viability, effectiveness, and accountability of global/regional governance architectures.

The Delphi technique is a forecasting “method for the systematic solicitation of informed judgments on a particular topic”. It is a tool to analyse complex issues and to gather information about the estimated impact of policy decisions.

The TRANSWORLD Policy Delphi has been designed, coordinated and conducted by a research team of the Department of Political, Social and Cognitive Sciences at the University of Siena, which was also responsible for data collection and analysis.
IdSurvey has been used for the implementation of the entire process, which has been articulated in different stages:

• Experts’ selection and recruitment (April – June 2014)
• Round 1 (October 2014-January 2015): “Exploring challenges and policy solutions”
• Round 2 (January 2015- March 2015): “Reaching consensus and identifying areas of polarisation”
• Round 3 (May 2015- July 2015): “Towards policy recommendations”

Experts’ selection and recruitment (April – June 2014)

The TRANSWORLD Delphi social research was designed to survey around 30 experts for each of the four areas of research, namely international security, global economy, global environment and climate change, human rights and democracy promotion.
Since April 2014, a list of around 300 experts’ names from the EU, the US, Turkey, Canada, Brazil, Russia, India and China was collected.

They were then ranked according to a number of criteria, such as:
a) scope of their expertise (international/regional vs. country-based approach, with preference given to the former);
b) seniority (with preference given to senior experts, where ‘senior’ is defined as experience exceeding 17 years, beginning with their first post-graduate or post-doctoral job);
c) number and quality of publications (with preference given to those who published relevant articles on the policy area of expertise).

This sociological research was designed to be representative with regards to country and role.
Accordingly, 266 experts (from 53 to 77 for each area) were shortlisted and formally invited to participate in the exercise between July and October 2014. Table 1 below shows the response rate (number of experts that accepted divided by the total number of invited people) in each of the four domains.

Round 1 (October 2014 – January 2015): “Exploring challenges and policy solutions”

The first round explored experts’ opinions about the main challenges and policy solutions in the four areas under study.

Experts were asked:
1) to list the main challenges in their field of expertise in terms of both impact and likelihood, on a scale from 0 to 100;
2) to express their opinion about the desirability and feasibility of a list of policy solutions to address them, on a scale from 1 to 10;
3) to state their opinion about a set of questions, which were intended to collect information about salient issues concerning the area of concern, on a Likert scale (agree/disagree options);
4) to suggest further challenges and solutions not included in the questionnaire, through comments.

The four questionnaires for the first round (one for each policy domain) have been drafted by relying on the research effort conducted under the broader TRANSWORLD project. Accordingly, working papers, reports and other research material produced by the TRANSWORLD project were analysed, in order to identify and list the core issues and themes to be addressed by the Delphi exercise.

The final versions of Round 1 questionnaires were administered to the experts by the end of October 2014 in CAWI (Computer Assisted Web Interview) mode. Each expert received a unique, personal link to the questionnaire related to his/her own field of expertise. They were given a deadline of two weeks to complete the survey. When needed, reminders were sent after the deadline expired, in order to increase the response rate.

The response rate for Round 1, accounting for those experts who filled the survey, ranged from 70% for Global Economy to 90% for Global Environment and Climate Change (see Table 1).

Round 2 (January 2015 – March 2015): “Reaching consensus and identifying areas of polarisation”

The second round of this social research identified the areas of consensus and those where disagreement existed over the main challenges and solutions. In the questionnaire of Round 2, items were clustered into thematic groups, based on factor analysis and grounded in the relevant theoretical literature.

Experts were then presented with the results of the previous round for each item, reported as average values and standard deviations and introduced by a short commentary. They could also see on the screen their own responses in the first round.

Then, they were invited to either confirm or revise their assessment. They were also strongly encouraged to comment on their personal assessments. Furthermore, a few new items and/or modified some old ones were introduced in the second round questionnaire.

In March 2015 the four questionnaires were sent out for completion to experts, with a deadline of two weeks to complete the survey. After three weeks we reached the 90-100% of response rate in all the four areas (Table 1).

Round 3 (May 2015 – July 2015): “Towards policy recommendations”

This round was designed to draw conclusions about near- and long-term future challenges to the international security, global economy, global environment and climate change, human rights and democracy promotion, and the solutions available to address them.

Experts received a summary report of the main findings and charts and we asked them to provide a commentary on their panel’s estimates. Only when large disagreement still existed, they were asked once again to either confirm or revise their judgments and to specify the reasons for their position.
A few new questions were introduced to verify results, explain controversial issues, and test general concepts in more concrete scenarios.

In Figure 1 there is the introduction to survey, where the task was briefly explained.
Following figures illustrate the case of a reiterated question: experts were preliminarily presented with the summary of their panel’s responses in the previous round (Figure 2). Then, the critical item where consensus was not reached was asked again, while the expert could see panel’s mean response and the standard deviation (in green).

In the box below, he/she could see his/her previous response and he/she could change or confirm it. In Round 3 of this social research, comments to new responses were compulsory.

Figure 1. Example of introduction to survey in Round 3

Figure 2. Summary of responses of previous round

Figure 3. New response to controversial item