Annex – 1. Monitoring Matrix methodology

Operationalization and data collection

The Monitoring Matrix on Enabling Environment for Civil Society Development is a detailed theoretical framework based on international human rights and freedoms and regulatory practices of European countries and the EU. The framework is built around three core areas: Basic Legal Guarantees of Freedoms; Framework for CSOs’ Financial Viability and Sustainability; Government – CSO Relationship, each divided in sub-areas. The areas are elaborated by standards, which are further specified through legal and practice indicators.

The legal indicators are measured by coding the presence or absence of rules, costs, procedures and obligations enshrined in legal regulation (primary and secondary) and policy frameworks enacted in the respective countries. To assure standardization and comparability of the data gathering process regarding the practice indicators, country researchers follow a methodology plan in which each of the 80 indicators are further operationalized in concrete mandatory and additional data types (i.e. operationalized dimensions of a practice indicator) to be reported across the countries. The mandatory data types tap into the core building blocks of a practice indicator as described in the Monitoring Matrix Toolkit. They mandate the reporting of optimal information without which one could not be able to evaluate the respective indicator. The additional data (operationalized dimensions) specified for each practice indicator are reported if country researchers want to deepen and further illustrate specific practice indicator (e.g. via case study; see next section on country-specific notes on methodology).

The data gathering strategy for the practice indicators is tailored to match the mandatory data types specified in the methodology plan. For each indicator there is a clear guidance on the data gathering strategy (instruments and sources) which should be utilized by country researchers. The specified data gathering instruments and sources follow an implicit hierarchy, in which publicly available factual data (e.g. official statistics) are the most important source of data for assessing practice indicators, followed by survey data from civil society organizations, which in turn is followed by relevant secondary sources (e.g. from CSOs reports, Ombudsman and media). Finally, at the end of the hierarchy are interview data, being subjective type of data, which covers smaller groups of respondents.

The primary factual data and secondary data are gathered through desktop research. Following the data gathering strategy, country researchers utilize three core data gathering instruments: Freedom of Information requests (FoI), survey questionnaire and interview topic guides. The questions in the data gathering instruments are tailored to match the mandatory data types (operationalised dimensions) of each practice indicator. The Freedom of Information requests (FoI requests) are used by researchers when public information and statistics on the state of civil society and their environment (primary factual data) are not readily and publicly available. The researchers can draw from a detailed bank of FoI questions tailored to match the operationalized practice indicators.

The survey questionnaire collects information on civil society organizations’ experiences and perceptions on the key aspects of the enabling environment for civil society for the year 2019. The organizational survey includes 50 questions matching the mandatory data types (operationalized dimensions) on basic rights and freedoms, organizational and financial sustainability and civil society’s cooperation with the state. The questionnaire dominantly consists of closed questions, and fewer follow-up open questions which require the respondents to elaborate on their experience. The same questionnaire is implemented across all countries, and only the formulation of few items is slightly adapted to the concrete country context to assure questions are understood by respondents. The survey is sent to lists of formal CSOs compiled and updated by country researchers on the basis of available registers or other alternative lists of active CSOs in the country (for more information see section on country-specific notes on methodology). The data collected from the survey is confidential and used strictly on anonymous basis. Individual responses are not connected to the organizations which answered the survey and reported only in an aggregate form.

The interview topic guides include questions which – similar to the survey questionnaire – matches specified mandatory data types and are used in all countries. In addition to these core questions, researchers formulate additional questions that capture contextual developments in the country. The four topic guides are used in semi-structured interviews with the following groups of respondents: representatives of associations of journalists and media professionals; representatives of organizations of volunteers; representatives of the institution or mechanism for CSO cooperation and representatives of informal civil society groups (e.g. citizen initiatives, social movements and online initiatives).

Data analysis and interpretation

To analyze and interpret the data, country researchers use a unified data collection template which provides the indicators description (including the mandatory and additional data types for the practice indicators) and five category descriptions ranging from fully enabling to fully disabling environment provided under each indicator. The five category descriptions are specified for each legal and practice indicator in the Monitoring Matrix Toolkit, to enable researchers – based on the reported data – to choose one code (score) which most accurately summarizes the state of enabling environment concerning the respective indicator. In a first step, the researcher reports the required data types collected through different sources in the template box. For example, they report factual data from primary sources complemented with descriptive statistics or cross tabulations based on survey data. In a second step, they choose one of five category descriptions specified for the respective indicator which best illustrates the reported data. The categories enable unified comparison of findings on the level of indicators across all country reports.