I – Methodology
The methodology chosen by KEIS to evaluate research entities is based on a few basic principles: a collective qualitative peer evaluation, an evaluation which, based on specific criteria, takes into account the variety of the entity’s missions, an evaluation which, for each criterion, is based on observable facts and results, in a qualitative assessment.
– The awareness of the objectives and the point of view of non-academic partners;
– The effective articulation between basic and applied research;
– The openness of academic and non-academic partnerships;
– The ability to adapt and change orientation in response to changes in the environment; the ability to adapt human resources to the strategic
objectives;
– The quality of self-evaluation (e.g. SWOT analysis);
III – Evaluation of multi-, inter and Transdisciplinarity Interdisciplinarity is a challenge for scientific evaluation and assessing interdisplinary entities requires specific procedures:
1. Evaluating interactions between disciplines KEIS distinguishes multidisciplinarity, interdisciplinarity and transdisciplinarity:
– Multidisciplinarity refers to the juxtaposition of disciplines that broadens the field of knowledge by increasing the amount of data, tools and methods available. The disciplinary components, in this case, keep their identities: a particular discipline, which generally steers the others, uses a methodology and tools of one or more other disciplines to address a question or make advances in a research project that is specific to its own disciplinary field.
– Interdisciplinarity refers to the cooperation between several disciplines in common projects. These projects open up research avenues for each discipline. The collaboration brings together data, methods, tools, theories or concepts from different disciplines and the role of the disciplinary 22 components goes beyond their mere juxtaposition. Indicators of this integration include:
– The combinations of models or representations that unify otherwise disparate approaches;
– A genuine collaborations rather than a mere exchanges of services, with coordinated and cooperative organisation;
– The creation of a common language, leading to the revision of initial hypotheses, a broader understanding of the initial scientific issue, the opening of new research avenues and the development of new knowledge.
– Transdisciplinarity refers to a scientific approach that goes beyond disciplinary points of view by offering a single approach to a scientific question. It shows an additional degree of integration in comparison with interdisciplinarity as it leads to the gradual emergence of a new discipline. Examples of transdiciplinarity are systems biology, synthetic biology, artificial intelligence and human ecology.
2. Criteria, observable facts and quality indicators
The evaluation criteria of multi-, inter- or Transdisciplinary entities are not different from those used in the evaluation of monodisciplinary labs. However, specific observable facts are used to assess the multi-, inter- or transdisciplinary dimension research. The level of multi-, inter- or transdisciplinary interaction varies between labs or groups and, within a lab, between various activities. Four types of interaction have been identified scientists of a leading discipline apply methods or use tools obtained from another discipline; scientists belonging to (at least) two different disciplines have a common research object; each group addresses its own questions and shares information and data with researchers of the other group. This type of cooperation is often driven by a common project; scientists belonging to (at least) two different disciplines have come up with a common 23 question, and research findings depend on progress made in each of the disciplines; scientists have a demonstrable experience in the aforementioned type of interdisciplinary projects. They are involved in one or more interdisciplinary networks and contribute to the coordination of a new research community. In addition to this distinction between Types of interaction, the proximity between disciplines should be indicated. The proximity will take into account epistemological factors : proximity of conceptual frames, paradigms and concepts, type of data, observation and measurement instruments used by the different disciplines. It will also assess the degree of interaction between disciplines in publications.
KEIS distinguishes
The following cases: partner disciplines are linked to the same disciplinary group (e.g. SHS 5: “Literature, language, art, philosophy, history of ideas”); partner disciplines fall within two different disciplinary groups (e.g. ST 2: “Physics” and ST 4: “Chemistry”), but within the same field (e.g. ST: “Science and technology” which is different from SVE fields: “Life and Environmental Sciences” and SHS: “Human and Social Sciences”); partner disciplines fall within two different fields (SHS and SVE etc.).
Criterion 1: Scientific production and quality
Observable facts The facts to be taken into account in this criterion include: In the case of multi-, inter- or transdisciplinary productions, it is possible to take into account: the publication of articles, book chapters etc., with multi-, inter- or transdisciplinary confirmed by the co- authors publishing in disciplines distinct from their discipline of origin, or in multi-, inter- or transdisciplinary journals; the oral presentations at multi-, inter- or transdisciplinary 24 conference; other outputs with a demonstrated multi-, inter- or transdisciplinary character;
Quality indicators Quality indicators include:
– The proportion of multi-, inter- or transdisciplinary outputs in the overall lab’s outputs; the type of interaction and proximity between disciplines in these multi-, inter- or transdisciplinary outputs;
– The novelty for the entity of these multi-, inter or transdisciplinary outputs, the originality in the scientific community;
– The impact of these outputs on disciplinary outputs (e.g. the use of new methodology taken from another discipline);
– The coherence between disciplinary and multi-, inter- or transdisciplinary outputs;
– Criterion 2: Academic influence and appeal Observable facts
The facts to be taken into account in this criterion include:
– The success rate when answering to multi-, interor transdisciplinarity calls for proposal;
– The involvement in multi-, inter- or transdisciplinarity networks;
– The participation of lab members in multi-, interor transdisciplinarity editorial committees;
– The visibility, in distinct disciplinary communities, of the conferences to which lab members are invited;
– Visiting senior researchers or postdoctoral students involved in multi-, inter- or transdisciplinary projects of the lab;
Quality indicators
The following quality indicators may be assessed:
– The driving role of the multi-, inter- or transdisciplinarity in the lab’s projects and networks;
– The international recognition of networks; – The reputation and level of scientists , visiting or recruited, who are part of the multi, inter- or 25 transdisciplinary projects;
– The quality of multi-, inter- or transdisciplinarity partnerships (are they productive? Are they reinforced, upgraded over time ?);
– Criterion 3: Interactions with the social,economic and cultural environment
Observable facts
The facts to be taken into account in this criterion include:
– The dissemination or communication of multi-, inter- or transdisciplinary knowledge (exhibitions, stands at cultural events, etc.);
– The reality of reviewing activities in multi-, interor transdisciplinary fields;
– The creation of multi-, inter or transdisciplinary small business and start-ups;
– Elements of local, regional or national public policies based on the lab’s multi-, inter- or transdisciplinary research;
Quality indicators
The following quality indicators may be assessed:
– The leading role of multi-, inter- or transdisciplinarity research in setting up an economic, social or cultural policy or in creating new business and employment, for example;
– The expert role of lab members in multi-, inter- or transdisciplinary business networks or “innovation cluster(s)”;
– The national or international reviewing of multi-, inter- or transdisciplinary applications, journal articles etc.; by lab members;
– Criterion 4: Organisation and life of the entity
Observable facts The facts to be taken into account in this criterion include:
– The existence and implementation of a multi-, inter- or transdisciplinary strategic plan, monitoring tools and procedures to reduce gaps 26 between objectives and achievement;
– The scientific coordination within the lab facilitating multi-, inter- or transdisciplinary project;
– The time and space dedicated to multi-, inter- or transdisciplinary interactions;
– The allocation of resources to multi-, inter- or transdisciplinarity projects;
– The existence of multi-, inter or transdisciplinary job offered by the lab;
Quality indicators The following quality indicators may be assessed:
– The ability to obtain support for the unit’s multi-, inter- or transdisciplinary strategy;
– The way the unit exploits a context favourable to multi-, inter- or transdisciplinarity or adapts to an unfavourable one;
– The adaptation of project management to collaborations between different scientific cultures;
– The dissemination of multi-, inter- or transdisciplinary approaches to the lab’s young researchers;
– The risk-taking and leadership of researchers in the construction of multi-, inter- or transdisciplinary projects;
– Criterion 5: Involvement in training through Research
Observable facts
The facts to be taken into account in this criterion include:
– Multi-, inter or transdisciplinary theses (co-) supervised by lab members; theses associating two doctoral students from different disciplines on the same project;
– Multi-, inter- or transdisciplinary seminars and summer schools;
– Involvement of the entity in multi-, inter- or transdisciplinary or courses; 27
Quality indicators
The following quality indicators may be assessed:
– The type of interaction and proximity between disciplines involved in multi-, inter- or transdisciplinary theses;
– The coherence of common thesis supervision, (the existence, for instance, of work sessions and presentations. where two distinct disciplinary components are involved);
– The recognition of theses by two disciplines;
– The interaction and proximity between disciplines in training, seminars and doctoral schools in which the entity is involved;
– The evolution of training and courses from multi- to interdisciplinarity, or even further to transdisciplinarity;
– The role of multi-, inter- or transdisciplinary training in the career of young doctors and in their job prospects.
– Criterion 6: Strategy and research perspectives for the next five years
Observable facts
The facts to be taken into account in this criterion include:
– The existence of a multi-, inter- or transdisciplinary scientific strategy to meet the following objectives such as, for example:
– Expanding the frontiers of a scientific discipline by opening it up to the approaches and methods of another discipline;
– Foreseeing possible inputs from a discipline into another (methods for observation or acquisition of data, method for representation of knowledge and modelling, formulation of new hypotheses, transfer of paradigms, etc.);
– Assessing the appropriateness of calling on several disciplines to address complex questions of social, economic or cultural importance;
– Creating multi-, inter or transdisciplinary training courses; 28
– The existence of a strategy to achieve these objectives.
Quality indicators
The following quality indicators may be assessed:
– As far as scientific strategy is concerned:
– The relevance of means used to obtain necessary support from external sources;
– The depth of interactions between disciplines and the potential to make multidisciplinarity advance towards interdisciplinarity or even further towards the emergence of a new discipline;
– The ability to obtain support from disciplinary components for multi-, trans- or interdisciplinary research perspectives;
– As far as management is concerned: – The ability to share resources (be they human, financial, material) for multi-, inter- or transdisciplinary research;
– The ability to define expected outputs (the gathering of existing knowledge, the production of new applications, the production of new knowledge, etc.) and their mode of dissemination;
– The ability to call on high-level competencies in each partner discipline of multi-, inter- or transdisciplinary research;
– The ability to gather relevant external competencies to implement multi-, inter- or transdisciplinary research;