Evaluating conceptual models and modeling languages

Transcription

1 2nd AQEMO Workshop Invited talk Evaluating conceptual models and modeling languages The quest for sensible evaluation criteria and methodic guidance Stefan Strecker Enterprise Modelling Research Group University of Hagen (FernUniversität in Hagen) Braunschweig, Germany,

2 Preamble : Personal motivation Why think about sensible criteria for evaluating conceptual models and modeling languages? fundamental to conceptual modeling better understand its foundations & limitations help and educate novice modellers 2

3 Preamble : Conceptual modeling as activity Viewed as an activity, conceptual modeling involves an intricate array of cognitive processes and performed actions including abstracting, conceptualizing, associating, contextualizing, visualizing, interpreting & sense-making, judging & evaluating, and, in group settings, communicating, discussing and agreeing. 3

4 Preamble : Teleological notion of a c.m. Conceptual models are purposeful (re-)constructions of statements about reality (teleological notion of model ; not images or pure representations of reality) 4

5 Preamble : Terminology model quality (in German eher Güte, not Modellqualität ) model adequacy / appropriateness (in German Modellangemessenheit ) evaluation criteria + methodic guidance 5

6 Objectives stimulate discussion rather than present results idiosyncratic selection of aspects pointed & rough treatment 6

7 An attempt to retrace aspects of the discourse Singular criteria proposals Driven by practical needs Fundamental reflections Framework development Driven by scientific considerations Fundamental reflections Wand/Weber 1990, 1993 Batini ea Batra/Davis 1992 Debates & refinements Ontology turn Reflecting & revisiting Empirical turn Ontology mainstream Rare reflecting Mendling ea Lindland ea Frank 2000, 2007 Evermann/Fang 2010 Moody/Shanks 1994 Poels ea zur Mühlen ea Reingruber/Gregory 1994 Moody/Shanks 2003 Milton ea Kung 1983 Levitin/Redman 1994,1995 Schütte vs. Kaschek 2000 Fettke ea Yeh ea Krogstie ea Nelson ea Claes ea. 2012, 2015 Roman 1985 Falkenberg ea Wyssusek Krogstie et al Juhn/Naumann 1985 Schütte/Rotthowe 1998 Gemino/Wand 2003,2005 Figl ea Stamper Frank 1997,1998 Moody 2009 Moreno-Montes ea

8 Singular criteria : Examples Lindland et al list 19 evaluation criteria compiled from sources dating back to 1983: Annotated, Appropriate, Complete, Conceptually clean, Consistent, Constructable, Correct, Executable, Expressive economy, Formal, Incompleteness tolerant, Minimal, Modifiable, Precise, Testable, Traceable, Unambiguous, Understandable, Verifiable Several alternative proposals, e.g., Batini ea. 1992: Expressiveness, Simplicity, Minimality, Formality, Graphic completeness, Ease of reading Moody/Shanks 1994: Simplicity, Completeness, Flexibility, Integration, Understandability, Implementability 8

9 Frameworks : Examples Semiotic Quality Framework (SEQUAL), Lindland et al. (1994), Krogstie ea. 1995, Grundsätze ordnungsmäßiger Modellierung (GoM), Becker et al. (1995); Schütte (1998) Bunge-Wand-Weber-Model (BWW) - represention theory, Wand/Weber (1990,1993,1995) Quality in Conceptual Modeling, Shanks/Darke (1997) Qualität von Datenmodellen, Maier (1998) Seven Process Modeling Guidelines (7PMG), Mendling et al. (2010) The 3QM Framework, Overhage et al. (2012) Conceptual Modeling Quality Framework (CMQF), Nelson et al. (2012) 9

10 Fundamental reflections : Examples What is the nature of a conceptual model? (e.g. Mahr, Thalheim, Frank) How to discoursively evaluate conceptual models? (e.g. Fettke,Loos,Frank) How do conceptual models relate to reality? (e.g. Schütte-Kaschek debate) Question epistemological / ontological assumptions (e.g. Wyssusek, Frank, Schütte, Kaschek) 10

11 Attempt to outline universe of discourse Semantics (meaning-making) Communication Sciences Cognitive Sciences (e.g. mental model), Sociolinguistics Konzeptuelles Modell Pragmatics (use context) Interpretation Rekonstruktion Aesthetics Modellbetrachter 2 Modellierer, Modellbetrachter Rekonstruktion Abstraktion Sprachkonzept (Beispiel): Entitätstyp Linguistics, Semiotics, Philosophy of Language, Diskurswelt Abstraktion Handlungssystem (Domäne) Reality Abstraktion Epistemological and ontological presuppositions Begriffe Namen Sätze (Wirklichkeit) Evaluations research Modelltheorie based on Strecker/Frank

12 Thoughts on present state of discourse -1- terminological variety & ambiguity / imprecision (repeatedly stated elsewhere) criteria often function as heuristic postulates (e.g. The final representation must accurately and completely capture the meaning of the physical domain ) but operationalization of judgement (rather than measurement) challenging (discoursive?) 12

13 Thoughts on present state of discourse -2- call for more empirical research (e.g. Moody 2015) fundamental epistemological obstacles when evaluating language, e.g. bias paradox (e.g. Frank 2000) methodological issues, e.g. w.r.t. prior mastery of a modeling language 13

14 Thoughts on present state of discourse -3- methodic guidance on process of evaluating? neglect of quality of modeling process? product quality vs. process quality? (e.g. Moody 2005, pp. 247ff.) fundamental reflections fragmented, scattered, non-consolidated, often not considered (e.g. reflections in Krogstie ea. 1995) 14

15 Considerations for future research -1- thoughtfully consider fundamental reflections (e.g. in framework development) make assumptions & basic positions explicit and build consistently on them consolidate essential reflections? 15

16 Considerations for future research -2- simplified, pragmatic approaches for teaching (and practitioners?) including pragmatic yet sensible methodic guidance incorporating research results 16

17 Stefan Strecker Enterprise Modelling Research Group University of Hagen, Germany Web:

18 References Full list of references: p Frank, U. (1997): Ein Bezugsrahmen zur Beurteilung objektorientierter Modellierungssprachen - veranschaulicht am Beispiel von OML und UML, Arbeitsberichte des Institut für Wirtschaftsinformatik der Universität Koblenz-Landau, Nr. 6, Frank, U. (1998): Evaluating Modelling Languages: Relevant Issues, Epistemological Challenges and a Preliminary Research Framework, Arbeitsberichte des Institut für Wirtschaftsinformatik der Universität Koblenz-Landau, Frank, U. (2000): "Modelle als Evaluationsobjekt: Einführung und Grundlegung" in Häntschel; Heinrich (Hrsg.): Evaluation und Evaluationsforschung in der Wirtschaftsinformatik, Oldenbourg, Frank, U. (2007) Evaluation of Reference Models. In: Fettke; Loos: Reference Modeling for Business Systems Analysis, Juhn, S., Naumann, J. (1985): The Effectiveness of Data Representation Characteristics on User Validation. In: Proceedings of the International Conference on Information Systems 1985, Krogstie, J., Sindre, G. & Lindland, O. I. (2013). 20 Years of Quality of Models.. In J. A. B. Jr., J. Krogstie, O. Pastor, B. Pernici, C. Rolland & A. Sølvberg (ed.), Seminal Contributions to Information Systems Engineering, Kung, C. H. (1983). An analysis of three conceptual models with time perspective. In Olle et al., editor, Information Systems Design Methodologies: A Feature Analysis, pages