Review Process - How to review

Transcription

1 Review Process - How to review Fausto Giunchiglia By Fausto Giunchiglia and Alessandro Tomasi Index: 1. Review Form 1 2. Review Form 2 3. Answer to the Reviews 4. Review Process Hannes Werthner

2 1. Review Form 1 1. Review Form 1 REVIEWER CODE: 1) Title 2) Author(s) 3) Paper Summary [short description of the message and maybe of how it has been developed. What if more than one message or no message?] Hannes Werthner

3 1. Review Form 1 4) Type of Paper/Research described [Chose one of the options below and explain] - Research (with original results) of which kind (Theory, expemerimental,...) - Application (case study,...) - Synthesis of recent advances - Other. Please specify 1. Review Form 1 5) General Ratings [Rate within Bad/Weak/Fair/Good/Excellent] [0/1/2/3/4/5] Put the score and some text motivating your score about: 5a) Relevance (with respect to the reference community) 5b) Originality (incremental, new work,...) 5c) Significance of the work (how big the gap from the state of the art) 3d) Technical soundness 3e) References 3f) Presentation Hannes Werthner

4 1. Review Form 1 6) Technical Soundness [Chose one of the options below and some text motivating the choice] - Technically correct - Minor errors (indicate them) - Major errors (indicate them) - Unsupported claims (provide a detailed explanation) 1. Review Form 1 7) Presentation [Rate within Yes/Somewhat/No] Put the score and some text motivating your score 7a) Are the title and abstract appropriate? 7b) Is the paper well-organized (discuss course and fine grained structure)? 7c) Is the paper easy to read and understand? 7d) Are figures/tables/illustrations sufficient? 7e) Is the English acceptable? 7f) Is the paper free of typographical/grammatical errors? 7g) Is the reference section complete? Hannes Werthner

5 1. Review Form 1 8) General Recommendation [Please, chose one of the options below] - Very strong accept (beautiful paper!) - Strong accept (excellent and important contribution) - Weak accept (good paper, some new interesting ideas) - Weak reject (marginal, weak content, would require a major revision) - Strong reject (unreadable, nothing new, premature, contains major errors) 1. Review Form 1 9) Main Reason for your Decision [For accept choices please indicate one of the options below] - accept because of the originality (good ideas, sound presentation) - accept because of the quality of the proposed synthesis (useful review on recent advances) - other [For reject choices please indicate one of the options below] - reject because it is not relevant for the conference - reject because of the presentation (unreadable, unstructured) - reject because the content is too premature for really making sense - reject because of the lack of originality (results already known, or similar overview already published) - reject because of major errors Hannes Werthner

6 1. Review Form 1 10) Your Level of Expertise (Compared to Level of Others) - I am an expert of the field and know the relevant literature - I understand the problem, I know some of the state of the art - I only have a superficial understandings of the issues 11) Does the paper qualify for the best paper award [Y/N] 12) Comments to the Author(s) [Please, provide here a clear justification of your ratings, in particular with regards to the overall recommendation] 1. Review Form 1 13) Additional Comments to the Author (after circulation of reviews among reviewers) (Can be empty, cannot change previous review) 14) Additional Comments as Answer to Author's Answer and Modifications (Can Be Very Short: - Evaluation Of Author Answer - Value Judgement - Final Score (Possibly Changed)) Hannes Werthner

7 2. Review Form 2 2. Review Form 2 REVIEWER CODE: 1) Title 2) Author(s) Hannes Werthner

8 2. Review Form 2 3) Main Message: Relevance: How relevant is the paper to the workshop? 0: not relevant at all 1: rather not relevant 2: relevant 3: very relevant Technical Quality: What is the technical quality of the paper? 0: really bad 1: bad 2: good 3: really good 2. Review Form 2 Presentation: What is the overall presentation of the paper? 0: really bad 1: bad 2: good 3: really good Overall Ranking: What is your overall recommendation? 0: strong reject 1: reject 2: weak reject 3: weak accept 4: accept 5: strong accept Hannes Werthner

9 2. Review Form 2 Confidence: Reviewer's expertise in the area 0: I know little about this area 1: I know enough about this area 2: I have good expertise in this area Why to accept? What are the most important reasons to accept this paper? (1-3 sentences) Why to not accept? What are the most important reasons NOT to accept this paper? (1-3 sentences) Comments: Detailed comments on the paper (primarily for the authors) 3. Answer to the Reviews Hannes Werthner

10 3. Answer to the Reviews <Brief introduction> LIST OF { <general comment quoted from reviews> <your answer arguing how you have accordingly modified the paper> } Moving now to the more specific comments: LIST OF { <specific comment quoted from reviews> <your answer arguing how you have accordingly modified the paper, providing detail but not too much> } <Concluding sentence> 4. Review Process Hannes Werthner

11 4. Review Process 0) Abstract (send it to not later than a week before the presentation) 1) Presentations (Fortunate situation where you may know of what the paper is about) 2) All Papers Submitted by June, 13th 3) Review Allocation (by Program Chair) by June, 20th 4) Reviews Due by July, 4th 5) Reviews circulated to Reviewers for additional comments 6) Reviews send to Author by July, 11th 7) Author sends back answer and modified paper by July, 23rd 8) Reviewers provide final answer 9) Exam Pass/No Pass (of Authors and Reviewers) by July, 29th 4. Notice!!! In the paper you have to use the same version of the stylefiles. It is available by downloading it from 0,11855, ,00.html Avoid using stylefiles acquired from other sources as these may not be correct Hannes Werthner

12 How to do presentations Structure 1. Presentation methods 2. Attitude towards the audience 3. How to select content 4. How to structure 5. Introduction 6. Stage fright 7. Main part Hannes Werthner

13 Structure 8. Using pictures 9. Visual aids 10. Interposed questions, interruptions 11. Final discussion 12. The end 13. Optical acoustic factors 14. Ancillary conditions 1. Presentation methods Rule No. 1: Control effect Rule No. 2: Integrate audience Hannes Werthner

14 1. Presentation methods Seminar Lecture Presentation with final discussion Presentation with intermediate discussion Comparison of some presentation methods What remains Time to prepare / h Open discussion 80% 4,0 Presentation with Intermediate discussion Presentation with Final discussion 60% 2,5 30% 1,7 Lecture 20% 1,0 Hannes Werthner

15 2. Attitude towards the audience Whois the audience What do they know What are they interested in What do they understand 2. Attitude towards the audience Contact with eye Simple and understandable languages Concentrate on the important Credibility Reply to objections Avoid techno language Hannes Werthner

16 3. How to select content How much? From where? What is interesting? What is the objective? What should the audience do? Do not want too much! 4. How to structure catch words Speak as you think follow the thinking Use written concept of catch words Hannes Werthner

17 5. Introduction Try to find common understanding with the audience Lead to the subject Initially: create attention 5. Introduction Say at the beginning The structure The length Invite to a dialogue But at the beginning: Who are you Hannes Werthner

18 attention Hope that it will end soon Length of presentation 6. Stage-fright Afraid to speak Is natural! Everybody has it! Is not forever! Hannes Werthner

19 Reduce Stage-fright Be well prepared! Learn important parts by hard! Relax! Look for a positive point Do some contacts before presentation! Stucked? Hannes Werthner

20 7. Main part Introduction Main part Current situation Potential solution How to realise End 8. Using pictures / figures Picture Take from the context of the audience Has to increase the message To explain the issue Does not be an end on itself Hannes Werthner

21 8. Metaphorical language To be a comparison Should be practial Story Citation Joke 9. Visual aids Black board Projector Film Videobeam Overhead-Display Hannes Werthner

22 Presentation programs GraficalLayout Colours Contrast Background Clipart-files Fonts (size) Produce hand outs What we keep 10% Reading 20% Hearing 30% Seeing 50% Hearing & Seeing 70% Reporting 90% Do it yourself Hannes Werthner

23 10. Questions Question Are welcome Should be answered immediatly Bit can also be delayed or forwarded to somebody else Interupptions ignore! 11. Final discussion Announce already in the introduction Write up issues of discussion Keep sequence of questions Repeat questions (if necessary) Draw conclusions Hannes Werthner

24 12. End Avoid New questions which do not help in the conclusions Main ideas Should be summarized 12. End Main idea + review objective + how to realise = Good end Hannes Werthner

25 13. Acoustic factors No speaking smoking" Speak loudly and slowly You should vary volume and speed Make breaks Try presentation beforehand 13. Optical factors Behave natural! With your body With your hands With your eyes Hannes Werthner

26 14. Other circumstanceas Room size Lighting Ventilation Sockets Other means (Microfon) Breaks Reading - How to read Fausto Giunchiglia Literature: Bruno Buchberger, Thinking Speaking Writing By Fausto Giunchiglia and Alessandro Tomasi Hannes Werthner

27 Index: 1. The Role of Literature 2. What is literature 2.1 Classify the results 3. The Documentation of Literature 4. Steps in the Use of Literature 1. The Role of Literature Hannes Werthner

28 1. The Role of Literature Keep Re-Inventing the wheel For many problems: the solutions is already in the literature the literature has solutions in similar problems 1. The Role of Literature It is necessary to know how the literature is organized: within your research community outside your research community [translate other concepts in our community] Necessary Useful but not Necessary Hannes Werthner

29 1. The Role of Literature There is an interleaving of: 1) Try to solve the problem yourself 2) Read the Literature 1. The Role of Literature Interactive Process of Reading Papers: Step 1 Quantity Step 2 Step 3 Depth Hannes Werthner

30 1. The Role of Literature How do I know when I can stop reading? When I know the 90% of the papers cited by relevant papers 1. The Role of Literature To keep Scholarship: go to conference talk to people (best opportunity in conference) journals proceedings web Hannes Werthner

31 2. What is Literature 2. What is Literature 1. Monographs / Books 2. Journal Articles 3. Articles in Collections 4. Papers in Proceedings of Conference 5. Papers in Proceedings of Workshop 6. Technical Reports 7. Grey Literature (Ex: Web Sites) Main Way to do Career More important than papers Hannes Werthner

32 2. What is Literature Each research community has: 1-2 Top Journals 1-2 Top Conference 2.1 Classify the results Hannes Werthner

33 2.1 Classify the results 1. Monographs / Books: Note: the research is finished Contents: extensive/complete description of research Originality: None (2/3 years old) Quality Control: Very Good 2.1 Classify the results 2. Journal Articles: Note: Come regularly Contents: Very good, very crisp, specialized Originality: Full of two years ago (in CS is obsolete) Quality Control: Very High (Many shots) Hannes Werthner

34 2.1 Classify the results 3. Articles in Collections: Note: Not regularly Contents: Very good, very crisp, specialized Originality: Full of two years ago (in CS is obsolete) Quality Control: Lower 2.1 Classify the results 4. Conferences: Note: Is Refereed Contents: New results, almost complete, semi-final Originality: Good (6 months ago) Quality Control: Medium (only one shot: accept or reject) Hannes Werthner

35 2.1 Classify the results 5. Workshop Note: Is Not Refereed Contents: New results, almost complete, semi-final Originality: Good (6 months ago) Quality Control: Medium (only one shot: accept or reject) 2.1 Classify the results 6. Technical Reports Note: --- Contents: Detailed about a specific topic Originality: Maybe Quality Control: Low (No Refereeing Process) Hannes Werthner

36 2.1 Classify the results 7. Grey Literature (Ex: Web Sites) Note: The main way to publish our results Contents: Random Originality: Random Quality Control: Random 3. The Documentation of Literature Hannes Werthner

37 3. The Documentation of Literature Library - author files - keywords files Review Journals - Computing Journal - Artificial Intelligence Review - Scientific Magazines - Scientific Magazines for Spreading Communications - Journal for Publish new Results Bibliographies Citation index Research index 3. The Documentation of Literature Research Community are organized into societies Produce: Organized Conferences, Journals, Magazines Examples of Societies: ACM, IEEE, ECCAI, VLDB, Hannes Werthner

38 3. The Documentation of Literature Citations Index: defines journals cites titles, authors, abstract of all paper Example: ISI, the most important for Science 3. The Documentation of Literature Research Index: the new game is on line how many times you are cited in Internet Example: Hannes Werthner

39 4. Steps in the Use of Literature 4. Steps in the Use of Literature 1. Use CiteSeer ( 2. Ask to Colleagues, Advisor, Friends, by 3. Consult References 4. Consult Review Journals 5. Ask Authors Hannes Werthner