slide1 n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Class 4 27.11.2006 PowerPoint Presentation
Download Presentation
Class 4 27.11.2006

Loading in 2 Seconds...

play fullscreen
1 / 71

Class 4 27.11.2006 - PowerPoint PPT Presentation


  • 68 Views
  • Uploaded on

Proposal Writing & The Refereeing Process. Class 4 27.11.2006. The Refereeing Process. Introduction Peer review process Journals Conferences Research programmes The tasks of a referee Reviewing a research paper Preparing the referee report & recommendations

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Class 4 27.11.2006' - yadid


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1

Proposal Writing & The Refereeing Process

Class 4

27.11.2006

Class 4, 27/11/2006

slide2

The Refereeing Process

Class 4, 27/11/2006

contents
Introduction

Peer review process

Journals

Conferences

Research programmes

The tasks of a referee

Reviewing a research paper

Preparing the referee report & recommendations

Evaluating a research proposal

Acting as an editor or program chairperson

How to become a referee?

Final words

Contents

Class 4, 27/11/2006

disclaimer
There is no fixed mechanism for refereeing

There are simple rules that help transforming a review in a constructive document

In time you will develop your own style of refereeing

Disclaimer

Class 4, 27/11/2006

introduction
A scientific paper is expected to provide a sufficient contribution to the knowledge base of its field

Number of scientific papers and articles (2000): > 600 000 (ISI)

About 50% in the fields of science and technology

The number of papers and articles submitted for publication is much larger

refereeing process selects the ones to be published

Examples of acceptance rates after refereeing:

Journals: ~10-20% (large variance)

Conferences: ~10-50%

Workshops: ~30%-90%

Refereeing is also used in selecting research projects to be funded

Introduction

Class 4, 27/11/2006

introduction1
What is a sufficient contribution?

new result, theoretical or experimental

new insight

novel synthesis of ideas

useful survey

useful tutorial

What is not a sufficient contribution

badly written

erroneous data

MPI = Minimum Publishable Increment depends on the forum

Introduction

Class 4, 27/11/2006

peer review process
Peer reviews are carried out by anonymous referees who evaluate the sufficiency of contribution

novelty, significance, correctness, readability

Refereeing is public service to the scientific community

professional obligation,

carried out on volunteer basis

requires high expertise

helps in improving one’s own expertise

ensures the integrity of science

Peer review process

Class 4, 27/11/2006

peer review process of a journal
Peer review process of a journal

submission

editor

author

publish

accept

reject

revise

selection of associate editor

reviews

recommendations

associate

editors

referees

selection of referees

checking of revised papers

Class 4, 27/11/2006

peer review process of a conference
Peer review process of a conference

submission

program

committee

program chair

author

accept/

reject/

accept with revisions

  • selection of the referees
  • checking of revisions

accept/reject/minor revision recommendations

referees

extra referees

Class 4, 27/11/2006

peer review process of a workshop
Peer review process of a workshop

submit

program

committee

program chair

author

accept/

reject

refereeing

checking of

revisions

extra referees

Class 4, 27/11/2006

peer review process of a research programme
Peer review process of a research programme

submission

steering

committee

proposer

accept with partial funding/

reject

referees

Notice: not representative of all research programmes

Class 4, 27/11/2006

the tasks of a referee

The reviewer grades a paper based on its novelty, significance,correctness, and readability

In case of substantial conflicts of interest or if the paper is out of the field of the reviewer, the editor must be informed promptly

Both positive and negative findings are summarized in a referee report

Confidential part only for the editor/program committee: Information that could reveal the identity of the reviewer or in minor conflicts of interest

non-confidential part for the author/program committee

Learn from the other reviews, if they are sent to you after the process

The tasks of a referee

Class 4, 27/11/2006

why do it
Several reasons

Enhance reputation (with editor/prog. committee)

Expedites processing of your own papers

Get on editorial board or program committee

Good practice

Increase your own critical appraisal ability

Your papers become better

Sometimes it gets preferential treatment for your papers

… but refereeing means more work!

Why do it?

Class 4, 27/11/2006

consideration
Most reviews have strict deadlines

By agreeing to review you take the responsibility of doing a thorough job

If you cannot commit to this, notify the editor asap

Editors understand you may not have the time, but are unforgiving if you commit and do a poor job

Good editors keep a list …

Consideration

Class 4, 27/11/2006

the right attitude i can learn something

Humbleness and an open mind needed; 100% self-confidence can be harmful

Early assumptions on the correctness of the paper or the sufficiency of its references should be avoided

an elegantly written paper may have zero actual contribution

a paper with broken English may contain a major new idea

The papers recommended for acceptance should have novelty and be correct

If the reviewer can’t check a fact or is unsure, this should be stated in the review report

But don’t waste your time on analysing in detail a paper that is never publishable

a single crucial error is enough

The right attitude: I can learn something!

Class 4, 27/11/2006

reviewing a research paper

The paper to be reviewed is typically accompanied with a review form

fill the five point scale questions last

it is most important to write an itemized review report

Relevance

[ ] poor [ ] marginal [ ] fair [ x ] good [ ] excellent

Originality

[ ] poor [ ] marginal [ ] fair [ x ] good [ ] excellent

Background knowledge of the subject and references

[ ] poor [ ] marginal [ ] fair [ x ] good [ ] excellent

Technical content

[ ] poor [ ] marginal [ ] fair [ x ] good [ ] excellent

Presentation

[ ] poor [ ] marginal [ ] fair [ x ] good [ ] excellent

Reviewing a research paper

Class 4, 27/11/2006

reviewing a research paper analysis

The analysis of a paper can be done by generating explanations to the following eight points (Smith 1990)

What is the purpose of the paper?

Is the problem clearly stated and have the key issues been pointed out?

Is it clear what has been accomplished?

Is the paper appropriate for the intended forum?

If it is not, what could be a better choice?

Is the goal significant = has the work been worth doing?

Are the results just trivial variations or extensions of previous results?

Are there any new ideas, or novelties in research methodology?

Citation analysis using electronic libraries are a big help!

Reviewing a research paper: analysis

Class 4, 27/11/2006

reviewing a research paper analysis cont d

Is the method of approach clear and valid?

Is there something fundamentally flawed in the approach?

Are the assumptions realistic and does that matter?

Is the method new? Can it be generalized to other problems?

Is the actual execution of the research correct?

Are the mathematics and statistics correct? Check!

Have the simulations been described in sufficient detail for replication?

What about the boundary conditions?

Do the results make sense?

This part may require considerable effort from the reviewer...

Reviewing a research paper: analysis (cont’d)

Class 4, 27/11/2006

reviewing a research paper analysis cont d1

Are the conclusions correct?

What are the applications or implications of the results and are the results analysed to an adequate depth?

Is the presentation satisfactory?

Is the paper readable? Is it structured according to the conventions of scientific publications?

What did you as the reviewer learn?

If you didn’t learn anything, then the paper is not publishable (provided that you understood the paper)

Reviewing a research paper: analysis (cont’d)

Class 4, 27/11/2006

reviewing a research paper analyzing the references
It is researcher’s professional obligation to cite prior work

the manuscript being reviewed includes claims of novelties; regularly citing prior research

the reviewer needs to check the validity of the claims

most efficient to carry out the analysis using electronic libraries

At minimum:

Check what is found using the key words of the article

Study the references you don’t know beforehand

Check which recent papers cite the same references

Check the references of those recent papers

Reviewing a research paper: analyzing the references

Class 4, 27/11/2006

slide21

Review structure

  • The actual refereeing form
  • General comments on the paper
  • Specific comments on the paper
  • Confidential note to editor
  • General idea: be professional and non-hostile: write the review in a style that you would like to receive for your paper

Class 4, 27/11/2006

slide22

The refereeing form

  • Forms might look quite different but basically ask the same things
  • Poorly designed ones just have yes/no answers, good ones prompt the referee to elaborate
  • Make sure you read and understand it well

Class 4, 27/11/2006

writing the referee report
No fixed rules exist, the following ones are according to (Smith 1990)

Most important: make your opinions clear; avoid ”perhaps” and ”maybe”; evaluate the paper, not the author; itemize the contributions

State the recommendation and its justification; the five point scale part of the evaluation form is not enough

Show with a few summarizing sentences that you have understood the paper. The editor may use this part and compare your summary to those of the other reviewers

Evaluate the significance and validity of the research goal

Evaluate the quality of methodology, techniques, accuracy and presentation; recommendations for revisions can be written here

Make a clear recommendation for or against publication with justifications

Writing the referee report

Class 4, 27/11/2006

compiling the recommendations
Compiling the recommendations
  • Classification of papers (Smith 1990)
  • Very significant; includes major results (<1% of all papers)
  • Interesting work, a good contribution (<10%)
  • Minor positive contribution (10-30%)
  • Elegant and technically correct, but useless
  • Neither elegant nor useful, but not wrong
  • Wrong and misleading
  • Unreadable, impossible to evaluate
  • The acceptance level of the journals and conferences vary; 1,2, and perhaps 3(-4)

Class 4, 27/11/2006

slide25

Outcome

  • Usually:
    • Accept the paper as it is
    • Paper requires minor changes
    • Paper requires major changes (with or without a new refereeing process)
    • Reject publication of the paper
  • You can only suggest, the choice is not yours
    • Decision is based on at least 3 reviews

Class 4, 27/11/2006

research proposals
Research proposals
  • A research proposal is a request for funding submitted to,
    • MCyT, MECD, GENCAT
    • European Commission
    • NIH, NASA, NSF, ESF
    • other funding organization such as a foundation
  • The key difference to reviewing research papers is that
  • the reviewers also evaluate the proposers
  • Not all organizations use peer review as a means for selecting proposals for funding

Class 4, 27/11/2006

evaluating research proposals
Evaluating research proposals
  • The evaluation criteria vary between funding organizations
  • Key criteria:
  • Is the research topic significant?
  • Are the goals realistic?
  • Has the proposer sufficient expertice and facilities to reach the goals?
  • Is the requested funding reasonable?

Class 4, 27/11/2006

ethics of refereeing
Objectivity

Judge paper on its own merits

Remove prejudice

If you are not able to review it, return it

Fairness

Author may have different point of view / methodology / arguments

Judge from their school of thought not yours

Speed

Be fast, but do not rush. Author deserves a fair hearing

Ethics of refereeing

Class 4, 27/11/2006

ethics of refereeing1
Professional treatment

Act in the best interest of the author and conference/journal

Specific rather than vague criticism

Confidentiality

Cannot circulate paper

Cannot use without permission

Conflict of interest

Discuss with editor

Ethics of refereeing

Class 4, 27/11/2006

ethics of refereeing2
Honesty

About your expertise and confidence in appraisal

Courtesy

Constructive criticism

Non-inflammatory language

Suggest improvements

Ethics of refereeing

Class 4, 27/11/2006

acting as an editor or program chairperson
Acting as an editor or program chairperson
  • The editor
    • maintains correspondence with authors and referees
    • finds new referees if the ones assigned fail to act in given time
    • decides on acceptance, rejection or a revision round based on 2-4 review statements.
    • should distribute all review statements to the referees
    • receives occasional negative feedback
  • Review is not a vote! The editor is likely to line himself according to
  • the best justified recommendations
  • Conference program committees often rely on the numerical
  • evaluations, occasionally resorting to vote

Class 4, 27/11/2006

how to become a referee
How to become a referee
  • Writing a publication that is cited is the most certain way to become a referee
  • Coordination or technical coordination of an EU RTD project is a direct road to proposal evaluations
  • Refereeing is very rewarding, helps to keep up-to-date and aware of developments in fields adjacent to ones own specialty

Class 4, 27/11/2006

final words
Final words
  • Good referee reports are valuable and free of charge
    • help in improving the paper
    • help in improving as a researcher
    • help in improving as a referee
  • Refereeing is a learning experience
  • Scientific progress rests heavily on peer reviews

Class 4, 27/11/2006

slide34
PROJECT THESIS EDITOR OFFICE

Using the articles prepared, we will set up an editorial office

Each student will act as an associate editor of one article and will review three articles

Each student will peer review his/her three allocated articles and will return the referee report to the assigned associate editor

The associate editor will compile the final report and will return the final report with the individual referee reports to the Editorial Office

Class 4, 27/11/2006

slide35

Proposal Writing

Class 4, 27/11/2006

proposal writing
In order to carry out research, in general financing is required.

There are several national and international sources of funding and the process for obtaining funding is realised through proposal submission and review.

The aim of this section is to INFORM you of the proposal process, proposal formats and existing funding bodies.

The homework of this class will be to draft a proposal of your PhD to assist you in your resaerch planning, but NOT with a view to preparing a formal proposal in the style of those submitted for financing!

Proposal Writing

Class 4, 27/11/2006

content
Why research ?

Why should this be in a competitive context ?

Why a research proposal ?

Getting started

What makes a good proposal ?

Writing your proposal

How to structure your proposal ?

The review process

Allocation of funding

What next ?

Getting help with your proposal ?

Quick TIPS for writing a good proposal

Content

Class 4, 27/11/2006

why research
Why is the development of research within universities a must ?

To maintain the quality of teaching programs.

Provide the basis for undergraduate and graduate thesis research projects.

Universities should be more than degree delivering institutions.

Universities should be the basket for new knowledge and developments.

Why research ?

Class 4, 27/11/2006

why should this be in a competitive context
Do universities have the financial capacity to develop and support research activities ?

Where can the money be found to develop and support research ?

How can society get the highest return on investment ?

Why should this be in a competitive context ?

Class 4, 27/11/2006

why a research proposal
Convince others the project you have designed is important, worth the effort.

Convince others that you have the ability to carry out the research design and report the findings.

Generate funds to sustain the research units operation.

Why a research proposal ?

Class 4, 27/11/2006

getting started
Know your subject. The reviewers will look for an up-to-date knowledge of the research area.

Know your funder. Be aware of the priorities and interests of the funder you approach, and know that funders are unlikely to support the same idea twice.

Getting started

Class 4, 27/11/2006

getting started1
Consult colleagues.

Don’t be afraid to discuss your proposal with colleagues, or even with the grants officer at the funding body.

Early discussions can ensure that your proposal is targeted appropriately.

Getting started

Class 4, 27/11/2006

what makes a good proposal
A well-prepared application should require minimal effort on the part of the reviewer.

Proposals must demonstrate high scientific quality.

The requested funds must be in proportion to the proposed project (cost-effectiveness).

What makes a good proposal ?

Class 4, 27/11/2006

writing your proposal
Allow plenty of time to prepare your proposal. A good starting point is to write a one-page summary of the whole project. This may take a while to get right, but once completed it will serve as an invaluable tool for writing your full proposal.

Use your proposal to show the need and then fill the gap.

Writing your proposal

Class 4, 27/11/2006

writing your proposal1
Present your proposal in terms of the aims and objectives of the funder and not just your own – make it clear how you will be helping them to fund their priorities.

Consider the questions the funder will be asking: Why fund you ? Why fund this ? Why now ? ... and make sure that the proposal answers them!

Writing your proposal

Class 4, 27/11/2006

writing your proposal2
Be aware that you will have limited to none opportunities to answer queries arising from a reading of your proposal.

Consult the funders website and read clearly the call for research proposals as well as the criteria against which your proposal will be judged.

Writing your proposal

Class 4, 27/11/2006

writing your proposal3
Although it is the content that matters, good presentation is often crucial to making your proposal accessible to reviewers and keeping their interest.

Use diagrams and tables to add clarity;

Bullet points and sections can break up text;

Keep to page, word and font size restrictions; and

Activate the spell checker while writing.

Writing your proposal

Class 4, 27/11/2006

how to structure your proposal
Check guidelines carefully – failing to meet the funder’s format and specifications is one of the most common reasons for applications being returned.

A common proposal structure normally consists of: title, abstract, background, aims and objectives, methodology, work program, resources, outcomes (outputs & dissemination), project management, reviewers.

How to structure your proposal ?

Class 4, 27/11/2006

how to structure your proposal1
Title: This is the first impression the reader gets.

The title should be short and clear, and the reviewer should be able to understand from the title the intentions of the research.

A catchy title posing a question or including an apparant contradiction or acronym may be more easily remembered by a reviewer.

How to structure your proposal ?

Class 4, 27/11/2006

how to structure your proposal2
Abstract: Should be a concise summary of the WHOLE project.

Use the abstract to identify the need for this research, state what you intend to do, and how you intend to do it.

Do not include unnecessary detail; make each phrase count.

And remember it is the first impression a reviewer gets of an applicant’s worth!

How to structure your proposal ?

Class 4, 27/11/2006

how to structure your proposal3
Background: This section should be used to put the work into context: what has been done before, and how will the proposed work add to it ?

What is the innovative aspect in the research project ?

Build your case by demonstrating your capability and familiarity in the area.

How to structure your proposal ?

Class 4, 27/11/2006

how to structure your proposal4
Aims and objectives: The aims should describe what you intend to achieve by doing this piece of work.

Your objectives are the small steps you need to reach in order to achieve your aim.

Aims and objectives should be realistic, consistent, and link them to methods, timetable, and outcomes.

How to structure your proposal ?

Class 4, 27/11/2006

how to structure your proposal5
Methodology: Methods should be detailed and well thought through.

Explain why you have chosen a particular method.

Base your explanation on literature references.

If your own experience of a methodology is limited, consider working with collaborators.

How to structure your proposal ?

Class 4, 27/11/2006

how to structure your proposal6
Work program: Make use of a Pert chart to illustrate the building blocks – work packages – of the research project. Be detailed in the description of the content of each work package (why, objectives, method(s), duration, when are you going to carry out each WP, partners involved in the realization, sequence of WP, etc.). How to structure your proposal ?

Class 4, 27/11/2006

slide55

WATERSHED LEVEL

IRRIGATION SCHEME LEVEL

Management system (CERES)

Water rights

Determination of rainfall and runoff

WP 1

WP 4

Determination of sediment load

Determination topography of reservoir

Water demand estimation

WP 5

WP 2

Irrigation infrastructure

WP 6

Modeling process

Calibration and validation of the methodology for the actual water management situation

Conveyance efficiency

Predicting the values of the irrigation and economic indicators for alternative scenarios of watermanagement

Irrigation indicators

Determination of sediment load

Gross water availability

WP 3

Life expectancy reservoir

INPUT for decision making

Topography

Land use

Soil

Climate

Economic indicators

Simplified management system

WP 7

WP 8

Example of a Pert chart

Class 4, 27/11/2006

how to structure your proposal7
Work program: This section contains also a diagrammatic work plan, called a Gannt chart.

The Gannt chart or diagrammatic work plan should also be accompanied by a written description.

How to structure your proposal ?

Class 4, 27/11/2006

slide57

deliverables

Example of a Gannt chart (= diagrammatic work plan)

Class 4, 27/11/2006

how to structure your proposal8
Resources: The proposal should contain a detailed budget.

The budget asked should be in proportion to the volume and complexity of the work activities.

Be aware that funders vary as to what they are prepared to pay in terms of direct project costs, such as staff and equipment, and indirect costs, such as overheads.

The funder might request to approve beforehand own inputs or inputs from other institutions participating in the project.

How to structure your proposal ?

Class 4, 27/11/2006

how to structure your proposal9
Outcomes, outputs (+ deliverables) and dissemination: In this section one should describe the contribution to knowledge and importance for future research, the benefits to users, and the broader relevance to beneficiaries.

Highlight how results will be disseminated (publications, conferences, commercial exploitation, websites, ....).

How to structure your proposal ?

Class 4, 27/11/2006

how to structure your proposal10
Project management: This might not be required for small projects.

However for projects in which several partners are involved sufficient information has to be provided on how the project will be managed (timescales, milestones, communication, criteria to measure progress, how crisis situations and conflicts will be handled, etc.).

How to structure your proposal ?

Class 4, 27/11/2006

how to structure your proposal11
Reviewers: Often requested to suggest name of referees.

Choose people who know you and your work;

Don’t use reviewers within your own institution;

Use international reviewers; and

Be aware that applicant’s own referees can write unfavourable reports.

How to structure your proposal ?

Class 4, 27/11/2006

the review process
Expert assessment: Traditionally applications will be assessed by 2 to 3 reviewers selected from the pool of experts.

Reviewers will make an independent assessment of the scientific quality of the proposal.

To be selected for funding at least 2 of the 3 reviewers should provide a positive assessment.

The review process

Class 4, 27/11/2006

the review process1
What are reviewers looking for ?

High scientific quality;

Proposals that meet the funder’s priorities or fill a knowledge gap;

Novelty and timeliness;

Value for money;

A clear and well thought out approach; and

An interesting idea – catch their attention!

The review process

Class 4, 27/11/2006

the review process2
Awards committee: Ranks the submitted proposals on the basis of the reviewer’s reports. Their operation and procedures can be very variable from funder to funder. They might for policy reasons of the funder deviate from the reviewer’s assessment. The review process

Class 4, 27/11/2006

allocation of funding
Position in the ranking is important – it could mean the difference between success and failure. Proposals are often ranked into the following categories:

Fund;

Fundable;

Invite resubmission (used by some funders); or

Reject.

Allocation of funding

Class 4, 27/11/2006

what next
If the project is retained for funding  OK.

If the project is found fundable  ???

If invited for resubmission  revise proposal  feedback from the reviewers panel.

If rejected, can be very frustrating  do not give up, try to get feedback  remember it is a learning process !

What next ?

Class 4, 27/11/2006

quick for writing a good proposal
Allow plenty of time;

Start by writing a summary of your proposed project;

Demonstrate an up-to-date knowledge of your field;

Present your proposal in terms of the aims and objectives of the funder;

Avoid jargon – say what you mean in clear, simple language;

Don’t be afraid to state the obvious;

Use graphics and diagrams to summarise what you are trying to communicate

Quick for writing a good proposal

Class 4, 27/11/2006

quick for writing a good proposal1
Allow a maximum of 4 charts (PERT, GANNT, PROJECT ORGANIZATION and BUDGET) - but include as many schematic representations of the concepts as possible;

Anticipate questions that may arise, before they arise;

Ask a colleague to review your proposal; and

Be enthusiastic about your idea – if you don’t sound interested, why should anyone else be ?

Quick for writing a good proposal

Class 4, 27/11/2006

funding sources
EUROPE

European Comission (www.cordis.lu) - 4 year programmes with identified priorities and objectives. Currently Framework 6 - Framework 7 soon begins this year

European Science Foundation (www.esf.org)

National Funding (www.medc.es, www.mcyt.es)

US

National Institute of Health (www.nih.gov)

DARPA (www.darpa.mil)

NASA

Department Of Energy

Department Of Agriculture

Funding Sources

Class 4, 27/11/2006

homework
Draft a proposal of your PhD project

The proposal should include:

Title Page

Table of Contents

Overall and sub-objectives

State-of-the-art and novelty of project

Workplan - divide into ‘workpackages’, for each WP describe the tasks and sub-tasks (a couple of paragraphs for each), the resources required, risk analysis and contingency plan, as well as deliverables and milestones

Pert Chart

Gantt Chart

Bibliography

(? Budget)

Homework

Class 4, 27/11/2006