The ARC Process and Writing a Rejoinder to an Assessment

1. The ARC Process and Writing a Rejoinder to an Assessment June 2007 I wish to acknowledge the two presenters from last year, whose presentations I have drawn on to create this one. Also Phyllis Tharenou from USA. Peter Sullivan

2. Handling Applications • Each application is read by two College of Experts memberswho read about 100 – no written comments • Each application is sent to two OzReaders, who read about 20, and make comments • Each application is sent to up to 4 IntReaders, who read about 3, and make comments • Applications are scored on • Significance 30% • Approach 20% • National Benefit 10% • Track record 40%

3. Handling applications • Some projects are assessed by someone (usually only 1) outside the immediate field – the RFCD codes are very important. • All scores are converted to ranks • CoE members are allocated particular projects, and sent all ranked scores, assessments, and rejoinders

4. There are two opportunities for CoE members to adjust scores • When they first receive all the above information. At this stage, the rejoinders important for projects for which there is a discrepancy in scores. • In the panel meeting when deciding which projects are funded. At this stage, the rejoinders can make the difference between equivalent projects near the funding cut off.

5. Crafting a rejoinder • The rejoinder is not sent to the assessors • The rejoinder is read by the specific CoE members for the application • Obtain feedback on the rejoinder before submitting it • Target specific claims/criticisms and address them: • To re-justify the original case in the application or • To make modifications to the case in light of the criticisms

6. Hints • However satisfying, it is a waste of time attacking the assessors • Write in a positive and conciliatory tone • Address criticisms directly • Provide an indication of which comment that you are responding to. • Help the College of Experts to see the proposal from your point of view.

7. I suspect that rejoinders are only read seriously where there is a (major) discrepancy in scores, and around the borderline. • In both of these cases, the most important thing is to address in some way any criticisms the influence the potential quality of the project, whether the criticisms are accurate or not. • If accurate, you need to say how the concern will be addressed. • If not accurate, you need to say why not.

8. Hints • Indicate that you have done some more work since submitting the proposal, confirming in a preliminary fashion your hypothesis, include references of papers which had been published since submitting the proposal. • Use the project as a learning experience on how the grant was written • You are allowed 5,000 character reply/comment on assessors’ comments

9. Example 1 (last year’s presenter) • “It is not clear that e-commerce would directly benefit as indicated”. • e-commerce would be directly effected if we can cluster more accurately web pages according to their structure, as well as content, as this will allow a user to access ``correct'' web pages using queries.

10. Example 1 (my approach) • “It is not clear that e-commerce would directly benefit as indicated”. • Assessor L1234 queried the benefits for e-commerce. The benefits provide the rationale this application. e-commerce will benefit by clustering more accurately web pages according to their structure, as well as content, as this will allow a user to access “correct'' web pages using queries.

11. Example 2 • “However, this proposal would not contribute anything significant to it. First, I doubt CIs’ knowledge in the field. There are several myths. Google is not updating their index in weeks. Page Ranking is not based on "the relative popularity of pages". It is the text analysis within a page. It’s called Page Ranking simply because one of authors surname is “Page”. Eq (1) is not "about the connectivity of Web pages", rather, the connectivity of phrases.”

12. Suggestion from last year • This assessor wishes to overturn evidence contained in a highly cited 1998 paper: ``An anatomy of a search engine'' by Page and Brin by saying that they used text analysis to perform page ranking. It is stated explicitly in the abstract of the paper that they are using connectivity of the web pages, rather than using text analysis to rank web pages.

13. My suggestion • Assessor L1234 challenges the basis of our page ranking process. Our process is based on the highly cited 1998 paper: ``An anatomy of a search engine'' by Page and Brin in which they state categorically that that they did not use text analysis to perform page ranking, but connectivity of web pages, as we have done. The opinions of this assessor can be discounted on this issue.