Tuesday, April 28, 2009

What Makes a Good Report?

What Makes a Good Report?

Key Features of a Good Report


dot Is expressed in language that differentiates the trainee’s level of competency;

dot Includes signatures from all the interested parties involved with the trainee;

dot Identifies areas of development in all areas;

dot Identifies progression;

dot Is evidence based;

dot Is contextualised to explain the status of the report and the date of the report;

dot Provides contextual information on the school and information on the nature of school based training;

dot Is linked to the QtT Standards;

dot Uses unambiguous language;

dot Avoids educational jargon;

dot Is professionally written in common language or OfSTED language;

dot Is informative;

dot Includes action points for the trainee’s continuing professional development and suggestions about how they can improve;

dot Makes clear what the trainee’s strengths and weaknesses are;

dot Conveys clear, visual message;

dot Involves the trainee and provides space for them to add their comments and signature;

dot Indicates who the report is for.


Other requirements…

dot Co-ordinating mentor needs oversight of all reports. Has a ‘quality assurance’ function in ensuring there is consistency/ parity in report writing.

dot Final summative report needs to inform CEDP

dot The structure of all reports across a consortium is based on a set of agreed criteria to ensure consistency.

Basic of report writing

How to Write a Good Report 

This short document describes how to write a good report. This is based on common mistakes I have observed over a period of time. 


I will first mention some general guidelines, then the structure of the report. Towards the end, I will also describe how to refine your writing, and how to give feedbackon others' writing. Based on these, I will recommend a possible strategy for producing high-quality reports which have high potential for being published.


General Guidelines

These are some general things you should know before you start writing. I will try to answer the questions of the purpose of report writing, and the overall approach as well.

Purpose of a report: writing to be read

A key thing to keep in mind right through your report writing process is that a report is written to be read, by someone else. This is the central goal of report-writing. A report which is written for the sake of being written has very little value.

Before you start writing your report, you need to have in mind the intended audience. In the narrowest of possibilities, your report is meant for reading by yourselves, and by your advisor/instructor, and perhaps by your evaluation committee. This has value, but only short-term. The next broader possibility is that your report is readable by your peers or your juniors down the line. This has greater value since someone else can continue on your work and improve it, or learn from your work. In the best case possibility, your report is of publishable quality. That is, readable and useful for the technical community in general.

Overall approach: top-down

Take a top-down approach to writing the report (also applies to problem solving in general). This can proceed in roughly three stages of continual refinement of details.

  1. First write the section-level outline,
  2. Then the subsection-level outline, and
  3. Then a paragraph-level outline. The paragraph-level outline would more-or-less be like a presentation with bulleted points. It incorporates the flow of ideas.

Once you have the paragraph-level flow of ideas, you can easily convert that into a full report, by writing out the flow of ideas in full sentences.

While doing the paragraph-level outline, think also about (a) figures, (b) tables, and (c) graphs you will include as part of the report at various stages. You will find that many things can be better explained by using simple figures at appropriate places.

Another thing to nail-down while doing the paragraph-level outline is the terminology you will be using. For instance, names of various protocols/algorithms/steps in your solution. Or names/symbols for mathematical notation.

The overall approach also includes multiple stages of refinement, and taking feedback from others (peers/advisor/instructor). I will talk about these in more detail after talking about the overall report structure.


Structure of a report

The following should roughly be the structure of a report. Note that these are just guidelines, not rules. You have to use your intelligence in working out the details of your specific writing.

  • Title and abstract: These are the most-read parts of a report. This is how you attract attention to your writing. The title should reflect what you have done and should bring out any eye-catching factor of your work, for good impact.

    The abstract should be short, generally within about 2 paragraphs (250 words or so total). The abstract should contain the essence of the report, based on which the reader decides whether to go ahead with reading the report or not. It can contain the following in varying amounts of detail as is appropriate: main motivation, main design point, essential difference from previous work, methodology, and some eye-catching results if any.

  • Introduction: Most reports start with an introduction section. This section should answer the following questions (not necessarily in that order, but what is given below is a logical order). After title/abstract introduction and conclusions are the two mainly read parts of a report.

    • What is the setting of the problem? This is, in other words, the background. In some cases, this may be implicit, and in some cases, merged with the motivation below.
    • What exactly is the problem you are trying to solve? This is the problem statement.
    • Why is the problem important to solve? This is the motivation. In some cases, it may be implicit in the background, or the problem statement itself.
    • Is the problem still unsolved? The constitutes the statement of past/related work crisply.
    • Why is the problem difficult to solve? This is the statement of challenges. In some cases, it may be implicit in the problem statement. In others, you may have to say explicitly as to why the problem is worthy of a MBA, or a semester project, as the case may be.
    • How have you solved the problem? Here you state the essence of your approach. This is of course expanded upon later, but it must be stated explicitly here.
    • What are the conditions under which your solution is applicable? This is a statement of assumptions.
    • What are the main results? You have to present the main summary of the results here.
    • What is the summary of your contributions? This in some cases may be implicit in the rest of the introduction. Sometimes it helps to state contributions explicitly.
    • How is the rest of the report organized? Here you include a paragraph on the flow of ideas in the rest of the report. For any report beyond 4-5 pages, this is a must.

    The introduction is nothing but a shorter version of the rest of the report, and in many cases the rest of the report can also have the same flow. Think of the rest of the report as an expansion of some of the points in the introduction. Which of the above bullets are expanded into separate sections (perhaps even multiple sections) depends very much on the problem.

  • Background: This is expanded upon into a separate section if there is sufficient background which the general reader must understand before knowing the details of your work. It is usual to state that "the reader who knows this background can skip this section" while writing this section.

  • Past/related work: It is common to have this as a separate section, explaining why what you have done is something novel. Here, you must try to think ofdimensions of comparison of your work with other work. For instance, you may compare in terms of functionality, in terms of performance, and/or in terms of approach. Even within these, you may have multiple lines of comparison -- functionality-1, functionality-2, metric-1, metric-2, etc.

    Although not mandatory, it is good presentation style to give the above comparison in terms of a table; where the rows are the various dimensions of comparison and the columns are various pieces of related work, with your own work being the first/last column. See the related work section of my MBA thesis for an example of such a table :-).

    While in general you try to play up your work with respect to others, it is also good to identify points where your solution is not so good compared to others. If you state these explicitly, the reader will feel better about them, than if you do not state and the reader figures out the flaws in your work anyway :-).

    Another point is with respect to the placement of related work. One possibility is to place it in the beginning of the report (after intro/background). Another is to place it in the end of the report (just before conclusions). This is a matter of judgment, and depends on the following aspect of your work. If there are lots of past work related very closely to your work, then it makes sense to state upfront as to what the difference in your approach is. On the other hand, if your work is substantially different from past work, then it is better to put the related work at the end. While this conveys a stronger message, it has the risk of the reader wondering all through the report as to how your work is different from some other specific related work.

  • Technical sections: The main body of the report may be divided into multiple sections as the case may be. You may have different sections which delve into different aspects of the problem. The organization of the report here is problem specific. You may also have a separate section for statement of design methodology, or experimental methodology, or proving some lemmas in a theoretical paper.

    The technical section is the most work-specific, and hence is the least described here. However, it makes sense to mention the following main points:

    • Outlines/flow: For sections which may be huge, with many subsections, it is appropriate to have a rough outline of the section at the beginning of that section. Make sure that the flow is maintained as the reader goes from one section to another. There should be no abrupt jumps in ideas.
    • Use of figures: The cliche "a picture is worth a thousand words" is appropriate here. Spend time thinking about pictures. Wherever necessary, explain all aspects of a figure (ideally, this should be easy), and do not leave the reader wondering as to what the connection between the figure and the text is.
    • Terminology: Define each term/symbol before you use it, or right after its first use. Stick to a common terminology throughout the report.
  • Results: This is part of the set of technical sections, and is usually a separate section for experimental/design papers. You have to answer the following questions in this section:

    • What aspects of your system or algorithm are you trying to evaluate? That is, what are the questions you will seek to answer through the evaluations?
    • Why are you trying to evaluate the above aspects?
    • What are the cases of comparison? If you have proposed an algorithm or a design, what do you compare it with?
    • What are the performance metrics? Why?
    • What are the parameters under study?
    • What is the experimental setup? Explain the choice of every parameter value (range) carefully.
    • What are the results?
    • Finally, why do the results look the way they do?

    The results are usually presented as tables and graphs. In explaining tables and graphs, you have to explain them as completely as possible. Identify trends in the data. Does the data prove what you want to establish? In what cases are the results explainable, and in what cases unexplainable if any?

    While describing a table, you have to describe every row/column. And similarly while describing a graph, you have to describe the x/y axes. If necessary, you have to consider the use of log-axes.

    If you are presenting a lot of results, it may be useful to summarize the main take-away points from all the data in a separate sub-section at the end (or sometimes even at the beginning) of the results section.

  • Future work: This section in some cases is combined along with the "conclusions" section. Here you state aspects of the problem you have not considered and possibilities for further extensions.

  • Conclusions: Readers usually read the title, abstract, introduction, and conclusions. In that sense, this section is quite important. You have to crisply state the main take-away points from your work. How has the reader become smarter, or how has the world become a better place because of your work?


Refinement

No report is perfect, and definitely not on the first version. Well written reports are those which have gone through multiple rounds of refinement. This refinement may be through self-reading and critical analysis, or more effectively through peer-feedback (or feedback from advisor/instructor).

Here are some things to remember:

  • Start early, don't wait for the completion of your work in its entirety before starting to write.
  • Each round of feedback takes about a week at least. And hence it is good to have a rough version at least a month in advance. Given that you may have run/rerun experiments/simulations (for design projects) after the first round of feedback -- for a good quality report, it is good to have a rough version at least 2 months in advance.
  • Feedback should go through the following stages ideally: (a) you read it yourself fully once and revise it, (b) have your peers review it and give constructive feedback, and then (c) have your advisor/instructor read it.

Feedback: evaluating someone else's report

Evaluation of a report you yourself have written can give benefits, but it usually is limited. Even in a group project, it is not good enough to have one person write the report and the other person read it. This is because all the group members usually know what the project is about, and hence cannot critique the paper from outside.

It is best to take feedback from your peer (and of course return favours!). The feedback procedure is quite simple. The one reading has to critically, and methodically see if each of the aspects mentioned above in the "structure of the report" are covered. It may even help to have a check-list, although with experience this becomes unnecessary.

  • Check if the title/abstract make sense, are effective/eye-catching.
  • Are all the relevant questions answered in the introduction?
  • Is the overall structure of the rest of the sections meaningful?
  • Is the difference from related/past work crisp and meaningful?
  • Are the technical sections understandable? Are the figures/tables explained properly? Is the terminology clear? Are the symbols used defined appropriately?
  • Are the results explained properly? Are the conclusions drawn from the graphs/tables sound? Or are there technical holes/flaws? Do the results show how the work presented is better/worse that the other cases of comparison?

When I give feedback on a peer's report or a student's report, I usually take a print-out and mark-up at various points in the paper. You may follow a similar procedure, or something suited to you. Be as critical as possible, but with the view that your peer has to improve his/her work, not with the view of putting him/her down. Your comments have to be impersonal. Likewise, while taking feedback from a peer, take the comments on their technical merit.


Recommended strategy for producing a high-quality report

Based on the above, I recommend the following strategy for students who want to produce a high-quality report, which would then have a high potential for being turned into a publication:

  • Think through the outline of the report even as you are working on the details of the problem. Such thinking will also lend focus to your work and you will end up optimizing the returns on the time invested.
  • Two months before the actual deadline, you have to have at least a paragraph-level outline of the report, with all details worked out.
  • After one round of critical analysis by yourselves (or by your group), have another student or another group review it, perhaps in exchange for you reviewing their work. Have them check your flow of ideas. While it may be good to get someone working in the same area, for much of the feedback, this may not really be necessary.
  • Now you are probably about 6-7 weeks from the deadline. At this point, have your advisor/instructor give feedback on the paragraph-level outline. Getting this early is important since, based on this, you may have to reorganize your report, rework your theorems, or rerun your experiments/simulations.
  • Have a pre-final version of the report ready 2 weeks before the deadline. Again, go through one round of self/peer-feedback, and then advisor/instructor feedback.
  • With these 3-4 rounds of revision and critical analysis, the quality of your report is bound to improve. And since many of the student theses are of good quality, quality of writing dramatically improves chances of publication.