Reviewer guidelines

Thank you for considering reviewing for the Journal of Visualization and Interaction (JoVI)! As with any peer-reviewed journal or conference, it is our volunteer reviewers who make it possible for us to publish valid and high-quality scientific papers. Reviewing for JoVI is in many ways the same as in reviewing for any other peer-reviewed venue. In this document, we will describe the overall review process, focusing in particular on topics that may be different from a typical journal or conference.

Before Agreeing to Review

Confidentiality

By agreeing to review for JoVI, you also agree to maintain the confidentiality of all materials shared with you during the review process. Do not use any of the ideas, concepts, or techniques disclosed to you as part of the review process in your own research until they have been made public.

Anonymity and Identity

Authors can choose to list their names or be anonymous during the reviewing process. We acknowledge the potential issues with unblinded review. For further discussion and feedback please see issue #16.

A reviewer is still eligible to review if they know who the authors are due to seeing the work on social media or seeing a prior version of the work. However, reviewers should not “play detective” and actively seek anonymous author identities. In other words, please do not search for paper titles, open-source project names, or preprints during the review.

All reviews for JoVI will be published along with the final article, grouped by review round.

External reviewers can choose to be named or to remain anonymous by simply choosing whether or not to sign their review (e.g., “signed, Jane Doe”). Signing reviews is encouraged, especially for thoughtful and thorough reviews.

For accountability, the action editor is not anonymous.

Conflicts of Interest

Peer review is based on unbiased and fair reviewing practices. Conflicts of interest (COIs) are situations where a reviewer is unable to (or is perceived to be unable to) uphold a lack of bias and fairness. To prevent COIs from impacting the process, JoVI abides by the general ACM guidelines for conflicts of interest. Generally speaking, if you feel unable to give an unbiased and fair opinion on a submission given its authors or their affiliations, you should not review it.

Example conflicts inappropriate for reviewing:

  • Ph.D. or postdoctoral mentor or mentee;
  • Family or close personal relationship;
  • Co-author on a peer-reviewed paper within the last 2 years;
  • Current collaborator on in-progress work or work under review; and
  • Anyone at the same institution (excluding different branches or campuses).
  • Professional friendship or animosity.

First Impression

Paper Length

JoVI enforces no page limits or writing style on papers, but papers which are overly long or difficult to read are wasteful of both readers’ and reviewers’ time. The page length of a paper should be commensurate with its contribution, so reviewers are encouraged to point out parts of the submission that can be trimmed. The length of a manuscript should not however be used to recommend its rejection — rather it is a reason to suggest revision.

The corollary is that there is also no minimum page count for JoVI papers; short papers are encouraged if they represent succinctly described and useful ideas of general value to the visualization and interaction community.

Key Assessment Criteria

Knowledge

The submission must prove its merit in advancing knowledge related to the fields of visualization and human-computer interaction. Knowledge advancement can be achieved in several ways, including but not limited to:

  • Empirical studies investigate research questions that have not been considered before
  • Replication studies improve or challenge the confidence of the field about existing knowledge through better methodology than the original studies (e.g., higher statistical power, improved methods, new population, etc)
  • Systems or design research expands the capacity of people to do useful things that were not previously possible
  • Systematic literature reviews enables the field to synthesize knowledge across existing work
  • Commentary enables the field to recognize unseen phenomena, see new perspectives, or challenge the status quo

Claims and evidence

When reviewing JoVI papers, focus on these questions about claims and evidence in the manuscript:

  • Are the claims and research questions clearly communicated?
  • Are the claims supported with evidence?
  • Match between claims and evidence: Does the provided evidence actually answer the question and support the claim? Does it support a stronger, weaker, or different claim? For example, if a paper only measures computational performance, it should not include claims about improved human behavioral performance.
  • Generalization and transfer: If the authors claim generalization or transfer beyond the evidence presented in the present studies: How strong are the arguments? How clear is the scope of generalization or transfer described?
  • Analytical validity: If there is any computational or statistical analysis, is it an appropriate match for the research question? Is it reported in an accurate and interpretable way?

Note: While lack of novelty is not a concern at this journal, failure to contrast with very similar work may be problematic.

Openness and Transparency

All JoVI submissions are required to include all material needed to properly scrutinize and build on the research. For any empirical components, manuscripts must make all data and reasoning available in ways that invite scrutiny, so that subsequent researchers can assess claims, reuse methods and materials, and build upon findings. For any computational components, all code needs to be reproducible within a reasonable effort. When ethical concerns prevent submissions to meet these requirements, the manuscript must clearly describe the characteristics of these artifacts and adequately justify the tradeoff between openness and ethics.

Examples of materials that should be included, except where justified as above: experiment code, software and hardware components of prototypes, stimuli, interview guides, transcripts, raw empirical data, analysis scripts, and URLs to preregistrations. For full details, see level 2 of the TOP guidelines (summary table and thefull guidelines).

As a reviewer, you are simply asked to check that the materials are present and accessible. You are encouraged (but not required) to check the validity of any part of the submission using its replication and reproducibility materials. If you do so, you should mention it in your review.

If you cannot access any of the material, you can stop the review to avoid wasting time. Tell the action editor immediately, and they will then contact the author to remedy the problem. The action editor will confirm, ask the authors to fix the problem, and reply to the reviewers once the review process can continue.

Under special circumstances, a submission may state the reason that material cannot be shared (e.g., for protecting participants’ identity). Such a statement must appear in the body of the paper. Reviewers and editors must make a judgment call as to whether the authors shared as much as possible and whether the rationale is acceptable.

Examples:

  • Acceptable: Sharing all of the data except subject-identifying columns.
  • Acceptable: Putting any necessary identifiable information, transcripts, and recordings in a protected-access repository that requires following an ethical-vetting procedure to access.
  • Unacceptable: Authors claim that material is their own intellectual property or will not share it due to commercial purposes.
  • Unacceptable: Only sharing aggregated data and omitting the raw data and data aggregation code without stating why.
  • Unacceptable: Authors told their IRB that they would not share the material despite it appearing fully anonymized.

Writing Your Review

JoVI reviews should be thorough but kind: aim for completeness and clarity of critique, but also think about how the authors will read your review (write the kind of review you would appreciate receiving). JoVI does not have a set rejection rate, so any work meeting our key criteria (above) should be a candidate for acceptance.

Modular Reviews

To facilitate understanding by the editor and responses by the authors, reviewers are encouraged to organize concerns into numbered sections. Each section should include:

  1. A description of the concern
  2. Where the issue occurs
  3. Severity: Is this a threat to validity? Or is it a suggestion?
  4. How to fix the issue, or how an editor would know if the issue has been fixed

Please put minor issues and typos into one “minor issues” section at the end of your review.

Intentional Citations

Each citation should have a clear purpose: to offload evidence or to build on existing progress.

  • References should be used to explain the rationale, methods, and claims.
  • Avoid telling authors to add references without stating what evidence or prior use that reference would support. A laundry list of references that “should be cited” is not helpful. If the reviewer wishes to suggest additional references, they must make a clear argument why.
  • References can also be used to contrast the article with other very similar articles. However, this style of citation should be kept to a minimum and only used to clarify differences.
  • A designated “Related Work” section is not required.
  • Missing citations that could be easily remedied in revision is generally not grounds for rejection.

Do’s and Don’ts

  • If you only have expertise in one facet of a multi-faceted paper, state what facets of the paper you were qualified to review. Example: My expertise is mostly in machine learning, so my review focuses on that part of the paper.
  • Avoid “struggling to understand”. The difficulties could be one of the following issues:
    • Technical terms, theories, conceptual frameworks, or methods that you have inadequate knowledge of: Explicitly state the limitation of your expertise in the review form.
    • Verbose or convoluted explanations: Precisely indicate which part(s) are verbose, convoluted, or confusing. Point out contradictions in the explanation, and suggest how the authors could improve them.
  • Typos and grammatical mistakes: Pointing out typos and grammatical mistakes is a greatly appreciated service, though not required. However, unless the article has severe readability issues, typos should not negatively impact how the article’s content is reviewed. And a long list of typos is no substitute for a thorough review.
  • Avoid assuming the nationality of the authors. Instead of saying “Have the paper proofread by a native speaker”, say that there are many grammatical mistakes and list a few examples.
  • Avoid arbitrary heuristics (e.g., subject count, color palettes, or cutoffs of statistical measures) without explaining why failing to abide by them calls conclusions into question.
  • Avoid prescribing preferred statistical approaches. If you say that a paper’s analysis or reporting needs to change, you should explain how their current approach fails to answer the questions the paper asks. Of course, you can request that a given approach is presented in a clear and transparent way.
  • Avoid statements about the authors. The review should comment on and objectively critique the paper’s contribution, not the authors, their background, or their intent. Instead of “The authors failed to…”, write that “The paper does not…”.
  • Limit the usage of opinionated statements (e.g., “I like”, “I dislike”) on issues that pertain to the validity of the paper. Instead, provide a substantial argument for your critique. Nevertheless, “I like” could still be used to complement stylistic and presentation choices.

Registered Reports

A registered report (RR) is a subtype of the empirical research articles with two stages: In the initial submission, the motivation, methods, and potential interpretation of methods are described, but the data collection has not yet been performed. The reviewers can evaluate and recommend revision prior to the authors collecting data. Once the initial stage is accepted, the authors can run the study knowing that the results will not impact whether the study is published. The second stage of review primarily checks that the authors stuck to the original plan, documented any deviations, and drew appropriate conclusions. For more information, see Registered Reports at COS. This type of submission might not be suited for all kinds of visualization or HCI papers (see discussions here). Each reviewing phase of a registered report has to focus on different aspects of the submissions.

Additional methodological points such as suggesting an additional analysis or data collection that does not directly fit with the authors’ research question are encouraged but cannot be used to assess the validity of the submission and thus decide to reject or accept it. Once authors, reviewers, and editor(s) agree on the methodological plan, the submission is accepted in the first round.

The second round of reviewing aims at verifying that the plan has been followed by the authors in their data collection, analysis and reporting. Reviewers will thus carefully examine the report of the results and the conclusions drawn from the analysis to ensure that they fit the strength of evidence that the authors have gathered. In some cases, authors might have had to deviate from the plan they agreed to in the first stage of the RR. How to treat such deviations is currently an open question in the scientific landscape of venues accepting RRs and can only be assessed on a case-by-case basis. As our community further embraces RRs, we might be able to extract guidelines on the nature and criticality of deviations. Reviewers should thus re-read the methodology section which will highlight any and all deviations and reflect on the criticality of these deviations with respect to the evidence brought forward by the authoring team. The final recommendation should only depend on the deviations (if any) and the strength of the claims made in the results, discussions, and conclusion. The direction of the results themselves (even if null or negative) cannot be used as an argument to reject the manuscript or suggest changes.

We expect, in the case of deviations, that editors and reviewers will have to thoroughly discuss the implications they have on the final decision.

Possible Review Outcomes

JoVI aims to advance knowledge. If the work is promising but still has not made the cut, reviewers and editors should encourage the authors to revise and resubmit. Rejections should be used sparingly. The following outcomes are possible at each stage:

Before sending out to external reviewers:

  • Desk reject: The editor-in-chief may choose to desk-reject manuscripts that are clearly outside the scope of the journal, or not written in English. We may set more specific desk rejection criteria in the future; for feedback and discussion on this see issue #17.
  • Pre-review revision: If a manuscript misses the required components described in the author guidelines, or if the associate editor has other concerns (such as length not being commensurate with contribution), the editor may request the authors to make some revisions prior to sending the manuscript out to reviewers.

After each round of review:

  • Reject: This decision will only be taken upon critical methodological, ethical, or technical flaws that cannot reasonably be fixed, even with major additional effort, or if the submission has not improved substantially over several revisions..
  • Major revision: In addition to critiques and suggestions, reviewers may also ask for clarifications in the major revision.
  • Minor revision: Reviewers have reached a consensus that the required revisions could be checked by the editor without requiring further input from the external reviewers.

After the final round of review:

  • Accept: The content of the manuscript is considered final. No further changes are allowed without approval from the editor.

Associate Editors

If you have been invited as an associate editor for JoVI, we also suggest looking through the following documents: