<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE article PUBLIC "-//TaxonX//DTD Taxonomic Treatment Publishing DTD v0 20100105//EN" "../../nlm/tax-treatment-NS0.dtd">
<article xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:tp="http://www.plazi.org/taxpub" article-type="research-article" dtd-version="3.0" xml:lang="en">
  <front>
    <journal-meta>
      <journal-id journal-id-type="publisher-id">109</journal-id>
      <journal-id journal-id-type="index">urn:lsid:arphahub.com:pub:3dc5f44e-8666-58db-bc76-a455210e8891</journal-id>
      <journal-title-group>
        <journal-title xml:lang="en">JUCS - Journal of Universal Computer Science</journal-title>
        <abbrev-journal-title xml:lang="en">jucs</abbrev-journal-title>
      </journal-title-group>
      <issn pub-type="ppub">0948-695X</issn>
      <issn pub-type="epub">0948-6968</issn>
      <publisher>
        <publisher-name>Journal of Universal Computer Science</publisher-name>
      </publisher>
    </journal-meta>
    <article-meta>
      <article-id pub-id-type="doi">10.3897/jucs.2020.045</article-id>
      <article-id pub-id-type="publisher-id">24089</article-id>
      <article-categories>
        <subj-group subj-group-type="heading">
          <subject>Research Article</subject>
        </subj-group>
        <subj-group subj-group-type="scientific_subject">
          <subject>J.4 - SOCIAL AND BEHAVIORAL SCIENCES</subject>
          <subject>L.4.0 - Security and Trust</subject>
          <subject>L.5 - SIMULATION/GAMES</subject>
        </subj-group>
      </article-categories>
      <title-group>
        <article-title>Evaluating Case Study and Action Research Reports: Real-world Research in Cybersecurity</article-title>
      </title-group>
      <contrib-group content-type="authors">
        <contrib contrib-type="author" corresp="yes">
          <name name-style="western">
            <surname>Vrhovec</surname>
            <given-names>Simon</given-names>
          </name>
          <email xlink:type="simple">simon.vrhovec@um.si</email>
          <uri content-type="orcid">https://orcid.org/0000-0002-6951-6369</uri>
          <xref ref-type="aff" rid="A1">1</xref>
        </contrib>
        <contrib contrib-type="author" corresp="no">
          <name name-style="western">
            <surname>Fujs</surname>
            <given-names>Damjan</given-names>
          </name>
          <xref ref-type="aff" rid="A2">2</xref>
        </contrib>
        <contrib contrib-type="author" corresp="no">
          <name name-style="western">
            <surname>Jelovčan</surname>
            <given-names>Luka</given-names>
          </name>
          <xref ref-type="aff" rid="A1">1</xref>
        </contrib>
        <contrib contrib-type="author" corresp="no">
          <name name-style="western">
            <surname>Mihelič</surname>
            <given-names>Anže</given-names>
          </name>
          <xref ref-type="aff" rid="A3">3</xref>
        </contrib>
      </contrib-group>
      <aff id="A1">
        <label>1</label>
        <addr-line content-type="verbatim">University of Maribor, Maribor, Slovenia</addr-line>
        <institution>University of Maribor</institution>
        <addr-line content-type="city">Maribor</addr-line>
        <country>Slovenia</country>
      </aff>
      <aff id="A2">
        <label>2</label>
        <addr-line content-type="verbatim">University of Ljubljana, Ljubljana, Slovenia</addr-line>
        <institution>University of Ljubljana</institution>
        <addr-line content-type="city">Ljubljana</addr-line>
        <country>Slovenia</country>
      </aff>
      <aff id="A3">
        <label>3</label>
        <addr-line content-type="verbatim">University of Maribor, Ljubljana, Slovenia</addr-line>
        <institution>University of Maribor</institution>
        <addr-line content-type="city">Ljubljana</addr-line>
        <country>Slovenia</country>
      </aff>
      <author-notes>
        <fn fn-type="corresp">
          <p>Corresponding author: Simon Vrhovec (<email xlink:type="simple">simon.vrhovec@um.si</email>).</p>
        </fn>
        <fn fn-type="edited-by">
          <p>Academic editor: </p>
        </fn>
      </author-notes>
      <pub-date pub-type="collection">
        <year>2020</year>
      </pub-date>
      <pub-date pub-type="epub">
        <day>28</day>
        <month>07</month>
        <year>2020</year>
      </pub-date>
      <volume>26</volume>
      <issue>7</issue>
      <fpage>827</fpage>
      <lpage>853</lpage>
      <uri content-type="arpha" xlink:href="http://openbiodiv.net/813B0AD5-89D3-5D3E-9D4F-E06CC5D36874">813B0AD5-89D3-5D3E-9D4F-E06CC5D36874</uri>
      <uri content-type="zenodo_dep_id" xlink:href="https://zenodo.org/record/5508557">5508557</uri>
      <history>
        <date date-type="received">
          <day>11</day>
          <month>01</month>
          <year>2020</year>
        </date>
        <date date-type="accepted">
          <day>13</day>
          <month>02</month>
          <year>2020</year>
        </date>
      </history>
      <permissions>
        <copyright-statement>Simon Vrhovec, Damjan Fujs, Luka Jelovčan, Anže Mihelič</copyright-statement>
        <license license-type="creative-commons-attribution" xlink:href="https://creativecommons.org/licenses/by-nd/4.0/" xlink:type="simple">
          <license-p>This is an open access article distributed under the terms of the Creative Commons Attribution License (CC BY-ND 4.0). This license allows reusers to copy and distribute the material in any medium or format in unadapted form only, and only so long as attribution is given to the creator. The license allows for commercial use.</license-p>
        </license>
      </permissions>
      <abstract>
        <label>Abstract</label>
        <p>There is a growing number of scientific papers reporting on case studies and action research published each year. Consequently, evaluating the quality of pilling up research reports is becoming increasingly challenging. Several approaches for evaluation of quality of the scientific outputs exist however they appear to be fairly time-consuming and/or adapted for other research designs. In this paper, we propose a reasonably light-weight structure-based approach for evaluating case study and action research reports (SAE-CSAR) based on eight key parts of a real-world research report: research question, case description, data collection, data analysis, ethical considerations, results, discussion and limitations. To evaluate the feasibility of the proposed approach, we conducted a systematic literature survey of papers reporting on real-world cybersecurity research. A total of N = 102 research papers were evaluated. Results suggest that SAE-CSAR is useful and relatively efficient, and may offer a thought-provoking insight into the studied field. Although there is a positive trend for the inclusion of data collection, data analysis and research questions in papers, there is still room for improvement suggesting that the field of real-world cybersecurity research did not mature yet. The presence of a discussion in a paper appears to affect most its citation count. However, it seems that it is not uniformly accepted what a discussion should include. This paper explores this and other issues related to paper structure and provides guidance on how to improve the quality of research reports.</p>
      </abstract>
    </article-meta>
  </front>
</article>
