Call for Artifacts

ACSOS 2024 will offer an artifact track. Our aim is to encourage the community to build high-quality artifacts that support evaluating novel approaches, methods, or applications for autonomic and self-* systems that can also be reused, shared and extended by other researchers. Our long-term goal is to establish a collection of reusable evaluation setups and case studies. Creating and maintaining a catalog of artifacts will also support discussing, comparing and assessing the research conducted in this community.

Types of Artifacts

ACSOS definition of artifacts is inspired by the ACM definition, which defines an artifact as “a digital object that was either created by the authors to be used as part of the study or generated by the experiment itself. For example, artifacts can be software systems, scripts used to run experiments, input datasets, raw data collected in the experiment, or scripts used to analyze results.”

In particular, we are interested but not limited to:

• Software systems, including implementation testbeds or exemplars. We are looking for both simulated software systems and software deployable on real hardware, as well as mobile applications. These systems should highlight the specificities in the requirements, specifications, design, implementation, testing, and validation relevant to engineering self-* and autonomic systems or present challenges that these systems address.
• Frameworks that potentially include tools and services that illustrate and implement new approaches, techniques, or algorithms for self-* and autonomic systems.
• Data and datasets that include, e.g., logging data, system traces, sensor data, or raw survey data that can support various phases of the engineering process of self-* and autonomic systems.

This list is not exhaustive; authors with questions about the suitability of an artifact should reach out to the chairs. Additionally, the published artifact papers will be associated with IEEE Reproducibility Badges in the conference program and conference proceedings. We explain more about the badging and the required processes for preparing the artifact in the Submission tab.

Submission Types

In ACSOS 2024, we call for two kinds of artifact submissions:

1. artifact standalone paper: consists of (1) an artifact paper (max 6 pages, including references) and (2) the artifact itself;
2. artifact from a full research paper accepted in the main ACSOS research track: only the artifact has to be submitted, and badges will be associated to the associated research paper.

The standalone artifact papers will be published in the conference proceedings companion. Also, a best artifact award will be issued.

Important Dates

Paper Submission Deadline: 	July 5th, 2024 
Notification to Authors: 	July 29th, 2024
Camera Ready Deadline: 	 	August 5th, 2024
ACSOS Conference:		September 26th - 20th, 2024

All times in Anywhere on Earth (AoE) timezone.

For more information on the artifacts submission to ACSOS 2024, please refer to the Submission tab.

Artifact Review Process and Selection Criteria

All submitted artifacts will be thoroughly evaluated by multiple reviewers of the artifact program committee. Each artifact will be evaluated according to the expectations set up 1) by the documentation of the artifact, and 2) by the artifact paper (in artifact standalone papers) or the full research paper (in artifact from a full research paper). In addition to just running the artifact, the reviewers may try to tweak provided inputs and create new ones, to test the limits of the system.

Artifacts will be evaluated according to the following criteria:

1. Understandability: Is the artifact easy to understand? Does the artifact identify and address a gap in the existing works and literature? Is the contribution of the artifact clearly explained?
2. Timeliness: Does the artifact address a problem that is timely? Is this value clearly explained in the paper?
3. Usability: Is the artifact executable: easy to download, install, configure and execute? Is the artifact operational: functional and usable? Can the artifact be reused by other researchers?
4. Quality of documentation: Is the artifact well-documented and structured? Is the readme of the artifact repository sufficient to run and execute the artifact? Does the documentation include the minimum system and environment requirements for usage (e.g., OS, CPU, RAM, GPU, Disk)? Are the limitations of the artifact identified and visibly documented?
5. Quality of code: Is the code of the artifact readable and well-commented?

The key benefit of investing the considerable work required to get an artifact published at ACSOS is the constant reuse of it by the community, which leads to continuously growing citations of the accompanying paper.

Submission

SUBMISSION LINK: https://easychair.org/conferences/?conf=acsos2024

In ACSOS 2024 we call submissions of two artifacts:

1. artifact standalone paper and
2. artifact from a full research paper.

An artifact standalone paper comprises two parts: an artifact paper (max 6 pages, including references) and the artifact itself. The artifact paper should summarize the research problem, the research challenges in relation to system autonomy or the concrete self-* properties that the artifact addresses. The artifact papers have to adhere to IEEE formatting instructions.

Additionally, in the Artifact Track, we also aim to review and promote research artifacts from papers that were previously accepted as part of the Research Track. Namely, authors that have a full paper accepted as part of ACSOS 2024 can submit only their artifact for review within the Artifact Track.

The artifacts in both cases are expected to be prepared according to the IEEE Artifact Reproducibility Badges Policy and have a well-documented usage of the artifact. The badges will also be visible in the electronic versions of the papers in the IEEE Xplore document on the top-right of the page to denote the availability of the artifact. Please note that in the case of an artifact from a full research paper, the badge will be assigned to the research paper. Depending on the type of artifact, the papers will be assigned with either Code Reviewed and Dataset Reviewed badges.

Papers should be submitted via EasyChair. The papers should also contain a link from where the reviewers can access and download the artifact. Please note that at this stage, it does not need to be a link to the permanent repository. In case of artifact submissions from full research papers, we request the authors, upon submission of the artifact, to submit the pre-print or the submitted version of their accepted paper in which the artifact is used and, ideally, detailed and described in some of the sections of the paper. Artifacts must not have been previously published or concurrently submitted elsewhere.

The standalone artifact papers will be published in the conference proceedings companion. Additionally, the accepted artifacts from both the standalone artifact papers and published research papers will be given a chance to present their artifact at ACSOS 2024. Lastly, we will acknowledge the most useful artifact to the community with the best artifact award.

Preparing the Artifact for Submission

Authors must perform the following steps to submit an artifact:

Preparing the Artifact

While preparing the artifact, the authors need to prepare an installation package so the artifact can be installed and executed in the environment used by the reviewer. It is important that the documentation, e.g., the README.md in the artifact repository, provide sufficient information and instructions so that the reviewers with computer science background and reasonable knowledge of scripting and build tools can install, build, and run the artifact. When preparing the artifact, it is important to keep in mind: the accessibility of the artifact to other researchers and the fact that the artifact reviewers have very limited time to assess each artifact. Therefore, the setup for your artifact should take less than 30 minutes, or it is unlikely to be endorsed simply because the committee will not have sufficient time to evaluate it.

Suppose the artifact contains or requires the use of special tools or software. In that case, the authors must provide the artifact with a working environment in a virtual image (e.g., using Oracle Virtual Box or VMware) or a Docker container image. Therefore, the code should run on all three major platforms (Mac, Win, Linux) either natively or using Docker/VM. If designed for mobile applications, instructions should be clear and concise to assist reproducibility on the appropriate platform during the artifact review process. Alternatively, authors can provide a link to a web service that allows reviewers to execute the artifact (e.g., using Jupyter notebooks). In any case, the artifact should be exercisable and appropriately validated; otherwise, they may be desk-rejected.

Making the Artifact Available

Upon submission, the authors need to make sure that the artifact is publicly available for download under an open-source license so that the reviewers can access it. We suggest a link to a public repository, e.g., on GitHub under GPL. If the artifact contains large datasets or VM images, we recommend using open (commercial) archival repositories that guarantee long-time storage, e.g., figshare and Zenodo. We encourage the authors to use repositories dedicated to data sharing where no registration is necessary for those accessing the artifacts (e.g., please avoid using services such as GoogleDrive).
The artifacts accepted to the ACSOS 2024 must be made permanently available to the public by the time the camera-ready version of the paper is due. ACSOS 2024 reserves the right to archive artifacts to a central repository or ACSOS artifact webpage in the future.

Documenting the Artifact

The artifact should be complete and carefully documented. We summarize the requirements for documenting the artifact in the following.

The artifact must be self-contained, meaning it contains the artifact itself, which may include source code, executables, data, a virtual machine image, documents, and other content deemed relevant by the authors.

The artifact must include documentation that describes the artifact. Namely, the authors need to write and submit documentation explaining how to obtain the artifact package, how to unpack the artifact, how to get started, and how to use the artifacts in more detail. The entry point to the documentation should be easy to identify (e.g., README.md or index.html file in the top-level directory of the artifact). The documentation should contain:

• a Getting Started section in which the authors summarize the key elements of the artifact, which ideally enable the reviewers to run, execute or analyze the artifact without any technical difficulty. The requirements and the artifact limitations should be discussed. If applicable, descriptions of how the artifact can be customized and extended to be reused in different research contexts.
• step-by-step instructions on how to download, install, run, configure and execute the artifact, including how to check the results of the execution. These instructions should also show how the authors propose evaluating the artifact and how a new user can get started with the artifact. If applicable, descriptions of how the artifact can be customized and extended to be reused in different research contexts. And lastly, if the installation of the artifact includes multiple steps, then we expect the authors to provide scripts that automate the build and install process or to provide a virtual image or a Docker container image.
• where appropriate, descriptions of different sub-components of the artifacts, architectures, design diagrams, and figures shall be provided that assist the comprehension and evaluation of the artifact.
• a license file or statement describing the distribution rights. Note that a reusable artifact requires some kind of open-source license.

Optionally, the authors are encouraged to include in the documentation a link to a short video (YouTube, max. 5 minutes) demonstrating the artifact.