Tutorial for Integrating and Hosting Experimental Lab Studies Through Amazon Web Services

Anthony Naranjo, University of Central Florida; Ezequiel Gioia, University of Central Florida; Samuel Donnelly, Saint Louis University; Dustin Jundt, Saint Louis University; Shiyang Su, University of Central Florida; and Mindy Shoss, University of Central Florida

Abstract

There has been a recent urgency to leverage modern technology to facilitate virtual experimental research and data collection across various social science disciplines. Various platforms have emerged to help transfer experimental lab software to virtual lab environments. However, the existing aids are limited by the degree of task complexity they can accommodate (e.g., dynamic simulation tasks), as well as the adherence of various data security governance they can provide to researchers. In an effort to alleviate these discrepancies, we present a comprehensive technical tutorial for transitioning software-based experimental tasks into the more dynamic and secure platform, Amazon Web Services. We also illustrate the strengths and weaknesses of administering virtual lab sessions and provide various solutions for optimizing security, controllability, user experience, and data collection speed. To support these objectives, we provide a detailed set of open source resources to aid researchers in extending experimental lab studies to geographically dispersed and, ideally, more generalizable sample populations.

Keywords. Amazon Web Services; experimental research; virtual lab studies; cloud computing; virtual machines    

Author’s Note: The current tutorial utilizes a windows-based application (.exe) hosted on a Windows virtual machine. However, AWS/Appstream 2.0 can host a number of different types of applications (.exe, .msi, .iso, etc.) and can be connected to with commonly available web browsers for PC and Mac that support HTML 5.

Tutorial for Integrating Experimental Lab Studies Online Through AWS

Recent efforts have sought to facilitate online experimental research when in-lab research is restricted (e.g., COVID-19; see http://www.movingresearchonline.info/) and to increase the generalizability of inferences beyond student samples (cf., Henrich et al., 2010). To these ends, we provide a technical tutorial for a flexible and secure virtual resource: Amazon Web Services (AWS). AWS requires minimal programming background, supports programs that run on a local device, and offers hundreds of security features. First, we demonstrate how to integrate experimental tasks into AWS. Second, we outline the operational steps needed to conduct virtual lab sessions using AWS in conjunction with communication services (e.g., Zoom). Third, we address potential decision points that researchers may face when employing these resources. Although this tutorial is designed to guide readers with varying levels of programming experience, we anticipate that individuals with at least some foundational programming knowledge will be able to follow along with this tutorial successfully. However, to the extent that researchers want to leverage this tutorial for various lab simulations or modify the nature of the task, this will require more advanced programming experience.

Amazon Web Services

AWS is a cloud computing platform that enables researchers to virtually host experimental tasks. Specifically, AWS contains a suite of applications, including AppStream 2.0, which allows users (e.g., participants) to access a lab task application (e.g., ATC-lab Advanced) using a web browser and Internet connection. The application runs on a virtual machine within AWS, which allows for bidirectional communication, thus sending input from the user’s mouse and keyboard actions to the streaming application.

Whereas other platforms generate tasks (e.g., PsychoPy, jsPsych) to be hosted and distributed on virtual networks (e.g., Pavlovia), AWS provides multiple benefits. AWS can accommodate any task or programming language (e.g., C++, Ruby, Java, Python) that runs on a local device rather than being limited to tasks written in JavaScript (in jsPsych and lab.js) or Python (in PsychoPy). AWS also offers higher base security and compliance with a number of data regulations (e.g., HIPPA, FISMA, GDPR). As such, AWS is a prime candidate for hosting tasks and storing data across a number of research disciplines with diverse research needs (e.g., psychology, management, economics). Given the extensive benefits provided by AWS, there is a cost associated with utilizing AWS, see https://aws.amazon.com/appstream2/pricing/. However, the monthly users fee can be discounted for universities and schools.

Running Example: Air Traffic Control Simulation Task

We will first provide a brief background on the task we’ll employ as a running example. ATC-lab Advanced (Fothergill et al., 2009) is a publicly available air traffic control simulation task that has been used in numerous research domains such as intelligence and working memory (e.g., Loft & Remington, 2013); ocular movements (e.g., Marchitto et al., 2012); motivation and decision making (e.g., Gee et al., 2018); affect (e.g., Yeo et al., 2014); and performance (e.g., Wilson et al., 2020). Its objectives are highly customizable but may include actions such as accepting/handing-off aircraft that enter/leave one’s designated air space and preventing conflicts between aircraft. The ATC-lab Advanced task environment and scenarios are created in customizable XML scripts, and data are stored in a working directory as a meta-data log and .csv file.

Tutorial for Integrating Lab Tasks Into AWS

This tutorial is designed to be referenced in conjunction with an integral vignette containing screenshots, code, tables, and more detailed instructions accessible via Open Science Framework (https://osf.io/4hqsm/?view_only=2747a3cf69c54297a0842651c262062f). To reference specific sections of the vignette in this tutorial, we denote text line numbers within brackets corresponding to numbered text lines in the vignette. For example, [4-18] contains a table describing common AWS terminology. Note, carrying out the tutorial will require an AWS account with administrative access. Instructions for creating an account are available here: https://aws.amazon.com/premiumsupport/knowledge-center/create-and-activate-aws-account/.

Creating S3 Buckets

We start by setting up Amazon’s S3 service, which houses the task software and stores task data. Two S3 buckets are created; the first bucket is to store the ATC-lab task application, its XML scenario files, and scripts that are part of this example solution. The second bucket is where the task data are stored, which is the output of the ATC-lab .csv and log files. We refer to these as the “app bucket” and “data bucket.” Buckets can be thought of as folders or “containers” to store files. S3 is a data storage service, which can be interacted with through the command line interface (see resource: https://aws.amazon.com/cli/), a programming language, or web browser. S3 can be thought of as somewhat analogous to services like DropBox or OneDrive. Two S3 buckets are necessary for separating the application and configuration from the data following the AWS security “least privilege access” best practice (https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#grant-least-privilege). With this approach, the task application can download the latest version of the application and configuration files from the app-bucket and write the data output into the data bucket (see Figure 1 below). 

Figure 1

Illustration of Security Permissions

Note: One policy defines permission to read the task (application, configuration files), whereas a second defines permission to write the output (data) in a separate S3 bucket. Separating where the task is accessed from where the data are stored, along with the permissions to access each, increases security so users (participants) are unable to make any modifications to the task or access data.

On the Amazon Web Console, search for “S3” and select the S3 service (accessible here; https://console.aws.amazon.com/) [19]. Once in the S3 management console, select “Create bucket” [20]. The default bucket settings are sufficient to start; these settings can be customized at a later point based on one’s requirements. Simply enter a name for the first bucket in the text box [21] (keeping the below default settings [22-24]), then scroll to the bottom of the page and select “Create bucket” [24], and repeat this same process for the second bucket. The following bucket names are used for the purpose of this tutorial: atc-lab-app and atc-lab-data. It is important to consider that S3 bucket names are global with regard to the AWS cloud; thus, if the proposed bucket name is already taken, a different name must be chosen. For instance, making a slight modification to your proposed bucket name to make it unique is a suitable approach (e.g., atc-lab-app-university).

Configure Permissions

Here we introduce the Identity and Access Management (IAM) service, which allows users, groups, and roles to be created and provides control to the access of resources and other services in AWS. The following sections will outline how to configure permissions of the S3 buckets by creating two IAM policies and one IAM role, both of which are described in detail below.  

Create IAM Policies

On the Amazon Web Console, search for the “IAM” service, select “IAM” and click on “Policies” on the left-hand side of the screen [25], and then select the “Create policy” option on the upper portion of the screen [25]. Once on the “Create policy” page [26], select the “JSON” tab [27]. Two policies must be created to independently control access to each bucket, one defining permission to read: atc-lab-app (application/task configuration or input to AppStream 2.0), and the second defining permission to write: atc-lab-data (data or output from AppStream 2.0). For the first policy [28], simply copy and paste the corresponding JSON scripts for the “read policy” (copy code from [31-47] into the JSON tab) and replace your unique read bucket names created in the S3 section [21] into the highlighted “Resource” lines [42-43]. After the JSON code has been updated, select “Next: Tags” (tags are optional so we skip ) [28], then select “Next: Review” [29], and last, fill in the name for your read IAM policy (e.g., lab-app-s3-read-policy) and click “Create policy” [30]. Then repeat the steps [25-30] for the write IAM policy, this time copy and pasting the code from [48-58] and replacing the S3 write bucket name into the highlighted text line [55]. The names chosen for the “read” and “write” policies in this tutorial are atc-lab-app-s3-read-policy and atc-lab-data-s3-write-policy, respectively. An IAM policy is a JSON (JavaScript Object Notation; https://www.json.org/json-en.html) document that defines the access to resources in AWS. Defining one S3 bucket for strictly running the task and a second for strictly storing data increases security and controllability.

Create an IAM Role for AppStream 2.0

Once the JSON policies are created, the IAM policies need to be associated to the streaming machines so that both S3 buckets can be accessed from AppStream 2.0. IAM roles will be used to create this association. An IAM role is an entity where permissions can be defined attaching policies to a specific IAM role and later a user or a service can assume this role, granting access to the resources specified in the policies. In the current solution, the role will be assumed by the AppStream 2.0 service to access the S3 buckets that were created above. One of the main advantages of using IAM roles is that it circumvents the necessity to embed credentials in our scripts or configuration files, which is beneficial as this is often a security concern. On the IAM console, click on “Roles” [59] on the left side of the screen, and then click “Create role” [59]. Follow the four steps in Table 1 below to create the IAM role.

Table 1

Steps for Creating IAM Role

Configure AppStream 2.0

Create an Image Builder

The next step is to create an Image Builder, which is a virtual machine where the task will be installed, configured, and tested. Once our sought-after results are achieved regarding configuration and testing, then the image can be created. This image will be the template that AppStream 2.0 service will use every time a new streaming instance is requested by experimenters. On the Amazon Web Console, access the “AppStream 2.0 service.” Then, on the navigation menu, click on “Images” [65], select the “Image Builder” tab [66], and click on the “Launch Image Builder” button. This launches the Image Builder wizard. Follow the four steps in Table 2 below to create an Image Builder.

Table 2

Steps for Creating Image Builder

Once you have selected “Launch,” it will take some time for the status to change from “pending” to “running.” While this process is ongoing, move on to the “Create an Image” section below.

Create an Image

Prepare the App Bucket

The “app bucket” must now be prepared by first cloning or downloading the repository template (e.g., download task source code). To aid in this process, we created a repository template that may be updated to accommodate any task or program, which is accessible on our OSF page linked above in a zip file titled “bucket-template.” The directory structure is displayed here [83-92]. Note that in order for the code provided below to function correctly, the folder names used must match the folder names provided in the repository template. The /app is simply the directory (folder) housing the executable file(s) for the task [83]. Once a copy of the repository template is obtained, the application files will need to be copied into the /app directory [93] (delete existing files used for ATC-lab in /app if using another task program). This conjunction of the application, configuration files, and scripts will be distributed to each streaming instance. Two of the script files will need to be modified to your custom bucket names. Open and edit install.bat and run.bat files from /scripts in a text editor (e.g., notepad on Windows, or TextEdit on Mac) [94]. In both of these files, replace the names of the bucket with your own bucket names into the highlighted sections [96, 97, 114, 115, 116]. Then in the run.bat file, replace the highlighted section in [125] with the executable file name of your task, and then replace the output file names for your task in the highlighted code chunks on line [133] (assuming you’re using a task other than ATC-lab). The download.bat file [137-148] does not require any modifications. If you are running more complex tasks, such as those utilizing other scripting languages (e.g., https://github.com/lukestrickland/ATC_XML_generator), this [125] is where you would specify those files. Last, make sure all files are saved as .bat files.

Once the two files have been configured (highlighted sections have been replaced), access the AWS S3 web console in order to upload all the files into the S3 application bucket (search and select your app bucket; e.g., atc-lab-app). You can drag and drop the directory directly into S3 [149], select “Upload” at the bottom of the screen [150], and confirm the upload was successful for all files [151].

Connect to Image Builder

Now go back to the AppStream 2.0 service, select Images, open the Image Builder tab, and once the status shows as “Running,” connect to it [152-153] by clicking “connect” (a new tab should now load, if not make sure your pop-up blocker is turned off). Note that once the Image Builder is running, you will start being charged by AWS. As a cost-saving suggestion, stop the Image Builder in the AppStream 2.0 service if you want to disconnect before finishing the image creation. Image Builder serves three different roles: (a) administrator, (b) template user, and (c) test user [154]. The “administrator” and “test user” roles will be utilized to install the application, its dependencies, create default Windows settings, and finally test the application. Upon accessing the Image Builder [154], select the administrator role [155]. Note you are now on a virtual machine desktop [156].

Install the App

The files that were previously uploaded to the app bucket in “prepare the app bucket” section will be used as a source to install the application, scripts, and scenarios into the Image Builder instance. For this, open a Windows command prompt as administrator (go to the start menu and search “cmd,” then right click, and select “Run as administrator”). Once in the command prompt, copy and paste the line of code on [157], and hit “enter” to execute the code. Next, copy and paste the line of code on [158]; however, make sure to replace the name of the bucket with your app-bucket name into the highlighted syntax on [158] before executing the code line. Last, copy, paste, and execute the code on [159] (once complete, command prompt should look like the screenshot) [160]. The first command [157] changes the directory to the root folder, and the second command [158] downloads the installation script from the app bucket. On the third command [159], we execute the installer, which will download the rest of the files (e.g., ATC-lab task, scripts, and scenarios) from S3 into C:\Apps directory.

Configure Image Assistant

Now, double click on the “Image Assistant” icon located on the Image Builder’s desktop (not your local desktop) [161]. This will launch a six-step wizard that communicates to AppStream 2.0, which application and application settings to stream.

Step 1. Add Apps. Click on “Add-App” [161], and follow the directory path to open the executable file in Apps/ (C:\Apps\ATCLab\run.bat) [162]. Then fill in the corresponding “Name,” “Display Name,” (note names are arbitrarily created for “Name” and “Display Name”), “Launch Path,” and “Working Directory,” and click “Save” [163] and “Next” [164].

Step 2. Configure Apps. Keeping the Image Assistant open to Step 2 [165], open a Windows File Explorer and browse to C:\Apps, right-click on the Apps/ folder, and then select “Properties” and the “Security” tab [166]. Then select “Edit” [166], and after making sure you have “Users” highlighted, grant “Modify” permission under the “allow” column [167] and click “OK.” Now back in the Image Assistant window under “configure apps” tab [165], click on “Next.”

Step 3: Test. Under the “test” tab [168], switch user to “Test User,” reopen the Image Assistant, and click on your application (e.g., ATC-lab) [169] to test the task [170]. Next, in the AppStream 2.0 Image Assistant window [169], switch users again to “Administrator” and click on “Next” under the “test” tab [168].

Step 4: Optimize. Under the “optimize” tab, select “Launch” [171], wait for the application to load (once script stops loading in background) before selecting “Continue” and then “Next” in the Image Assistant.

Step 5: Configure Image. In step 5, enter a name and display name (e.g., atc-lab-image) for the image, and click “Next” [172].

Step 6: Review. Last, under the “Review” tab, click on “Disconnect and Create Image” [173]. The connection to the Image Builder will be closed (this will cause the image assistant to fail; this should happen), and the Image Builder status will transition to “Snapshotting” (note this process will take up to an hour and needs to be completed before proceeding). Now the Image Builder is finally creating the image, and it will be stopped once the image is ready.

Create a Fleet

A group of streaming instances is called a “fleet,” and each streaming instance in the fleet is ephemeral. The streaming instances will be discarded after use and cannot be shared between users. This design decision of AppStream 2.0 helps in maximizing security. On the AppStream 2.0 main menu, click on “Fleets” and then on the “Create Fleet” button [174]. AWS AppStream 2.0 Fleets are created through the following five steps.

Step 1: Provide Fleet Details. Add a name and description (e.g., atc-lab-fleet), and click “Next” to advance to Step 2 [175].

Step 2: Choose an Image. This is where we are going to select the image (e.g., ATC-lab-image) [176] created in the previous section and then click “Next.”

Step 3: Configure Fleet. Select the “streaming.standard.small” type of general purpose instance [177]. After scrolling down, make sure “On-Demand” is selected for “Fleet type” [178]. Then under “User session details,” specify the desired durations in minutes of the session, disconnect timeout, and idle disconnect timeout (see our example inputs in [178]). These parameters are referring to the actual task to be accessed in URLs, so be sure to provide adequate time based on the duration of the experiment. Next specify the capacity for the fleet [179]. Capacity refers to the number of users (participants) within each virtual lab session. At the bottom of the “Step 3: Configure Fleet” page [180], make sure “Application” is selected for the “Stream view” and that the IAM role created earlier is selected before clicking “Next.”

Step 4: Configure Network. Make sure “Default Internet Access” is checked and that the same VPC as the one selected in “Create an Image Builder” section is selected from the drop-down list [181]. Both subnets may be chosen arbitrarily, and the security group created previously (e.g., atc-lab-private) should be selected from the drop-down menu before selecting “Next.”

Step 5: Review. Last, review the fleet configuration and click on “Create” [182]. A pop-up message will alert us regarding how the pricing model works (see [183-186] for more details on pricing). The fleet is created with two default scaling policies; delete both following the steps described in [187-191] because we will manage the number of streaming instances manually.

Create a Stack

Stacks are the last main component of AppStream 2.0; these are configuration sets to specify the fleet, the user access policies, and the storage that will be used in our solution. It is also where one may customize the appearance of AppStream 2.0 to apply any specific branding. Click on the “Stacks” option on the AppsStream 2.0 main menu [192] and then on the “Create Stack” button. From here follow the four steps outlined below in Table 3.

Table 3

Steps for Creating a Stack

Generating and Distributing Streaming URLs on AWS

In order to access the AWS console and AppStream 2.0 application, researchers will need to login into the AWS web console using their provided credentials. Once at the home screen, researchers will need to search for the AppStream 2.0 application using the search bar located at the top of the home screen. On the AppStream 2.0 home screen, configure the desired capacity for the AppStream 2.0 fleet. Capacity refers to the number of virtual machines (e.g., number of participants) that AppStream 2.0 will launch. For example, if you are hosting two simultaneous sessions (two experimenters) each consisting of three participants, then the desired capacity should be configured to six. This can be done under the “Fleets” tab by selecting your task under the “Items” list and editing desired capacity under the fleet details section.

Once the desired capacity has been set, under the same Fleets tab, researchers will start the fleet by selecting the “Actions” drop-down menu and selecting “Start.” This startup process takes approximately 10 minutes and researchers will know that the fleet is running via the status column, which will display “Running.” The web page can be refreshed periodically to check the status of fleet startup. After the fleet is running, the task simulation streaming URLs can now be created. Researchers will navigate to the “Stacks” tab on the left-hand side of the web page, select their lab task under the “Items” list, click the “Actions” drop-down menu, and select “Create streaming URL.” A menu box will pop up prompting for a “User ID” and URL expiration time. Researchers should follow a standardized naming convention for user IDs (e.g., user-01, user-02) and reuse these IDs across different lab sessions as the number of IDs used is one pricing dimension considered in the AWS pricing model. With regard to URL expiration time (how long the URL link is active), researchers should select a time that best suits their research needs (e.g., 1 hour, 2 hours, etc.). Once this information is provided, click “Get URL,” and a URL will be produced that may then be distributed directly to participants. The process of creating URLs will have to be repeated to accommodate the number of participants in a single lab session. For instance, if a researcher has three participants in a single lab session, then they should plan on creating three separate streaming URLs to share with each participant.

Prior to sharing the URLs with participants, researchers should be sure to launch the URLs in their personal web browser to start the actual simulation, as this takes about 2 minutes. At this point, researchers can begin their study and will be able to have access to all streaming URLs.

Considerations for Conducting Virtual Lab Sessions

Although both AWS and video conference software can support dozens of users simultaneously, technical, situational, and cognitive demands may limit the number of participants a single experimenter can practically run at a time. For instance, when participants and experimenters are connected via virtual conferencing software (e.g., Zoom), having numerous participants with unique technical or comprehension issues that require remediations one at a time would substantially increase session length. Experimenters using video conference software also need to be cognizant of and plan for dealing with potential environmental limitations (e.g., participants may not be alone, anonymity may be difficult to preserve). Potential solutions include requiring video cameras to be turned on, encouraging participants to be in a well-lit room by themselves, communicating contingency plans to participants if they were to experience technological issues (e.g., Internet connection is lost), and requiring participants to change their Zoom name within the video conference to an anonymous research identifier prior to joining lab sessions. In order to successfully monitor and interact with participants, we have found the ideal number of participants per experimenter should be kept relatively low (approximately 3–4 participants, although this may vary with the complexity of the chosen task). However, given that AWS allows for multiple experimenters to be running simultaneous virtual lab sessions, this provides a substantial benefit for studies with multiple research assistants.  

Conclusion

The current paper provides a technical tutorial for integrating lab simulation tasks into Amazon Web Services, complete with an online example, using a highly generalizable example task.  It is our hope that the current manuscript serves as a useful blueprint as researchers conduct the technical integration and design of an effective virtual experimental lab study that ideally reaches more diverse and generalizable populations.

Declarations

Funding

Research was sponsored by the Army Research Institute for the Behavioral and Social Sciences (ARI) and was accomplished under Grant Number W911NF-19-1-0453. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Institute for the Behavioral and Social Sciences (ARI) or the U.S. government. The U.S. government is authorized to reproduce and distribute reprints for government purposes notwithstanding any copyright notation herein.

Conflicts of interest/Competing interest: We disclose no conflicts or competing interests.

Availability of data and materials: All supporting materials are available to the public via OSF. https://osf.io/4hqsm/?view_only=2747a3cf69c54297a0842651c262062f

Code availability: All code is available to the public in a knit R markdown on OSF. https://osf.io/4hqsm/?view_only=2747a3cf69c54297a0842651c262062f

References

Fothergill, S., Loft, S., & Neal, A. (2009). ATC-lab Advanced: An air traffic control simulator with realism and control. Behavior Research Methods, 41(1), 118–127. https://doi.org/10.3758/BRM.41.1.118

Gee, D. G., Bath, K. G., Johnson, C. M., Meyer, H. C., Murty, V. P., van den Bos, W., & Hartley, C. A. (2018). Neurocognitive development of motivated behavior: Dynamic changes across childhood and adolescence. Journal of Neuroscience, 38(44), 9433–9445. https://doi.org/10.1523/JNEUROSCI.1674-18.2018

Henrich, J., Heine, S. J., & Norenzayan, A. (2010). The weirdest people in the world? Behavioral and Brain Sciences, 33(2–3), 61–83. https://doi.org/10.1017/S0140525X0999152X

Loft, S., & Remington, R. W. (2013). Wait a second: Brief delays in responding reduce focality effects in event-based prospective memory. Quarterly Journal of Experimental Psychology, 66(7), 1432–1447. https://doi.org/10.1080/17470218.2012.750677

Marchitto, M., Di Stasi, L. L., & Cañas, J. J. (2012). Ocular movements under taskload manipulations: Influence of geometry on saccades in air traffic control simulated tasks. Human Factors and Ergonomics in Manufacturing & Service Industries, 22(5), 407–419. https://doi.org/10.1002/hfm.20320

Wilson, M. D., Strickland, L., Farrell, S., Visser, T. A., & Loft, S. (2020). Prospective memory performance in simulated air traffic control: Robust to interruptions but impaired by retention interval. Human Factors, 62(8), 1249–1264. https://doi.org/10.1177/0018720819875888

Yeo, G. B., Frederiks, E. R., Kiewitz, C., & Neal, A. (2014). A dynamic, self-regulatory model of affect and performance: Interactions between states, traits and task demands. Motivation and Emotion, 38(3), 429–443. https://doi.org/10.1007/s11031-013-9376-z