Data
Data
The Cloud Platform's Data management module offers researchers efficient data storage, sharing, and collaboration solutions. It addresses issues such as underlying engineering complexities and difficulties with the transfer and reproduction of research findings. By providing functions such as upload, download, deletion, archiving, thawing, and metadata management, the platform promptly and accurately caters to various data service needs across different scenarios, creating a one‑stop data solution to support data‑driven research and collaborative innovation.
Project data supports both file and table management. Files primarily handle data files within the project, while tables focus on managing and organizing the metadata of project data, making it convenient for users to batch submit data.
Files
On the Files page, there are three default system folders: RawData, ResultData, and ManualData. The RawData folder is used for storing original FASTQ files generated by sequencing, image files, and mask files output by image quality control. In the ResultData folder, there are two subfolders; the Notebook folder for storing result data output by Notebooks, and Workflow folder for storage of result files output by workflow analysis. The ManualData folder is used to store manually processed image files (such as IPRs produced by Tissuecut or preprocessed IPRs uploaded by users) and matrix files of gene expression (gef).
Add files
Used for uploading files to the cloud platform through various methods, including Website upload, Cluster upload, Tool upload, and Copying from projects.
Add files—Website upload
This feature is used to transfer local files to the cloud platform using web-based transmission.
On the data management files page, after clicking on the "Website upload" option in the dropdown menu under the "Add files" button, you will be directed to the web upload interface. Simply drag the local file(s) to the designated area, then click "OK" to initiate the upload.
Note
When using Website upload, it is recommended that individual file sizes do not exceed 100MB for the Shenzhen or Chongqing regions; for the Alibaba Cloud region, individual file sizes should not exceed 2GB.
Batch uploads are supported, and uploaded files are placed in the current folder by default.
Files within the current folder must have unique names; otherwise, the upload will fail.
Add files— Cluster upload (Online editing)
For projects using BGI sequencing, data can be swiftly uploaded to the cloud platform through cluster upload.
Note
Currently, cluster upload supports the upload of sequencing data and also supports the upload of other data.
On the File page, click the "Add files - Cluster upload" button to enter the cluster upload interface. In this interface, select "Sequencing data" edit the relevant information, and click "OK" to upload the FastQ file(s) to the current folder.
Note
Except for FastQ1, all other information on the above interface is optional. Move the data to the header field to view annotations for each field.
During cluster upload, data with filled SN will automatically synchronize to chip management.
If uploading FastQ files with library thawing, ensure that the FastQ path filled in has been granted "read" permissions to cluster accounts st_stereonote, st_notebook, and bigdata_autoanalysis before uploading.
At least one row of data must be filled in on the upload interface before saving.
This feature is only available in Shenzhen and Chongqing regions.
Add files—Cluster upload(Other data)
Other data supports filling in the full path and other optional information of all other format files in the cluster and then uploading it to the cloud platform. The following are the specific operations for cluster upload-other data:
Click the "Add File-Cluster Upload" button on the "Data Management-File" page to enter the cluster upload interface;
Select "Other Data" on this interface, edit the relevant information and click the "OK" button to upload the filled-in files to the current folder.
Tips
Make sure that the path of the file has been st_stereonote to the cluster account, and st_notebook bigdata_autoanalysis authorize the read permission.
The file path is required, others are optional, please fill in at least 1 line of data. If you want to upload a file again, you must delete the existing file on the Data Management - Files page. Currently, only full-path input files are supported, please fill in the standard format, for example: /dell2/test/1.txt.
Please check the access permissions of the uploaded file path and the file duplicate name, as the data without access permissions and the data with the duplicate name will not be supported.
Add files—Cluster upload(Batch import)
On the File page, click "Add files - Cluster upload"to enter the cluster upload interface. Select "Batch Import," download the Excel template, fill in the relevant information, save it, drag the Excel file to the batch import upload area, and click "OK."
Note
Please download the template first, fill in the template and upload the data, do not modify the header information in the template.
Only one excel file can be uploaded at a time, and multiple file uploads will replace the original file
Make sure that the sequencing data and other data paths are authorized to the cluster account bigdata_autoanalysis, st_notebook, st_stereonote, megabolt, and st_dcscloud_odms to read.
Please check the access permissions of the upload file path, the data without access will not be supported.
Sequencing data: Only the sequencing data filled in with SN will be automatically synchronized to the chip management and table-spatiotemporal chip table.
Sequencing data: Please fill in the FastQ path of BGI Glims dismounting, FastQ1 is required, others are optional, please fill in at least 1 line of data;
Other data: Currently, only full-path input files are supported, please fill in the standard format, for example: /dell2/test/1.txt.
Sequencing Data Reference Template: fastqImport.xlsx
Additional data reference templates :otherdata_Import.xlsx
This feature is only available in Shenzhen and Chongqing regions.
Add files—Tool upload
The Tool upload feature includes ossutil, Raysync tool, and AWS CLI tools. The ossutil tool is suitable for uploading and downloading data in Alibaba Cloud regions, the Raysync tool is designed for data upload and download in the BGI‑Shenzhen region, and the AWS CLI tool is applicable for data upload and download on overseas cloud platforms.
On the Files page, click the "Add files" button, and select "Tool upload" to access the tool upload interface.
ossutil tool
This feature is available only in Alibaba Cloud regions. Utilize this tool to directly upload local files or folders to Alibaba Cloud OSS buckets via command‑line operations.
- For Windows users, the steps to upload using the ossutil tool are as follows:
Download the ossutil tool package (for first‑time use of the ossutil tool).
Extract the compressed package, and after extraction, double‑click to open the ossutil tool (on Windows, double‑click the ossutil.bat file) to access the command prompt interface.
On the Cloud platform's tool upload interface, input the local folder or file path that you wish to upload (if uploading a file, make sure to provide the full path with the file format, for example: D:\upLoad\test.txt).
Click the "Generate upload command" button. After the command is generated on the page, click the "Copy command" button. Paste the command into the command prompt interface and press Enter to initiate the file upload to the current folder on the cloud platform. After the upload is completed, make sure to refresh the cloud platform page.
Note
The folder name for upload must only contain letters, numbers, "_", or ".", otherwise, the upload will fail.
- For macOS users, the steps to upload using the ossutil tool are as follows:
Download the ossutil tool package (for first‑time use of the ossutil tool).
After opening the terminal, navigate to the downloaded folder (or drag the folder directly into the terminal), and grant authorization to the ossutil tool using the following command: chmod 755 ossutilmac64.
On the Cloud platform's tool upload interface, input the local folder or file path that you wish to upload (if uploading a file, make sure to provide the full path with the file format, for example: /Users/lixiaohong/Downloads/).
Click the "Generate upload command" button. After the command is generated on the page, click the "Copy command" button. Paste the command into the command prompt interface and press Enter to initiate the file upload to the current folder on the cloud platform. After the upload is completed, make sure to refresh the cloud platform page.
Note
If you encounter the message "macOS cannot verify the developer of ossutilmac64" when executing the upload command, you need to go to Apple Menu > System Preferences > Security & Privacy > General tab, and select "App Store and identified developers" (as shown in the figure below). After selecting this option, proceed with the steps starting from the second step mentioned above.
Raysync tool
This functionality is exclusive to the Shenzhen region. It utilizes the Raysync client to select local files for upload, and the files are then uploaded to the BGI‑Shenzhen storage via the Raysync server. The steps are as follows:
Download the Raysync client (for first‑time use). After installation, click the "Install and start the transport client" button to start the application.
Click the "Upload" button. In the pop‑up window, select the file or folder (hold Ctrl to select multiple files), then click the "OK" button.
Note
While the Raysync client indicates the upload is complete, the cloud platform backend still needs to copy files from Raysync to BGI‑Shenzhen storage. This process may take several minutes to hours depending on the file size. Therefore, you may not be able to immediately view the uploaded files on the system page after Raysync completes the transfer.
AWS CLI tool
This feature is exclusive to the overseas AWS environment of the cloud platform. It allows users to utilize AWS tools to upload local files or folders to the cloud platform.
On the Data Management Files page, click the "Add file" button. Select "Tool upload" to access the tool upload interface. The steps are as follows:
Download the AWS CLI tool to your local machine and double‑click the installation file (for first‑time AWS CLI tool users).
Open the command prompt on your local computer (on Windows, run 'cmd'; on macOS or Linux, access the terminal).
On the Cloud platform's tool upload interface, input the local folder or file path that you want to upload (if uploading a file, make sure to provide the full path with the file format, for example: D:\upLoad\test.txt).
Click the "Generate upload command" button. After the command is generated on the page, click the "Copy command" button. Paste the command into the command prompt. If you are uploading a file, remove the "‑‑recursive" flag from the command. Press Enter to begin uploading the file to the current folder on the cloud platform. After the upload is complete, make sure to refresh the cloud platform page.
Note
The folder name for upload must only contain letters, numbers, "_", or ".", otherwise, the upload will fail.
Add files—Project
This feature is used to copy data from other projects to the current project. On the Files page, click the "Add files ‑ project" button to enter the Project interface. On this interface, select the files or folders you want to copy and click the "Copy to project" button. In the pop‑up, click "Confirm" to proceed.
File Information Modification:
This feature is used to modify the attributes of files for better file retrieval. On the file list interface, select one or more files, click the "Edit" button, and a dialog for editing metadata will appear. After filling in the metadata information, click the "Save edit" button to apply the changes.
File Download
File Download—Raysync download
This feature is exclusive to the Shenzhen region. It allows users to download files, excluding fastq and bam files, to their local devices using the Raysync tool. Here are the steps:
Select one or more files, then click the "Download ‑ Raysync download" button to enter the Raysync download interface. The interface displays the selected file names, sizes, transfer status, and operation buttons.
For files with a "Transfer Completed" status, you can directly click the download button to download the file to your local device.
For files with a "Pending Transfer" status, you need to click the "Transfer" button first. After the data transfer is complete, you can proceed to download.
Operation Instructions:
"?" Button: Used to view download guides. Click this button to install the Raysync client according to the download guide.
"Transfer" Button: Used to transfer files to the downloadable area. After clicking this button, the system will copy the file to the downloadable area, and you can only download it after the copy is complete.
"Download" Button: Used to download files to the user's local device. After clicking this button, the system will initiate the Raysync client to store the file locally.
"Refresh" Button: Used to update the transfer status.
"Install and Launch Transfer Client" Button: Used to install and launch the Raysync client.
Note
Files can be directly downloaded within the first three months after their creation. After three months, you need to transfer them first before downloading.
Before downloading, please follow the "Download Guide" to install and launch the Raysync client.
File Download—Web download
This feature is exclusive to the Alibaba Cloud region. It allows users to download Alibaba Cloud files to their local devices through a browser. Here are the steps:
On the file management page, select one or more files that you want to download.
Click the "Download ‑ Web download" button.
File Download—Ossutil tool download
This feature is exclusive to the Alibaba Cloud region. It allows users to download Alibaba Cloud files to their local devices through the Ossutil tool.
- Here are the steps for Windows users:
Download the Ossutil tool zip file (for the first‑time use).
Extract the downloaded zip file and double‑click to open the Ossutil tool, entering the tool interface.
On the cloud platform file page, select the files you want to download, click the "Download ‑ Tool download" button, then click "Generate download command," and finally click "Copy command." Paste the copied command into the tool interface. The files will be automatically downloaded to the current folder.
- Here are the steps for Linux users:
Download the Ossutil tool zip file (Linux version) for the first‑time use.
Copy the downloaded zip file to the cluster server and unzip it using the command: unzip ossutil‑v1.7.16‑linux‑amd64.zip.
Grant permission to the unpacked Ossutil tool using the command: chmod +x ossutil*.
On the cloud platform file page, select the files you want to download, click the "Download ‑ Tool download" button, then click "Generate download command," and finally click "Copy command." Paste the copied command into the tool interface, add ./ before the command, and execute the command. The files will be automatically downloaded to the current folder.
Note
The prerequisite for downloading files using the above method is that the local cluster can access the external network.
File Download—AWS CLI tool download
This feature allows users to download files from the overseas AWS environment of the cloud platform to their local devices using AWS tools. Here are the steps:
Download the AWS tools and install them by double‑clicking the file (for the first‑time use).
Open the console on your local computer (execute cmd for Windows, or enter the terminal interface for macOS or Linux).
On the cloud platform file page, select the files you want to download, click the "Download ‑ Tool download" button, then click "Generate download command," and finally click "Copy command." Paste the copied command into the console. The files will be automatically downloaded to the current folder.
Copying File Full Path
Used to copy the file path to make it easier to fill out tables.
On the file page, hover over the name of the file whose path you want to copy and click the 
button to copy the full path ending with the file format.
File Deletion
To delete a file or folder, select the file(s) or folder(s) and click the Deletebutton. Then, click Confirm in the pop‑up box to proceed with deletion.
Note: Once deleted, files cannot be restored through the cloud platform, so this action should be performed with caution.
Display Selected Files Only
After selecting multiple files across folders, if you wish to quickly deselect some files, check the "Show only selected" button at the top‑left corner of the list. This will show only the selected records on the page, allowing for quick deselection as illustrated below:
File Online Viewing
The platform integrates file‑opening tools for online viewing of file content.
After entering the project data file page, click on the file name to enter the online file viewing page. Current support for online viewing includes file formats such as PDF, PNG, CSV, TXT, and HTML.
Exporting File Information
This function is for exporting Excel format file metadata information. The exported file can then be adjusted and used as a table.
After entering the project data‑file page, select one or multiple files and click the Export button as shown below:
Operation Records
Used for viewing records of file addition and downloading.
After clicking the Operation records button, a data upload page will pop up, recording upload file names, methods, statuses, file paths, upload times, etc.
Table
Table are primarily used to manage and organize data, facilitating the batch submission of analysis tasks.
When a project is created, the system automatically generates chip tables, sequencing tables, and image tables, which are displayed on the Data-Table page.
The STOmics chip table displays information related to chips, including chip number, specifications, size, mask, as well as corresponding sequencing data, image data, etc. For spatial-temporal commercial projects, the chip table will automatically generate its values; for individual projects, after associating chips in the chip management page, the chip table will automatically generate relevant information. Click "Table" on the Data page to enter the tables page, then click on the Chip Data table to open the chip table.
The STOmics sequencing data displays sequencing-related information.
Click "Table" on the Data page to enter the table page, then click on the sequencing data table to open the sequencing table. FastQ uploaded in chip details will be automatically synchronized with this table.
The STOmics image data displays spatial project image-related information, including original TAR and IPR, as well as preprocessed IPR. Click "Tables" on the Data page to enter the table page, then click on the STOmics image data to open the image table. IPR uploaded in chip details will be automatically synchronized with this table.
Add table
The icon 
is used to download spreadsheet templates.
The icon 
is used to add a new spreadsheet.
How to fill out the form
- Scenario 1: The form is created by yourself or filled in by using the form provided by the process. Please carefully read the parameter description under each parameter. In order to avoid errors in the task, please fill in the corresponding fields correctly according to the file format specified by the parameter. Special attention should be paid to whether the characters and values are accurately filled in,And how to fill in the array of fq files.
- Scenario 2: The form is a template provided by the data model. You can view the sample format below the form. Based on the provided sample format, you can complete the form by matching the real data of the individual.
Q: How should I fill in the format of arrays involved in the sequencing data path related to fq?
A: For different processes of different OMICS (single cell, spatiotemporal OMICS, genomics), the parameters of the offline _.fq.gz file will be used as input. The filling method in the form is based on SampleID. If there are multiple pairs of fq.gz for the same sample, only one pair can be filled in one line _.fq.gz information, multiple pairs of \*.fq.gz need to be filled in by branch, ensure that the samples of different lines are the same sample ID.
Download table
When a table is focused on by the mouse, icons for downloading table will appear. The icon
is used to download the table locally.
Delete table
This allows users to delete an entire table at once.
When a table is focused on by the mouse, icons for deleting the table will appear, click the icon
and then confirm in the popup box.
Note:Deletion is irreversible, so this should be done with caution.
Add data
This allows for the continuous addition of data to an existing table.
On the Data‑Table page, click on the tale to which you want to add data, click the Add data button in the top right corner, download the template, fill in information, and import it back.
Delete data
This allows for the deletion of some data within a table.
On the Data‑Table page, click on the table from which you want to delete data, select the data points to be removed, click the Delete button and then confirm in the popup box.
Note
As the deletion is irreversible, proceed with caution.
Download data
This is for downloading selected data from the table.
On the Data‑Table page, click on the table from which you wish to download data, select the relevant data points, and click Download to save an Excel file locally.
