Merging Datasets in BRIGHT

How to Merge Datasets in BRIGHT. Learn how to merge datasets in BRIGHT to streamline your data management process. This guide covers everything from uploading a placeholder file to setting up automatic refresh schedules.

In this guide, we will walk you through the process of merging datasets in BRIGHT to ensure your data remains up-to-date with new fields and records. This is essential for creating a unified dataset that can be updated automatically or manually refreshed after changes are made.

Before proceeding, ensure that a folder containing all BRIGHT datasets has been shared with you under your assigned operator ID.

The new datasets adhere to the naming convention: dataset name_instance_operatorNumber, where the first part represents the dataset name, the second part denotes the instance, and the third part is the operator number, which is for internal use only (eg. flight_prod_OP5).

Important note: Please note that access to the new datasets is still in the beta testing phase. If you do not currently have access to the datasets and would like to participate in the beta testing program, please contact our support team.

Step 1: Upload a Placeholder CSV File

1. Create or Use a Dummy File: Begin by either creating an empty .csv file or using any existing .csv file from your computer. This file serves as a placeholder and has no effect on the end result.

2. Navigate to Datasets in BRIGHT:
   Open BRIGHT, and from the QS user interface, click on "Datasets" in the left panel. In the top-right corner, click on the blue "New Dataset" button.
   

3. Upload Your CSV File:
   Choose the first option, "Upload a file", and select your dummy .csv file.
   

4. Preview and Edit Data:
   Click "Next" and then "Edit/Preview Data" to proceed.
   
   

5. Save and Publish:
   Once the preview appears, click on "Save and Publish" in the top-right corner.
   

Your new Quicksight dataset has been created and will act as a staging area for merging other datasets.

Step 2: Merging the Datasets

Now that you have an initial dataset, you can begin merging your actual data.

1. Remove the CSV File:
   Locate the dummy .csv file, click the three dots next to it, and choose "Remove". This step clears the placeholder.
   

2. Add Your First Dataset:
   Click on the "Add Data" button and select the first dataset you wish to merge (e.g., the Booking dataset). The dataset will appear on the preview page.
   

3. Add Your Second Dataset:
   Again, click "Add Data" and select the second dataset (e.g., the Flight dataset).

4. Join the Datasets:
   Now, you can join the datasets. Click on the two red dots connecting them and select the type of join (e.g., inner, outer, left). Specify the join keys (e.g., booking identifier) to combine the data on common fields.
   
   

5. Apply and Save:
   After setting the join, click "Apply". Lastly, click "Save and Publish" to complete the process and create your merged dataset.

Alternatively, you can click "Save and Visualize" to proceed directly to analysis with your new dataset.

Step 3: Set Up a Refresh Schedule

To ensure your merged dataset stays up to date, it's crucial to set up a refresh schedule.

1. Open the Dataset:
   Once the dataset is saved, click on it to open the settings.
   

2. Set the Refresh Schedule:
   Navigate to the "Refresh" tab, then click "ADD NEW SCHEDULE". Here, you can define when and how often the data should be refreshed, ensuring that any new records or fields are automatically included.
   

Important Notes:

• The merged dataset will be able to handle updates both from newly added fields (via Python scripts) and new records (via manual or scheduled refreshes).

• Regularly monitor your refresh schedule to avoid outdated datasets and ensure continuous, accurate reporting.

By following these steps, you can efficiently create, manage, and update merged datasets in BRIGHT, enabling you to maintain accurate and up-to-date aviation management data.