How to Onboard data using AWS Bucket Sync to INCRMNTAL

This article explains how to enable and onboard raw or aggregated data to INCRMNTAL using Amazon AWS Bucket Sync

There are two options for sharing data with INCRMNTAL via AWS. As our partner, you can choose whichever method suits you best. 

  • Allowing INCRMNTAL to read from a bucket created.
    • Using your company's AWS role
    • Using an AWS role at INCRMNTAL
    • Using an IAM user, provided by your company
  • Data transfer by the client to a bucket created by INCRMNTAL (Not Recommended)
    • Using your company's AWS role
    • Using an AWS role at INCRMNTAL
    • Using an IAM user, provided by INCRMNTAL

First Step: How to define the integration type

  1. Go to Configuration > Integration on the INCRMTNAL dashboard.
  2. Click the S3 icon. Amazon S3
  3. Click "Create New Adapter".
  4. Choose the integration type you'd like to implement. Choose the bucket's location (your own or INCRMNTAL's), and the authentication method you'd like to use (as detailed above).

The ideal settings would be using INCRMNTAL's bucket, and creating an IAM user for your company to write into the bucket.

Allowing INCRMNTAL to Read from a Bucket Created

Option A: Providing INCRMNTAL AWS IAM credentials to access your bucket

1. Create an IAM limited access user credentials for INCRMNTAL in your AWS account

The IAM user permissions should include: "s3:ListBucket" and "s3:GetObject"

Instructions to creating an IAM user with AWS can be found here.

2. Enter the bucket name, Secret Key, and Access Key to the respectable fields in the UI


3. Click "Validate"

Option B: Providing INCRMNTAL with your own AWS role

On the INCRMTNAL dashboard, navigate to Configuration > Integrations > S3 > Select "Your AWS S3 bucket" and "Using your own AWS role".


  1. The steps required for granting INCRMNTAL access to your bucket are outlined below (for a more detailed description please go to the AWS example walkthrough page)
  2. Your AWS administrator attaches a bucket policy, granting cross-account permission to INCRMNTAL to perform specific bucket operation.

    INCRMNTAL grants permission to its users and verifies that the objects in the client’s bucket are accessible.

    Note: Your administrator will need the INCRMNTAL Account ID, which is provided in the dashboard, under Configuration > Integrations > S3 > Select "Your AWS S3 bucket" and "Using your own AWS role".

    Then, on the INCRMNTAL dashboard (see navigation above):

    1. Enter your bucket name in which you defined the new policy.
    2. Enter your ARN.
    3. Click "validate". This will test the connection to your bucket.
    4. Enter the path to the dumped files within the S3 bucket, under "Adapter Details"
    5. Click "Save".


    Option C: INCRMTNAL provides their own role in your bucket

    1. On the S3 configuration dashboard, choose " Your AWS S3 environment" and " Using an AWS Role at INCRMNTAL"


    2. Enter your bucket name
    3. Click "Validate"

    Below is an example of a Bucket Policy, providing read only permissions.

    {

          "Version": "2012-10-17", 

          "Statement": [

          {

                 "Sid": "Example permissions", 

                 "Effect": "Allow",

                 "Principal": {

                        "AWS": "arn:aws::iam::[INCRMNTAL_ACCOUNT_ID]:root" 

                  },

                   "Action": [ 

                            "s3:ListBucket", 

                            "s3:GetObject"

                   ], 

                   "Resource": [

                             "arn:aws:s3:::BUCKET_NAME/*",

                             "arn:aws:s3:::BUCKET_NAME" 

                   ]

             }

         ]

    }

     

    Data Transfer to a Bucket Created by INCRMNTAL

    (Not Recommended)

    Option A: INCRMNTAL provides access to your company AWS account

    1. Select "INCRMNTAL S3 environment" and " Using your own AWS Role"

    2. Insert your ARN in the respectable row
    3. Click "Create Bucket at INCRMTNAL"

    Note: When copying data please add the following option to the copy comment: --acl bucket-owner-full-control

    This will ensure INCRMNTAL has access to the copied files when are copied or moved between buckets

     

    Option B: INCRMNTAL provides a AWS IAM user 

    1. Select "INCRMNTAL S3 environment" and " Using INCRMNTAL's IAM user" in the configuration screen
    2. Click "Create Bucket at INCRMTNAL"

    For all options of integration above

    Once the connection is set on the dashboard, we'd love hopping on a call with you to further understand how the bucket is built and how the provided data looks like, to map the data correctly to the dashboard.

    Reach out to your dedicated CSM to schedule such a call.

     

    The following parameters are sensitive to changes, and in case changed, might break the integration process with your S3 bucket and files:

    • Date format (e.g. changes from 1/1/2022 to 01/01/22)
    • Column names
    • File names
    • File scheme (e.g. if an additional column has been added, such as an adgroup column, please inform us to include in the current integration).
    • File location (e.g. files outside of the integrated folder will not be automatically ingested).



    For any question please contact onboarding@incrmntal.com , or open a support Ticket here