Skip to main content

Create URL File Uploads

Creating a URL file upload is a two-step process using the createSignedUrlFileUpload mutation.

Step 1: Create the upload and get a pre-signed URL​

Call the createSignedUrlFileUpload mutation. This creates the upload record and returns a pre-signed S3 URL for uploading the file.

mutation CreateSignedUrlFileUpload(
  $projectId: ObjectID!
  $crawlTypeCode: CrawlType!
  $fileName: String!
  $projectUploadType: ProjectUploadType!
) {
  createSignedUrlFileUpload(
    input: {
      projectId: $projectId
      crawlTypeCode: $crawlTypeCode
      fileName: $fileName
      projectUploadType: $projectUploadType
    }
  ) {
    signedS3UploadUrl
    urlFileUpload {
      id
      fileName
      status
    }
  }
}

Try in explorer

The required input fields are:

  • projectId - the Test Suite ID (Test Suites are a type of Project).
  • crawlTypeCode - the crawl type, typically List for URL list uploads.
  • fileName - the name of the file being uploaded (must end with .csv or .txt for List type).
  • projectUploadType - the upload type, typically ListTxt for URL list uploads.

Optional fields:

  • enabled - whether the upload is active (defaults to true).
  • uploadBaseDomain - a base domain for resolving relative URLs in the file.

Step 2: Upload the file to S3​

Use the signedS3UploadUrl from the response to upload your file directly to S3 via an HTTP PUT request:

curl -X PUT --data-binary @url_list.txt "<signedS3UploadUrl>"

The pre-signed URL is valid for 15 minutes, so make sure to upload the file promptly.

Step 3: Poll for processing status​

After the file is uploaded to S3, it is automatically picked up for processing. Poll the upload status to track progress (see Retrieve URL File Uploads).

The upload transitions through the following statuses: Draft → Processing → Processed (or Errored).

caution
  • It may take a while to process a URL file upload. Poll the status field to check progress.
  • If the status changes to Errored, check the errorMessage field for details.