SQS Listener

This feature is available in v42.3 and later.

With the SQS Listener Input Block, you can configure connections to Amazon SQS queues. This block replaces Message Queue (MQ) Listener Blocks that have Amazon SQS connections, as those blocks will be removed from the system in v45 of Hyperscience.

If you would like to use an SQS Listener connection in a SaaS instance and your SQS service is not public, you should create a firewall pinhole for the connection. Contact your Hyperscience representative for more information.

Sample use cases

• I send submission info from external systems to an Amazon SQS queue. I want to ingest this submission info into Hyperscience for processing.

• My customers will post relevant documents and metadata to my Amazon SQS queue. I want to then ingest these documents and metadata for processing into Hyperscience.

Message format

A JSON object must be passed to the queue in order for Hyperscience to read the appropriate input image files. It should have the same format as the request payload for the Submission Creation API endpoint, with one key difference: in the message payload, the files element is an array of file URLs and does not contain individual file_url elements.

An example message payload is shown below.

{
  "flow_uuid": "a89d6440-a2c2-423b-8c95",
  "machine_only": "true",
  "files": [
    "s3://s3-bucket/input/demo-file.pdf",
    "https://example.com/files/demo-file.pdf",
    "ocs://2021",
    "gs://bucket/example.pdf",
    "abs://container-name/blob.pdf",
    "abs://container-name/blob.pdf?sas=secret-token"
  ]
}

For more information about the Submission Creation payload, see the Submission Creation section of our API documentation.

Migrating from the Message Queue Listener Block

Message Queue Listener Blocks with Amazon SQS connections will continue to work in v42.3, v43, and v44. However, they will be removed in v45 of Hyperscience. If you are using the Message Queue Listener Block to connect to Amazon SQS queues, you will need to migrate to the SQS Listener Block before upgrading your application to v45.

Other connection types (i.e., ActiveMQ, IBM MQ, or RabbitMQ) can continue to be used in Message Queue Listener Blocks, as no migration is required to maintain those connection types in future versions.

To migrate to the SQS Listener Block, follow the steps below.

1. Identify affected flows.

Look for warnings/errors mentioning MQ_LISTENER with MQ_TYPE=AMAZON_SQS.

2. In each affected flow, add an SQS Listener Block.

See Setting up a connection in Hyperscience to learn how to add the SQS Listener Block.

3. Transfer you settings from the Message Queue Listener Block to the SQS Listener Block.

Transfer your Amazon SQS configuration to the new dedicated SQS block, including the queue URL, AWS credentials/authentication settings, and any other SQS-specific settings.

The table below provides a mapping of settings from the Message Queue Listener Block to the SQS Listener Block.

The Consume Timeout and Consume Num Messages settings are new in the SQS Listener.

To learn more about these settings, see Block settings.

4. Disable the Amazon SQS Message Queue Listener connection and enable the SQS Listener connection.

Disabling the Message Queue Listener connection before enabling the SQS Listener connection processing of duplicate messages.

To enable or disable a connection, click on Inputs in Flow Studio, then toggle the switch for the connection to the on (blue) or off (gray) position.

For more information on managing connections, see Managing Connections.

5. Test the updated flow.

1. Deploy the updated flow in your lower environment.

2. Verify that message consumption / notification behavior works as expected.

3. Confirm that no deprecation warnings appear for the flow.

6. Deploy the updated flow in production.

After testing is complete in your lower environment, deploy the updated flow in production before upgrading to v45.

If you need additional assistance with the migration or with migration planning, contact your Hyperscience representative.

Block settings

In addition to the settings outlined below, you can also configure the settings described in Universal Integration Block Settings. 

Setting up SQS Listener connections

SQS Listener connections require configuration in both Hyperscience and AWS.

Setting up a connection in Hyperscience

To set up the SQS Listener Block in Hyperscience, follow these steps. 

1. Log in to your Hyperscience instance.

2. Go to Flows, and click on the flow you want to add the SQS Listener to.

3. Click Edit Flows.

4. In Flow Studio, click on Inputs.

5. Click Add and select SQS Listener from the list.

6. Click the Add Connection button.

7. Enter the block settings described above.

Setting up a connection in AWS

1. Go to your Amazon Simple Queue Service.

2. Choose or create the relevant input queue. The SQS Listener supports standard and FIFO queues.

3. Grant the following permissions via the AWS Interface:

   • sqs:ReceiveMessage

   • sqs:GetQueueUrl

   • sqs:DeleteMessage

   • sqs:ChangeMessageVisibility

Visibility timeout

To avoid duplicate processing, configure the queue's visibility timeout to at least 60 seconds. Doing so ensures that the timeout window is longer than the maximum 30-second response time from StrongDM (SDM) HTTP requests. Making the timeout window longer minimizes the possibility of duplicate processing due to a failure to acknowledge messages during the window.

For more information, see AWS's Amazon SQS visibility timeout.

Message size

Note that Amazon imposes certain limitations on the size of messages sent to queues as documented in AWS’s Amazon SQS quotas. At the time of writing this article, the maximum message size allowed is 256KB. Depending on the specifics of your use case, this limit may prevent you from using the SQS Listener connection effectively.

Dead-letter queues

A dead-letter queue (DLQ) is a specialized message queue designed to temporarily hold messages that encounter processing errors within a software system. DLQs allow producer-consumer systems to operate without interruptions in cases when a malformed message can’t be consumed. The message is isolated into a different queue and is processed by a different system, reviewed manually, or both. In this way, setting up a DLQ prevents issues with blocked submission pipelines. 

Hyperscience recommends setting the maximum number of retry attempts for a message to be between 3 and 5. When a message goes above this number of retry attempts, it should be redirected to the DLQ. 

Setting up DLQs

To set up a DLQ, see the instructions in AWS’s Configuring a dead-letter queue redrive.

If you’re setting up a DLQ, make sure to set the DISCARD_MALFORMED_RESOURCES “.env” file variable to false.