This feature is available in v42.3 and later.
With the SQS Notifier Output Block, you can configure connections to Amazon SQS queues. This block replaces Message Queue (MQ) Notifier Blocks that have Amazon SQS connections, as those blocks will be removed from the system in v45 of Hyperscience.
If you would like to use an SQS connection in a SaaS instance and your SQS service is not public, you should create a firewall pinhole for the connection. Contact your Hyperscience representative for more information.
Sample use cases
I have extracted content from insurance forms and want to send the JSON downstream to my SQS queue in order to populate other systems.
I have transformed submission data and want to send it to my Amazon SQS queue, where it will be picked up by internal software to be further processed.
Migrating from the Message Queue Notifier Block
Message Queue Notifier Blocks with Amazon SQS connections will continue to work in v42.3, v43, and v44. However, they will be removed in v45 of Hyperscience. If you are using the Message Queue Notifier Block to connect to Amazon SQS queues, you will need to migrate to the SQS Notifier Block before upgrading your application to v45.
Other connection types (i.e., ActiveMQ, IBM MQ, or RabbitMQ) can continue to be used in Message Queue Notifier Blocks, as no migration is required to maintain those connection types in future versions.
To migrate to the SQS Notifier Block, follow the steps below.
1. Identify affected flows.
Look for warnings/errors mentioning MQ_NOTIFIER with MQ_TYPE=AMAZON_SQS.
2. In each affected flow, add an SQS Notifier Block.
See Setting up a connection in Hyperscience to learn how to add the SQS Notifier Block.
3. Transfer you settings from the Message Queue Notifier Block to the SQS Notifier Block.
Transfer your Amazon SQS configuration to the new dedicated SQS block, including the queue URL, AWS credentials/authentication settings, and any other SQS-specific settings.
The table below provides a mapping of settings from the Message Queue Notifier Block to the SQS Notifier Block.
Message Queue Notifier setting | SQS Notifier setting |
|---|---|
Queue URL | Queue URL |
AWS Region | AWS Region |
Use AWS EC2 Instance IAM Role Credentials | Use Default Credential Chain |
Access Key ID | AWS Access Key ID |
Secret Access Key | AWS Secret Access Key |
Group ID for FIFO Queues | Message Group ID |
The Message Deduplication ID setting is new in the SQS Notifier.
To learn more about these settings, see Block settings.
4. Disable the Amazon SQS Message Queue Notifier connection and enable the SQS Notifier connection.
Disabling the Message Queue Notifier connection before enabling the SQS Notifier connection processing of duplicate messages downstream.
To enable or disable a connection, click on Outputs in Flow Studio, then toggle the switch for the connection to the on (blue) or off (gray) position.
For more information on managing connections, see Managing Connections.
5. Test the updated flow.
Deploy the updated flow in your lower environment.
Verify that message consumption / notification behavior works as expected.
Confirm that no deprecation warnings appear for the flow.
6. Deploy the updated flow in production.
After testing is complete in your lower environment, deploy the updated flow in production before upgrading to v45.
If you need additional assistance with the migration or with migration planning, contact your Hyperscience representative.
Block settings
In addition to the settings outlined below, you can also configure the settings described in Universal Integration Block Settings.
Name | Required? | Description |
|---|---|---|
Use Default Credential Chain | Yes | If selected, credentials are obtained from the EC2 instance directly, and the AWS Access Key ID and AWS Secret Key are not present. This option applies only to the following deployment types:
Selected by default. |
Queue URL | Yes | The queue URL for receiving messages. |
AWS Region | Yes | The AWS region of the data's destination. |
AWS Access Key ID | Yes, if Use Default Credential Chain is not selected | The access key ID allows access to the message queue. This setting is only available if Use Default Credential Chain is not selected. |
AWS Secret Access Key | Yes, if Use Default Credential Chain is not selected | The secret access key allows access to the message queue. To edit the key, click Edit value, modify the key, and then click Done. This setting is only available if Use Default Credential Chain is not selected. |
Additional SQS metadata | No | Additional metadata that you'd like to send to Amazon SQS. |
Message Group ID | Yes, if sending messages to FIFO queues | The group ID that will be added to messages, which ensures that AWS processes messages in the correct order. |
Message Deduplication ID | Yes, if sending messages to FIFO queues and | The token used for deduplication of sent messages. If a message with a particular Message Deduplication ID is sent successfully, any messages sent with the same ID are accepted successfully but aren't delivered during the 5-minute deduplication interval. |
Setting up SQS Notifier connections
SQS Listener connections require configuration in both Hyperscience and AWS.
Setting up a connection in Hyperscience
Log in to your Hyperscience instance.
Go to Flows, and click on the flow you want to add the SQS Notifier to.
Click Edit Flows.
In Flow Studio, click on Outputs.
Click Add and select SQS Notifier Output Block from the list.
Click the Add Connection button.
Enter the block settings described above.
Setting up a connection in AWS
Go to your Amazon Simple Queue Service.
Choose or create the relevant input queue. We support standard and FIFO queues.
Grant the following permissions via the AWS Interface:
sqs:GetQueueUrlsqs:SendMessage
If you are using a FIFO queue, take note of the the Group ID so you can enter it in the Message Group ID field in Hyperscience.
We provide maximum flexibility in authentication and access. As long as Hyperscience has access to the queue, this integration should automatically work.
Message size
Note that Amazon imposes certain limitations on the size of messages sent to Amazon SQS message queues as documented in Amazon’s Amazon SQS quotas. At the time of writing this article, the maximum message size allowed is 256KB. Depending on the specifics of your use case, this limit may prevent you from using the SQS Notifier connection effectively.
For more information on the structure of the JSON object that should be passed to the message queue in order for Hyperscience to read the appropriate input image files, refer to the Submission Creation section of our API documentation.