S3 - List Object(s)

Declaration

<!--<AMAWSS3 ACTIVITY="get_objects_dataset" BUCKETNAME="text" RESULTDATASET="text" SESSION="text" />-->

See Also

Description

Retrieves all objects within a bucket and stores the information in a dataset.

IMPORTANT: The S3 activities in AutoMate use the Amazon S3 engine to perform their work. Because of this, you must have a valid Amazon S3 service account to use these activities.

Practical Usage

Used to get information about a list of objects contained in a bucket.

Parameters

Bucket Properties

Property

Type

Required

Default

Markup

Description

Bucket Name

Text

Yes

(Empty)

BUCKETNAME="myBucket"

The name of the bucket you want to list keys on.

Prefix (Optional)

Text

No

(Empty)

PREFIX='photos/2006/"

Restricts the response to only contain results that begin with the specified prefix.

Marker (Optional)

Text

No

(Empty)

MARKER="6"

Specifies where in the result set to begin listing. It restricts the response to only contain results that occur alphabetically after the value of the marker. This optional parameter enables pagination of large result sets.

Max Keys (Optional)

Text

No

(Empty)

MAXKEYS="10"

Limits the number of keys returned in response to your query.

Create and populate dataset with S3 object information

Text

Yes

(Empty)

RESULTDATASET="myDataset"

Indicates the name of the dataset in which to create and populate with the S3 object information. More on the dataset that this activity creates can be found below under Notes).

 

Advanced Properties

Each Amazon S3 object has a set of key-value pairs with which it is associated called Headers or Metadata. Metadata provides important details about an object, such as file name, type, date of creation/modification etc. There are two kinds of metadata in S3; system metadata, and user metadata. System metadata is used and processed by Amazon S3. User metadata (also known as custom header) is specified by you, the user. Amazon S3 simply stores it and passes it back to you upon request.

AutoMate lets you to store your personal information as custom headers or user metadata like name, company name, and phone numbers etc, so that you can distinguish specific files. Using this option, you can add new custom header/user metadata to existing S3 objects, edit default S3 metadata on a bucket or store/upload new objects with custom header or metadata.

Property

Type

Required

Default

Markup

Description

Name

Text

No

(Empty)

HEADER NAME="myHeader"

Specifies the "key" in a key-value pair. This is the handle that you assign to an object.

Value

Text

No

(Empty)

VALUE="theValue"

Specifies the "value" in a key-value pair. This is the content that you are storing for an object.

 

Credentials Properties

These properties allow you to enter a custom set of credentials specific to this activity or link this activity to an existing session.  

Property

Type

Required

Default

Markup

Description

Provide Credentials

 

 

 

 

Indicates where this activity's credentials should originate from. Different properties apply depending on the option selected.

The available options are:

  • Custom (Default) - Specifies that a custom set of credentials will be entered for this S3 activity. Select this option if performing a single S3 activity.

  • Session Based - Specifies that credentials should derive from a session created in a previous step with the use of the Create Session activity. This allows several S3 activities to be linked to a specific session.

Access Key

Text

Yes

(Empty)

ACCESSKEY="MyAccessKey"

Specifies the 20-character, alphanumeric string that uniquely identifies a user who owns an s3 account (i.e. 022QF06E7MXBSH9DHM02). This along with a Secret Access Key forms a secure information set that S3 uses to confirm a valid user's identity.

This property is active only if Custom is selected under the Provide Credentials property.

Secret Access Key

Text

Yes

(Empty)

SECRETKEY="SecretKey"

A 40-character string that serves the role as Password to access S3 (i.e. kWcrlUX5JEDGM/LtmEENI/aVmYvHNif5zB+d9+ct). This along with an associated Access Key forms a secure information set that S3 uses to confirm a valid user's identity.

This property is active only if Custom is selected under the Provide Credentials property.

Protocol

Text

No

HTTP

PROTOCOL = "HTTPS"

The protocol that should be used to make requests to the S3 service.

The available options are:

  • HTTP (Default)

  • HTTPS

This property is active only if Custom is selected under the Provide Credentials property.

User Agent

Text

No

AutoMate

USERAGENT="AutoMate"

The value of the HTTP User-Agent header. The User-Agent request-header field contains information about the user agent originating the request. The default User Agent is AutoMate.

This property is active only if Custom is selected under the Provide Credentials property.

Service URL

Text

No

(Empty)

SERVICEURL=

"https://eu-west-1.ec2.amazonaws.com"

The Service URL used to make requests to the S3 service. The Service URL provides the service endpoint.

This property is active only if Custom is selected under the Provide Credentials property.

Maximum retry on error

Number

No

(Empty)

MAXERRORRETRY="4"

Signifies how many times the Amazon S3 engine should retry the request before returning an error.

This property is active only if Custom is selected under the Provide Credentials property.

Proxy Host

Text

No

(Empty)

PROXYHOST="proxy.host.com"

The hostname (server.domain.com) or IP address (xxx.xxx.xxx.xxx) of the proxy server.

This property is active only if Custom is selected under the Provide Credentials property.

Proxy Port

Number

No

(Empty)

PROXYPORT="1028"

The port that should be used to connect to the proxy server.

This property is active only if Custom is selected under the Provide Credentials property.

Session Name

Text

Yes

(Empty)

SESSION="SessionName"

The session name that this activity should originate from. This allows several S3 activities to be linked to a specific session. Numerous sessions can be used within a single task.

This property is active only if Session Based is selected under the Provide Credentials property.

NOTE: Use the Create Session activity to create and name an S3 session.

 

Description Properties

The Description tab allows you to customize the text description of any step when it appears in the Steps Pane.

More on setting custom step description

Error Causes Properties

The Error Causes tab is part of AutoMate's Error Handling functionality which allows you to select / omit specific errors that will cause a particular step to fail.

More about error causes properties

On Error Properties

The On Error tab is part of AutoMate's Error Handling functionality which allows you to determine what the task should do if a particular step encounters an error.

More about on error properties

Notes

Datasets

A dataset is a multiple column, multiple row container object. This activity creates and populates a dataset containing a specific set of fields in addition to the standard dataset fields. The table below describes these fields (assuming the dataset name assigned was theDataset).

Name

Type

Return Value

theDataset.KeyName

Text

The Key Name of the object.

theDataset.LastModified

Date

The last modified date/time of the object.

theDataset.ETag

Text

The object's entity tag. This is a hash of the object that can be used to do conditional operations. The ETag only reflects changes to the contents of an object, not its metadata.

theDataset.Size

Number

The size of the object, in bytes.

theDataset.Owner

Text

The owner assigned to the object.

theDataset.StorageClass

Text

The storage class of the object (i.e. STANDARD)

 

Variables and Expressions

All text fields allow the use of expressions, which can be entered by surrounding the expression in percentage signs (example: %MYVARIABLE%, %Left('Text',2)%). To help construct these expressions, you can open Expression Builder from these fields by pressing F2.

More on variables
More on expressions

More on the expression builder

Example

NOTE: The code below can be copied and pasted directly into the Steps pane of the Task Builder.

 

<AMAWSS3 ACTIVITY="create_session" SESSION="mySession" ACCESSKEY="AKIAJU2LHLMG6BA6BZTA" SECRETKEY="AM1B2xUP4i/Z1KM5ySNpiipKpe+rOm+wdDFaME" />

<AMAWSS3 ACTIVITY="get_objects_dataset" BUCKETNAME="myBucket" RESULTDATASET="test" SESSION="mySession" />

<AMLOOP TYPE="DATASET" DATASET="test">

<AMMESSAGEBOX>Owner - %test.ETag%

Key Name - %test.KeyName%

Last Modified Date - %test.LastModified%

Owner - %test.Owner%

Size - %test.Size%

Storage Class - %test.StorageClass%</AMMESSAGEBOX>

</AMLOOP>

<AMAWSS3 ACTIVITY="end_session" SESSION="mySession" />

↑ Top