Supported Functionality and Known Limits for S3 in Qumulo Core

This section documents Qumulo Core support for S3 API functionality and S3 API limits.

Supported S3 API Actions

The following table lists the S3 API actions that Qumulo Core supports and the version from which support begins. For the full list of S3 API actions, see Actions in the Amazon Simple Storage Service API Reference.

Note

The S3 API became generally available in Qumulo Core 5.3.3. This guide doesn't document enabling or using API actions that became available with preview functionality in versions of Qumulo Core lower than 5.3.3.
The Qumulo S3 protocol creates data that supports all file system functionality, such as quotas, snapshots, replication, and Global Namespace functionality.

Click to expand

API Action	Supported from Qumulo Core Version
`AbortMultipartUpload`	5.3.3
`CompleteMultipartUpload`	5.3.3
`CopyObject`	5.3.3
`CreateBucket`	5.2.3
`CreateMultipartUpload`	5.3.3
`DeleteBucket`	5.2.4
`DeleteBucketPolicy`	7.0.1.1
`DeleteBucketVersioning`	7.1.2
`DeleteObject`	5.2.1
`DeleteObjects`	5.2.2
`DeleteObjectTagging`	6.3.2
`GetBucketAcl`	6.1.1
`GetBucketLocation`	5.1.2
`GetBucketPolicy`	7.0.0.1
`GetBucketVersioning`	7.1.2
`GetObject`	5.0.4
`GetObjectAcl`	6.1.1
`GetObjectLegalHold`	7.2.3.2
`GetObjectLockConfiguration`	7.0.0.1
`GetObjectRetention`	7.2.1.1
`GetObjectTagging`	7.1.2
`HeadBucket`	5.1.2
`HeadObject`	5.0.4
`ListBuckets`	5.0.4
`ListMultipartUploads`	5.3.3
`ListObjects`	5.0.5
`ListObjectsV2`	5.0.4
`ListParts`	5.3.3
`PutBucketPolicy`	7.0.1.1
`PutBucketVersioning`	7.1.2
`PutObject`	5.2.1
`PutObjectLegalHold`	7.2.3.2
`PutObjectLockConfiguration`	7.2.3.2
`PutObjectRetention`	7.2.1.1
`PutObjectTagging`	6.3.2
`UploadPart`	5.3.3
`UploadPartCopy`	6.0.2

Unsupported S3 Functionality

The following table lists some of the S3 API functionality that Qumulo Core doesn’t support.

Unsupported Feature	Description
BitTorrent	—
Bucket ACLs	For comparable functionality, use inheritable access control entries (ACEs).
Bucket lifecycle configurations	—
Bucket notifications	—
Control of server-side encryption	All Qumulo Core data is encrypted at rest. You can’t control this functionality by using the S3 API.
Object Lock in governance mode	Qumulo Core supports only compliance mode.
Logging controls	—
Multi-chunk payload signing	Qumulo Core doesn’t support the streaming version of Amazon Signature Version 4 (SigV4), only the single-chunk version.
Signature Version 2	Qumulo Core supports only SigV4 signatures.
Storage classes	Qumulo Core doesn’t use the storage class concept. All objects have the same storage class status.
Temporary access credentials	—
Virtual-hosted bucket addressing	Qumulo Core supports only path-style bucket addressing.
Web hosting configuration	—

S3 API Limitations

This section describes the most important S3 API limitations in Qumulo Core.

Bucket Addressing Style

Because Qumulo Core supports only path-style bucket addressing, you must configure your client applications to use path-style addressing to send S3 API requests to a Qumulo cluster. For more information, see Configuring the AWS CLI for Use with Qumulo Core.

ETags

RESTful APIs, such as the S3 API, use HTTP ETags to identify different resource versions.

Qumulo Core uses a proprietary mechanism to generate an object’s ETag.
Amazon S3 uses the MD5 checksum of an object’s contents as its ETag.

Important
Well-behaved applications shouldn’t attempt to interpret the contents of an ETag. However, certain applications do assume that S3 object ETags contain the MD5 checksum of the object’s contents. Such applications might not function properly with the Qumulo S3 API.

Listing Objects

The S3 API supports listing objects in a bucket by using the ListObjects and ListObjectsV2 API actions.

Function	Qumulo Core	Amazon S3
Returning results	Consistent but non-alphabetical order	Alphabetical order, by object key
Arbitrary prefix	Partial support for `Prefix`, only if `Prefix` is a path to a file or directory under the bucket root directory	`Prefix` limits results to object keys that begin with the prefix
Arbitrary delimiter	Only the slash (`/`) character can act as `Delimiter`	`Delimiter` groups results into common prefixes

Note
Although Qumulo Core supports Prefix and Delimiter partially, it supports the most common use case—listing the contents of S3 buckets as a hierarchical file tree—fully.)))

Request Authentication

Qumulo Core supports authenticating requests by using only Amazon Signature Version 4. Most S3 client applications support this authentication type.

If your application attempts to use a previous Amazon signature version, you receive a 400 Bad Request response with the error code AuthorizationHeaderMalformed.

Versioning

Object Version Limits: In Qumulo Core, S3 bucket versioning is consistent with that of Amazon S3, with the exception of individual object versions. Qumulo Core limits directories to approximately 4.3 billion child files. The approach that Qumulo Core takes to indexing files in a directory might cause object creation commands to output the QumuloDirectoryEntryLimitReached error when a directory gets close to its capacity. Because Qumulo Core gives object versions unique identifiers, it might be possible to retry the command successfully. However, if you begin to observe this error, we recommend removing previous object versions from your system.
Creating Empty Versioned Directories: Qumulo Core doesn’t support creating empty, versioned directories.
Deleting Versioned Objects: If you don’t specify an object version ID, the DeleteObject and DeleteObjects S3 API actions create a deletion marker for an object but don’t delete any file system data. Because currently Qumulo Core doesn’t support bucket lifecycle policies, the data remains accessible by using S3 API actions and the object version ID. To delete a specific object version permanently, specify its version ID when you use either of these API actions.

Comparison of Known Limits between S3 in Qumulo and Amazon

This section compares the Qumulo Core S3 API limits with native Amazon S3 limits.

Limits for S3 Buckets

Limit	Qumulo Core	Amazon S3
Maximum number of buckets	16,000	1,000
Maximum number of objects in one bucket	Nominally unlimited	Unlimited
Minimum bucket name length	3 characters
Maximum bucket name length	63 characters

Note
If all objects in a bucket are under the same directory—none of the object keys have the slash (/) character in them—the maximum number of objects in the bucket is limited to the maximum number of files in a directory. For more information, see Supported Configurations and Known Limits for Qumulo Core.

Limits for S3 Objects

Limit	Qumulo Core	Amazon S3
Minimum object size	0 bytes
Maximum object size (by using `PutObject`)	5 GiB
Maximum object size (by using `MultipartUpload`)	48.8 TiB (10,000 * 5 GiB)	5 TiB
Minimum object key length	1 character
Maximum object key length	1,530 characters, if there are no slash (`/`) characters in the key	1,024 characters
Maximum object versions	4,294,967,296 (theoretical)	Unlimited

Limits for S3 Multipart Uploads

Limit	Qumulo Core	Amazon S3
Minimum part ID	1
Maximum part ID	10,000
Minimum number of parts for each upload	1
Maximum number of parts for each upload	10,000
Minimum part size	5 MiB (except for the last part of an upload)
Maximum part size	5 GiB
Additional part size requirements	Must be a multiple of 4 KiB (4,096 bytes), except for the last part of an upload	—

Limits for S3 API Requests

Maximum Limit	Qumulo Core	Amazon S3
Object keys that `DeleteObjects` specifies	Nominally unlimited	1,000
Buckets that `ListBuckets` returns	16,000	1,000
Objects that `ListObjects` and `ListObjectsV2` return	1,000
Parts that `ListParts` returns	Unlimited	1,000
Uploads that `ListMultipartUploads` returns	1,000

Note
DeleteObjects is subject to a 10 MiB request payload limit in Qumulo Core. This provides a practical upper limit on the number of object keys that the API action can specify.

In addition, the following API actions have the Qumulo-specific maximum payload size limit of 10 MiB.

CompleteMultipartUpload
CreateBucket
DeleteObjects