This section documents Qumulo Core support for S3 API functionality and S3 API limits.
Supported S3 API Actions
The following table lists the S3 API actions that Qumulo Core supports and the version from which support begins. For the full list of S3 API actions, see Actions in the Amazon Simple Storage Service API Reference.
- The S3 API became generally available in Qumulo Core 5.3.3. This guide doesn't document enabling or using API actions that became available with preview functionality in versions of Qumulo Core lower than 5.3.3.
- The Qumulo S3 protocol creates data that supports all file system functionality, such as quotas, snapshots, and replication.
API Action | Supported from Qumulo Core Version |
---|---|
AbortMultipartUpload |
5.3.3 |
CompleteMultipartUpload |
5.3.3 |
CopyObject |
5.3.3 |
CreateBucket |
5.2.3 |
CreateMultipartUpload |
5.3.3 |
DeleteBucket |
5.2.4 |
DeleteBucketVersioning |
7.1.2 |
DeleteBucketPolicy |
7.0.1.1 |
DeleteObject |
5.2.1 |
DeleteObjects |
5.2.2 |
DeleteObjectTagging |
6.3.2 |
GetBucketAcl |
6.1.1 |
GetBucketLocation |
5.1.2 |
GetBucketVersioning |
7.1.2 |
GetBucketPolicy |
7.0.0.1 |
GetBucketVersioning |
7.0.0.1 |
GetObject |
5.0.4 |
GetObjectLockConfiguration |
7.2.1.1 |
GetObjectRetention |
7.2.1.1 |
GetObjectTagging |
7.1.2 |
GetObjectAcl |
6.1.1 |
GetObjectLockConfiguration |
7.0.0.1 |
HeadBucket |
5.1.2 |
HeadObject |
5.0.4 |
ListBuckets |
5.0.4 |
ListMultipartUploads |
5.3.3 |
ListObjects |
5.0.5 |
ListObjectsV2 |
5.0.4 |
ListParts |
5.3.3 |
PutBucketPolicy |
7.0.1.1 |
PutBucketVersioning |
7.1.2 |
PutObject |
5.2.1 |
PutObjectLockConfiguration |
7.2.1.1 |
PutObjectRetention |
7.2.1.1 |
PutObjectTagging |
6.3.2 |
UploadPart |
5.3.3 |
UploadPartCopy |
6.0.2 |
Unsupported S3 Functionality
The following table lists some of the S3 API functionality that Qumulo Core doesn’t support.
Unsupported Feature | Description |
---|---|
BitTorrent | — |
Bucket ACLs | For comparable functionality, use inheritable access control entries (ACEs). |
Bucket lifecycle configurations | — |
Bucket notifications | — |
Control of server-side encryption | All Qumulo Core data is encrypted at rest. You can’t control this functionality by using the S3 API. |
Logging controls | — |
Multi-chunk payload signing | Qumulo Core doesn’t support the streaming version of Amazon Signature Version 4 (SigV4), only the single-chunk version. |
Object locks | — |
Signature Version 2 | Qumulo Core supports only SigV4 signatures. |
Storage classes | Qumulo Core doesn’t use the storage class concept. All objects have the same storage class status. |
Retention policies | — |
Temporary access credentials | — |
Virtual-hosted bucket addressing | Qumulo Core supports only path-style bucket addressing. |
Web hosting configuration | — |
S3 API Limitations
This section describes the most important S3 API limitations in Qumulo Core.
Bucket Addressing Style
Because Qumulo Core supports only path-style bucket addressing, you must configure your client applications to use path-style addressing to send S3 API requests to a Qumulo cluster. For more information, see Configuring the AWS CLI for Use with Qumulo Core.
ETags
RESTful APIs, such as the S3 API, use HTTP ETags to identify different resource versions.
-
Qumulo Core uses a proprietary mechanism to generate an object’s ETag.
-
Amazon S3 uses the MD5 checksum of an object’s contents as its ETag.
Well-behaved applications shouldn’t attempt to interpret the contents of an ETag. However, certain applications do assume that S3 object ETags contain the MD5 checksum of the object’s contents. Such applications might not function properly with the Qumulo S3 API.
Listing Objects
The S3 API supports listing objects in a bucket by using the ListObjects
and ListObjectsV2
API actions.
Function | Qumulo Core | Amazon S3 |
---|---|---|
Returning results | Consistent but non-alphabetical order | Alphabetical order, by object key |
Arbitrary prefix | Partial support for Prefix , only if Prefix is a path to a file or directory under the bucket root directory |
Prefix limits results to object keys that begin with the prefix |
Arbitrary delimiter | Only the slash (/ ) character can act as Delimiter |
Delimiter groups results into common prefixes |
Although Qumulo Core supports
Prefix
and Delimiter
partially, it supports the most common use case—listing the contents of S3 buckets as a hierarchical file tree—fully.)))Request Authentication
Qumulo Core supports authenticating requests by using only Amazon Signature Version 4. Most S3 client applications support this authentication type.
If your application attempts to use a previous Amazon signature version, you receive a 400 Bad Request
response with the error code AuthorizationHeaderMalformed
.
Versioning
-
Object Version Limits: In Qumulo Core, S3 bucket versioning is consistent with that of Amazon S3, with the exception of individual object versions. Qumulo Core limits directories to approximately 4.3 billion child files. The approach that Qumulo Core takes to indexing files in a directory might cause object creation commands to output the
QumuloDirectoryEntryLimitReached
error when a directory gets close to its capacity. Because Qumulo Core gives object versions unique identifiers, it might be possible to retry the command successfully. However, if you begin to observe this error, we recommend removing previous object versions from your system. -
Creating Empty Versioned Directories: Qumulo Core doesn’t support creating empty, versioned directories.
-
Deleting Versioned Objects: If you don’t specify an object version ID, the DeleteObject and DeleteObjects S3 API actions create a deletion marker for an object but don’t delete any file system data. Because currently Qumulo Core doesn’t support bucket lifecycle policies, the data remains accessible by using S3 API actions and the object version ID. To delete a specific object version permanently, specify its version ID when you use either of these API actions.
Comparison of Known Limits between S3 in Qumulo and Amazon
This section compares the Qumulo Core S3 API limits with native Amazon S3 limits.
Limits for S3 Buckets
Limit | Qumulo Core | Amazon S3 |
---|---|---|
Maximum number of buckets | 16,000 | 1,000 |
Maximum number of objects in one bucket | Nominally unlimited | Unlimited |
Minimum bucket name length | 3 characters | |
Maximum bucket name length | 63 characters |
If all objects in a bucket are under the same directory—none of the object keys have the slash (
/
) character in them—the maximum number of objects in the bucket is limited to the maximum number of files in a directory. For more information, see Supported Configurations and Known Limits for Qumulo Core.Limits for S3 Objects
Limit | Qumulo Core | Amazon S3 |
---|---|---|
Minimum object size | 0 bytes | |
Maximum object size (by using PutObject ) |
5 GiB | |
Maximum object size (by using MultipartUpload ) |
48.8 TiB (10,000 * 5 GiB) | 5 TiB |
Minimum object key length | 1 character | |
Maximum object key length | 1,530 characters, if there are no slash (/ ) characters in the key |
1,024 characters |
Maximum object versions | 4,294,967,296 (theoretical) | Unlimited |
Limits for S3 Multipart Uploads
Limit | Qumulo Core | Amazon S3 |
---|---|---|
Minimum part ID | 1 | |
Maximum part ID | 10,000 | |
Minimum number of parts for each upload | 1 | |
Maximum number of parts for each upload | 10,000 | |
Minimum part size | 5 MiB (except for the last part of an upload) | |
Maximum part size | 5 GiB | |
Additional part size requirements | Must be a multiple of 4 KiB (4,096 bytes), except for the last part of an upload | — |
Limits for S3 API Requests
Maximum Limit | Qumulo Core | Amazon S3 |
---|---|---|
Object keys that DeleteObjects specifies |
Nominally unlimited | 1,000 |
Buckets that ListBuckets returns |
16,000 | 1,000 |
Objects that ListObjects and ListObjectsV2 return |
1,000 | |
Parts that ListParts returns |
Unlimited | 1,000 |
Uploads that ListMultipartUploads returns |
1,000 |
DeleteObjects
is subject to a 10 MiB request payload limit in Qumulo Core. This provides a practical upper limit on the number of object keys that the API action can specify.In addition, the following API actions have the Qumulo-specific maximum payload size limit of 10 MiB.
CompleteMultipartUpload
CreateBucket
DeleteObjects