Object storage vs block and file

Block storage (EBS) exposes sectors; your OS builds a filesystem. File storage (EFS) exposes paths and locks across NFS clients. Object storage exposes immutable blobs addressed by key, with rich metadata and HTTP semantics. There is no “seek to sector 42” API—you PUT, GET, LIST, and DELETE whole objects (or byte ranges within them).

That abstraction is why S3 scales: AWS can shard and replicate behind a key without you provisioning capacity. The trade-off is latency and access pattern. Millisecond API round trips suit backups, assets, analytics, and event-driven pipelines—not a transactional database’s random 8 KiB writes. For how S3 compares to EBS and EFS on AWS, see EBS, S3, and EFS compared.

Core building blocks

Bucket

A bucket is a container for objects. Bucket names are globally unique across all AWS accounts (DNS-style names like my-company-logs-prod). You create a bucket in a Region; data at rest stays in that Region unless you replicate or copy elsewhere. Buckets are the unit for most policies, logging, replication rules, and public-access settings.

Object and key

An object consists of:

• Key — UTF-8 identifier, often path-like: raw/events/year=2026/month=05/day=20/part-00042.parquet
• Body — Up to 5 TiB per object in general-purpose storage classes (multipart upload required above 5 GiB for best practice)
• Metadata — System metadata (size, ETag, storage class) and user-defined key/value pairs
• Version ID — Present when versioning is enabled

“Folders” in the console are a visual convenience: S3 is a flat key space. Prefixes (logs/app/) power LIST filtering and lifecycle rules.

ARNs and access endpoints

Objects and buckets have ARNs for IAM and AWS Backup. Requests hit Regional endpoints (s3.ap-south-1.amazonaws.com) or path-style / virtual-hosted-style URLs. From VPCs, a gateway endpoint for S3 keeps traffic on the AWS network and avoids NAT charges for high-volume private-subnet access—see AWS network architecture.

Durability, availability, and consistency

S3 Standard is designed for 99.999999999% (11 nines) durability over a given year—meaning AWS spreads copies across multiple facilities in the Region. Availability is a separate SLA (e.g. 99.99% for Standard); durability does not mean your application never sees 503s during extreme events.

Consistency: Today, S3 provides strong read-after-write consistency for PUTs and DELETEs of objects in all Regions—including LIST and HEAD after writes. Designs written around “eventual consistency for new keys” are outdated; still plan for application-level idempotency and retry logic on 5xx responses.

Storage classes (choose on purpose)

Each object has a storage class that drives durability, availability, minimum storage duration, retrieval fees, and retrieval time.

Class	Access pattern	Availability	Notes
S3 Standard	Frequent	99.99%	Default for hot data; lowest latency
S3 Intelligent-Tiering	Unknown or changing	Same as underlying tier	Auto-moves between frequent and infrequent tiers; small monitoring fee per object
S3 Standard-IA	Infrequent, needs rapid access	99.9%	30-day minimum; retrieval charge per GB
S3 One Zone-IA	Infrequent, recreatable	99.5% (single AZ)	Lower cost; not for sole copy of irreplaceable data
S3 Glacier Instant Retrieval	Archive, ms retrieval	99.9%	90-day minimum; higher storage $/GB than Standard-IA
S3 Glacier Flexible Retrieval	Archive, minutes–hours	99.99% (metadata)	Formerly “Glacier”; expedited retrieval costs more
S3 Glacier Deep Archive	Archive, 12+ hours	99.99% (metadata)	Lowest $/GB; compliance and tape-replacement workloads
S3 Express One Zone	Very frequent, latency-sensitive	Single AZ	Directory buckets; single-digit ms; different API/bucket type

Lifecycle rules transition objects between classes (e.g. Standard → Glacier after 90 days) or expire them. Combine with S3 Intelligent-Tiering when access patterns are messy; use explicit transitions when patterns are predictable and you want predictable bills.

Upload and download mechanics

Single PUT vs multipart upload

For objects larger than a few hundred MiB, use multipart upload: parallel parts, resume on failure, and completion with CompleteMultipartUpload. Failed or abandoned multipart uploads leave parts that still bill—automate abort rules in lifecycle configuration.

Checksums and integrity

Clients can send additional checksum headers (CRC32, CRC32C, SHA-1, SHA-256). S3 validates on upload and stores checksums for later verification—valuable for compliance and large dataset pipelines where bit rot must be detected, not assumed away.

Performance at scale

• Horizontal scale — S3 scales request rate with your keyspace; avoid artificial single-key hotspots for write-heavy workloads.
• Prefix design — Use meaningful prefixes for LIST and lifecycle, not because “more slashes” magically multiplies throughput on modern S3.
• Transfer Acceleration — CloudFront edge uploads for distant clients.
• S3 Select / Glacier Select — SQL-like filtering on object contents to reduce data movement.
• Byte-range GET — Parallel downloads of large objects.

Security model (defense in depth)

S3 security is IAM identities, resource policies, optional ACLs (legacy), encryption, and network path—layered together.

Block Public Access

Enable Block Public Access at the organization and account level first. It overrides bucket policies that would expose data. Public buckets remain a top breach pattern; SCPs plus this setting are baseline hygiene—aligned with cloud security foundations.

IAM policies and bucket policies

• Identity-based (IAM) — What a role or user may do across buckets.
• Resource-based (bucket policy) — What principals may do to this bucket; required for cross-account access without assuming a role in the other account.
• Condition keys — Enforce TLS (aws:SecureTransport), source VPC (aws:SourceVpce), IP ranges, prefix, storage class, and encryption headers.

Prefer least privilege actions: s3:GetObject on arn:.../bucket/prefix/* beats s3:*. For policy anatomy practice, see IAM policy JSON anatomy.

ACLs

S3 ACLs are largely legacy. New designs should use bucket policies and IAM; AWS recommends disabling ACLs with “Bucket owner enforced” object ownership.

Encryption

Option	Who holds keys	When to use
SSE-S3 (AES-256)	AWS	Default encryption; simple compliance checkbox
SSE-KMS	AWS KMS CMK	Audit trail per key use; separation of duties; watch KMS request costs at scale
SSE-C	Customer-provided per request	Rare; you manage key material and rotation
DSSE-KMS	KMS with dual-layer	Regulated workloads needing extra algorithmic layer

Set default encryption on every bucket. For buckets with strict compliance, deny PUTs without encryption headers via bucket policy.

Access Points and Object Lambda

S3 Access Points give each application or team a dedicated hostname and policy—simplifying shared buckets with different prefix permissions. Multi-Region Access Points route clients to the nearest replica with failover. S3 Object Lambda runs Lambda on GET to transform objects (resize images, redact fields) without changing how clients address the bucket.

Versioning, replication, and compliance

Versioning

With versioning enabled, DELETE adds a delete marker; overwrites retain prior versions. Protects against operator mistakes and ransomware-style overwrites when combined with tight IAM and MFA Delete on versioned buckets. Cost grows with churn—lifecycle rules to expire noncurrent versions are essential.

Replication

• Same-Region Replication (SRR) — Log aggregation, environment sync, ownership separation.
• Cross-Region Replication (CRR) — Disaster recovery, locality for global users; watch replication time control (RTC) SLA and transfer costs.
• Batch Replication — Backfill existing objects when enabling replication mid-life.
• Replication to S3 Glacier — DR copies land in cheaper tiers automatically.

Replication is asynchronous. Failover runbooks must define how applications switch endpoints and whether conflict resolution is needed.

S3 Object Lock and legal hold

Object Lock (WORM) supports compliance modes: governance (admins can override with permission) and compliance (no one deletes until retention expires). Legal hold blocks deletion regardless of retention. Pair with versioning for SEC 17a-4, HIPAA archive, and ransomware-resilient backup targets. Enable Object Lock only at bucket creation.

Event-driven architecture

S3 can notify downstream systems when objects are created, removed, or restored:

• Amazon EventBridge — Preferred for rich filtering and multiple targets.
• Lambda — Thumbnail generation, virus scan, ETL kickoff.
• SQS — Buffering and decoupling for high burst ingest.
• SNS — Fan-out alerts.

Design for at-least-once delivery: idempotent consumers, deduplication keys, and dead-letter queues. S3 → Lambda → downstream is a classic data-lake ingestion pattern used in data engineering on AWS.

Observability and governance

• Server access logging — Detailed request logs to another bucket (mind recursion and cost).
• CloudTrail data events — Who called GetObject on sensitive prefixes (volume pricing—sample or scope prefixes).
• S3 Inventory — Scheduled CSV/Parquet reports of objects and metadata for audits.
• S3 Storage Lens — Organization-wide metrics, anomalies, and recommendations.
• S3 Storage Class Analysis — Suggests lifecycle transitions from access patterns.
• AWS Config — Rules for encryption, public access, versioning.

Mounting and “filesystem” patterns

Standard S3 is not POSIX. Options when you need file-like access:

• Application SDK — Best for new code; explicit object semantics.
• Mountpoint for Amazon S3 — AWS-supported FUSE-style mount; read-heavy, sequential access; not a database data directory.
• Third-party adapters (s3fs, goofys) — Understand caching, consistency, and failure modes before production.
• Amazon EMR / Spark — Native s3a:// with Hadoop-compatible semantics; see EMR and Hadoop in depth.

If you need shared POSIX files across EC2 instances, use EFS or FSx—not S3 pretending to be NFS.

How S3 fits the wider AWS platform

Integration	Role of S3
CloudFront	Origin for static sites and signed URLs
Athena, Glue, Redshift Spectrum	Query layer over data lake prefixes
EBS snapshots, RDS backups	Durable backend you do not administer as buckets
SageMaker, Bedrock	Training data, model artifacts, batch inference I/O
AWS Backup	Centralized backup plans including S3
Storage Gateway	Hybrid cache with on-premises NFS/SMB/iSCSI

Cost and FinOps levers

S3 bills for storage (per class), requests, data transfer out to the internet, replication, retrieval (IA/Glacier), and features (Inventory, analytics). Common waste:

• Noncurrent versions never expired
• Incomplete multipart uploads
• Small objects in Intelligent-Tiering with monitoring overhead
• Standard class for data accessed once a quarter
• Cross-Region replication without business need
• Heavy LIST operations in analytics anti-patterns

Lifecycle to Glacier, tighten prefixes in Inventory, and use Storage Lens dashboards—see also FinOps: the invisible bill.

Operations: CLI patterns

# Create bucket (Region required)
aws s3api create-bucket --bucket my-app-data-prod-ap-south-1 \
  --region ap-south-1 --create-bucket-configuration LocationConstraint=ap-south-1

# Block public access (account or bucket level)
aws s3api put-public-access-block --bucket my-app-data-prod-ap-south-1 \
  --public-access-block-configuration BlockPublicAcls=true,IgnorePublicAcls=true,\
BlockPublicPolicy=true,RestrictPublicBuckets=true

# Default encryption with KMS
aws s3api put-bucket-encryption --bucket my-app-data-prod-ap-south-1 \
  --server-side-encryption-configuration '{"Rules":[{"ApplyServerSideEncryptionByDefault":{
  "SSEAlgorithm":"aws:kms","KMSMasterKeyID":"arn:aws:kms:..."},"BucketKeyEnabled":true}]}'

# Sync build artifacts (deploy pattern)
aws s3 sync ./dist s3://my-app-assets-prod/ --delete --sse aws:kms

# Presigned URL (time-limited download for clients)
aws s3 presign s3://my-app-assets-prod/reports/q1.pdf --expires-in 3600

Example lifecycle fragment (transition + abort stale multipart):

{
  "Rules": [{
    "ID": "tier-and-tidy",
    "Status": "Enabled",
    "Filter": { "Prefix": "logs/" },
    "Transitions": [{ "Days": 90, "StorageClass": "STANDARD_IA" }],
    "NoncurrentVersionExpiration": { "NoncurrentDays": 30 },
    "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 }
  }]
}

Architect checklist

1. Classify data: hot, warm, archive, compliance-bound.
2. One bucket per security/domain boundary; use prefixes for environments, not one giant public bucket.
3. Encryption by default; KMS where audit matters.
4. Versioning + lifecycle on anything operators touch.
5. Replication and Object Lock only where RPO/RTO or regulation requires—rehearse failover.
6. Events idempotent; monitor 4xx/5xx and replication lag.
7. VPC endpoints for private workloads; CloudFront for global read-heavy assets.

Common mistakes

• Public bucket policy drift — "Principal": "*" with s3:GetObject on sensitive prefixes.
• Using S3 as primary database storage — Wrong latency and semantics; use RDS, DynamoDB, or OpenSearch by access pattern.
• Ignoring delete markers and versions — Storage grows “invisibly” after incidents.
• Assuming replication is backup — Corrupt or encrypted objects replicate too; use versioning, Object Lock, and isolated accounts.
• KMS without bucket key — High request volume → KMS throttling and cost; enable S3 Bucket Keys.
• Listing huge buckets naively — Use Inventory or S3 Inventory reports instead of paginating millions of keys in app startup.

Further reading

• AWS — Amazon S3 User Guide (security, performance, storage classes)
• AWS — Well-Architected Framework — Security and Cost Optimization pillars
• AWS — S3 best practices for security and for data lakes
• AWS — Mountpoint for Amazon S3 documentation

Blog index · EBS, S3, and EFS · Data engineering on AWS · Cloud security foundations

Back to blog list