AWS Unveils Amazon S3 Files, Bridging the Gap Between Object Storage and File Systems

Amazon Web Services (AWS) has announced the launch of Amazon S3 Files, a groundbreaking new feature that fundamentally redefines how users interact with Amazon Simple Storage Service (Amazon S3). This innovation effectively transforms S3 buckets into fully functional, high-performance file systems, offering seamless integration with virtually any AWS compute resource. This development promises to streamline data management, enhance application development, and unlock new possibilities for workloads ranging from machine learning to agentic AI systems.

The announcement marks a significant evolution in cloud storage, addressing a long-standing distinction between object storage, known for its durability and cost-effectiveness, and file systems, prized for their interactive capabilities and familiar hierarchical structure. Historically, developers and IT professionals often had to make trade-offs, choosing between the scalability and resilience of S3 or the direct manipulation and compatibility of traditional file systems. Amazon S3 Files appears to bridge this divide, presenting S3 data as a native file system accessible via standard Network File System (NFS) v4.1+ operations.

Bridging the Object Storage and File System Divide

For over a decade, the distinction between object storage and file systems has been a cornerstone of cloud architecture discussions. AWS trainers often employed analogies, such as comparing S3 objects to immutable books in a library—requiring replacement of the entire object for any modification—versus files on a local drive, which allow for granular, in-place editing. This fundamental difference dictated how data was architected and accessed for various applications. With the introduction of Amazon S3 Files, this distinction becomes significantly more fluid.

Amazon S3 Files positions S3 as the first and only cloud object store to offer comprehensive, high-performance file system access. This capability makes existing S3 buckets accessible as file systems. Any modifications made to data through the file system interface are automatically reflected in the corresponding S3 bucket, with fine-grained control over synchronization. Furthermore, S3 Files can be attached to multiple compute resources concurrently, enabling efficient data sharing across clusters without the need for data duplication, a common challenge in distributed computing environments.

This new offering effectively eliminates the traditional trade-off between the cost-efficiency, durability, and broad service integration of Amazon S3, and the interactive, real-time capabilities of file systems. S3 now serves as a central repository for an organization’s data, directly accessible from any AWS compute instance, container, or function. This includes Amazon Elastic Compute Cloud (EC2) instances, containers managed by Amazon Elastic Container Service (ECS) or Amazon Elastic Kubernetes Service (EKS), and serverless functions like AWS Lambda.

Technical Underpinnings and Performance

Amazon S3 Files presents S3 objects as files and directories, supporting a full suite of NFS v4.1+ operations, including creating, reading, updating, and deleting files. When data is accessed or modified through the file system, associated metadata and file content are intelligently managed by the file system’s high-performance storage layer. For data requiring low-latency access, files are stored and served from this optimized storage. Conversely, for workloads involving large sequential reads, S3 Files can serve data directly from Amazon S3, maximizing throughput. The system also supports byte-range reads, ensuring that only the necessary portions of data are transferred, thereby minimizing data movement and associated costs.

Intelligent pre-fetching capabilities are built into the system to anticipate data access patterns, further enhancing performance. Users retain granular control over what data is cached on the file system’s high-performance storage, allowing them to optimize for specific access patterns, whether it involves loading full file data or merely metadata.

Under the hood, Amazon S3 Files leverages Amazon Elastic File System (EFS) technology to deliver low latencies, reportedly around 1 millisecond for active data. The file system supports concurrent access from multiple compute resources with NFS close-to-open consistency. This makes it particularly well-suited for interactive, shared workloads that involve data mutation. Potential use cases include agentic AI systems collaborating via file-based tools or machine learning (ML) training pipelines processing large datasets.

Getting Started: A Practical Demonstration

AWS has provided a clear pathway for users to begin leveraging Amazon S3 Files. The process of creating and mounting an S3 file system on an EC2 instance is described as straightforward. A typical workflow involves an existing EC2 instance and a general-purpose S3 bucket.

Step 1: Create an S3 File System
Users can initiate the process through the AWS Management Console. Within the Amazon S3 section, they navigate to "File systems" and select "Create file system." The console prompts for the name of the S3 bucket to be exposed as a file system.

Step 2: Discover the Mount Target
A mount target, a network endpoint residing within the user’s virtual private cloud (VPC), is automatically created. This endpoint facilitates access from EC2 instances to the S3 file system. The Mount Target IDs are noted from the "Mount targets" tab. For users preferring the AWS Command Line Interface (CLI), separate commands, create-file-system and create-mount-target, are required.

Step 3: Mount the File System on an EC2 Instance
Once connected to an EC2 instance, the file system can be mounted using standard Linux commands. For example:
sudo mkdir /home/ec2-user/s3files
sudo mount -t s3files fs-0aa860d05df9afdfe:/ /home/ec2-user/s3files

After mounting, users can interact with their S3 data directly through the mounted file system using familiar file operations within the specified directory, such as /home/ec2-user/s3files.

Synchronization and Data Flow
Updates made to files within the mounted file system are automatically synchronized back to the S3 bucket, appearing as new objects or new versions of existing objects within minutes. Conversely, changes made directly to objects in the S3 bucket become visible in the mounted file system within seconds, though this can occasionally take longer.

A practical demonstration showed the creation of a file named hello.txt on the EC2 file system. The command echo "Hello S3 Files" > s3files/hello.txt created the file. A subsequent ls -al s3files/hello.txt verified its presence and attributes. Verification through the AWS CLI confirmed the file’s existence in the S3 bucket (aws s3 ls s3://s3files-aws-news-blog/hello.txt) and its content (aws s3 cp s3://s3files-aws-news-blog/hello.txt . && cat hello.txt displayed "Hello S3 Files").

Differentiating S3 Files from Other AWS File Services

AWS acknowledges the potential for confusion among its various file storage offerings. The company has provided guidance on choosing the appropriate service for different workloads:

• Amazon S3 Files: This service is ideal for scenarios requiring interactive, shared access to data residing in Amazon S3 via a high-performance file system interface. It excels in workloads where multiple compute resources need to collaboratively read, write, and mutate data. Benefits include shared access across compute clusters without data duplication, sub-millisecond latencies for active data, and automatic synchronization with S3 buckets.

• Amazon FSx: For organizations migrating from on-premises Network Attached Storage (NAS) environments, Amazon FSx offers familiar features and compatibility. It is also a strong choice for high-performance computing (HPC) and GPU cluster storage, particularly with Amazon FSx for Lustre. FSx is also recommended when applications require specific file system capabilities offered by Amazon FSx for NetApp ONTAP, Amazon FSx for OpenZFS, or Amazon FSx for Windows File Server.

This clarification aims to help cloud architects and developers select the most suitable storage solution based on their specific application requirements and existing infrastructure.

Pricing and Availability

Amazon S3 Files is available starting today in all commercial AWS Regions. Pricing is based on several factors:

• The amount of data stored within the S3 file system.
• The number of small file read operations and all write operations performed on the file system.
• S3 requests incurred during data synchronization between the file system and the S3 bucket.

Detailed pricing information is available on the Amazon S3 pricing page.

Broader Implications and Future Outlook

The introduction of Amazon S3 Files is poised to significantly simplify cloud architectures by breaking down data silos and reducing the complexity associated with data synchronization and manual data movement between object and file formats. For organizations developing agentic AI systems that rely on file-based Python libraries and shell scripts, or preparing datasets for machine learning training, this feature allows these interactive, shared, and hierarchical workloads to access S3 data directly. This eliminates the need to choose between the inherent durability and cost benefits of Amazon S3 and the interactive capabilities of a file system.

By enabling Amazon S3 to serve as a unified data repository accessible from any AWS compute instance, container, or function, AWS is empowering businesses to build more agile, efficient, and scalable solutions. The ability to leverage familiar file system interfaces for data stored in S3 opens up new avenues for innovation and simplifies the development and deployment of a wide range of applications.

AWS encourages users to explore the S3 Files documentation for further details and guidance on getting started. Feedback on this new capability is welcomed, suggesting an ongoing commitment from AWS to refine and enhance its storage offerings based on user experience and evolving industry needs.