When learning cloud computing, it's tempting to jump straight into individual services like Amazon S3, DynamoDB, or Lambda. While understanding each service is important, I found that the bigger lesson wasn't about the services themselves—it was about how applications are designed to evolve over time.

To explore this, I built a simple document processing application locally using FastAPI, Floci (an open-source AWS emulator), Docker, and the AWS SDK for Python (boto3).

The goal wasn't just to upload files. It was to understand how good backend architecture makes it easier to replace one implementation with another without changing the application's public interface.

Starting Simple

The application began with a single responsibility:

Accept a file upload through an API.

The initial architecture was straightforward:

Client
   │
   ▼
FastAPI
   │
   ▼
uploads/

The uploaded file was simply written to a local directory.

While this worked, it tightly coupled the API endpoint to the storage implementation.

Introducing a Service Layer

Instead of handling file storage directly inside the API route, the upload logic was extracted into a dedicated storage service.

The architecture became:

Client
   │
   ▼
FastAPI
   │
   ▼
Storage Service
   │
   ▼
Local Storage

Although the functionality remained the same, this small refactor introduced an important software engineering principle:

Separation of Concerns.

The API became responsible for handling HTTP requests, while the storage service became responsible for managing files.

Swapping Local Storage for Amazon S3

Once the storage logic was isolated, replacing the implementation became surprisingly simple.

Instead of saving files locally, the storage service was updated to use Amazon S3 through the AWS SDK (boto3).

The architecture changed to:

Client
   │
   ▼
FastAPI
   │
   ▼
S3 Storage Service
   │
   ▼
Amazon S3 (Floci)

The API endpoint itself didn't need to change.

Only the storage implementation changed.

That was one of the biggest takeaways from the project.

Making Infrastructure Self-Initializing

Another improvement was avoiding manual infrastructure setup.

Rather than assuming an S3 bucket already existed, the application checks for it during startup and creates it if necessary.

Conceptually:

Application Starts
        │
        ▼
Check Bucket
        │
        ▼
Create If Missing

This small addition makes the application easier to run in a fresh environment and reduces manual setup steps.

Adding DynamoDB

Uploading files solved only part of the problem.

The application also needed to store metadata about each uploaded document.

A dedicated DynamoDB service was introduced with responsibilities such as:

• Generating a unique document ID
• Recording the upload timestamp
• Storing document metadata

The architecture now looked like this:

Client
        │
        ▼
FastAPI
   ├──────────────┐
   ▼              ▼
S3 Storage     DynamoDB Service
   ▼              ▼
Amazon S3     Amazon DynamoDB

Separating these responsibilities keeps the codebase modular and easier to maintain.

More Than Learning AWS

Although the project involved services like Amazon S3 and DynamoDB, the most valuable lessons weren't AWS-specific.

It reinforced several software engineering concepts:

• Separation of Concerns
• Service Layer Pattern
• Dependency Isolation
• Modular Backend Design
• Infrastructure Initialization

These ideas apply regardless of whether the backend eventually uses Amazon S3, Azure Blob Storage, Google Cloud Storage, or even local files.

The Bigger Lesson

One realization stood out throughout this project:

Cloud engineering isn't just about knowing cloud services.

It's about designing applications so that implementations can change without affecting the rest of the system.

For example:

LocalStorage
        │
        ▼
S3Storage

The API doesn't need to know which implementation is being used.

As long as the interface remains consistent, the underlying storage mechanism can evolve over time.

That flexibility is what makes production systems easier to extend, test, and maintain.

Final Thoughts

Building applications has been one of the most effective ways for me to learn cloud engineering.

Working through real architectural decisions made concepts like Amazon S3, DynamoDB, and the AWS SDK feel much more intuitive than simply reading documentation.

More importantly, it highlighted that good cloud applications are built not just on cloud services, but on sound software engineering principles.

GitHub Repository

The complete project is available here:

Repository: https://github.com/micheal000010000-hub/aws-document-processing-pipeline/tree/Document_Processing_Pipeline

Feedback and suggestions are always welcome.