Secure Docs

CDK • Comprehend • Bedrock • LLM • AWS Lambda • DynamoDB • S3 • SNS • SES • GitHub Actions • Serverless • IaC • Event-driven architecture

GitHub

Project Description

SecureDocs is a demonstration project for automating the initial verification of uploaded documents. It is designed to encapsulate the full business workflow for validating and screening incoming documents before they are accepted for further processing.

Project Context

The primary challenge SecureDocs addresses is the necessity for a unified method to handle the uploading and primary screening of incoming documents across various company departments. To guarantee isolation between different organizational units, the architecture assumes that a dedicated construct is created and deployed for each individual department.

Architecture & Technical Solution

The following diagram illustrates the key components and their interaction within the SecureDocs project.

To maintain strict isolation of documents across different stages of their lifecycle, the construct provisions several specialized Amazon S3 buckets:

Upload: Designed to hold documents that have been uploaded from external sources.
Quarantine: Serves as a repository for documents that the antivirus scanner has flagged as infected or suspicious.
Internal: Stores cleared documents that are accessible to all employees within the department.
Restricted: Secures documents containing Personally Identifiable Information (PII) or sensitive business data, restricting access strictly to senior employees.

End-users interact with a web form to upload their files. This form makes an API call to retrieve a pre-signed URL, allowing the client to securely upload the document directly into the Upload bucket.

Document Verification Flow

Upon successful upload to the Upload bucket, an Amazon EventBridge event initiates an orchestration workflow implemented using an AWS Step Functions state machine, as shown in the diagram below:

show diagram

Antivirus Scanning: The initial step performs an antivirus scan on the document. For this demonstration project, the scan is simulated using an AWS Lambda mock. If a document is found to be infected, it is immediately relocated to the Quarantine bucket, and an alert is broadcast to administrators via an Amazon SNS topic. If the document is clean, it proceeds to the next validation stage.
PII Detection: An AWS Lambda function invokes Amazon Comprehend to inspect the document's contents for PII. Upon detecting confidential information, the system moves the document to the Restricted bucket and dispatches an SNS notification to senior managers. If no sensitive data is identified, the workflow advances.
Content Classification: The system then evaluates whether the document contains financial or business secrets. An AWS Lambda function interfaces with an LLM via Amazon Bedrock to categorize the content. The model outputs a classification result of Internal, Business, or Ambiguous. If the model successfully determines the category (Internal or Business), the file is routed to the corresponding Internal or Restricted bucket, and an SNS message alerts the appropriate employees or administrators.

Manual Review & Error Handling

In cases where the LLM returns an "Ambiguous" classification, the automated flow is paused and the document is assigned to a manager for manual review. An AWS Lambda function sends a message to the managers containing a token required to continue the workflow. Once the manager determines the correct category and completes the task, the document is transferred to either the Internal or Restricted bucket, triggering a final SNS notification. The Step Functions execution then completes.

Additionally, if an error is encountered during either the PII scanning phase or the LLM categorization phase, the task is automatically routed to a manager for manual evaluation.

Data Management and Security

Throughout the processing pipeline, documents are enriched with specific metadata tags reflecting their verification status:

org:AVStatus: Tagged as Unverified, Clean, or Infected.
org:PIIStatus: Tagged as NoPii or PiiFound.
org:Category: Tagged as Internal or Business.

To enforce robust data governance, strict resource-based IAM policies are attached to the Internal and Restricted S3 buckets. These policies explicitly prohibit the placement of any documents into these buckets unless they possess an org:AVStatus tag of "Clean", along with the appropriate combination of org:PIIStatus and org:Category tags.

Audit Log

Throughout the document lifecycle, the system records audit events in a dedicated Amazon DynamoDB table. These events capture key actions such as document upload, malware scan results, classification decisions, and manual review outcomes.

Upon workflow completion, all audit events for the document are consolidated into a single audit log and stored in an Amazon S3 bucket with S3 Object Lock (WORM) enabled, ensuring the log remains immutable and tamper-resistant for compliance.

Deployment

To ensure consistent and reproducible environments, the SecureDocs project is deployed directly from its repository utilizing automated GitHub Actions workflows.