Making an Enterprise Document Pipeline Observable and Self-Service
An internal pipeline moved engineering drawings, specifications, and part files from Amazon S3 into OpenSearch. It worked, but to most users it was a black box: they could not confirm whether a document had completed the journey, understand why it was blocked, or manage files without involving the infrastructure team.
I helped turn that invisible workflow into one focused internal product: a secure storage browser for document operations and a monitoring dashboard for ingestion health. The challenge was not simply building two screens. It was delivering useful software across Azure AD, an on-premises .NET API, and shared AWS infrastructure while the backend data contract was still evolving.
1. The operational gap
The existing workflow left both business users and support teams dependent on manual investigation.
The product goal was straightforward: reduce the distance between an operational question and a trustworthy answer, without weakening the security boundary around enterprise documents.
2. Constraints that shaped the system
- Azure AD was the company identity standard; the AWS Cognito-based reference application could inform the UX, but not the architecture.
- The API ran under IIS on premises, so it had no EC2, ECS, or Lambda instance role and could not rely on ambient AWS credentials.
- The DynamoDB schema belonged to another team and was not finalized when frontend work began.
- The S3 bucket also served a legacy tool, so policy and CORS changes had to remain additive, scoped, and reversible.
- The team needed to ship incrementally, without waiting for every upstream decision to settle.
3. Architecture: control through the API, bytes around it
System boundaries
Architecture flowThe central decision was to keep the API in the control path but out of the data path. The server authenticates the caller, authorizes the action, and creates a short-lived presigned URL. The browser then transfers the file directly to or from S3. Large file bytes never pass through IIS.
4. A storage browser designed for the existing identity model
The storage experience uses familiar file-manager patterns—breadcrumbs, folders, upload progress, download, and recursive delete—while preserving server-side control over every privileged operation.
- Browse uses S3 ListObjectsV2 with a delimiter to present object prefixes as folders.
- Uploads and downloads use ten-minute presigned PUT and GET URLs.
- Up to four uploads run concurrently, each with independent progress feedback.
- Folders are represented by zero-byte objects with trailing-slash keys.
- Recursive deletion paginates through a prefix and removes objects in batches.
This avoided turning the on-premises API into a file proxy and made upload throughput depend on the browser-to-S3 connection rather than server memory and bandwidth.
5. Authorization with deliberately different failure modes
Authentication and authorization were split into two explicit layers. Azure AD login is mandatory, but signing in alone does not grant access. An administrator assigns approved employees to the application's Azure AD group, and an internal app-roles service verifies that membership before protected data is available.
This supports least-privilege access: group membership determines whether someone can enter the application, while the assigned role determines whether that person can only view data or can also upload, modify, and manage documents. The API enforces these permissions for every operation; hiding a button in Angular is never treated as authorization.
The asymmetry is intentional. Bypassing or breaking a client-side guard can never grant access to backend data because every controller action independently revalidates the user. The required group remains environment configuration, so enabling the final stakeholder-approved group does not require an application rewrite.
6. Auditable changes and recoverable documents
Document access is only one part of the control model. Every mutating action—upload, replacement, folder creation, metadata update, and deletion—is recorded as an audit event with the authenticated user, action, object key, timestamp, and relevant version reference. This creates a searchable history of who changed what and when.
S3 versioning retains earlier object versions when a document is replaced or deleted. The database audit trail explains the business action, while the S3 version ID points to the corresponding object state. Together, they support later investigation, comparison, and recovery without presenting version history as a substitute for authorization.
7. Building the dashboard before the data contract was final
Waiting for the final DynamoDB schema would have blocked the entire dashboard. Instead, I defined the service contract around the questions the UI needed to answer, then built a realistic mock implementation behind it. That allowed the filters, KPI cards, table states, pagination, and errors to be developed against stable behavior.
When the real table shape was confirmed, the production implementation replaced the mock through dependency injection. The frontend contract did not change, and the mock stayed useful for development and failure-state testing.
8. Turning pipeline data into operational answers
The dashboard reduces two backend fields—upload state and search-index sync state—into three user-facing outcomes: Synced, Pending, and Blocked. A blocked state can represent failure in either stage, with the original error visible for diagnosis.
Data is loaded in paginated batches rather than sending the full table to the browser. The table had no suitable secondary index, so the first production implementation uses a DynamoDB Scan with date, filename, deletion, and status filters. Each response includes the current rows and an opaque page token wrapping LastEvaluatedKey. The client returns that token to load the next batch without needing to understand the DynamoDB key structure.
This is a pragmatic first version, not a claim that Scan is the ideal access pattern. DynamoDB applies the page limit before the filter expression, so a page may contain fewer visible rows even when more matches exist. The behavior is documented and the UI can continue loading until the token is empty. If data volume or traffic grows, a status-and-date access pattern backed by a secondary index is the natural next optimization.
9. Production-minded decisions beyond the happy path
- Runtime configuration loads before Angular bootstraps, allowing one build artifact to serve multiple environments.
- AWS credentials support local developer secrets while remaining compatible with the SDK default provider chain for a future managed-identity approach.
- Shared-bucket IAM and CORS changes were documented with their prior state and rollback steps before deployment.
- Database audit events and S3 version IDs connect user actions to recoverable document states.
- MSAL-triggered HTTP updates explicitly re-enter Angular's zone so authenticated requests reliably update the view.
- The dashboard favors searchable rows and visible errors over decorative charts because the primary job is finding what is stuck and why.
10. Outcome and engineering value
The resulting tool works end to end: authorized group members can access capabilities appropriate to their assigned permissions, document changes remain attributable and recoverable, and the monitoring dashboard reads paginated ingestion state from DynamoDB without requiring AWS console access.
More importantly, the delivery strategy removed avoidable blockers. Presigned URLs respected the on-premises hosting constraint. Interface-first development decoupled UI progress from an evolving schema. Layered authorization kept convenience checks separate from real enforcement. Reversible infrastructure changes protected a shared production dependency.
The case study demonstrates the kind of work I value most: translating ambiguous operational pain into a secure, understandable product—and making explicit tradeoffs so the next engineer can evolve it safely.
11. What I would evolve next
- Replace long-lived service credentials with the organization’s approved workload-identity mechanism.
- Introduce a DynamoDB index or purpose-built read model when usage justifies moving beyond Scan.
- Add trend metrics for processing latency and failure rate once enough history exists to make them meaningful.
- Complete the app-group configuration after the directory owner confirms the production group.
Company, environment, tenant, account, and document identifiers have been generalized to preserve confidentiality.