TL;DR

This is Phase 9 of the FSx for ONTAP S3 Access Points serverless pattern library. Building on Phase 8, Phase 9 delivers:

• VPC Endpoint auto-detection: deploy_generic_ucs.sh now checks VPC Endpoint status before deploying and automatically enables creation when none exist — removing the missing-endpoint Connect timeout from the default deployment path
• OutputDestination parameters for UC6/7/8: The last 3 Athena-using UCs now have OutputDestination, OutputS3APAlias, and OutputS3APPrefix CloudFormation parameters, completing the full 17-UC OutputDestination unification
• cfn-guard IAM security rules: 4 new CloudFormation Guard rules integrated into the validator CI workflow
• Lambda performance tuning: UC1-14 Discovery Lambdas now default to 512MB+ with extended timeouts, and UC1's AclCollection Map state parallelism is configurable at deploy time
• CDK migration evaluation: A weighted-scoring evaluation concluded No-Go for this repository at this maturity level
• Full environment cleanup: 3 stale stacks, 6 retained DynamoDB tables, and 1 versioned S3 bucket (2118 objects) cleaned up during verification

In short: Phase 8 built the operational tooling. Phase 9 rolled it out across the pattern library and removed the biggest deployment foot-gun: assuming that shared VPC Endpoints already exist.

Repository: github.com/Yoshiki0705/FSx-for-ONTAP-S3AccessPoints-Serverless-Patterns

1. The VPC Endpoint ownership problem

Phase 9's most impactful change came from a failure, not a plan.

During Phase 9 AWS verification, UC7 was deployed with OutputDestination=FSXN_S3AP to verify the new template parameters. The execution failed immediately with:

Connect timeout on endpoint URL:
"https://eda-demo-s3ap-...ext-s3alias.s3.ap-northeast-1.amazonaws.com/?list-type=2"

Root cause: during Phase 8 cleanup, the fsxn-eda-uc6 stack was deleted. That stack happened to "own" all VPC Endpoints in the shared VPC (S3 Gateway, Secrets Manager, FSx, CloudWatch, SNS). When it was deleted, CloudFormation dutifully removed those endpoints. Every subsequent VPC Lambda deployment — with EnableVpcEndpoints=false (the old default) — had no path to AWS services.

This turned the deployment model from "the operator must remember who owns the shared endpoints" into "the script checks the environment and chooses the safe default." The important change is not just endpoint creation — it is removing hidden infrastructure ownership from the deployment path.

The fix: auto-detection

The deploy script now runs a pre-flight check:

EXISTING_ENDPOINTS=$(aws ec2 describe-vpc-endpoints \
  --filters "Name=vpc-id,Values=${VPC_ID}" "Name=state,Values=available" \
  --query 'length(VpcEndpoints)' --output text)

if [[ "$EXISTING_ENDPOINTS" == "0" ]]; then
    ENABLE_S3_GATEWAY_EP="true"
    ENABLE_VPC_ENDPOINTS="true"
fi

The defaults changed from false to auto. When auto:

• If no VPC Endpoints exist → create them (first deploy or post-cleanup)
• If endpoints already exist → skip creation (avoids private-dns conflict)
• S3 Gateway Endpoint is checked separately because it has different semantics from interface endpoints and does not create a private DNS conflict

The current logic intentionally optimizes for the two common states: a clean environment with no endpoints (first deployment or post-cleanup), and a steady-state environment where endpoints already exist. Partially drifted endpoint sets — where some interface endpoints are missing but others remain — are expected to be handled by explicit deployment parameters (ENABLE_VPC_ENDPOINTS=true) or infrastructure repair. This is documented in docs/operational-runbooks/deployment-troubleshooting.md Failure Mode 7.

2. Completing OutputDestination for UC6/7/8

Phase 8 standardized non-discovery AI/ML output handlers on OutputWriter.from_env(). Discovery manifest writers remain on the separate s3ap_output.put_object helper and are covered by the S3AP IAM validator. But UC6, UC7, and UC8 templates still lacked the CloudFormation parameters to control the output destination at deploy time.

Phase 9 adds to each template:

Parameters:
  OutputDestination:
    Type: String
    Default: "STANDARD_S3"
    AllowedValues: ["STANDARD_S3", "FSXN_S3AP"]

  OutputS3APAlias:
    Type: String
    Default: ""

  OutputS3APPrefix:
    Type: String
    Default: "ai-outputs/"

Conditions:
  UseStandardS3: !Equals [!Ref OutputDestination, "STANDARD_S3"]
  UseFsxnS3AP: !Equals [!Ref OutputDestination, "FSXN_S3AP"]
  HasOutputS3APAlias: !Not [!Equals [!Ref OutputS3APAlias, ""]]

And passes OUTPUT_DESTINATION, OUTPUT_S3AP_ALIAS, OUTPUT_S3AP_PREFIX environment variables to every OutputWriter-using Lambda function.

AWS verification: UC7 deployed with OutputDestination=FSXN_S3AP and executed successfully. AI outputs landed on FSxN S3AP under the ai-outputs/ prefix. Athena results stayed on the S3 bucket (AWS constraint, as designed).

With this change, all 17 UCs now support OutputDestination switching at deploy time. The full Pattern B migration — from handler code to template parameters — is complete.

3. Security validation with cfn-guard

Phase 8's check_s3ap_iam_patterns.py catches the S3AP alias-only IAM bug via Python regex. Phase 9 adds CloudFormation Guard rules that catch broader IAM anti-patterns:

These run in advisory mode (|| true) in CI — they report violations without blocking the pipeline. Advisory mode lets the project introduce the rule set without breaking existing PRs while the remaining warnings are reviewed. Once all templates are fully compliant, the || true will be removed to make them blocking.

The CI workflow now has 6 blocking/advisory validator jobs in validators.yml:

1. S3AP IAM pattern check (Python)
2. Handler undefined-name sweep (pyflakes)
3. Conditional resource ref check
4. Python quality (pyflakes CRITICAL gate)
5. cfn-guard IAM security (new, advisory mode)
6. Sensitive leak scan (OCR, optional — requires SENSITIVE_STRINGS_PY GitHub Secret; no-op when unconfigured)

These complement the existing lint.yaml workflow (cfn-lint + ruff) and ci.yml (cfn-lint + unit tests + cfn-guard + bandit) that run on the same triggers. The existing ci.yml continues to run the broader cfn-guard suite against all YAML files, while validators.yml adds the same IAM-focused rules scoped specifically to template-deploy.yaml files in advisory mode.

The sensitive leak scan uses a GitHub Secret to materialise the pattern set at CI runtime; the private _sensitive_strings.py file remains local-only for contributor-specific secrets. This avoids committing environment-specific sensitive strings while still allowing the main repository CI to run a meaningful OCR leak scan. On forks and external PRs where the secret is unavailable, the scan runs with an empty pattern set and effectively becomes a no-op.

4. Lambda performance tuning across the fleet

UC1 MaxConcurrency parameter

The AclCollection Map state processed 549 files with MaxConcurrency: 10 in Phase 8, taking 2:20 of the 2:38 total execution time. Phase 9 parameterizes this:

AclCollectionMaxConcurrency:
  Type: Number
  Default: 10
  MinValue: 1
  MaxValue: 1000
  Description: |
    AclCollection Map state の並列度。
    ONTAP REST API の rate limit (100 req/s) を超えないよう注意。

Operators can now raise AclCollectionMaxConcurrency — for example, from 10 toward 100 — to reduce wall-clock time, trading ONTAP API rate budget for throughput. Actual speedup depends on ONTAP REST API latency, throttling behavior, and Step Functions overhead.

Fleet-wide Lambda tuning

In practice, the higher memory setting gave the Discovery Lambdas more CPU and network headroom for initialization and SDK calls, while the 900s timeout removed the premature failure mode observed during VPC cold starts and large ONTAP volume scans. The longer timeout is paired with CloudWatch alarms and Step Functions visibility; it is intended to avoid premature failure during initialization, not to hide hung workflows. UC15-17 retain their 120s timeout because their ONTAP volumes contain fewer files and Discovery completes well within that window.

5. CDK migration evaluation: No-Go

The evaluation is documented in docs/cdk-migration-evaluation.md. The weighted scoring:

The decision was not "YAML is better than CDK." It was narrower: for this repository, at this maturity level, the existing validators now provide most of the practical safety benefit without a migration tax. The templates are stable (Phase 8 was the last structural change). Migration cost (34+ hours) exceeds the benefit for a library that adds ~1 UC per phase.

Revisit conditions: 25+ UCs, CDK-native team member, or AWS shipping "import from existing CFn" that preserves logical names.

6. Event-driven trigger status

As of May 2026, FSxN S3AP does not support GetBucketNotificationConfiguration for S3 access points attached to FSx for ONTAP volumes (AWS documentation: Access point compatibility). The UC1 EventBridge rule + DynamoDB idempotency table remain staged and ready to be wired into the native event source once AWS ships FR-2.

Alternative path (ONTAP FPolicy → SQS → EventBridge) is documented as a Phase 10 candidate for environments that need event-driven processing before FR-2 ships.

7. Environment cleanup

Phase 9 verification also included a full cleanup pass for stale resources left from Phases 6-8: three stacks, six retained DynamoDB tables, and one versioned S3 bucket containing 2,118 objects.

After cleanup: only fsxn-s3ap-guard-hooks (Phase 6 shared infra, intentionally kept) and fsxn-eda-deploy-* (Lambda package bucket, always needed) remain.

Full stats

• Templates updated: 13 (UC1-8 + UC10-14 performance tuning; UC6-8 OutputDestination)
• New cfn-guard rules: 4 (in security/cfn-guard-rules/s3ap-iam-dual-format.guard)
• CI jobs: 6 validator jobs in validators.yml (was 5) + existing cfn-lint and test workflows
• Deploy script: VPC Endpoint auto-detection (30 lines of pre-flight logic)
• Documentation: docs/cdk-migration-evaluation.md, docs/verification-results-phase9.md, docs/guides/observability-snippet.md
• AWS verification: UC7 SUCCEEDED with OutputDestination=FSXN_S3AP + EnableCloudWatchAlarms=true
• Cleanup: 3 stacks + 6 DDB tables + 1 S3 bucket (2118 objects)

Looking Forward to Phase 10

Phase 9 completed the production rollout. Phase 10 candidates:

1. ONTAP FPolicy event-driven path: For environments that need event-driven processing before AWS ships FR-2 (native S3AP events)
2. Observability alarm tuning: Per-UC threshold customization based on actual traffic patterns
3. Multi-account deployment: Cross-account S3AP access patterns for enterprise environments
4. Cost optimization automation: Auto-scaling Map state MaxConcurrency based on ONTAP API response latency

Who should care about Phase 9?

• Platform teams get safer deployments because shared VPC Endpoint assumptions are no longer hidden.
• Security teams get cfn-guard IAM checks in CI, complementing the Python S3AP validator.
• Data teams get OutputDestination parameters for the final Athena-constrained UCs.
• Operations teams get tuned Lambda defaults and cleanup of stale resources.
• Architects get a documented CDK No-Go decision with revisit conditions.
• Partners and SIs get reduced demo and PoC friction by removing the assumption that shared VPC Endpoints already exist.
• Contributors get a unified baseline for new UCs: OutputDestination parameters, tuned Discovery defaults, validator compliance, and no hidden endpoint assumptions (see docs/guides/new-uc-checklist.md).

Operational notes for production adoption

The auto-detection logic removes the hidden dependency from the default deployment path, but it does not replace shared-infrastructure ownership. In production, VPC Endpoints should still have an explicit owner — ideally a shared-infra stack or platform team — for cost allocation and lifecycle management.

The cfn-guard rules are intentionally narrow and based on recurring failure patterns observed across Phases 7-9. They complement, but do not replace, IAM Access Analyzer or full least-privilege review.

Even after OutputDestination unification, Athena results remain on STANDARD_S3 by AWS service constraint. Operators should manage those result buckets separately with encryption, lifecycle, and access policies appropriate to their compliance requirements.

Phase 9 does not solve multi-account deployment yet. It makes the single-account baseline safer first, so Phase 10 can address cross-account and shared-network ownership from a more stable foundation.

A future contributor guide should document which validator jobs are blocking, which are advisory, and how to reproduce each check locally before opening a PR.

Conclusion

Phase 9 did not add a flashy new use case. It made the existing pattern library safer to deploy, easier to operate, and more honest about its infrastructure dependencies.

Phase 8 built the operational tooling — cleanup scripts, validators, observability, OutputWriter unification. Phase 9 rolled it out to every UC and made the deployment experience resilient to environment state.

The most impactful change was the simplest: checking whether VPC Endpoints exist before deploying. Every previous phase had at least one "why did this fail?" moment caused by missing endpoints. Phase 9 removes that failure mode from the default deployment path for clean and steady-state environments.

The CDK evaluation confirmed what the project's history suggests: YAML with 6 CI validators catches the bug classes that actually hurt this repository so far — without requiring a rewrite of 8 phases of documentation.

The pattern library now has a consistent operational baseline across all 17 UCs:

• OutputDestination switching where applicable
• CloudWatch Alarms and EventBridge failure notifications as deploy-time options
• Tuned Discovery Lambda defaults for long-running VPC workloads (UC1-8, UC10-14: 512MB/900s; UC9: 1024MB/600s; UC15-17: 512MB/120s)
• OutputWriter-standardized AI/ML output paths
• cfn-guard IAM security validation in CI

Phase 10 will focus on event-driven processing alternatives and multi-account patterns. The operational baseline is complete.

Repository: github.com/Yoshiki0705/FSx-for-ONTAP-S3AccessPoints-Serverless-Patterns
Previous phases: Phase 1 · Phase 6A/6B · Phase 7 · Phase 8