Scenario

The VCF 9.1 upgrade path has been selected. The planning tool has been reviewed. The target version is understood. The team has a maintenance window on the calendar.

That does not mean the environment is ready.

A VCF 9.1 upgrade runbook needs to prove more than task order. It needs to prove that the organization can execute the upgrade, validate the result, stop safely when needed, and hand the platform back to operations without ambiguity.

This is where many upgrade plans become too lightweight.

They contain the sequence, but not the gates.

They mention backups, but not restore confidence.

They include prechecks, but not application validation.

They describe rollback, but not what rollback means for each phase.

VCF 9.1 deserves a runbook that treats testing, fallback, and readiness as first-class workstreams.

Why This Matters Operationally

VCF 9.1 is not a single-appliance patch.

The upgrade path touches VCF Operations, SDDC Manager, VCF Management Services, licensing, NSX, vCenter, ESXi, NSX Edge, workload domains, and Day-N operational services.

That creates three practical risks:

1. A phase can be technically complete but operationally incomplete.
2. A component can upgrade successfully while observability, licensing, identity, or application acceptance remains unresolved.
3. A failed phase may not have a clean “undo” path unless the fallback decision was defined before the work began.

The runbook should make those risks visible before the window starts.

Symptoms of a Weak VCF 9.1 Upgrade Runbook

A weak runbook usually sounds confident until you ask who owns the evidence.

Common warning signs include:

Symptom	Why It Is a Problem
“We will validate after the upgrade.”	Validation must be designed before execution, not improvised after change.
“Backups are handled by the backup team.”	The upgrade team still needs backup evidence and recovery ownership.
“Rollback is restore from backup.”	That may be true for one component and wrong for another.
“Monitoring will be checked at the end.”	VCF Operations is part of the sequencing model, not a final dashboard check.
“The platform team owns it.”	Platform teams often do not own DNS, IPAM, PKI, identity, network policy, or application testing.
“Workload domains can be done later.”	Later still needs an owner, sequence, acceptance model, and Day-N backlog.

If these answers show up in the planning meeting, the runbook is not ready.

The Runbook Model: Gates Before Tasks

A VCF 9.1 runbook should be organized around gates.

Each gate answers a simple question:

“Do we have enough evidence to proceed to the next phase?”

The task list still matters, but the gate controls the risk.

This model keeps the runbook honest.

The team cannot hide missing DNS records inside a task list. It cannot bury unclear rollback language under a generic change record. It cannot treat application testing as a courtesy check after the platform team leaves.

Every gate needs an owner, evidence, and a stop/go decision.

Prerequisites and Safety Checks

Before the runbook enters the execution window, the team should complete the following checks.

Area	Required Evidence
Upgrade path	Approved path based on current Broadcom documentation, release notes, KBs, interoperability checks, and compatibility guidance.
Current-state inventory	SDDC Manager, vCenter, ESXi, NSX, NSX Edge, Aria Operations, VCF Operations, workload domains, and adjacent management components documented.
VCF Operations readiness	Aria Operations version and patch level validated, VCF Operations path confirmed, Cloud Proxy requirements understood, dashboards and policies reviewed.
Management services readiness	IP ranges reserved, DNS records planned, internal IP overlap checked, management network reachability confirmed.
Licensing readiness	License server design, DNS requirements, and post-upgrade licensing validation documented.
Backup evidence	SDDC Manager, vCenter, NSX, VCF Operations, and other critical management appliances covered by known backup/recovery procedures.
Change control	Window, owner, bridge, escalation path, support case strategy, communication plan, and stop/go criteria approved.
Application validation	Representative workloads and application owner smoke tests identified before the window.

The point is not to create paperwork.

The point is to avoid discovering a blocking dependency while the environment is already in motion.

Runbook Stage 0: Source Validation

The first stage is source validation.

This is where the team confirms the runbook is based on current information, not stale assumptions.

Capture:

• VCF 9.1 upgrade documentation.
• VCF 9.1 release notes.
• Broadcom KBs for known upgrade issues.
• Interoperability Matrix results.
• VCF compatibility and hardware guidance.
• VCF Upgrade Planning Tool output.
• Environment-specific assumptions.
• Vendor or partner recommendations.

This stage should produce a source pack attached to the change record.

Do not rely on memory for version-sensitive claims. VCF 9.1 sequencing, Aria Operations patch-level behavior, licensing, management services, and supported upgrade paths can change.

Runbook Stage 1: Current-State Inventory

The inventory stage answers the question:

“What are we actually upgrading?”

Document the following:

Component Area	What to Capture
VCF baseline	Current VCF version, SDDC Manager version, workload domain list.
vSphere	vCenter versions, ESXi versions, cluster lifecycle model, vSAN state, distributed switch versions.
NSX	NSX Manager version, transport nodes, TEP health, edge clusters, routing, federation if applicable.
Operations	Aria Operations version, VCF Operations state, policies, dashboards, collectors, adapters, alerts.
Management services	Existing fleet management state, identity services, licensing services, log management services.
External dependencies	DNS, IPAM, NTP, PKI, identity provider, backup platform, syslog, monitoring integrations.
Workload dependencies	Critical workloads, application owners, smoke tests, maintenance constraints.

This inventory should be reviewed by the owners, not only exported by the platform team.

A version report is useful. A validated dependency map is better.

Runbook Stage 2: VCF Operations Readiness

VCF Operations is the first major readiness gate.

Before core components move forward, the runbook should verify:

• whether Aria Operations is already deployed,
• the exact Aria Operations version and patch level,
• whether the environment must first move to Aria Operations 8.18,
• whether the exact 8.18.x patch level has known upgrade-path caveats,
• whether VCF Operations 9.1 will be upgraded in place or deployed,
• whether Cloud Proxy placement is required,
• whether collectors and integrations are healthy,
• whether dashboards, alerts, policies, and custom content are documented,
• whether post-upgrade observability acceptance criteria are defined.

The output of this stage should be an operations readiness record.

That record should not simply say “monitoring healthy.”

It should say what was checked, who checked it, what changed, what remains at risk, and who accepts the operational state.

Runbook Stage 3: SDDC Manager and Management Services Readiness

After VCF Operations readiness is complete, the runbook moves into SDDC Manager and VCF Management Services.

This stage needs more than SDDC Manager prechecks.

It should validate:

Dependency	Readiness Check
SDDC Manager	Backup evidence, precheck output, upgrade bundle availability, administrative access, certificate/password health.
VCF Management Services	IP range plan, management network reachability, DNS requirements, internal network overlap review.
License services	License server deployment plan, A/PTR records, connectivity, post-upgrade license validation.
DNS/IPAM	Reserved addresses, FQDNs, reverse lookup, change timing, owner availability.
Security	Firewall paths, certificate trust, privileged access, audit requirements.
Support	Support entitlement, case process, log bundle procedure, escalation owner.

This is the stage where many upgrade windows get derailed.

The platform workflow may be ready, but DNS, IPAM, routing, firewall, or licensing may not be. The runbook should expose those blockers before the window.

Runbook Stage 4: Core Platform Upgrade

The core platform stage includes the remaining management-domain components such as NSX, vCenter, ESXi, and NSX Edge.

The exact sequence must follow the supported Broadcom guidance for the environment’s starting version. The runbook should not flatten this into “upgrade the core stack.”

Break it into explicit work packages:

Work Package	Required Validation
NSX upgrade	NSX backup, manager health, edge health, transport node health, routing validation, federation checks if applicable.
vCenter upgrade	Backup evidence, temporary IP requirements, RDU or in-place decision, service health, plugin compatibility.
ESXi remediation	Cluster lifecycle model, image readiness, evacuation capacity, vSAN health, host maintenance behavior.
NSX Edge finalization	Edge version alignment, routing checks, north/south connectivity, service impact validation.
Management-domain acceptance	SDDC Manager health, VCF Operations visibility, license state, monitoring state, backup resumption.

The key principle is simple:

A component upgrade is not complete until its control plane, data plane, and operational visibility are validated.

Runbook Stage 5: Validation

Validation should be layered.

A green platform dashboard is useful, but it is not enough.

Platform Validation

Confirm the infrastructure state:

• SDDC Manager reports expected versions and health.
• VCF Operations is reachable and collecting expected data.
• VCF Management Services are healthy.
• NSX managers are healthy.
• NSX Edge nodes are healthy.
• vCenter services are running.
• ESXi hosts are connected and remediated.
• vSAN and storage health are clean.
• Backup jobs resume successfully.
• License assignment is valid.

Operations Validation

Confirm the platform can still be operated:

• Alerts are firing where expected.
• Dashboards and policies are intact or intentionally replaced.
• Logs are flowing to the expected destination.
• Admin access works through the expected identity path.
• Support bundle generation is understood.
• Certificate and password workflows are documented.
• Lifecycle views show the expected state.
• Service desk routing is ready for post-upgrade noise.

Workload Validation

Confirm business impact is acceptable:

• Representative VMs remain available.
• Critical applications pass smoke tests.
• North/south connectivity works.
• East/west connectivity works.
• Backup and restore confidence checks complete where applicable.
• Monitoring accurately reflects workload state.
• Application owners provide acceptance.

This is the difference between technical completion and operational acceptance.

Runbook Stage 6: Rollback and Fallback

The rollback plan should not pretend the whole platform has one undo button.

For VCF 9.1, use the term fallback deliberately.

In some phases, fallback may mean restoring a specific appliance. In other phases, it may mean stopping the sequence, stabilizing the environment, collecting logs, opening a support case, and waiting for a supported recovery path.

Define fallback by phase.

Phase	Fallback Question
Before VCF Operations upgrade	Can we restore Aria Operations state, content, and integrations if the upgrade fails?
Before SDDC Manager upgrade	Do we have a valid SDDC Manager backup and documented recovery procedure?
Before VCF Management Services deployment	Can DNS, IPAM, firewall, or deployment changes be backed out cleanly if deployment fails?
Before NSX upgrade	Do we have NSX backups, edge validation, routing state, and a supported recovery plan?
Before vCenter upgrade	Do we have vCenter backup evidence, temporary IP planning, and a supported RDU or upgrade recovery path?
Before ESXi remediation	Can workloads evacuate safely, and are cluster capacity/admission constraints understood?
Before workload-domain upgrades	Are application owners available for validation and incident response?

The runbook should also define stop conditions.

Examples:

• VCF Operations upgrade fails and observability cannot be trusted.
• SDDC Manager precheck fails with unresolved blockers.
• VCF Management Services deployment cannot complete due to DNS/IP/network issues.
• NSX manager health is degraded before upgrade.
• NSX Edge routing validation fails.
• vCenter upgrade fails or services do not stabilize.
• ESXi remediation creates cluster capacity or vSAN health risk.
• Application validation fails for a critical service.

The important decision is not whether the team wants to continue.

The important decision is whether the team has evidence that continuing is safer than stopping.

Runbook Stage 7: Day-N Handoff

The upgrade is not finished when the first window closes.

The Day-N handoff should include:

Handoff Item	Owner
Remaining workload-domain upgrades	VCF platform owner
VCF Operations cleanup	Operations / monitoring owner
License validation and cleanup	Licensing / platform owner
Identity transition tasks	Security / identity owner
Log management plan	Operations / logging owner
Documentation updates	Platform architecture owner
Application validation follow-up	Application owners
Known issues and support cases	Change owner
Next maintenance windows	Platform owner and change manager

This stage prevents the common problem where the project celebrates the management-domain upgrade but leaves the operating team with undocumented changes, half-finished cleanup, and unclear ownership.

VCF 9.1 Readiness Scorecard

Use a scorecard before the change advisory meeting.

Do not proceed with red items in:

• VCF Operations readiness,
• supported path validation,
• DNS/IP/network readiness,
• backup/recovery evidence,
• or stop/go decision ownership.

Those are not cosmetic gaps. They are upgrade control failures.

Common Gotchas to Put in the Runbook

The following items should be explicit in the runbook, not left to memory.

Conclusion: A Good Runbook Controls the Upgrade, Not Just the Tasks

A VCF 9.1 upgrade runbook should do more than list steps.

It should control the upgrade.

That means every major phase has:

• an owner,
• a readiness gate,
• evidence,
• validation,
• fallback criteria,
• and a handoff path.

VCF 9.1 makes this especially important because VCF Operations, management services, licensing, NSX, vCenter, ESXi, and Day-N workload-domain planning are all connected.

A patch-window checklist may get the team into the upgrade.

A real runbook is what gets the platform back into a supportable operating state.

External References

• VMware Cloud Foundation Blog: How to Upgrade to VMware Cloud Foundation 9.1
  https://blogs.vmware.com/cloud-foundation/2026/06/18/how-to-upgrade-to-vmware-cloud-foundation-9-1/
• Broadcom KB 440630: Upgrade Sequence and Related Issues for VMware Cloud Foundation and vSphere Foundation 9.1
  https://knowledge.broadcom.com/external/article/440630/upgrade-sequence-and-related-issues-for.html
• Broadcom KB 430778: Unable to Upgrade Aria Operations 8.18.6 to VCF Operations 9.0, 9.0.1, or 9.0.2 from Aria Suite Lifecycle Manager
  https://knowledge.broadcom.com/external/article/430778/unable-to-upgrade-aria-operations-8186-t.html
• VMware Cloud Foundation 9.1 Release Notes
  https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-1/release-notes/vmware-cloud-foundation-9-1-0-0-release-notes.html
• VCF 9.1 Interactive Upgrade Planning Tool
  https://williamlam.com/2026/05/vcf-9-1-interactive-upgrade-planning-tool.html

The post Building a VCF 9.1 Upgrade Runbook: Testing, Fallback, and Readiness Gates appeared first on Digital Thought Disruption.