8.0.60 Release Notes

Fixed an issue where performing a Controller backup restore could cause a temporary traffic outage of approximately 40 seconds due to all routes being deleted and re-added during the etcd route reconvergence process. Routes are now preserved during backup restore to prevent traffic disruption.

Fixed an issue where Edge-as-a-Transit (EaT) gateways with HPE peering to transit using many-to-one IP addressing could fail to send tunnel status reports due to duplicate tunnel ping IP pairs causing an exception. The tunnel status monitoring logic now correctly handles duplicate ping address pairs.

Fixed an issue where gateways with FQDN tags attached were not visible in the Egress FQDN Gateway View tab after upgrading the Controller from version 7.2.x to 8.0 or later. The FQDN gateway data migration has been corrected to properly display gateways in the UI and API.

Fixed an issue where certain license types were unable to enable Distributed Cloud Firewall (DCF) on Controller versions 8.0.10 through 8.0.50. A license validation check that was removed in 8.1+ had not been backported to the 8.0.x branch, causing a controller error when users with affected license types attempted to enable DCF.

Fixed an issue where the database migration timeout during Controller upgrade was hard-coded at 15 minutes, causing upgrades to fail and roll back in large-scale deployments with thousands of gateways and tunnels. The migration timeout is now user-configurable.

Azure gateway image upgrades may fail when the Controller does not have the required Azure image subscription access. During the upgrade, the system deletes the existing gateway before validating subscription availability, which can result in gateway deletion without a replacement being created. This leaves dangling gateways in the Controller and can cause potential service outages.

Impact: Existing gateways may be deleted during image upgrade. Replacement gateway creation fails due to missing subscription. Customers may experience connectivity loss and dangling gateway entries in the Controller. Manual intervention required, leading to support escalations.

Workaround: None. To avoid outages, ensure the Controller subscription includes access to the required Azure image before attempting upgrades.

When upgrading from Controller version 7.1 to 7.2 or 8.0, Spoke Gateways with routing through a Public Subnet Filtering (PSF) Gateway may fail to upgrade and become unreachable if the PSF Gateway has not been upgraded first. This issue affects AWS environments where Spoke Gateway route tables are configured to point to a PSF Gateway.

To avoid this issue, follow the correct upgrade sequence: . Upgrade the PSF Gateway first. . Wait for the PSF Gateway upgrade to complete successfully. . Then upgrade the dependent Spoke Gateways.

During a gateway software upgrade, traffic matching DCF WebGroup rules may be briefly dropped during the upgrade. This impacts both Layer 7 (HTTP/HTTPS) and Layer 4 traffic and occurs across all supported cloud providers (AWS, Azure, and GCP). The disruption typically lasts a few seconds but may vary depending on gateway load and policy complexity.

Workaround: None

Recommendations: - Schedule upgrades during maintenance windows or low-traffic periods. - Use HA deployments and upgrade gateways one at a time in HA pairs. - Monitor logs for policy reload failures.

In Controller release 8.0, gateway software upgrades take longer to complete compared to earlier versions. On average, the upgrade rate drops from approximately 14 gateways per minute in version 7.2 to approximately 11 gateways per minute in 8.0, which is an increase of about 20% in execution time.

Affected Scenarios: Upgrading from version 7.2.x to 8.0.x, Upgrading between 8.0.x versions

Impact: Only the upgrade duration is affected. Gateway functionality remains unaffected after a successful upgrade.

Recommendations: - Allocate approximately 20% more time for gateway upgrades. - For large environments (e.g., 1,000+ gateways), plan for 90–120 minutes of upgrade time. - Schedule upgrades during maintenance windows to accommodate the longer duration.

In some scenarios involving rapid VRRP state transitions, the keepalived VRRP state may not be reported accurately to the Controller. This can result in temporary discrepancies between the actual VRRP status and what is displayed in the Controller UI, leading to confusion and difficulties during troubleshooting.

Impact: - Controller UI may show incorrect VRRP status such as both peers reporting Primary or Initializing - No impact on actual VRRP traffic handling or failover behavior

Workaround: Use diagnostic logs to verify actual VRRP state.

In some environments, the Firewall state may not recover from Unaccessible after the first vendor integration failure. This issue has been observed when integrating with third-party firewall vendors, leaving the gateway firewall state stuck even after the environment stabilizes.

Impact: - Firewall integration appears stuck in Unaccessible state - Recovery does not occur automatically after initial failure - May require manual intervention to restore proper firewall state reporting

Workaround: Contact Aviatrix Support for manual correction.

When editing HPE spoke-transit peering tunnel count via API while Customized SNAT is configured, approximately 100–120 seconds of traffic outage occurs per gateway. During tunnel re-creation on the primary spoke, the transit gateway falls back to HA routes, but the HA spoke gateway loses the metric 200 route for the primary SNAT IP, causing traffic to black-hole. The same issue recurs when the HA gateway subsequently re-creates its tunnels.

Impact: Traffic outage of 100–120 seconds per gateway during tunnel count changes. Total outage of 3–4 minutes for affected VPC workloads. Consistently reproducible.

Affected Configuration: HPE spoke-transit peering with Customized SNAT configured; tunnel count modification via API.

Workaround: Schedule tunnel count changes during maintenance windows. Avoid modifying tunnel counts on HPE spoke gateways with active Customized SNAT rules during production hours.

Transit gateways with large-scale tunnel deployments (1300+ tunnels) may experience extended traffic loss during image upgrades. Although the image upgrade completes successfully, traffic may remain down for several minutes afterward due to delayed tunnel reconfiguration.

Impact: - Traffic loss may persist after image upgrade completes - Route service startup is blocked until all tunnels are sequentially reconfigured - Configuration push may time out with "Context cancelled during Phase 1 Create" error

Workaround: - Schedule maintenance windows to account for potential traffic loss beyond upgrade completion - Consider staggering upgrades across transit gateways to reduce impact - Monitor tunnel and route service status post-upgrade through the CoPilot UI

When processing high logging volumes, DCF triggers rsyslogd rate-limiting, dropping messages that exceed 500 per 5-second interval.

Affected Scenario: High-traffic environments with DCF enabled.

Impact: - Log message drops during high-volume periods - Potential gaps in audit trails - Reduced monitoring visibility

Workaround: Monitor rsyslogd logs and implement log aggregation strategies across multiple collection points.

Dry-run validation may fail when upgrading the Controller from version 8.0.10 to 8.1.0 due to a gateway version mismatch error. This occurs when the upgrade path starts from 8.0.0, progresses to 8.0.10 successfully, but encounters a dry-run failure when proceeding to 8.1.0.

OpenVPN clients cannot connect to VPN gateways on OCI configured with DUO multi-factor authentication, failing with ECONNREFUSED errors. Only affects OCI deployments with DUO MFA; other authentication methods work normally.

Workaround: Switch to OKTA or LDAP authentication if feasible.

When upgrading the Controller from version 8.0.30 to 8.1.10, the UI may display a misleading "Service temporarily unavailable" error message immediately after the upgrade begins. This message can persist for 5–10 minutes but does not indicate upgrade failure. The upgrade continues normally in the background and the Controller becomes accessible again once the upgrade finishes.

Impact: Users may believe the upgrade has failed. Error message persists for 5–10 minutes, especially in larger deployments (50+ gateways). No effect on upgrade success or Controller functionality.

Workaround: Ignore the message during upgrade. Wait 10–15 minutes for the process to complete. Refresh the browser and verify the new Controller version after reconnection.

In large-scale deployments with 1300+ gateways, enabling Distributed Cloud Firewall Site-to-Cloud (DCF S2C) can cause gateway configurations to become out of sync with the Controller. Even after disabling DCF S2C, the issue may persist and lead to elevated Controller resource usage.

Impact: Gateway configurations may show as out of sync in the Controller UI. Controller CPU utilization (conduit process) increases significantly. Performance degradation may occur during DCF S2C operations. Issue may persist after disabling DCF S2C.

Workaround: Monitor Controller CPU usage before enabling DCF S2C in large-scale environments. Consider enabling DCF S2C during scheduled maintenance windows. For deployments with 1300+ gateways, evaluate the necessity of DCF S2C functionality.

When attaching VPN users to profiles via API, CoPilot or Controller UI may display user profile as "N/A" despite successful attachment completion. VPN profile is correctly assigned in backend.

Impact: Display inconsistency only; no functional impact. Users connect as expected.

Workaround: None.

After Controller experiences out-of-memory conditions followed by upsizing and restart, duplicate resource ID entries may be created in the database, preventing Controller startup and blocking web UI access.

Impact: Controller fails to start. Web UI inaccessible.

Workaround: Manually remove duplicate resource ID entries from the Controller database.

The Controller does not properly handle withdrawn route advertisements, causing stale routes to remain in the routing table.

Impact: - Stale routes may persist after route withdrawal - Routing table may contain outdated entries

Workaround: Contact Aviatrix Support for assistance with manual route cleanup.

HA-enabled spoke gateways with DPI/IDS or Layer 7 policies using "Destination: Anywhere" with private CIDR smart groups become invalid, causing traffic drops.

Impact: Egress traffic matching policy rules gets dropped. Network connectivity loss for affected flows.

Workaround: Modify destination from Anywhere to specific targets excluding conflicting CIDRs, or disable HA.

TrafficServer incorrectly reuses origin connections with HTTP/2 clients in MITM SNI verification scenarios.

Impact: Inappropriate connection sharing. Improper MITM SNI verification.

Workaround: Configure records.yaml to match both IP address and SNI instead of IP alone.

Gateway downsizing with IPS enabled causes L7 traffic (HTTP/HTTPS) to be dropped instead of allowed through.

Impact: - Complete HTTP/HTTPS traffic blockage - DCF error notifications - Service disruption for affected flows

Workaround: Resize gateway back to adequate specifications supporting IPS functionality, or temporarily disable L7 policies.

After Identity Provider rotates SAML signing certificate, Aviatrix Controller may fail to fetch the updated certificate from the metadata URL, causing signature verification errors.

Impact: SAML authentication fails. Users unable to access Controller via SAML login.

Workaround: Contact Aviatrix Support for manual SAML certificate update.

Gateway software upgrade from 7.2 to 8.0.30 causes VRRP state file to become empty on AEP edge gateways in active-active HA pairs.

Impact: Loss of VRRP state information. Potential HA failover disruption.

Workaround: Reconfigure VRRP settings post-upgrade to repopulate the state file.

DCF log support for end-session events does not emit IDS match logs when Decryption is not enabled for Intrusion Analysis.

Impact: - End-session IDS match logs are not generated when decryption is disabled - Reduced visibility into intrusion analysis events

Workaround: Enable decryption for Intrusion Analysis to receive complete end-session logs.

Controller processes inventory data by inserting new entries instead of updating existing records, causing database bloat with millions of redundant rows.

Impact: Database storage consumption increases over time. Query performance degrades.

Workaround: Monitor database size and perform periodic cleanup of old inventory entries during maintenance windows.

Azure controllers using the default P6 disk tier (240 IOPS) may experience performance degradation, particularly with 8.x containerized controllers.

Impact: - Controller performance degradation on Azure with default disk tier - I/O bottlenecks during high-load operations

Workaround: Manually upgrade the Azure Controller disk tier to P10 (500 IOPS) or higher.

Gateways included in custom syslog profiles may be removed from those profiles after a gateway image upgrade, causing syslog forwarding configurations to be lost.

Impact: - Syslog forwarding stops after image upgrade - Custom syslog profile associations are lost

Workaround: Re-add gateways to custom syslog profiles after image upgrade.

The database migration may fail if the HA gateway entry appears before the primary gateway entry in the database collection. This can occur during Controller upgrade to 8.0 or later versions.

Impact: - Controller upgrade may fail during database migration - Migration must be corrected before upgrade can proceed

Workaround: Contact Aviatrix Support for assistance.

The database migration may fail during Controller upgrade when cloud_type field values are stored as an unexpected type instead of an integer.

Impact: - Controller upgrade may fail during database migration - Migration must be corrected before upgrade can proceed

Workaround: Contact Aviatrix Support for assistance.

During upgrades from older Aviatrix Controller versions to versions 8.1.x, NetflowMode migration may fail due to incomplete NetflowMode records stored in databases. Older Controller versions allowed NetflowMode records with missing or empty fields, such as an empty port value. In newer releases, the migration enforces stricter validation and fails when encountering these incomplete records.

Impact: - NetflowMode migration may fail during Controller upgrade - Upgrade process may be blocked until the invalid records are addressed

Workaround: Contact Aviatrix Support for assistance.

Creating a new gateway with the same name as an existing gateway may cause local files of the existing gateway to be deleted when the creation fails. The existing gateway name disappears from the Controller CLI.

Impact: - SSH access to the existing gateway may stop working - Gateway recovery may require manual intervention

Workaround: Do not reuse gateway names. Contact Aviatrix Support if recovery is required.

When using the update_spoke_vpc_route_table API to onboard an Azure route table in environments where a Spoke Gateway learns the default route (0.0.0.0/0) from an attached egress transit, the default route is not programmed in the spoke VNET route table.

Impact: - Default route (0.0.0.0/0) is not installed in the onboarded Azure spoke VNET route table - Traffic that depends on the default route through the transit egress path may not be routed correctly

Workaround: Manually add the default route to the Azure route table. Contact Aviatrix Support for assistance.

The VPC name field may be overwritten with incorrect data during the database migration, causing affected VPC records to become unfindable via index lookups.

Impact: - VPC records may not be found through normal Controller operations - Affected VPCs may appear missing in the Controller UI

Workaround: Contact Aviatrix Support for assistance.

In Site-to-Cloud deployments with single IP HA tunnels, the Controller may send stale IPsec session teardown messages to gateways that temporarily lost connectivity. Additionally, the Controller may fail to stop failover of pending tunnels to the HA gateway when the active gateway reconnects quickly, causing unnecessary tunnel downtime.

Impact: - Unnecessary tunnel teardown and re-establishment - Extended tunnel downtime during brief connectivity interruptions

Workaround: Contact Aviatrix Support for assistance.

CoPilot deployments and migrations may fail with "Unsupported instance size" errors for valid instance types.

Impact: - CoPilot deployment or migration fails for supported instance types - Users may need to select alternative instance sizes

Workaround: Try a different supported instance size for CoPilot deployment.

The Controller does not properly handle BGP route updates during transit gateway failover, causing stale routes to persist in the routing table.

Impact: - Stale BGP routes may persist after transit gateway failover - Routing inconsistencies may occur until routes are manually cleared

Workaround: Contact Aviatrix Support for assistance with route cleanup after failover.

Users are unable to modify tags on firewall instances when those tags contain values with multiple colons (for example, team:iac:module.version:v1.5.3). Attempts to update tags after deployment fail with a "too many values to unpack" error. Initial deployment is unaffected because tags are passed via a different code path during creation.

Impact: - Firewall instance tag updates fail when tag values contain multiple colons - Initial deployment with multi-colon tags is not affected

Workaround: Avoid using multiple colons in tag values when modifying tags after deployment. Use alternative delimiters such as hyphens or underscores.

A memory leak in the TrafficServer (ATS) process on gateways with DCF intrusion analysis and decryption enabled under high-concurrency traffic conditions may cause the ATS process to crash and enter a restart loop.

Impact: - ATS process may crash on gateways with DCF intrusion analysis and decryption - Traffic disruption during crash and restart cycle

Workaround: Contact Aviatrix Support to remove intrusion detection rules from the affected gateway’s DCF profile to prevent the memory leak from occurring. Upgrade to version 8.2.10 or 9.0.0 for permanent fix.

Due to a temporary Oracle Cloud Infrastructure (OCI) Marketplace partner agreement issue, new OCI gateway images cannot be published for this release. OCI gateway pointers have been reverted to previous versions to ensure stability. As a result, OCI gateways will not receive patched images as part of this release.

Impact: - OCI gateways will not receive the latest patched images in this release - Customers using OCI gateways will remain on previous image versions until the OCI Marketplace issue is resolved

Workaround: Contact Aviatrix Support for assistance with OCI gateway image updates.

In Azure environments, when Distributed Cloud Firewall (DCF) Security Group Orchestration attaches a Network Security Group (NSG) to a subnet, the subnet name is changed to all lowercase. Although Azure resource names are generally case-insensitive, this modification causes issues with infrastructure-as-code tools such as Terraform, which treat resource names as case-sensitive. Terraform may flag affected subnets for replacement, potentially disrupting existing deployments.

Impact: - Subnet names in Azure are modified to lowercase when Security Group Orchestration attaches NSGs - Terraform plans may show unexpected resource replacements for affected subnets - Customer naming conventions in the cloud may be altered without consent

Workaround: In Terraform, add a lifecycle block with ignore_changes for subnet_id and id in the azurerm_subnet resource to prevent forced resource replacement. Note that this workaround does not restore the original subnet name casing.

In Azure environments, when a custom IAM policy blocks resource creation with a RequestDisallowedByPolicy error, the Aviatrix Controller unnecessarily retries the operation instead of failing immediately. Since this error requires manual policy changes to resolve, the repeated retries congest the Controller’s event handler, causing new gateway deployments to be delayed and spoke-to-transit attachments to fail.

Impact: - Gateway deployments may be significantly delayed due to unnecessary retry attempts - Spoke-to-transit attachments may fail while gateways remain in a waiting state - Controller event handler performance may be degraded

Workaround: Update the Azure IAM policy to allow the required permissions for Aviatrix Controller resource operations before deploying new gateways.

Gateway launch may fail due to a circular dependency when the spiffe-helper-gateway container image is not pre-baked in the gateway AMI. When the bootstrap SVID (SPIFFE Verifiable Identity Document) expires before the image can be pulled, the gateway cannot authenticate to the container registry to download the required image, resulting in a launch failure.

Impact: - Gateway creation fails completely when SVID expires during image pull - Affects environments where gateway launch takes longer than the 15-minute bootstrap SVID TTL

Workaround: Restart the appserver service on the affected gateway. Contact Aviatrix Support for assistance.