8.0.60 Release Notes

Fixed an issue where FireNet deployment with bootstrap configuration failed in Google Cloud due to a change in how the credential JSON file is read.

Fixed an issue where the default access control rules did not properly allow ICMP traffic used for debugging. The updated rules ensure ICMP-based troubleshooting continues to work after upgrades.

Fixed an issue where the VRRP state file became empty on AEP edge gateways configured in active-active HA pairs after upgrading gateway software from version 7.2 to 8.0.30. The VRRP state file now correctly retains primary/backup state information after upgrades.

Resolved an issue where incorrect eBPF filters could be applied to the eth1 interface on Azure gateways with Accelerated Networking enabled during upgrades from versions earlier than 7.2.2994. The upgrade process now properly handles interface filter configuration to prevent unintended traffic drops.

Resolved an issue where the avx-gw-state-sync service leaked D-Bus connections on gateways. The implementation has been updated to use a shared D-Bus connection that is properly managed and cleaned up, preventing connection exhaustion during gateway operations.

Fixed an issue where mapped standby tunnels had cleanup issues with mangle rules and route tables during tunnel failover operations.

Fixed an issue where the gateway state sync service required a restart prior to upgrading gateways to version 7.2 or 8.0, which could cause upgrade failures if not performed.

Fixed an issue where performing a Controller backup restore could cause a temporary traffic outage of approximately 40 seconds due to all routes being deleted and re-added during the etcd route reconvergence process. Routes are now preserved during backup restore to prevent traffic disruption.

Fixed an issue where the database migration timeout during Controller upgrade was hard-coded at 15 minutes, causing upgrades to fail and roll back in large-scale deployments with thousands of gateways and tunnels. The migration timeout is now user-configurable.

Fixed an issue where Edge-as-a-Transit (EaT) gateways with HPE peering using many-to-one IP addressing could fail to report tunnel status to the Controller due to duplicate tunnel ping IP pairs in the monitoring job.

Fixed an issue where FQDN gateway data was not correctly displayed after upgrading from version 7.2.x to 8.0 or later, causing the Egress FQDN Gateway View to appear empty. Gateways with FQDN tags now display correctly in the UI and are returned properly by the list_fqdn_gateways API.

Fixed an issue where certain license types were unable to enable Distributed Cloud Firewall (DCF) on Controller versions 8.0.10 through 8.0.50. A license validation check that was removed in 8.1+ had not been backported to the 8.0.x branch, causing a controller error when users with affected license types attempted to enable DCF.

Fixed an issue where migrating an existing Controller to a new Controller VM could temporarily disrupt datapath on gateways with NAT configured due to configuration changes being applied too early during database migration.

Azure gateway image upgrades may fail when the Controller does not have the required Azure image subscription access. During the upgrade, the system deletes the existing gateway before validating subscription availability, which can result in gateway deletion without a replacement being created. This leaves dangling gateways in the Controller and can cause potential service outages.

Impact:

Workaround:

None. To avoid outages, ensure the Controller subscription includes access to the required Azure image before attempting upgrades.

When upgrading from Controller version 7.1 to 7.2 or 8.0, Spoke Gateways with routing through a Public Subnet Filtering (PSF) Gateway may fail to upgrade and become unreachable if the PSF Gateway has not been upgraded first. This issue affects AWS environments where Spoke Gateway route tables are configured to point to a PSF Gateway.

To avoid this issue, follow the correct upgrade sequence:

During a gateway software upgrade, traffic matching DCF WebGroup rules may be briefly dropped during the upgrade. This impacts both Layer 7 (HTTP/HTTPS) and Layer 4 traffic and occurs across all supported cloud providers (AWS, Azure, and GCP). The disruption typically lasts a few seconds but may vary depending on gateway load and policy complexity.

Workaround:

None

Recommendations:

In some scenarios involving rapid VRRP state transitions, the keepalived VRRP state may not be reported accurately to the Controller. This can result in temporary discrepancies between the actual VRRP status and what is displayed in the Controller UI, leading to confusion and difficulties during troubleshooting.

Impact:

Workaround:

In some environments, the Firewall state may not recover from Unaccessible after the first vendor integration failure. This issue has been observed when integrating with third-party firewall vendors, leaving the gateway firewall state stuck even after the environment stabilizes.

Impact:

Firewall integration appears stuck in Unaccessible state

Recovery does not occur automatically after initial failure

May require manual intervention to restore proper firewall state reporting

Workaround:

Contact Aviatrix Support for manual correction.

Transit gateways with large-scale tunnel deployments (1300+ tunnels) may experience extended traffic loss during image upgrades. Although the image upgrade completes successfully, traffic may remain down for several minutes afterward due to delayed tunnel reconfiguration.

Impact:

Workaround:

Dry-run validation may fail when upgrading the Controller from version 8.0.10 to 8.1.0 due to a gateway version mismatch error. This occurs when the upgrade path starts from 8.0.0, progresses to 8.0.10 successfully, but encounters a dry-run failure when proceeding to 8.1.0.

OpenVPN clients could not connect to VPN gateways configured with DUO multi-factor authentication (MFA) in Oracle Cloud Infrastructure (OCI) environments due to ECONNREFUSED errors during tunnel establishment.

Workaround:

Contact Aviatrix Support for assistance.

In Controller version 8.0.10, Spoke-to-Transit attachments initiated through Terraform may fail with a decode check_task_status failed error during gateway creation.

This issue occurs when multiple API calls are executed in rapid succession, causing the Controller to occasionally return an empty response body for the check_task_status API.

Affected Scenario:

Impact:

Workaround:

In large-scale deployments with 1300+ gateways, enabling Distributed Cloud Firewall Site-to-Cloud (DCF S2C) can cause gateway configurations to become out of sync with the Controller. Even after disabling DCF S2C, the issue may persist and lead to elevated Controller resource usage.

Impact:

Gateway configurations may show as out of sync in the Controller UI

Controller CPU utilization (conduit process) increases significantly

Performance degradation may occur during DCF S2C operations

Issue may persist after disabling DCF S2C

Workaround:

Monitor Controller CPU usage before enabling DCF S2C in large-scale environments.

Consider enabling DCF S2C during scheduled maintenance windows.

For deployments with 1300+ gateways, evaluate the necessity of DCF S2C functionality.

Traffic loss may occur through AEP Edge-as-Spoke gateways during a gateway software upgrade.

Impact:

Workaround:

Schedule gateway software upgrades during maintenance windows. Contact Aviatrix Support for assistance.

On Azure Controllers with Controller Security Group Management enabled, existing gateway might go to keepalive fail status while creating a new gateway.

Due to this bug, while creating a new gateway the Controller may automatically disable Controller Security Group Management, which impacts the connectivity between gateway and controller.

Impact:

Workaround:

Enable "Controller Security Group Management" manually, or contact Aviatrix Support for assistance.

When attaching VPN users to profiles using the attach_vpn_user_to_profile API, the CoPilot or Controller UI may continue to display the user profile as N/A even though the attachment operation completes successfully.

Impact:

VPN user profile assignments may appear unsuccessful in the UI, which can cause confusion during profile management. There is no functional impact: the VPN profile is correctly assigned in the backend, and users can connect to the VPN as expected.

Workaround:

None.

The migration dry-run EIP accounting does not include public IPs that are not part of the Elastic IP quota, potentially producing inaccurate dry-run results.

Impact:

Workaround:

Manually verify EIP allocation and quotas before performing the migration. Contact Aviatrix Support for assistance.

The CoPilot UI may not accurately reflect real-time Controller migration progress, potentially showing stale or incomplete status information.

Impact:

Workaround:

Monitor migration progress through Controller logs or API. Contact Aviatrix Support for assistance.

The Controller failed to fetch and update new SAML signing certificates after Identity Provider (IdP) certificate rotation, causing SAML single sign-on (SSO) authentication failures.

Workaround:

Contact Aviatrix Support for assistance.

The AM4.0 database migration may fail when the tunnel database timestamp field contains string values instead of the expected integer type. This can block Controller upgrades that include the AM4.0 migration step.

Impact:

Workaround:

Contact Aviatrix Support for assistance with correcting the database field values before retrying the upgrade.

The traffic_ctl process may consume 100% CPU, potentially causing traffic processing disruptions on affected gateways.

Impact:

Workaround:

Contact Aviatrix Support for assistance.

Upgrading the Controller to version 8.1 could fail during database migration if the tunnel rtt_avg field contained None values. The migration logic now correctly handles these values, preventing conversion errors and allowing the upgrade to complete successfully.

Workaround:

Contact Aviatrix Support for assistance.

When deploying a load balancer–enabled VPN gateway with an overlapping VPN CIDR on Controller versions 8.0, 8.1, or 8.2, the gateway creation fails.

Impact:

Workaround:

Ensure that the VPN CIDR does not overlap with existing gateways behind the load balancer before deployment. Contact Aviatrix Support for assistance.

Upgrading OpenVPN gateways to Controller version 8.1 with profiles containing FQDN-based policies resulted in service disruption due to a DNS resolution limitation. Users can now access whitelisted FQDNs in OpenVPN Profiles after upgrading.

Workaround:

Contact Aviatrix Support for assistance.

When multiple syslog profiles are configured on the Controller, performing a gateway image upgrade results in gateways being removed from syslog profiles that have only a subset of gateways specified in the include list.

Impact:

Workaround:

Manually re-add the required gateways to the affected syslog profiles after the upgrade.

The SAML Endpoint field is not displayed when creating a VPN user for GeoVPN configurations with a SAML-enabled gateway.

Impact:

Workaround:

Contact Aviatrix Support for assistance.

The database migration during Controller upgrade to 8.1 could fail if a High Availability Gateway (HAGW) entry appeared before its corresponding primary gateway in the vpc_info database collection. The migration logic now correctly handles HAGW records regardless of document ordering in the database.

Workaround:

Contact Aviatrix Support for assistance.

Controller software upgrade to version 8.1 may fail with the error "Please reload the page in order to upgrade" due to an issue with the database migration when the database contains string values instead of integers for the cloud_type field.

Impact:

Workaround:

Contact Aviatrix Support for assistance in correcting the database values before retrying the upgrade.

Upgrading Spoke Gateways to version 8.1.20 in environments using Transit FireNet with Egress through Firewall and customized SNAT policies could result in loss of the default route, causing traffic disruption.

Workaround:

Contact Aviatrix Support for assistance.

The Cloud Asset Inventory (CAI) service has a memory leak in its L1 cache. When cloud instances such as VMs are removed from the cloud provider, the associated network interfaces remain cached and are never cleaned up.

Impact:

Workaround:

Contact Aviatrix Support for assistance with periodic CAI service restarts to reclaim memory.

After changing rx_queue_size on a gateway interface, creating a new interface does not apply the updated rx_queue_size value.

Impact:

Workaround:

Contact Aviatrix Support for assistance.

When using the update_spoke_vpc_route_table API to onboard an Azure route table in environments where a Spoke Gateway learns the default route (0.0.0.0/0) from an attached egress transit, the default route is not programmed in the spoke VNET route table. This affects deployments using transit with egress functionality where spoke gateways need to propagate the learned default route to their associated Azure route tables.

Impact:

Workaround:

Manually add the default route to the Azure route table. Contact Aviatrix Support for assistance.

The VPC name field could be overwritten with incorrect data during the AM4.0 migration, causing affected VPC records to become unfindable via index lookups. The migration logic now correctly preserves the VPC name field.

Workaround:

Contact Aviatrix Support for assistance.

Two issues with single IP HA tunnel failover in Site-to-Cloud deployments. The Controller no longer sends stale IPsec session teardown messages to gateways that temporarily lost connectivity, and now stops failover of pending tunnels to the HA gateway if the active gateway reconnects quickly, reducing unnecessary tunnel downtime.

Workaround:

Contact Aviatrix Support for assistance.

DUO-integrated OpenVPN users may experience intermittent connection failures to VPN gateways due to a deprecated DUO client version.

Impact:

Workaround:

Contact Aviatrix Support for assistance with updating the DUO client configuration.

CoPilot deployments and migrations may fail with "Unsupported instance size" errors when selecting valid instance types. The instance type validation incorrectly blocks supported sizes during CoPilot deployment or migration operations.

Impact:

CoPilot deployment or migration may fail when selecting certain valid instance types

Error message "Unsupported instance size" is displayed even for supported sizes

Workaround:

Contact Aviatrix Support for assistance with CoPilot deployment using the affected instance types.

Aviatrix HPE gateway (including HA gateway) creation failed in OCI VCNs with DNS disabled. Gateways can now be created regardless of VCN DNS configuration.

Workaround:

Contact Aviatrix Support for assistance.

Users are unable to modify tags on third-party firewall instances when those tags contain values with multiple colons (for example, team:iac:module.version:v1.5.3). Attempts to update tags after deployment fail with a too many values to unpack error. Initial deployment is unaffected because tags are passed via a different code path during creation.

Impact:

Workaround:

Avoid using multiple colons in tag values when modifying tags after deployment. Use alternative delimiters such as hyphens or underscores.

In OCI environments, security list rules may not be restored when a spoke gateway re-joins a transit gateway after a leave operation.

Impact:

Workaround:

Manually verify and restore security list rules after re-joining. Contact Aviatrix Support for assistance.

During a Controller upgrade, a locking race condition between the initial setup process and post-upgrade actions can cause post-upgrade configuration steps to fail or require manual intervention.

Impact:

Contact Aviatrix Support to manually complete the post-upgrade configuration.

Unable to configure more than one OpenVPN gateway behind a UDP Load Balancer. Only the first gateway is retained while additional gateways are incorrectly excluded. This issue affects versions 6.9, 7.x, 8.x, and 9.0.0.

Impact:

Workaround:

Contact Aviatrix Support for assistance in applying the workaround.

During routine gateway operations (such as resize or image upgrade) in GCP global VPC environments, traffic may be blackholed due to incorrect route handling.

Impact:

Workaround:

Schedule gateway operations during maintenance windows. Contact Aviatrix Support for assistance.

When the avx-ctrl-state-sync service starts up and finds gateways stored as "Down" in etcd, the controller reprograms the network as if those gateways are down, without allowing time for the gateways to connect and prove they are up. This can occur when the controller has previously lost connectivity to gateways (for example, during a transient network issue) and the service then restarts while the gateways themselves remain healthy and continue forwarding traffic.

Impact:

A temporary dataplane disruption occurs while the controller has the gateways marked as down. The controller automatically detects the gateways are up and reprograms the network correctly immediately afterward. In a large-scale user deployment (~1,400 gateways), full recovery completed in approximately 6 minutes. The underlying behavior has existed for 4+ years and has been observed in a user environment only once.

Workaround:

Not applicable. The system recovers automatically, no user action required. Recovery typically completes within minutes (~6 min observed in a ~1,400-gateway deployment; smaller deployments recover faster).

On Site-to-Cloud (S2C) gateways running 8.0.x, customized NAT rules configured on S2C interfaces may not be applied by the gateway’s NAT translator.

Impact:

Contact Aviatrix Support for assistance.