8.0.70 Release Notes

Fixed a locking race condition during Controller upgrade where the initial setup process and post-upgrade actions could acquire conflicting locks, causing post-upgrade configuration steps to fail or require manual intervention.

Fixed an issue where Controller restore operations could stall for approximately 30 minutes during the scheduler shutdown phase due to connectivity issues with AWS API endpoints or invalid/expired IAM credentials.

Fixed an issue on Site-to-Cloud (S2C) gateways where customized NAT rules configured on S2C interfaces were not honored by the gateway’s NAT translator, leading to incorrect or missing address translation on traffic traversing those tunnels.

Azure gateway image upgrades may fail when the Controller does not have the required Azure image subscription access. During the upgrade, the system deletes the existing gateway before validating subscription availability, which can result in gateway deletion without a replacement being created. This leaves dangling gateways in the Controller and can cause potential service outages.

Impact:

Workaround:

None. To avoid outages, ensure the Controller subscription includes access to the required Azure image before attempting upgrades.

When upgrading from Controller version 7.1 to 7.2 or 8.0, Spoke Gateways with routing through a Public Subnet Filtering (PSF) Gateway may fail to upgrade and become unreachable if the PSF Gateway has not been upgraded first. This issue affects AWS environments where Spoke Gateway route tables are configured to point to a PSF Gateway.

To avoid this issue, follow the correct upgrade sequence:1. Upgrade the PSF Gateway first.

During a gateway software upgrade, traffic matching DCF WebGroup rules may be briefly dropped during the upgrade. This impacts both Layer 7 (HTTP/HTTPS) and Layer 4 traffic and occurs across all supported cloud providers (AWS, Azure, and GCP). The disruption typically lasts a few seconds but may vary depending on gateway load and policy complexity.

Workaround:

None

Recommendations:

In some scenarios involving rapid VRRP state transitions, the keepalived VRRP state may not be reported accurately to the Controller. This can result in temporary discrepancies between the actual VRRP status and what is displayed in the Controller UI, leading to confusion and difficulties during troubleshooting.

Impact:

Workaround:

Transit gateways with large-scale tunnel deployments (1300+ tunnels) may experience extended traffic loss during image upgrades. Although the image upgrade completes successfully, traffic may remain down for several minutes afterward due to delayed tunnel reconfiguration.

Workaround:

Impact:

Dry-run validation may fail when upgrading the Controller from version 8.0.10 to 8.1.0 due to a gateway version mismatch error. This occurs when the upgrade path starts from 8.0.0, progresses to 8.0.10 successfully, but encounters a dry-run failure when proceeding to 8.1.0.

In Oracle Cloud Infrastructure (OCI) environments, OpenVPN clients cannot connect to VPN gateways configured with DUO multi-factor authentication (MFA). Connection attempts fail with ECONNREFUSED errors during tunnel establishment, preventing authentication from completing.

Impact:

Workaround: No current workaround. Users may temporarily switch to OKTA or LDAP authentication if feasible.

In Controller version 8.0.10, Spoke-to-Transit attachments initiated through Terraform may fail with a decode check_task_status failed error during gateway creation.

This issue occurs when multiple API calls are executed in rapid succession, causing the Controller to occasionally return an empty response body for the check_task_status API.

Affected Scenario:

Impact:

Workaround:

In large-scale deployments with 1300+ gateways, enabling Distributed Cloud Firewall Site-to-Cloud (DCF S2C) can cause gateway configurations to become out of sync with the Controller. Even after disabling DCF S2C, the issue may persist and lead to elevated Controller resource usage.

Impact:

Gateway configurations may show as out of sync in the Controller UI

Controller CPU utilization (conduit process) increases significantly

Performance degradation may occur during DCF S2C operations

Issue may persist after disabling DCF S2C

Workaround:

Monitor Controller CPU usage before enabling DCF S2C in large-scale environments.

Consider enabling DCF S2C during scheduled maintenance windows.

For deployments with 1300+ gateways, evaluate the necessity of DCF S2C functionality.

On Azure Controllers with Controller Security Group Management enabled, existing gateway might go to keepalive fail status while creating a new gateway.

Due to this bug, while creating a new gateway the Controller may automatically disable Controller Security Group Management, which impacts the connectivity between gateway and controller.

Impact:

Workaround: Enable "Controller Security Group Management" manually, or contact Aviatrix Support for assistance.

When attaching VPN users to profiles using the attach_vpn_user_to_profile API, the CoPilot or Controller UI may continue to display the user profile as N/A even though the attachment operation completes successfully.

In some cases, users later reappear as active but still show no profile association in the UI. This results in a display inconsistency between the UI and the backend state.

Impact: VPN user profile assignments may appear unsuccessful in the UI, which can cause confusion during profile management. There is no functional impact: the VPN profile is correctly assigned in the backend, and users can connect to the VPN as expected.

Affected Scenario: OpenVPN profile management operations that use API-based user-to-profile attachment.

Workaround: None.

The migration dry-run EIP accounting does not include public IPs that are not part of the Elastic IP quota, potentially producing inaccurate dry-run results.

Impact:

Workaround:

Manually verify EIP allocation and quotas before performing the migration. Contact Aviatrix Support for assistance.

In some environments, after the Identity Provider (IdP) rotates its SAML signing certificate, the Aviatrix Controller may fail to fetch and update the new certificate from the configured metadata URL.

As a result, the Controller continues to use a stale certificate, which causes signature verification errors during SAML authentication.

Impact: SAML single sign-on (SSO) authentication fails. Users may experience repeated login failures or timeouts and are unable to access the Controller dashboard using SAML.

Workaround: Contact Aviatrix Support to manually update the SAML certificate on the Controller.

When upgrading to Controller 8.1, the database migration may fail if VPC tunnel records contain string values in the timestamp field instead of numeric values.

During migration, the process attempts to convert these non-numeric timestamp strings to floating-point values, which results in a conversion error and causes the upgrade to fail.

Impact: The upgrade to Controller 8.1 cannot complete and the system remains on the previous version.

Workaround: Contact Aviatrix Support for assistance.

When upgrading the Controller to version 8.1, the database migration may fail if the tunnel rtt_avg field contains None values. The migration logic expects either a numeric value or the string "N/A", and encountering a None value causes the upgrade to stop.

Impact:

Workaround: Contact Aviatrix Support for assistance in correcting the database values before retrying the upgrade.

When deploying a load balancer–enabled VPN gateway with an overlapping VPN CIDR on Controller versions 8.0, 8.1, or 8.2, the gateway creation fails.

Impact:

Affected Scenario: Load balancer–enabled VPN gateway deployments on Controller versions 8.0, 8.1, and 8.2.

Workaround:

Ensure that the VPN CIDR does not overlap with existing gateways behind the load balancer before deployment. Contact Aviatrix Support for assistance.

When upgrading OpenVPN gateways to Controller version 8.1 (including 8.1.11 and 8.1.20), where profiles that contain FQDN-based policies may result in service disruption due to a DNS resolution limitation.

Affected Scenario:

Impact:

Workaround:

When multiple syslog profiles are configured on the Controller, performing a gateway image upgrade results in gateways being removed from syslog profiles that have only a subset of gateways specified in the include list.

Impact:

Workaround:

Manually re-add the required gateways to the affected syslog profiles after the upgrade.

When upgrading to Controller 8.1, the database migration may fail if a High Availability Gateway (HAGW) entry appeared before its corresponding primary gateway in the database collection. During migration, the process relies on a strict ordering — primary gateway data must be initialized before the HAGW is processed to correctly populate gateway group data. The migration may fail or leave behind gateway entries in the wrong order in the database, which can lead to further issues.

Impact:

Workaround:

Contact Aviatrix Support for assistance.

Controller software upgrade to version 8.1 may fail with the error "Please reload the page in order to upgrade" due to an issue with the database migration when the database contains string values instead of integers for the cloud_type field.

Impact:

Workaround:

Contact Aviatrix Support for assistance in correcting the database values before retrying the upgrade.

In environments where Spoke Gateways are configured with customized SNAT policies pointing to Transit Gateway VTI interfaces, upgrading Controller to version 8.1.20 may result in loss of the default route. This issue is not limited to Transit FireNet with Egress through Firewall setups, it can also occur when a default route is learned over BGP by the Transit Gateway and propagated to Spoke Gateways, as long as customized SNAT is in use.

This issue may remove the default route in the spoke gateway pointing to the Transit Gateway tunnels, resulting in traffic egressing through unintended interfaces.

Affected Scenario: Spoke gateways configured with customized SNAT policies pointing to Transit Gateway VTI interfaces, upgrading Controller to version 8.1.20

This includes but is not limited to Transit FireNet with Egress through Firewall setups and environments using BGP-learned default routes propagated from Transit to Spoke Gateways.

Impact:

Traffic may bypass the Transit Gateway, leading to traffic disruption or loss.

Workaround:

No workaround is currently available. Contact Aviatrix Support for assistance.

The Cloud Asset Inventory (CAI) service has a memory leak in its L1 cache. When cloud instances such as VMs are removed from the cloud provider, the associated network interfaces remain cached and are never cleaned up.

Impact:

Workaround:

Contact Aviatrix Support for assistance with periodic CAI service restarts to reclaim memory.

During a Controller software upgrade from version 8.0 to 8.1, the database migration may overwrite the VPC name field with incorrect data when an old VPC record contains a pre-existing 'name' key set to the gateway name. Affected VPC records become unfindable via index lookups, which can disrupt Controller operations that reference those VPCs.

Impact:

Workaround:

Contact Aviatrix Support for assistance.

In environments where Duo Authentication is enabled for Client VPN, Duo-authenticated users may intermittently fail to connect to the VPN Gateway.

The gateway may log the following error message:

Duo OpenVPN: Received 403 Client duo_openvpn version 2.4 is deprecated and no longer supportedThis occurs because the gateway uses an older Duo OpenVPN client library version that is no longer supported by the Duo service.

Impact:

Users configured with Duo authentication may fail to establish VPN connections. In some cases, bypass users may connect intermittently.

Workaround:

Update the Duo OpenVPN client version on the gateway by modifying the version in the duo_openvpn.py file from 2.4 to 3.0.

CoPilot deployments and migrations may fail with "Unsupported instance size" errors when selecting valid instance types. The instance type validation incorrectly blocks supported sizes during CoPilot deployment or migration operations.

Impact:

CoPilot deployment or migration may fail when selecting certain valid instance types

Error message "Unsupported instance size" is displayed even for supported sizes

Workaround:

Contact Aviatrix Support for assistance with CoPilot deployment using the affected instance types.

Aviatrix HPE gateway (including HA gateway) creation failed in OCI VCNs with DNS disabled. Gateways can now be created regardless of VCN DNS configuration.

Workaround:

Contact Aviatrix Support for assistance.

Users are unable to modify tags on third-party firewall instances when those tags contain values with multiple colons (for example, team:iac:module.version:v1.5.3). Attempts to update tags after deployment fail with a too many values to unpack error. Initial deployment is unaffected because tags are passed via a different code path during creation.

Impact:

Workaround:

Avoid using multiple colons in tag values when modifying tags after deployment. Use alternative delimiters such as hyphens or underscores.

Unable to configure more than one OpenVPN gateway behind a UDP Load Balancer. Only the first gateway is retained while additional gateways are incorrectly excluded. This issue affects versions 6.9, 7.x, 8.x, and 9.0.0.

Impact:

Workaround:

Contact Aviatrix Support for assistance in applying the workaround.

Description: When the avx-ctrl-state-sync service starts up and finds gateways stored as "Down" in etcd, the controller reprograms the network as if those gateways are down, without allowing time for the gateways to connect and prove they are up. This can occur when the controller has previously lost connectivity to gateways (for example, during a transient network issue) and the service then restarts while the gateways themselves remain healthy and continue forwarding traffic.

Impact: A temporary dataplane disruption occurs while the controller has the gateways marked as down. The controller automatically detects the gateways are up and reprograms the network correctly immediately afterward. In a large-scale user deployment (~1,400 gateways), full recovery completed in approximately 6 minutes. The underlying behavior has existed for 4+ years and has been observed in a user environment only once.

Workaround: Not applicable. The system recovers automatically, no user action required. Recovery typically completes within minutes (~6 min observed in a ~1,400-gateway deployment; smaller deployments recover faster).

When the Use SSL to connect option is enabled for LDAP configuration, VPN gateway configuration updates may not apply correctly.

Impact:

VPN gateway authentication using LDAP with TLS may not function as expected

Manual intervention may be required to ensure VPN authentication is properly configured

Workaround:

Contact Aviatrix Support for assistance.