8.1.40 Release Notes

Fixed an issue where the firewall state did not recover from the Unaccessible state after the first vendor integration failure.

Fixed an issue where performing a Controller backup restore could cause a temporary traffic outage of approximately 40 seconds due to all routes being deleted and re-added during the etcd route reconvergence process. Routes are now preserved during backup restore to prevent traffic disruption.

Fixed an issue where the database migration timeout during Controller upgrade was hard-coded at 15 minutes, causing upgrades to fail and roll back in large-scale deployments with thousands of gateways and tunnels. The migration timeout is now user-configurable.

Fixed an issue where tunnel status report processing on the Controller was slower after upgrading from version 8.0 to 8.1 due to increased database query overhead. The database queries have been optimized to restore processing performance.

Fixed an issue where FQDN gateway data was not correctly displayed after upgrading from version 7.2.x to 8.0 or later, causing the Egress FQDN Gateway View to appear empty. Gateways with FQDN tags now display correctly in the UI and are returned properly by the list_fqdn_gateways API.

Fixed an issue where a background service responsible for collecting network topology data experienced unbounded memory growth, which could eventually cause the process to be terminated due to excessive memory consumption.

Fixed an issue where a Controller upgrade could fail during a database migration step if a corrupted VPC record was present in the Controller database, causing the upgrade to fail and trigger a rollback.

Fixed an issue where the Aviatrix Controller retried Azure RequestDisallowedByPolicy errors unnecessarily, causing event handler congestion and delaying gateway deployments.

Fixed a locking race condition during Controller upgrade where the initial setup process and post-upgrade actions could acquire conflicting locks, causing post-upgrade configuration steps to fail or require manual intervention.

Fixed an issue in GCP global VPC environments where the Controller removed all gateway routes during routine gateway operations such as resize or image upgrade, causing extended traffic blackholing.

Fixed an issue where Controller restore operations could stall for approximately 30 minutes during the scheduler shutdown phase due to connectivity issues with AWS API endpoints or invalid/expired IAM credentials.

Fixed an issue where the gateway state synchronization service (avx-gw-state-sync) could crash if the gateway’s DNS configuration was not yet fully initialized, temporarily interrupting state synchronization between the Controller and gateways.

Fixed an issue where editing legacy FQDN domain name filters on a gateway could cause all FQDN filtering processes to stop simultaneously, resulting in a brief filtering outage until the processes were automatically restarted.

Azure gateway image upgrades may fail when the Controller does not have the required Azure image subscription access. During the upgrade, the system deletes the existing gateway before validating subscription availability, which can result in gateway deletion without a replacement being created. This leaves dangling gateways in the Controller and can cause potential service outages.

Impact:

Existing gateways may be deleted during image upgrade

Replacement gateway creation fails due to missing subscription

Customers may experience connectivity loss and dangling gateway entries in the Controller

Manual intervention required, leading to support escalations

Workaround:

None. To avoid outages, ensure the Controller subscription includes access to the required Azure image before attempting upgrades.

When upgrading from Controller version 7.1 to 7.2 or 8.0, Spoke Gateways with routing through a Public Subnet Filtering (PSF) Gateway may fail to upgrade and become unreachable if the PSF Gateway has not been upgraded first. This issue affects AWS environments where Spoke Gateway route tables are configured to point to a PSF Gateway.

To avoid this issue, follow the correct upgrade sequence:1. Upgrade the PSF Gateway first.

During a gateway software upgrade, traffic matching DCF WebGroup rules may be briefly dropped during the upgrade. This impacts both Layer 7 (HTTP/HTTPS) and Layer 4 traffic and occurs across all supported cloud providers (AWS, Azure, and GCP). The disruption typically lasts a few seconds but may vary depending on gateway load and policy complexity.

Workaround:

None

Recommendations:

Schedule gateway upgrades during maintenance windows or low-traffic periods.

Use HA deployments and upgrade gateways one at a time in HA pairs.

Monitor logs for "Failed to load policy" messages to confirm when policies are reloaded.

In some scenarios involving rapid VRRP state transitions, the keepalived VRRP state may not be reported accurately to the Controller. This can result in temporary discrepancies between the actual VRRP status and what is displayed in the Controller UI, leading to confusion and difficulties during troubleshooting.

Impact:

Controller UI may show incorrect VRRP status such as both peers reporting Primary or Initializing

No impact on actual VRRP traffic handling or failover behavior.

Workaround:

Use diagnostic logs to verify actual VRRP state.

Transit gateways with large-scale tunnel deployments (1300+ tunnels) may experience extended traffic loss during image upgrades. Although the image upgrade completes successfully, traffic may remain down for several minutes afterward due to delayed tunnel reconfiguration.

Workaround:

Impact:

Dry-run validation may fail when upgrading the Controller from version 8.0.10 to 8.1.0 due to a gateway version mismatch error. This occurs when the upgrade path starts from 8.0.0, progresses to 8.0.10 successfully, but encounters a dry-run failure when proceeding to 8.1.0.

In some environments running Aviatrix Controller version 8.1.x, the Controller UI may become inaccessible after the Controller is restarted. When this issue occurs, API requests fail with RequestRefused errors and the UI cannot be accessed, although SSH access to the Controller remains available.

Impact:

Users are unable to log in to the Controller UI or perform API operations.

Workaround:

Restart the VM where the Aviatrix Controller is running.

When upgrading the Controller from version 8.0.30 to 8.1.10, the UI may display a misleading "Service temporarily unavailable" error message immediately after the upgrade begins. This message can persist for 5–10 minutes but does not indicate upgrade failure. The upgrade continues normally in the background and the Controller becomes accessible again once the upgrade finishes.

Impact:

Users may believe the upgrade has failed.

Error message persists for 5–10 minutes, especially in larger deployments (50+ gateways).

No effect on upgrade success or Controller functionality.

Workaround:

Ignore the message during upgrade.

Wait 10–15 minutes for the process to complete.

Refresh the browser and verify the new Controller version after reconnection.

In large-scale deployments with 1300+ gateways, enabling Distributed Cloud Firewall Site-to-Cloud (DCF S2C) can cause gateway configurations to become out of sync with the Controller. Even after disabling DCF S2C, the issue may persist and lead to elevated Controller resource usage.

Impact:

Gateway configurations may show as out of sync in the Controller UI

Controller CPU utilization (conduit process) increases significantly

Performance degradation may occur during DCF S2C operations

Issue may persist after disabling DCF S2C

Workaround:

Monitor Controller CPU usage before enabling DCF S2C in large-scale environments.

Consider enabling DCF S2C during scheduled maintenance windows.

For deployments with 1300+ gateways, evaluate the necessity of DCF S2C functionality.

When attaching VPN users to profiles using the attach_vpn_user_to_profile API, the CoPilot or Controller UI may continue to display the user profile as N/A even though the attachment operation completes successfully.

In some cases, users later reappear as active but still show no profile association in the UI. This results in a display inconsistency between the UI and the backend state.

Impact: VPN user profile assignments may appear unsuccessful in the UI, which can cause confusion during profile management. There is no functional impact: the VPN profile is correctly assigned in the backend, and users can connect to the VPN as expected.

Affected Scenario: OpenVPN profile management operations that use API-based user-to-profile attachment.

Workaround: None.

The migration dry-run EIP accounting does not include public IPs that are not part of the Elastic IP quota, potentially producing inaccurate dry-run results.

Impact:

Workaround:

Manually verify EIP allocation and quotas before performing the migration. Contact Aviatrix Support for assistance.

When deploying a load balancer–enabled VPN gateway with an overlapping VPN CIDR on Controller versions 8.0, 8.1, or 8.2, the gateway creation fails.

Impact:

Affected Scenario: Load balancer–enabled VPN gateway deployments on Controller versions 8.0, 8.1, and 8.2.

Workaround:

Ensure that the VPN CIDR does not overlap with existing gateways behind the load balancer before deployment. Contact Aviatrix Support for assistance.

Creating a new gateway with the same name as an existing gateway may cause local files of the existing gateway to be deleted when the creation fails. The existing gateway name disappears from the Controller CLI once we get into this issue.

This can break SSH access (sshgw) for the existing gateway.

Impact:

Affected Scenario:

Workaround:

When using the update_spoke_vpc_route_table API to onboard an Azure route table, the default route (0.0.0.0/0) is not programmed in the spoke VNET route table if the Spoke Gateway has learned the default route from any of the following sources:

In all of the above cases, although the Spoke Gateway has successfully learned and installed the default route in its own routing table, the route is not re-programmed into the associated Azure VNET route table during the onboarding operation.

Impact:

Workaround:

Manually add the default route to the Azure route table. Contact Aviatrix Support for assistance.

In some high-traffic environments using FQDN filtering, the NFQ process may stall due to a deadlock. If the signal interrupts a thread that is already executing a non-reentrant function, the signal handler may attempt to acquire the same lock, causing a deadlock.

Impact:

The avx-nfq process may stall and stop processing traffic until the service is restarted.

Workaround:

Restart the instance to continue processing traffic.

In environments where Duo Authentication is enabled for Client VPN, Duo-authenticated users may intermittently fail to connect to the VPN Gateway.

The gateway may log the following error message:

Duo OpenVPN: Received 403 Client duo_openvpn version 2.4 is deprecated and no longer supported

This occurs because the gateway uses an older Duo OpenVPN client library version that is no longer supported by the Duo service.

Impact:

Users configured with Duo authentication may fail to establish VPN connections. In some cases, bypass users may connect intermittently.

Workaround:

Update the Duo OpenVPN client version on the gateway by modifying the version in the duo_openvpn.py file from 2.4 to 3.0.

CoPilot deployments and migrations may fail with "Unsupported instance size" errors when selecting valid instance types. The instance type validation incorrectly blocks supported sizes during CoPilot deployment or migration operations.

Impact:

CoPilot deployment or migration may fail when selecting certain valid instance types

Error message "Unsupported instance size" is displayed even for supported sizes

Workaround:

Contact Aviatrix Support for assistance with CoPilot deployment using the affected instance types.

Users are unable to modify tags on third-party firewall instances when those tags contain values with multiple colons (for example, team:iac:module.version:v1.5.3). Attempts to update tags after deployment fail with a too many values to unpack error. Initial deployment is unaffected because tags are passed via a different code path during creation.

Impact:

Third-party firewall instance tag updates fail when tag values contain multiple colons

Initial deployment with multi-colon tags is not affected

Workaround:

Avoid using multiple colons in tag values when modifying tags after deployment. Use alternative delimiters such as hyphens or underscores.

Gateway launch may fail with a tls: bad certificate error when pulling container images. The Controller’s registry TTL eviction (garbage collection) may fire while images are being downloaded to the gateway, corrupting in-flight blob transfers. The gateway’s container initialization cannot complete, and the apache-spiffe-helper service exits with code 125.

Impact:

Restart the avx-ctrl-appserver on the Controller to reset the registry state, then retry gateway creation. Contact Aviatrix Support for assistance.

Unable to configure more than one OpenVPN gateway behind a UDP Load Balancer. Only the first gateway is retained while additional gateways are incorrectly excluded. This issue affects versions 6.9, 7.x, 8.x, and 9.0.0.

Impact:

Workaround:

Contact Aviatrix Support for assistance in applying the workaround.

Description:

When the avx-ctrl-state-sync service starts up and finds gateways stored as "Down" in etcd, the controller reprograms the network as if those gateways are down, without allowing time for the gateways to connect and prove they are up. This can occur when the controller has previously lost connectivity to gateways (for example, during a transient network issue) and the service then restarts while the gateways themselves remain healthy and continue forwarding traffic.

Impact:

A temporary dataplane disruption occurs while the controller has the gateways marked as down. The controller automatically detects the gateways are up and reprograms the network correctly immediately afterward. In a large-scale user deployment (~1,400 gateways), full recovery completed in approximately 6 minutes. The underlying behavior has existed for 4+ years and has been observed in a user environment only once.

Workaround:

Not applicable. The system recovers automatically, no user action required. Recovery typically completes within minutes (~6 min observed in a ~1,400-gateway deployment; smaller deployments recover faster).

When the Use SSL to connect option is enabled for LDAP configuration, VPN gateway configuration updates may not apply correctly.

Impact:

VPN gateway authentication using LDAP with TLS may not function as expected

Manual intervention may be required to ensure VPN authentication is properly configured

Workaround:

Contact Aviatrix Support for assistance.

Running Controller diagnostics may report NTP-related errors that do not reflect the actual NTP synchronization status of the Controller.

Impact:

Controller diagnostics output may show NTP errors even when time synchronization is functioning correctly.

These spurious diagnostic messages may cause unnecessary concern when reviewing Controller health output.

Workaround:

Verify Controller time synchronization through other means before acting on NTP-related diagnostic errors. Contact Aviatrix Support for assistance.