8.2.10 Release Notes

Fixed an issue where during a gateway software upgrade, traffic matching DCF WebGroup rules may be briefly dropped during the upgrade. This impacts both Layer 7 (HTTP/HTTPS) and Layer 4 traffic and occurs across all supported cloud providers (AWS, Azure, and GCP). The disruption typically lasts a few seconds but may vary depending on gateway load and policy complexity.

Fixed an issue where OpenVPN clients could not connect to VPN gateways configured with DUO multi-factor authentication (MFA) in Oracle Cloud Infrastructure (OCI) environments due to ECONNREFUSED errors during tunnel establishment.

Fixed an issue where spoke-to-transit gateway attachment could fail with a check_task_status decode error, preventing successful attachment completion.

Fixed an issue where when upgrading the Controller from version 8.0.30 to 8.1.10, the UI may display a misleading "Service temporarily unavailable" error message immediately after the upgrade begins. This message can persist for 5–10 minutes but does not indicate upgrade failure. The upgrade continues normally in the background and the Controller becomes accessible again once the upgrade finishes.

Fixed an issue where enabling Distributed Cloud Firewall Site-to-Cloud (DCF S2C) in large-scale deployments with 1300+ gateways caused gateway configurations to become out of sync with the Controller and elevated Controller CPU utilization.

Fixed an issue where Azure Controller Security Group management operations could fail, preventing proper network security group rule updates.

Fixed an issue where duplicate resource ID entries could be created in the database when a Controller experienced out-of-memory conditions followed by upsizing and restart, preventing the Controller from starting properly. The database now handles resource IDs correctly to avoid duplicates.

Fixed an issue where FireNet deployment with bootstrap configuration failed in Google Cloud due to a change in how the credential JSON file is read.

Fixed an issue with Controller access control ICMP handling that could cause incorrect ICMP traffic behavior on Controllers running version 8.0 and 8.1.

Fixed an issue in some environments where the Aviatrix Controller failed to fetch and update a rotated SAML signing certificate from the configured Identity Provider (IdP) metadata URL, which caused SAML single sign-on (SSO) authentication failures. The Controller now correctly retrieves and applies updated SAML certificates after IdP rotation.

Resolved an issue where upgrading to Controller 8.1 failed during database migration if VPC tunnel records contained non-numeric values in the timestamp field. The migration logic now correctly handles timestamp values, preventing conversion errors and allowing the upgrade to complete successfully.

Fixed an issue where the VRRP state file became empty on AEP edge gateways configured in active-active HA pairs after upgrading gateway software from version 7.2 to 8.0.30. The VRRP state file now correctly retains primary/backup state information after upgrades.

Resolved an issue where incorrect eBPF filters could be applied to the eth1 interface on Azure gateways with Accelerated Networking enabled during upgrades from versions earlier than 7.2.2994. The upgrade process now properly handles interface filter configuration to prevent unintended traffic drops.

Fixed an issue where upgrading the Controller to version 8.1 could fail during database migration when the tunnel rtt_avg field contained None values. The migration logic now correctly handles None values, allowing the upgrade to complete successfully.

Fixed an issue where ICMP traffic passing through Suricata inspection on gateways only triggered alert rules once until the Suricata process restarted. Alert rules now fire consistently for all matching traffic.

Fixed an issue where decrypted POST traffic passing through the ATS tee plugin could cause PSF gateway crashes during request body processing. The tee plugin now correctly handles POST request bodies without crashing.

Fixed an issue where the packet mark eBPF program was not loaded on some gateways, potentially causing incorrect traffic classification.

Fixed an issue where in Aviatrix software versions 8.1.x and 8.2.0, the VRRP state file /etc/localgateway/vrrp_state.json, may be empty on AEP and self-managed Edge-as-Spoke gateways configured in active-active HA pairs. This prevents VRRP state updates from being sent from the edge gateways to the Aviatrix Controller, and Aviatrix CoPilot will not display the updated VRRP states. This is a cosmetic issue and there will be no disruption to traffic.

Fixed an issue where upgrading OpenVPN gateways to Controller version 8.1 with profiles containing FQDN-based policies resulted in service disruption due to a DNS resolution limitation. Users can now access whitelisted FQDNs in OpenVPN Profiles after upgrading.

Fixed an issue where gateways included in custom syslog profiles were removed from those profiles after a gateway image upgrade, causing syslog forwarding configurations to be lost.

Fixed an issue where the database migration during Controller upgrade to 8.1 could fail if a High Availability Gateway (HAGW) entry appeared before its corresponding primary gateway in the vpc_info database collection. The migration logic now correctly handles HAGW records regardless of document ordering in the database.

Fixed an issue where the avx-gw-state-sync service leaked D-Bus connections over time, which could lead to resource exhaustion and degraded gateway state synchronization.

Fixed an issue where upgrading the Controller from version 8.0.x to 8.1.x could fail with the error "Please reload the page in order to upgrade" due to a database migration issue with incorrectly typed field values. The migration logic now includes proper type checking to handle these records.

Fixed an issue where during upgrades from older Aviatrix Controller versions to versions 8.1.20 and 8.2.0, the NetflowMode migration may fail due to incomplete NetflowMode records in the database. Older Controller versions allowed NetflowMode records with missing or empty fields, such as an empty port value. Newer releases enforce stricter validation and fail when encountering these incomplete records.

Fixed an issue where avx-nfq processes on FQDN-enabled gateways were killed and restarted during gateway software upgrade, causing a traffic outage of approximately 10 minutes after the upgrade completed. The upgrade process now correctly handles the nfq service transition without extended traffic interruption.

Fixed an issue where the VPC name field could be overwritten with incorrect data during the AM4.0 migration, causing affected VPC records to become unfindable via index lookups. The migration logic now correctly preserves the VPC name field.

Fixed two issues with single IP HA tunnel failover in Site-to-Cloud deployments. The Controller no longer sends stale IPsec session teardown messages to gateways that temporarily lost connectivity, and now stops failover of pending tunnels to the HA gateway if the active gateway reconnects quickly, reducing unnecessary tunnel downtime.

Fixed an issue where duplicate iptables mangle table MARK rules could remain on gateways during mapped Site-to-Cloud tunnel failover, gateway image upgrade, or rollback scenarios.

Fixed an issue where CoPilot deployments and migrations could fail with "Unsupported instance size" errors for valid instance types. Instance type validation no longer incorrectly blocks supported sizes.

Fixed an issue where the Controller did not properly handle BGP route updates during transit gateway failover, causing stale routes to persist in the routing table. The Controller now correctly clears and repropagates routes after failover completes.

Fixed an issue where Aviatrix HPE gateway (including HA gateway) creation failed in OCI VCNs with DNS disabled. Gateways can now be created regardless of VCN DNS configuration.

Fixed an issue where performing a Controller backup restore could cause a temporary traffic outage of approximately 40 seconds due to all routes being deleted and re-added during the etcd route reconvergence process. Routes are now preserved during backup restore to prevent traffic disruption.

Fixed an issue where the database migration timeout during Controller upgrade was hard-coded at 15 minutes, causing upgrades to fail and roll back in large-scale deployments with thousands of gateways and tunnels. The migration timeout is now user-configurable.

Fixed an issue where controller software upgrade from version 8.0.40/8.0.50 to 8.1.20 may cause Controller CPU utilization to spike due to a schema migration being skipped during the upgrade. This can result in sluggish Controller UI performance.

Fixed a memory leak in the TrafficServer (ATS) process on gateways with DCF intrusion analysis and decryption enabled under high-concurrency traffic conditions that could cause the ATS process to crash and enter a restart loop.

Fixed an issue where tunnel status report processing on the Controller took longer after upgrading from version 8.0 to 8.1 due to increased database query overhead, with average processing time increasing from approximately 150ms to 230ms per report.

Fixed an issue where FQDN gateway data was not correctly displayed after upgrading from version 7.2.x to 8.0 or later, causing the Egress FQDN Gateway View to appear empty. Gateways with FQDN tags now display correctly in the UI and are returned properly by the list_fqdn_gateways API.

Azure gateway image upgrades may fail when the Controller does not have the required Azure image subscription access. During the upgrade, the system deletes the existing gateway before validating subscription availability, which can result in gateway deletion without a replacement being created. This leaves dangling gateways in the Controller and can cause potential service outages.

Impact:

Workaround:

None. To avoid outages, ensure the Controller subscription includes access to the required Azure image before attempting upgrades.

When upgrading from Controller version 7.1 to 7.2 or 8.0, Spoke Gateways with routing through a Public Subnet Filtering (PSF) Gateway may fail to upgrade and become unreachable if the PSF Gateway has not been upgraded first. This issue affects AWS environments where Spoke Gateway route tables are configured to point to a PSF Gateway.

To avoid this issue, follow the correct upgrade sequence:

In some scenarios involving rapid VRRP state transitions, the keepalived VRRP state may not be reported accurately to the Controller. This can result in temporary discrepancies between the actual VRRP status and what is displayed in the Controller UI, leading to confusion and difficulties during troubleshooting.

Impact:

Workaround:

In some environments, the Firewall state may not recover from Unaccessible after the first vendor integration failure. This issue has been observed when integrating with third-party firewall vendors, leaving the gateway firewall state stuck even after the environment stabilizes.

Impact:

Firewall integration appears stuck in Unaccessible state

Recovery does not occur automatically after initial failure

May require manual intervention to restore proper firewall state reporting

Workaround:

Contact Aviatrix Support for manual correction.

When editing the HPE spoke-transit peering tunnel count via API on gateways with customized SNAT configured, a traffic outage of approximately 100-120 seconds per gateway may occur. During the tunnel count change, the primary spoke gateway deletes and recreates tunnels, causing the transit gateway to temporarily remove the metric 100 route for the primary SNAT IP. The HA spoke gateway loses the metric 200 route for the primary SNAT IP during this period, leading to a traffic blackhole until tunnels are fully recreated.

Affected Scenario:

Impact:

Workaround:

Schedule tunnel count changes during a maintenance window when traffic disruption is acceptable. Avoid editing tunnel counts during peak traffic periods.

Transit gateways with large-scale tunnel deployments (1300+ tunnels) may experience extended traffic loss during image upgrades. Although the image upgrade completes successfully, traffic may remain down for several minutes afterward due to delayed tunnel reconfiguration.

Impact:

Workaround:

Dry-run validation may fail when upgrading the Controller from version 8.0.10 to 8.1.0 due to a gateway version mismatch error. This occurs when the upgrade path starts from 8.0.0, progresses to 8.0.10 successfully, but encounters a dry-run failure when proceeding to 8.1.0.

Users may receive a RequestRefused error when attempting to log on to the Controller UI under certain conditions.

Impact:

Workaround:

Retry the login attempt. If the issue persists, contact Aviatrix Support for assistance.

Traffic loss may occur through AEP Edge-as-Spoke gateways during a gateway software upgrade.

Impact:

Workaround:

Schedule gateway software upgrades during maintenance windows. Contact Aviatrix Support for assistance.

When attaching VPN users to profiles using the attach_vpn_user_to_profile API, the CoPilot or Controller UI may continue to display the user profile as N/A even though the attachment operation completes successfully.

Impact:

VPN user profile assignments may appear unsuccessful in the UI, which can cause confusion during profile management. There is no functional impact: the VPN profile is correctly assigned in the backend, and users can connect to the VPN as expected.

Workaround:

None.

The migration dry-run EIP accounting does not include public IPs that are not part of the Elastic IP quota, potentially producing inaccurate dry-run results.

Impact:

Workaround:

Manually verify EIP allocation and quotas before performing the migration. Contact Aviatrix Support for assistance.

When DPI/IDS or Layer7 policies are configured with "Destination: Anywhere" on HA-enabled spoke gateways where the destination smart group contains private CIDRs, the policies become invalid and cause traffic drops.

Impact:

All egress traffic matching the policy rules gets dropped

Network connectivity loss for affected traffic flows

Workaround:

Modify the policy destination from "Anywhere" to specific target destinations that exclude conflicting private CIDR ranges, or disable HA on the affected spoke gateway if operationally acceptable.

DCF smart group push configuration may fail with a context deadline exceeded error, causing DFW configuration to not be applied correctly to gateways.

Impact:

Workaround:

Contact Aviatrix Support for assistance.

When clients use HTTP/2 connections, TrafficServer incorrectly reuses origin connections, potentially causing connection handling issues in MITM SNI verification scenarios.

Impact:

Workaround:

Use both IP address and SNI instead of IP alone to ensure proper connection isolation.

Layer 7 (L7) traffic may be dropped when a DCF policy cannot be enforced due to gateway sizing constraints. The system blocks traffic that matches policies it cannot enforce rather than allowing it to pass uninspected.

Impact:

Workaround:

Ensure gateways meet the minimum sizing requirements for L7 policy enforcement. Contact Aviatrix Support for assistance.

The CoPilot UI may not accurately reflect real-time Controller migration progress, potentially showing stale or incomplete status information.

Impact:

Workaround:

Monitor migration progress through Controller logs or API. Contact Aviatrix Support for assistance.

Additional Distributed Cloud Firewall (DCF) log support records end-session events for IDS and IPS signature matches. These logs include a reason field indicating the match type (IPS_POLICY_DENY or IDS_POLICY_ALERT) along with the Signature ID (SID) of the matched rule.

Due to a bug, the end-session log is omitted when Decryption is not enabled for Intrusion Analysis.

Impact:

Workaround: Enable Decryption under Intrusion Analysis to ensure end-session logs are generated.

The traffic_ctl process may consume 100% CPU, potentially causing traffic processing disruptions on affected gateways.

Impact:

Workaround:

Contact Aviatrix Support for assistance.

Source IP translation may not work correctly for policy-based Site-to-Cloud connections with SNAT configuration when traffic is sent from cloud to site.

Impact:

Workaround:

Contact Aviatrix Support for assistance.

The traffic_ctl process may consume 100% CPU, potentially causing traffic processing disruptions on affected gateways.

Impact:

Workaround:

Contact Aviatrix Support for assistance.

Source IP translation may not work correctly for policy-based Site-to-Cloud connections with SNAT configuration when traffic is sent from cloud to site.

Impact:

Workaround:

Contact Aviatrix Support for assistance.

When upgrading gateways from version 8.1.20 to 8.2.0 in rare cases, the gateway could enter an infinite retry loop attempting to download a non-existent configuration file from the Controller, causing the upgrade process to fail completely.

Impact:

Workaround: Retry the gateway upgrade operation from the Controller UI or CoPilot UI. If the issue persists, perform an image upgrade of the impacted gateway.

Azure HPE Transit HA Gateway resize may fail when there is insufficient secondary IP address space available in the subnet.

Impact:

Workaround:

Ensure sufficient IP address space is available in the gateway subnet before attempting a resize. Contact Aviatrix Support for assistance.

Azure HPE Transit HA Gateway resize may fail when there is insufficient secondary IP address space available in the subnet.

Impact:

Workaround:

Ensure sufficient IP address space is available in the gateway subnet before attempting a resize. Contact Aviatrix Support for assistance.

When CoPilot Asset Inventory (CAI) performs queries on the inventory table, the existing database indexes are not utilized effectively, causing performance degradation during inventory operations.

Impact:

Workaround: None.

When performing a batch gateway image upgrade in Azure, some gateways may fail during the upgrade process.

Impact:

Workaround:

Retry the gateway image upgrade for the failed gateways individually rather than as a batch operation. Contact Aviatrix Support for assistance.

Azure Controllers using disks with IOPS less than 500 may experience performance issues. This limitation can lead to system instability and processing delays during high I/O operations.

Affected Scenario:

Impact:

Workaround:

Upgrade the Azure controller disk tier to minimum 500 IOPS through Azure portal disk configuration settings.

When performing a batch gateway image upgrade in Azure, some gateways may fail during the upgrade process.

Impact:

Workaround:

Retry the gateway image upgrade for the failed gateways individually rather than as a batch operation. Contact Aviatrix Support for assistance.

Azure Controllers using disks with IOPS less than 500 may experience performance issues. This limitation can lead to system instability and processing delays during high I/O operations.

Affected Scenario:

Impact:

Workaround:

Upgrade the Azure controller disk tier to minimum 500 IOPS through Azure portal disk configuration settings.

When deploying a load balancer–enabled VPN gateway with an overlapping VPN CIDR on Controller versions 8.0, 8.1, or 8.2, the gateway creation fails.

Impact:

Workaround:

Ensure that the VPN CIDR does not overlap with existing gateways behind the load balancer before deployment. Contact Aviatrix Support for assistance.

Enabling oslogin in GCE instance metadata could break SSH access to GCP gateways by interfering with the gateway SSH service configuration.

Workaround:

Contact Aviatrix Support for assistance.

The SAML Endpoint field is not displayed when creating a VPN user for GeoVPN configurations with a SAML-enabled gateway.

Impact:

Workaround:

Contact Aviatrix Support for assistance.

Creating a new gateway with the same name as an existing gateway may cause local files of the existing gateway to be deleted when the creation fails. The existing gateway name disappears from the Controller CLI once we get into this issue.

This can break SSH access (sshgw) for the existing gateway.

Impact:

Affected Scenario:

Workaround:

After changing rx_queue_size on a gateway interface, creating a new interface does not apply the updated rx_queue_size value.

Impact:

Workaround:

Contact Aviatrix Support for assistance.

When using the update_spoke_vpc_route_table API to onboard an Azure route table, the default route (0.0.0.0/0) is not programmed in the spoke VNET route table if the Spoke Gateway has learned the default route from any of the following sources:

In all of the above cases, although the Spoke Gateway has successfully learned and installed the default route in its own routing table, the route is not re-programmed into the associated Azure VNET route table during the onboarding operation.

Impact:

Workaround:

Manually add the default route to the Azure route table. Contact Aviatrix Support for assistance.

In some high-traffic environments using FQDN filtering, the NFQ process may stall due to a deadlock. If the signal interrupts a thread that is already executing a non-reentrant function, the signal handler may attempt to acquire the same lock, causing a deadlock.

Impact:

Workaround:

Restart the instance to continue processing traffic.

DUO-integrated OpenVPN users may experience intermittent connection failures to VPN gateways due to a deprecated DUO client version.

Impact:

Workaround:

Contact Aviatrix Support for assistance with updating the DUO client configuration.

Users are unable to modify tags on third-party firewall instances when those tags contain values with multiple colons (for example, team:iac:module.version:v1.5.3). Attempts to update tags after deployment fail with a too many values to unpack error. Initial deployment is unaffected because tags are passed via a different code path during creation.

Impact:

Workaround:

Avoid using multiple colons in tag values when modifying tags after deployment. Use alternative delimiters such as hyphens or underscores.

Traffic loss may occur during gateway software upgrade from version 8.1.30 to 8.2.10.

Impact:

Workaround:

Schedule gateway upgrades during maintenance windows and monitor traffic after upgrade completion. Contact Aviatrix Support for assistance.

Traffic loss may occur when gateways are upgraded from version 8.1.30 to 8.2.10.

Impact:

Workaround:

Schedule gateway upgrades during maintenance windows. Upgrade gateways one at a time in HA pairs to minimize impact. Contact Aviatrix Support for assistance.

Due to a temporary Oracle Cloud Infrastructure (OCI) Marketplace partner agreement issue, new OCI gateway images cannot be published for this release. OCI gateway pointers have been reverted to previous versions to ensure stability. As a result, OCI gateways will not receive patched images as part of this release.

Impact:

Workaround:

Contact Aviatrix Support for assistance with OCI gateway image updates.

In Azure environments, when Distributed Cloud Firewall (DCF) Security Group Orchestration attaches a Network Security Group (NSG) to a subnet, the subnet name is changed to all lowercase. Although Azure resource names are generally case-insensitive, this modification causes issues with infrastructure-as-code tools such as Terraform, which treat resource names as case-sensitive. Terraform may flag affected subnets for replacement, potentially disrupting existing deployments.

Impact:

Workaround:

In Terraform, add a lifecycle block with ignore_changes for subnet_id (as well as id in the azurerm_subnet resource) to prevent forced resource replacement. Note that this workaround does not restore the original subnet name casing.

In Azure environments, when a custom IAM policy blocks resource creation with a RequestDisallowedByPolicy error, the Aviatrix Controller unnecessarily retries the operation instead of failing immediately. Since this error requires manual policy changes to resolve, the repeated retries congest the Controller’s event handler, causing new gateway deployments to be delayed in reaching an operational state and spoke-to-transit attachments to fail.

Impact:

Workaround:

Update the Azure IAM policy to allow the required permissions for Aviatrix Controller resource operations before deploying new gateways.

When using Terraform to create gateways and immediately attach them to transit gateways, a race condition can occur where the Terraform provider attempts to create the spoke-to-transit attachment before the gateway has fully transitioned to the "Up" state. The gateway creation API returns before the gateway is operationally ready, causing subsequent attachment operations to fail.

Impact:

Workaround:

Add a delay or polling mechanism in Terraform configurations between gateway creation and spoke-to-transit attachment resources. Use depends_on with a time_sleep resource to allow the gateway to reach the "Up" state before attempting attachment.

Gateway launch may fail with a tls: bad certificate error when pulling container images. The Controller’s registry TTL eviction (garbage collection) may fire while images are being downloaded to the gateway, corrupting in-flight blob transfers. The gateway’s container initialization cannot complete, and the apache-spiffe-helper service exits with code 125.

Impact:

Workaround:

Restart the avx-ctrl-appserver on the Controller to reset the registry state, then retry gateway creation. Contact Aviatrix Support for assistance.

In OCI environments, security list rules may not be restored when a spoke gateway re-joins a transit gateway after a leave operation.

Impact:

Workaround:

Manually verify and restore security list rules after re-joining. Contact Aviatrix Support for assistance.

A locking race condition between initial setup and post-upgrade actions may cause gateway configuration issues after upgrade.

Impact:

Workaround:

Contact Aviatrix Support for assistance.

Deploying more than one OpenVPN gateway behind a UDP load balancer is not supported and may fail.

Impact:

Workaround:

Use a single OpenVPN gateway behind each UDP load balancer. Contact Aviatrix Support for assistance.

On Edge-as-a-Transit (EaT) gateways with IPsec peering, if the primary and secondary IP addresses on a network interface are swapped following network events such as VM reboot, cloud provider host maintenance, or gateway replacement/re-deployment, the gateway identifies itself with an incorrect source IP address to the Aviatrix Controller. As a result, the Controller generates a new pre-shared key (PSK) keyed to the incorrect IP identifier for the IPsec tunnel pair. Since the peer gateway retains the PSK associated with the original correct IP identifier, the IKE authentication exchange (IKE_AUTH) fails with an AUTHENTICATION_FAILED error, causing IPsec peering sessions to fail to establish.

Impact:

Workaround: Contact Aviatrix Support for assistance.

During routine gateway operations (such as resize or image upgrade) in GCP global VPC environments, traffic may be blackholed due to incorrect route handling.

Impact:

Workaround:

Schedule gateway operations during maintenance windows. Contact Aviatrix Support for assistance.

After a Controller restart or recovery, gateways that were temporarily unreachable may not be given sufficient time to reconnect before being marked as permanently down.

Impact:

Workaround:

None.