8.1.20 Release Notes

Fixed an issue where the traffic count displayed in the Controller interface could be inaccurate when using Distributed Cloud Firewall (DCF) with external groups that include multiple IP ranges.

The Controller now reports traffic statistics correctly for DCF rules involving external groups, providing accurate visibility for monitoring, analysis, and validation of firewall policy behavior.

Resolved an issue where Spoke-to-Transit attachments initiated through Terraform could fail with a decode check_task_status failed error.

The Controller now returns a valid response consistently, preventing Terraform failures during Spoke-to-Transit attachment operations.

Resolved an issue where Edge gateway upgrades from version 8.1 to 8.1.10 could cause temporary traffic disruption due to service restarts during the upgrade process.

The upgrade workflow now handles service restarts more effectively, reducing traffic impact during Edge gateway upgrades, including in large-scale deployments.

Resolved an issue on Azure Controllers where Controller Security Group Management could cause gateway deployments to fail.

The issue occurred when security group rules were not updated correctly while creating multiple gateways or HA gateways, which led to duplicate rule errors and prevented additional gateways from being deployed.

Resolved an issue where the ESTABLISHED rule disappeared after a Public Subnet Filtering (PSF) gateway image upgrade.

This issue affected PSF gateways using the legacy stateful firewall on Controller versions 7.1 and later, and could result in traffic disruption after the upgrade. The rule is now preserved during PSF gateway image upgrades.

Fixed an issue with database schema type definitions that could trigger migration errors during the Controller upgrade process.

The schema now uses the correct database type definition, ensuring compatibility with migration logic and preventing upgrade failures.

Fixed an issue where FireNet deployments with bootstrap enabled could fail in Google Cloud due to changes in how GCP credentials were handled during the bootstrap process.

The bootstrap workflow has been updated to correctly retrieve and use GCP credentials, ensuring FireNet deployments with bootstrap complete successfully in Google Cloud environments.

Fixed an issue where the default access control rules did not properly allow ICMP traffic used for debugging. The updated rules ensure ICMP-based troubleshooting continues to work after upgrades.

Resolved an issue where the VRRP state file could become empty when upgrading AEP and self-managed Edge-as-Spoke gateways in active-active HA configurations from version 7.2 to 8.0.30.

With this fix, VRRP primary and backup state information is preserved during the upgrade, and newly created Edge-as-Spoke gateways with VRRP configuration no longer remain in the Initializing state.

Resolved an issue where eBPF packet marking could fail on transit gateways with Network Segmentation enabled, causing traffic to be associated with incorrect network domains.

The packet marking logic has been corrected to ensure Network Segmentation policies are enforced consistently without requiring service restarts.

Azure gateway image upgrades may fail when the Controller does not have the required Azure image subscription access. During the upgrade, the system deletes the existing gateway before validating subscription availability, which can result in gateway deletion without a replacement being created. This leaves dangling gateways in the Controller and can cause potential service outages.

Impact:

Workaround:

None. To avoid outages, ensure the Controller subscription includes access to the required Azure image before attempting upgrades.

When upgrading from Controller version 7.1 to 7.2 or 8.0, Spoke Gateways with routing through a Public Subnet Filtering (PSF) Gateway may fail to upgrade and become unreachable if the PSF Gateway has not been upgraded first. This issue affects AWS environments where Spoke Gateway route tables are configured to point to a PSF Gateway.

To avoid this issue, follow the correct upgrade sequence:

During a gateway software upgrade, traffic matching DCF WebGroup rules may be briefly dropped during the upgrade. This impacts both Layer 7 (HTTP/HTTPS) and Layer 4 traffic and occurs across all supported cloud providers (AWS, Azure, and GCP). The disruption typically lasts a few seconds but may vary depending on gateway load and policy complexity.

Workaround:

None

Recommendations:

In some scenarios involving rapid VRRP state transitions, the keepalived VRRP state may not be reported accurately to the Controller. This can result in temporary discrepancies between the actual VRRP status and what is displayed in the Controller UI, leading to confusion and difficulties during troubleshooting.

Impact:

Workaround:

In some environments, the Firewall state may not recover from Unaccessible after the first vendor integration failure. This issue has been observed when integrating with third-party firewall vendors, leaving the gateway firewall state stuck even after the environment stabilizes.

Impact:

Workaround:

Contact Aviatrix Support for manual correction.

Transit gateways with large-scale tunnel deployments (1300+ tunnels) may experience extended traffic loss during image upgrades. Although the image upgrade completes successfully, traffic may remain down for several minutes afterward due to delayed tunnel reconfiguration.

Impact:

Workaround:

In Oracle Cloud Infrastructure (OCI) environments, OpenVPN clients cannot connect to VPN gateways configured with DUO multi-factor authentication (MFA). Connection attempts fail with ECONNREFUSED errors during tunnel establishment, preventing authentication from completing.

Impact:

Workaround:

No current workaround. Users may temporarily switch to OKTA or LDAP authentication if feasible.

When upgrading the Controller from version 8.0.30 to 8.1.10, the UI may display a misleading "Service temporarily unavailable" error message immediately after the upgrade begins. This message can persist for 5–10 minutes but does not indicate upgrade failure. The upgrade continues normally in the background and the Controller becomes accessible again once the upgrade finishes.

Impact:

Workaround:

In large-scale deployments with 1300+ gateways, enabling Distributed Cloud Firewall Site-to-Cloud (DCF S2C) can cause gateway configurations to become out of sync with the Controller. Even after disabling DCF S2C, the issue may persist and lead to elevated Controller resource usage.

Impact:

Workaround:

When attaching VPN users to profiles using the attach_vpn_user_to_profile API, the CoPilot or Controller UI may continue to display the user profile as N/A even though the attachment operation completes successfully.

In some cases, users later reappear as active but still show no profile association in the UI. This results in a display inconsistency between the UI and the backend state.

Impact: VPN user profile assignments may appear unsuccessful in the UI, which can cause confusion during profile management. There is no functional impact: the VPN profile is correctly assigned in the backend, and users can connect to the VPN as expected.

Affected Scenario: OpenVPN profile management operations that use API-based user-to-profile attachment.

Workaround: None.

When a Controller experiences out-of-memory conditions followed by upsizing and restart, duplicate resource ID entries may be created in the database. This prevents the Controller from starting properly and blocks access to the web UI.

Impact:

Affected Scenario: Controllers that have experienced memory issues, been upsized, and restarted may encounter this database corruption.

Workaround: Connect directly to the Controller database and manually remove the duplicate resource ID entries to restore normal operation.

When DPI/IDS or Layer7 policies are configured with "Destination: Anywhere" on HA-enabled spoke gateways where the destination smart group contains private CIDRs, the policies become invalid and cause traffic drops.

Affected Scenario: Spoke gateways with HA enabled using DPI/IDS or Layer7 policies that have destination smart groups containing private CIDR ranges and "Destination: Anywhere" configuration.

Impact:

Workaround: Modify the policy destination from "Anywhere" to specific target destinations that exclude conflicting private CIDR ranges, or disable HA on the affected spoke gateway if operationally acceptable.

When clients use HTTP/2 protocol, Trafficserver incorrectly reuses origin connections, which can cause SSL/TLS verification issues and potential security concerns with SNI (Server Name Indication) handling.

Affected Scenario: HTTP/2 client connections through Trafficserver proxy

Impact:

Workaround: Configure records.yaml to match on both IP address and SNI to ensure proper connection handling.

When a gateway is downsized in environments with IPS (Intrusion Prevention System) enabled, L7 traffic (HTTP/HTTPS) is dropped instead of being allowed through. The system blocks traffic when it detects that security policies cannot be properly enforced due to insufficient gateway resources, preventing the traffic-server from running.

Affected Scenario: Gateways with IPS enabled that undergo downsizing operations.

Impact:

In some environments, after the Identity Provider (IdP) rotates its SAML signing certificate, the Aviatrix Controller may fail to fetch and update the new certificate from the configured metadata URL.

As a result, the Controller continues to use a stale certificate, which causes signature verification errors during SAML authentication.

Impact: SAML single sign-on (SSO) authentication fails. Users may experience repeated login failures or timeouts and are unable to access the Controller dashboard using SAML.

Workaround: Contact Aviatrix Support to manually update the SAML certificate on the Controller.

On Azure Aviatrix gateways with accelerated networking enabled and using Distributed Cloud Firewall (DCF) features, intermittent traffic drops may occur after upgrading from a version earlier than 7.2.2994 to 7.2.2994 or later.

This issue is caused by incorrect eBPF filters being applied to the slave eth1 interface during the upgrade process.

Affected Scenario:

Contact Aviatrix Support for assistance.

When ICMP traffic passes through Suricata inspection on gateways, alert rules trigger only once until the Suricata process restarts. This limitation affects the eBPF → proxyPcap → Suricata traffic path and likely impacts UDP and other non-TCP protocols as well.

Impact:

Workaround: Contact Aviatrix Support to restore the alert functionality for ICMP traffic.

When processing decrypted POST traffic through the ATS tee plugin, PSF gateways may experience crashes during request body processing. This occurs specifically with decrypted traffic that contains POST requests being processed by the tee plugin’s request body handling path.

Impact:

Affected Scenario: PSF gateways processing decrypted POST traffic through ATS tee plugin

Workaround: Avoid routing decrypted POST traffic through affected PSF gateways until the fix is implemented. Consider using alternative routing paths or temporarily disabling tee plugin functionality for POST request processing if operationally feasible.

When deploying a load balancer–enabled VPN gateway with an overlapping VPN CIDR on Controller versions 8.0, 8.1, or 8.2, the gateway creation fails.

Impact:

Affected Scenario: Load balancer–enabled VPN gateway deployments on Controller versions 8.0, 8.1, and 8.2.

Workaround:

Ensure that the VPN CIDR does not overlap with existing gateways behind the load balancer before deployment. Contact Aviatrix Support for assistance.

In Aviatrix software versions 8.1.x and 8.2.0, the VRRP state file /etc/localgateway/vrrp_state.json, may be empty on AEP and self-managed Edge-as-Spoke gateways configured in active-active HA pairs. This prevents VRRP state updates from being sent from the edge gateways to the Aviatrix Controller, and Aviatrix CoPilot will not display the updated VRRP states. This is a cosmetic issue and there will be no disruption to traffic.

Impact:

Affected Scenario: - AEP and self-managed Edge-as-Spoke gateways in active-active HA deployments with VRRP enabled upgrade from 8.0 to 8.1 or created in 8.1 and 8.2

Workaround: Please contact Aviatrix Support for assistance to help you fix the incorrect display of VRRP states in Aviatrix CoPilot.

When upgrading OpenVPN gateways to Controller version 8.1 (including 8.1.11 and 8.1.20), where profiles that contain FQDN-based policies may result in service disruption due to a DNS resolution limitation.

Affected Scenario:

Impact:

Workaround:

When multiple syslog profiles are configured, performing a gateway image upgrade results in gateways being removed from syslog profiles that use subset include lists. After the upgrade, remote syslog logs are no longer forwarded for the affected gateways.

Impact:

Gateways are removed from syslog profiles with subset include lists after image upgrade.

+ Remote syslog log forwarding stops for affected gateways.

+ Manual re-addition of gateways to syslog profiles is required.

Workaround:

After performing a gateway image upgrade, manually re-add the affected gateways to their syslog profiles to restore log forwarding.

Controller software upgrade to version 8.1 may fail with the error "Please reload the page in order to upgrade" due to an issue with the AM4.0 database migration. The migration fails when the database contains string values instead of integers for the cloud_type field in certain records.

Impact:

Controller upgrade to 8.1 may fail to complete.

+ Controller remains on the previous version until the issue is resolved.

Workaround:

Contact Aviatrix Support for assistance with correcting the database values before retrying the upgrade.

Creating a new gateway with the same name as an existing gateway may cause local files of the existing gateway to be deleted when the creation fails. The existing gateway name disappears from the Controller CLI once we get into this issue.

This can break SSH access (sshgw) for the existing gateway.

Impact:

Affected Scenario:

Workaround:

In environments where Spoke Gateways are configured with customized SNAT policies pointing to Transit Gateway VTI interfaces, upgrading Controller to version 8.1.20 may result in loss of the default route. This issue is not limited to Transit FireNet with Egress through Firewall setups, it can also occur when a default route is learned over BGP by the Transit Gateway and propagated to Spoke Gateways, as long as customized SNAT is in use.

This issue may remove the default route in the spoke gateway pointing to the Transit Gateway tunnels, resulting in traffic egressing through unintended interfaces.

Affected Scenario:

Spoke gateways configured with customized SNAT policies pointing to Transit Gateway VTI interfaces, upgrading Controller to version 8.1.20

This includes but is not limited to Transit FireNet with Egress through Firewall setups and environments using BGP-learned default routes propagated from Transit to Spoke Gateways.

Impact:

Traffic may bypass the Transit Gateway, leading to traffic disruption or loss.

Workaround:

No workaround is currently available. Contact Aviatrix Support for assistance.

The Cloud Asset Inventory (CAI) service has a memory leak in its L1 cache. When cloud instances such as VMs are removed from the cloud provider, the associated network interfaces remain cached and are never cleaned up.

Impact:

Workaround:

Contact Aviatrix Support for assistance with periodic CAI service restarts to reclaim memory.

Agent certificate renewal may fail for 8.0.x gateways managed by 8.1.x Controllers due to differences in SPIRE versions.

Controller 8.1.x uses SPIRE v1.12.0, while 8.0.x gateways use SPIRE v1.0.1. In mixed-version deployments, 8.0.x gateways may fail to renew the agent_SVID certificate.

As a result, users may begin receiving repeated PKI agent expiry alert emails indicating that the agent certificate could not be renewed.

Impact:

Controller is running 8.1.x * Gateway is running 8.0.x Workaround:

When using the update_spoke_vpc_route_table API to onboard an Azure route table, the default route (0.0.0.0/0) is not programmed in the spoke VNET route table if the Spoke Gateway has learned the default route from any of the following sources: an attached Egress Transit Gateway (Transit with egress functionality enabled); a Transit Gateway that learned the default route via an external Site-to-Cloud (S2C) connection (for example, from an on-premises network or third-party appliance advertising 0.0.0.0/0 over IPSec/BGP); another Spoke Gateway that is propagating the default route within the Aviatrix network. In all of the above cases, although the Spoke Gateway has successfully learned and installed the default route in its own routing table, the route is not re-programmed into the associated Azure VNET route table during the onboarding operation.

Impact:

Workaround:

Manually add the default route to the Azure route table. Contact Aviatrix Support for assistance.

In some high-traffic environments using FQDN filtering, the NFQ process may stall due to a deadlock. If the signal interrupts a thread that is already executing a non-reentrant function, the signal handler may attempt to acquire the same lock, causing a deadlock.

Impact:

The avx-nfq process may stall and stop processing traffic until the service is restarted.

Workaround:

Restart the instance to continue processing traffic.

When upgrading the Controller from version 8.0 to 8.1, the AM4.0 database migration may overwrite the VPC name field with incorrect data if old VPC records contain a pre-existing 'name' key with the gateway name. This causes affected VPC records to become unfindable via index lookups, potentially impacting Controller operations that reference those VPCs.

Impact:

VPC records may become unfindable after Controller upgrade.

+ Controller operations referencing affected VPCs may fail.

+ Any cloud service provider can be affected.

Workaround:

Contact Aviatrix Support for assistance.

In some environments running 8.0.30 or later builds, duplicate iptables mangle table MARK rules may remain on gateways during mapped Site-to-Cloud tunnel failover, gateway image upgrade, or rollback scenarios.

These rules may accumulate due to incomplete cleanup during tunnel role transitions.

Impact:

No traffic impact has been observed. The issue only affects residual rule cleanup on the gateway.

Workaround:

No workaround is required. The extra rules do not affect traffic forwarding.

For Controllers with large-scale deployments (for example, several thousand gateways and tunnels), the database migration during upgrade can exceed the current hard-coded 15-minute timeout, causing the Controller upgrade to fail and roll back. The migration timeout is not configurable in affected versions, so customers with very large environments are more likely to encounter this issue during Controller upgrades.

Impact:

Contact Aviatrix Support for assistance with adjusting the migration timeout for large-scale deployments.

On Edge-as-a-Transit (EaT) gateways with HPE peering to transit using many-to-one IP addressing (multiple source private IPs peering to a single transit IP), the tunnel status monitoring job may raise an exception due to duplicate tunnel ping IP pairs. This causes the tunnel status report to fail periodically.

Impact:

Workaround:

None. This is a monitoring-only issue with no traffic impact.

Controller software upgrade from version 8.0.40/8.0.50 to 8.1.20 may cause Controller CPU utilization to spike due to a schema migration being skipped during the upgrade. This can result in sluggish Controller UI performance.

Impact:

Workaround:

Contact Aviatrix Support for assistance with applying the workaround to fix the skipped schema migration.

After upgrading the Aviatrix Controller from version 7.2.x to 8.0 or later, gateways with FQDN tags attached may no longer be visible in the Egress FQDN Gateway View tab. The list_fqdn_gateways API returns an empty list despite the gateways being present and properly associated with their FQDN tags.

Impact:

Contact Aviatrix Support for assistance.

After a Controller restart or recovery, gateways that were temporarily unreachable may not be given sufficient time to reconnect before being marked as permanently down.

Impact:

Workaround:

None.

On Controller and gateway running 8.1.x or 8.2.x, editing legacy FQDN domain name filters can cause all FQDN filtering processes on the gateway to stop simultaneously. Gateway monitoring restarts the processes automatically, but a brief filtering outage may occur during the restart.

Impact:

Schedule edits to legacy FQDN filters during a maintenance window. Contact Aviatrix Support for assistance.