Migrating infrastructure from AWS to Google Cloud Platform (GCP) was a major transformation for our team at Shadowfax. While such moves promise improved performance, scalability, and cost efficiency, they also come with a learning curve. In our case, this journey was shaped by some key lessons around Network Address Translation (NAT) configuration.

In this blog, we'll take you through the issues we faced, the solutions we implemented, and the long-term optimizations that followed. Our goal is to provide practical insights to teams planning a similar cloud migration or refining their network architecture in GCP.

Port Utilization Differences Between AWS and GCP

One of the first surprises came from how NAT is handled differently in GCP compared to AWS. In AWS, NAT Gateways have strict port and IP limits, which helped us plan capacity accordingly. In GCP, however, each NAT IP can support approximately 50,000 concurrent ports, but there are no clearly defined upper limits, unlike in AWS.

During the early stages of migration, we assigned just one NAT IP to our infrastructure. We believed it would be sufficient based on AWS standards. But in reality, our services were consuming around 260,000 to 280,000 ports before any optimization. This led to:

Exhausted NAT ports
Dropped packets
Connection failures
Elevated error rates across microservices

This clearly pointed to a misalignment between AWS-style planning and GCP’s NAT behavior.

Scaling NAT IPs and Handling Whitelisting Complexities

To fix the port exhaustion, we scaled our NAT configuration and assigned 8 NAT IPs. This brought stability back into the system, but it introduced a new challenge—whitelisting.

Many of our clients and partners had previously whitelisted our single NAT IP. Now, with 8 new IPs, we had to coordinate across teams to get all of them whitelisted. It was a time-consuming and operationally sensitive task, but necessary to ensure reliable communication with external systems.

Despite the delay, we managed to get this done with clear communication and tight coordination.

A Costly Misstep: TCP Timeout Misconfiguration

While we were solving the port exhaustion issue, we also made another change: we reduced the NAT TCP timeout from the default 1200 seconds to 30 seconds. This was intended to free up unused ports faster.

Unfortunately, this change backfired. Our systems had long-lived TCP connections that were now being closed too early. As a result, critical sessions were dropped, and the platform experienced a full-day outage.

Lesson learned: Always evaluate the full impact before tuning low-level infrastructure parameters. A small change in timeout settings can break essential workflows and affect customer experience.

Root Cause Analysis and Recovery

We dove into the GCP documentation and closely analyzed metrics to find the cause of the outage. It became clear that the short TCP timeout setting was the culprit.

We reverted the timeout to its original 1200 seconds and monitored the system closely. Traffic and application behavior normalized within hours, validating our hypothesis. This reinforced the importance of data-backed debugging and careful change management.

Infrastructure Optimizations to Reduce NAT Usage

Once stability was restored, we began optimizing our cloud infrastructure to reduce unnecessary NAT usage. This involved several focused initiatives:

a. Kafka Schema Registry Cleanup

Problem: Kafka’s Schema Registry was keeping old and unused connections alive, occupying a huge number of NAT ports.

Action: We removed outdated schemas and cleared stale connections.

Result: NAT port usage decreased significantly, and connection handling became more efficient.

b. Enabling Private Google Access

Problem: API calls to Google services were unnecessarily routed through NAT.

Action: We enabled Private Google Access within our VPC.

Result:

Google API calls now route through Google’s private backbone
Reduced NAT traffic
Faster and more reliable API calls

Measurable Impact of Our Optimizations

After these changes, our NAT port usage dropped by 38%—from about 260,000 to 160,000.

Final Thoughts

Our migration to GCP came with its share of surprises, especially around NAT behavior. But through careful investigation, collaboration, and optimization, we emerged with a more efficient and resilient system.

Here’s a quick recap of our key takeaways:

GCP handles NAT differently than AWS—plan accordingly.
Don't underestimate the impact of small configuration changes.
Infrastructure hygiene (like cleaning stale connections) goes a long way.
Use cloud-native features like ILBs and Private Access to reduce overhead.
Ensure that all inter-service communication between microservices uses Internal Load Balancers instead of External Load Balancers. This change will help optimize and significantly reduce NAT data usage.

We hope our journey offers useful guidance for others going through similar transitions. The effort was intense—but the gains in performance, reliability, and scalability have been well worth it.

FAQs

1. What is Google Cloud Platform (GCP)?

Google Cloud Platform is a suite of cloud computing services by Google. It provides infrastructure tools like storage, computing power, and machine learning to build and scale applications.

2. What is AWS?

Amazon Web Services (AWS) is a cloud platform provided by Amazon. Like GCP, it offers a wide range of services for computing, storage, databases, and machine learning.

3. What is Network Address Translation (NAT)?

NAT is a method used in networks to allow multiple devices to access the internet using a single IP address. It helps improve security and conserve IP addresses.

4. Is GCP better than AWS?

Both GCP and AWS are powerful platforms with their strengths. GCP excels in data analytics and machine learning, while AWS has a broader range of services and global reach. The choice depends on your business needs.

5. Why do we need AWS to GCP migration?

Migration from AWS to GCP might be needed for reasons like cost optimization, better integration with Google services, improved analytics capabilities, or strategic cloud diversification.