This incident was the result of an infrastructure change that was made to our load balancers to prepare us for IPv6 enablement of GitHub.com. This change was deployed to a subset of our global edge sites.
The change had the unintended consequence of causing IPv4 addresses to start being passed as an IPv4-mapped IPv6-compatible address to our IP Allow List functionality.
For example 10.1.2.3 became ::ffff:10.1.2.3. While our IP Allow List functionality was developed with IPv6 in mind, it wasn't developed to handle these mapped addresses, and hence started blocking requests as it deemed these to be not in the defined list of allowed addresses. Request error rates peaked at 0.23% of all requests.
We have so far identified three remediation items here:
- Update the IP Allow List functionality to handle IPv4-mapped addresses.
- Audit the rest of our stack to confirm there are no further places this IPv4-mapped IPv6 addresses flaw exists.
- Improve our testing and monitoring processes to better catch these issues in the future.
https://bxt.org/j2s3p