Fixing GitHub API Rate Limit Exceeded (403/429) & Authentication (401) Errors

Understanding GitHub API Errors

When building automation, CI/CD pipelines, or internal developer portals, integrating with the GitHub API is practically mandatory. However, as your infrastructure scales, you will inevitably encounter API restrictions. These restrictions manifest as HTTP 403 (Rate Limit), HTTP 429 (Secondary Limits), HTTP 401 (Authentication failures), or HTTP 502/Timeout errors.

Understanding the exact error payload and HTTP headers returned by GitHub is the first step toward building resilient integrations.

The Core Errors

1. HTTP 403: API Rate Limit Exceeded (Primary Limit)

This is the most common error engineers face. The GitHub API enforces a hard cap of 60 requests per hour for unauthenticated clients (tracked by IP address) and 5,000 requests per hour for authenticated clients.

Exact Error Message:

{
  "message": "API rate limit exceeded for 192.0.2.1. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)",
  "documentation_url": "https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting"
}

2. HTTP 403 / 429: Secondary Rate Limits (Abuse Protection)

Even if you are well within your 5,000 requests/hour budget, GitHub employs secondary rate limits to prevent abuse, scraping, and degradation of their infrastructure. These are triggered by making too many concurrent requests, requesting computationally expensive endpoints (like massive search queries), or polling a single endpoint too rapidly.

Exact Error Message:

{
  "message": "You have exceeded a secondary rate limit. Please wait a few minutes before you try again.",
  "documentation_url": "https://docs.github.com/rest/overview/resources-in-the-rest-api#secondary-rate-limits"
}

3. HTTP 401: Bad Credentials

When you attempt to authenticate but provide an invalid token, a token that lacks the required scopes, or an expired token, GitHub returns a 401 Unauthorized.

Exact Error Message:

{
  "message": "Bad credentials",
  "documentation_url": "https://docs.github.com/rest"
}

4. HTTP 502 Bad Gateway & API Timeouts

These occur when the upstream GitHub servers fail to process your request in a timely manner. This typically happens during complex GraphQL queries, massive repository clones via API, or during global GitHub incidents.

Step 1: Diagnosing the Bottleneck

Before implementing a fix, you must inspect the HTTP response headers. GitHub explicitly tells you your limits, how many requests you have remaining, and when your quota resets.

Execute a diagnostic curl command to inspect your headers:

curl -i https://api.github.com/users/octocat

Look for the following crucial headers in the response:

• x-ratelimit-limit: The maximum number of requests you're permitted to make per hour.
• x-ratelimit-remaining: The number of requests remaining in the current rate limit window.
• x-ratelimit-reset: The time at which the current rate limit window resets, in UTC epoch seconds.
• x-ratelimit-used: The number of requests you have made in the current window.
• retry-after: (Only present during secondary rate limits) The number of seconds you must wait before making another request.

Step 2: Strategic Fixes and Mitigation

Fix 1: Implement Proper Authentication (Fixes 403/401)

If you are hitting the 60 requests/hour limit, the immediate fix is to authenticate.

Do not use basic authentication (username/password) as it is deprecated. Instead, use a Personal Access Token (PAT) or a GitHub App token.

Ensure you are passing the token correctly using the Authorization: Bearer or Authorization: token header:

curl -H "Accept: application/vnd.github+json" \
     -H "Authorization: Bearer YOUR_TOKEN_HERE" \
     -H "X-GitHub-Api-Version: 2022-11-28" \
     https://api.github.com/user

If you receive a 401 Bad credentials error:

1. Verify the token has not expired in your GitHub Developer Settings.
2. Check if SSO (Single Sign-On) enforcement is enabled on your Organization. You may need to click "Authorize" next to the token in the GitHub UI to allow it to access Org resources.
3. Ensure the token has the correct scopes (e.g., repo, read:org). Fine-grained PATs require explicit repository permissions.

Fix 2: Respect Secondary Limits & Implement Exponential Backoff (Fixes 429/403)

If you are hitting secondary rate limits, your script is too aggressive. GitHub explicitly mandates the following best practices:

• Do not make concurrent requests. Process API calls sequentially.
• Pause between mutating requests. Wait at least 1 second between POST, PATCH, PUT, or DELETE requests.
• Respect the Retry-After header. If GitHub sends this header, sleep your thread for exactly that many seconds.

Here is an architectural pattern for implementing a robust retry mechanism with exponential backoff in Python:

import time
import requests

def github_api_request(url, headers, max_retries=5):
    backoff = 1
    for attempt in range(max_retries):
        response = requests.get(url, headers=headers)
        
        # Handle standard rate limit exhaustion
        if response.status_code == 403 and 'x-ratelimit-remaining' in response.headers:
            if int(response.headers['x-ratelimit-remaining']) == 0:
                reset_time = int(response.headers['x-ratelimit-reset'])
                sleep_duration = max(reset_time - time.time(), 0) + 1
                print(f"Primary rate limit hit. Sleeping for {sleep_duration} seconds.")
                time.sleep(sleep_duration)
                continue
                
        # Handle secondary rate limits
        if response.status_code in [403, 429] and 'retry-after' in response.headers:
            retry_after = int(response.headers['retry-after'])
            print(f"Secondary rate limit hit. Sleeping for {retry_after} seconds.")
            time.sleep(retry_after)
            continue
            
        # Handle 5xx Server Errors (Bad Gateway / Timeout)
        if response.status_code >= 500:
            print(f"Server error {response.status_code}. Retrying in {backoff} seconds...")
            time.sleep(backoff)
            backoff *= 2 # Exponential backoff
            continue
            
        response.raise_for_status()
        return response.json()
    
    raise Exception("Max retries exceeded")

Fix 3: Utilize Conditional Requests (Saves Quota)

The GitHub REST API supports conditional requests, which is a massive optimization for polling scripts. By leveraging the ETag and If-None-Match headers, or the Last-Modified and If-Modified-Since headers, you can ask GitHub: "Has this data changed since I last checked?"

If the data has not changed, GitHub returns a 304 Not Modified status code. Crucially, 304 responses do not count against your API rate limit.

How to use it:

1. Make an initial request and save the ETag header from the response (e.g., ETag: "W/123456789").
2. On your next request, send that ETag back in the If-None-Match header:

curl -i -H "If-None-Match: \"W/123456789\"" https://api.github.com/users/octocat

Fix 4: Migrate to GraphQL to Prevent 502 Timeouts

If you are consistently experiencing API timeouts (502 Bad Gateway) or hitting secondary rate limits due to high payload sizes, you are likely over-fetching data. The REST API often returns massive JSON objects when you only need a single field.

Migrating to the GitHub GraphQL API (v4) allows you to specify exactly what data you need, vastly reducing the database overhead on GitHub's side and eliminating timeout errors.

Instead of fetching 100 Pull Requests via REST just to get their titles, use GraphQL:

query {
  repository(owner: "octocat", name: "Hello-World") {
    pullRequests(last: 10) {
      nodes {
        title
        state
      }
    }
  }
}

Note that GraphQL has a different rate-limiting structure based on a point-scoring system rather than raw request counts. A complex query costs more points than a simple one, up to 5,000 points per hour.

Conclusion

Building resilient applications on top of the GitHub API requires defensive programming. Never assume a request will succeed. Always inspect headers, implement conditional caching, use proper authentication scopes, and build robust exponential backoff retry mechanisms to handle secondary rate limits and gateway timeouts gracefully.