GitHub’s trending page updates every few hours using an opaque ranking algorithm. Developers who want different signals (velocity over absolute stars, language-specific trends, spam filtering) hit a wall. Building an alternative means solving three hard problems: staying under API rate limits while tracking thousands of repositories, making velocity queries fast enough for real-time ranking, and detecting gaming without GitHub’s internal fraud signals.
Shadowgraph is a Show HN project that tackles these constraints head-on. The creator posted about frustration with GitHub’s trending algorithm and built a custom discovery layer that exposes repository velocity metrics through a public dashboard. The Hacker News discussion surfaced common pain points: developers want transparency in ranking logic, better filtering by language and time window, and access to historical velocity data that GitHub does not expose.
The plumbing decisions behind any repository discovery system apply equally to agent-driven code search, automated dependency monitoring, or competitive intelligence dashboards.
The Rate Limit Wall
GitHub’s REST API gives authenticated users 5,000 requests per hour (official docs). The GraphQL API shares the same budget but counts differently: each query costs points based on complexity, with a maximum of 5,000 points per hour.
Tracking 10,000 repositories naively requires 10,000 API calls. At 5,000 requests per hour, you need two hours for a single refresh cycle. Real-time trending requires sub-hour updates.
Strategies that work:
- Batch GraphQL queries: Request 100 repositories per call using aliases. Reduces 10,000 calls to 100 calls, but each query costs more points.
- Conditional requests with ETags: GitHub returns a 304 Not Modified response that does not count against your quota if repository data has not changed.
- Webhooks for high-value repos: Subscribe to push events, stars, and forks for repositories you care most about. Webhooks do not count against rate limits.
- Multiple tokens with round-robin: Rotate between API tokens tied to different accounts. This violates GitHub’s Terms of Service if done at scale, but small projects use 2-3 tokens for development.
A realistic architecture uses GraphQL batching for bulk updates, ETags for unchanged repos, and webhooks for the top 500 repositories by current velocity.
Caching and Indexing for Velocity Queries
Repository velocity means stars per hour, forks per day, or commits per week. Calculating velocity requires historical snapshots. Storing every API response in a time-series database works but makes ranking queries expensive.
Snapshot storage:
- Store repository metrics (stars, forks, watchers, open issues) every 15 minutes.
- Use PostgreSQL with a
repo_snapshotstable indexed on(repo_id, timestamp). - Partition by day to keep query performance predictable as data grows.
Precomputed velocity table:
Instead of calculating velocity on every page load, maintain a repo_velocity table updated every 15 minutes:
CREATE TABLE repo_velocity (
repo_id INTEGER PRIMARY KEY,
stars_1h INTEGER,
stars_24h INTEGER,
forks_1h INTEGER,
forks_24h INTEGER,
velocity_score FLOAT,
updated_at TIMESTAMP
);
CREATE INDEX idx_velocity_score ON repo_velocity(velocity_score DESC);The velocity_score is a weighted combination of short-term and long-term growth. A simple formula:
velocity_score = (stars_1h * 10) + (stars_24h * 1) + (forks_1h * 5)This table supports fast ranking queries without recalculating deltas on every request. The trade-off: stale data between updates. Fifteen-minute refresh cycles balance freshness and database load.
Ranking Algorithm Design
GitHub’s trending algorithm is not public, but reverse engineering suggests it weighs recent activity more heavily than absolute star count and applies language-specific normalization.
A transparent alternative needs explicit rules:
| Factor | Weight | Rationale |
|---|---|---|
| Stars in last hour | 10x | Captures breakout moments |
| Stars in last 24 hours | 1x | Smooths out spikes |
| Forks in last hour | 5x | Signals developer intent to use |
| Commit frequency (7 days) | 2x | Filters abandoned projects |
| Issue close rate | 1x | Proxy for maintenance quality |
Anti-gaming heuristics:
- Star velocity ceiling: Repos gaining more than 1,000 stars per hour get flagged for manual review. Legitimate viral projects hit this threshold, but so do vote brigades.
- Account age filter: Stars from accounts created in the last 30 days count at 50% weight.
- Fork-to-star ratio: Repos with fewer than 5% forks relative to stars get downweighted. Real projects get forked; spam repos get starred.
These rules are not foolproof. Determined actors create aged accounts and fake forks. The goal is raising the cost of gaming, not perfect detection.
How Shadowgraph Implements This
Shadowgraph runs a polling service that fetches repository data every 15 minutes using GraphQL batching. The system tracks approximately 5,000 repositories across multiple language ecosystems. Velocity calculations happen server-side and get cached in a precomputed table similar to the schema above.
The dashboard exposes three time windows (1 hour, 24 hours, 7 days) and allows filtering by programming language. Unlike GitHub’s trending page, Shadowgraph shows the actual velocity numbers (stars gained per hour, forks per day) alongside each repository. This transparency lets developers understand why a repository ranks where it does.
The creator noted in the Show HN discussion that the hardest part was not building the initial polling loop. It was handling edge cases: repositories that get deleted mid-poll, API responses that return null values for private repos that later go public, and rate limit exhaustion during traffic spikes when users refresh the dashboard repeatedly.
Polling Architecture
A production-grade discovery system needs a polling service, a ranking service, and a caching layer.
Polling service:
- Runs every 15 minutes.
- Fetches repository data using GraphQL batching.
- Writes snapshots to
repo_snapshotstable. - Updates
repo_velocitytable with calculated deltas.
Ranking service:
- Runs every 15 minutes after polling completes.
- Reads from
repo_velocitytable. - Applies ranking formula and anti-gaming filters.
- Writes top 500 results to a
trending_cachetable.
API layer:
- Serves requests from
trending_cachetable. - No real-time calculation on the hot path.
- Cache TTL matches polling interval (15 minutes).
This architecture decouples data collection from ranking from serving. Each component can scale independently. Polling can run on a single worker. Ranking can run on a separate cron job. The API layer can scale horizontally behind a load balancer.
Observability and Failure Modes
Metrics to track:
- API rate limit remaining (per token).
- Polling job duration and success rate.
- Velocity calculation errors (missing snapshots, null values).
- Cache hit rate on the API layer.
Common failure modes:
- Rate limit exhaustion: Polling job runs long and exhausts the hourly quota before completing. Solution: split polling into smaller batches with rate limit checks between batches.
- Stale velocity data: Ranking job fails silently, leaving old data in
trending_cache. Solution: add alast_updatedtimestamp to the cache table and alert if it falls behind by more than 30 minutes. - Missing snapshots: API call fails for a subset of repositories, leaving gaps in velocity calculation. Solution: backfill missing snapshots from the next successful poll or mark velocity as unavailable.
Alerting on rate limit thresholds (below 500 remaining requests) gives time to pause polling before hitting zero.
When to Build Your Own Discovery Layer
Use cases that justify custom infrastructure:
- Agent-driven code search: LLMs need repository metadata (activity, language, license) to recommend libraries. GitHub’s trending page does not expose structured data.
- Competitive intelligence: Track velocity of competitor repositories, related projects, or ecosystem growth. GitHub’s API supports this, but you need historical snapshots.
- Internal discovery for monorepos: Large organizations with hundreds of internal repositories need custom ranking (by team, by service criticality, by deployment frequency).
When to skip it:
- You only need a daily digest of trending repos. Use existing aggregators or GitHub’s RSS feed.
- You care about absolute popularity, not velocity. GitHub’s star count is good enough.
- You do not want to manage polling infrastructure, rate limits, and database growth.
Technical Verdict
Building a GitHub trending alternative is a good forcing function for learning API polling patterns, time-series data modeling, and ranking algorithm design. The plumbing (rate limit management, snapshot storage, precomputed velocity tables) transfers directly to any system that tracks external state at scale.
Use this approach when you need custom ranking logic, historical velocity data, or agent-driven discovery. Avoid it if you just want a better UI on top of GitHub’s existing trending page. The infrastructure cost (database, polling workers, observability) only pays off when you need signals GitHub does not expose.
In practice, the hardest part is not the initial build. It is maintaining data quality as GitHub’s API evolves, repositories get deleted, and bad actors find new ways to game velocity metrics. Shadowgraph demonstrates that the technical challenges are solvable, but the operational burden of running a discovery service at scale requires ongoing attention to edge cases and API changes.