incremental-fetch
Build resilient data ingestion pipelines from APIs. Use when creating scripts that fetch paginated data from external APIs (Twitter, exchanges, any REST API) and need to track progress, avoid duplicates, handle rate limits, and support both incremental updates and historical backfills. Triggers: 'ingest data from API', 'pull tweets', 'fetch historical data', 'sync from X', 'build a data pipeline', 'fetch without re-downloading', 'resume the download', 'backfill older data'. NOT for: simple one-shot API calls, websocket/streaming connections, file downloads, or APIs without pagination.
Installation
Suivez les instructions du dépôt pour votre agent. Consultez les fichiers et dépendances avant l’installation.
Voir la source ↗GuidesInstructions
Fonctions et usage
Build resilient data ingestion pipelines from APIs. Use when creating scripts that fetch paginated data from external APIs (Twitter, exchanges, any REST API) and need to track progress, avoid duplicates, handle rate limits, and support both incremental updates and historical backfills. Triggers: 'ingest data from API', 'pull tweets', 'fetch historical data', 'sync from X', 'build a data pipeline', 'fetch without re-downloading', 'resume the download', 'backfill older data'. NOT for: simple one-shot API calls, websocket/streaming connections, file downloads, or APIs without pagination.
Démarrage rapide
Suivez les instructions du dépôt pour votre agent. Consultez les fichiers et dépendances avant l’installation.
Guides ↗Exemples
Consultez les exemples du dépôt source.
Voir la source ↗Historique source
- Date d’indexation
- 2026-01-20
- Mise à jour de l’index
- 2026-02-24
- Branch
- main
Les dates concernent ce répertoire. Consultez le dépôt pour l’historique du code.
GitHub ↗chat Comments (0)
Sign in to join the discussion and leave a comment.
No comments yet. Be the first to share your thoughts!