finviz-crawler

# finviz-crawler

## Why This Skill?

📰 **Your own financial news database** — most finance skills just wrap an API for one-shot queries. This skill runs continuously, building a local archive of every headline and article from Finviz. Query your history anytime — no API limits, no missing data.

🆓 **No API key, no subscription** — scrapes finviz.com directly using Crawl4AI + RSS. Bloomberg, Reuters, Yahoo Finance, CNBC articles extracted automatically. Zero cost.

🤖 **Built for AI summarization** — the query tool outputs clean text/JSON optimized for LLM digests. Pair with an OpenClaw cron job for automated morning briefings, evening wrap-ups, or weekly investment summaries.

💾 **Auto-cleanup** — configurable expiry automatically deletes old articles from both the database and disk. Set `--expiry-days 30` to keep a month of history, or `0` to keep everything forever.

🔄 **Daemon architecture** — runs as a background service that starts/stops with OpenClaw. No manual intervention needed after setup. Works with systemd (Linux) and launchd (macOS).

## Install

```bash
python3 scripts/install.py
```

Works on **macOS, Linux, and Windows**. Installs Python packages (`crawl4ai`, `feedparser`), sets up Playwright browsers, creates data directories, and verifies everything.

### Manual install
```bash
pip install crawl4ai feedparser
crawl4ai-setup  # or: python -m playwright install chromium
```

## Usage

### Run the crawler
```bash
# Default: ~/workspace/finviz/, 7-day expiry
python3 scripts/finviz_crawler.py

# Custom paths and settings
python3 scripts/finviz_crawler.py --db /path/to/finviz.db --articles-dir /path/to/articles/

# Keep 30 days of articles
python3 scripts/finviz_crawler.py --expiry-days 30

# Never auto-delete (keep everything)
python3 scripts/finviz_crawler.py --expiry-days 0

# Custom crawl interval (default: 300s)
python3 scripts/finviz_crawler.py --sleep 600
```

### Query articles
```bash
# Last 24 hours of headlines
python3 scripts/finviz_quer...

# finviz-crawler

A continuous financial news crawler that scrapes headlines and articles from [Finviz](https://finviz.com), stores them in SQLite, and provides a query interface.

Powers the **📰 Market News** app in [PrivateApp](https://github.com/camopel/PrivateApp).

## Install

```bash
python3 scripts/install.py
```

This will:
- Install Python dependencies (`crawl4ai`, `feedparser`)
- Create data directory at `~/workspace/finviz/`
- Set up a SQLite database with default tickers
- Create a background service (systemd on Linux, launchd on macOS)

## Usage

### Run the crawler

```bash
# Crawl once
python3 scripts/finviz_crawler.py

# Crawl and delete articles older than 30 days
python3 scripts/finviz_crawler.py --expiry-days 30
```

### Query articles

```bash
# List tickers being tracked
python3 scripts/finviz_query.py --list-tickers

# Recent headlines (last 24h)
python3 scripts/finviz_query.py --hours 24 --titles-only

# Add/remove tickers
python3 scripts/finviz_query.py --add-ticker NVDA:nvidia,jensen
python3 scripts/finviz_query.py --remove-ticker MSFT

# Output JSON
python3 scripts/finviz_query.py --hours 48 --format json
```

## Default Tickers

The installer seeds these tickers by default:

| Symbol | Keywords |
|--------|----------|
| QQQ | qqq, nasdaq |
| AMZN | amazon, amzn, aws |
| GOOGL | google, googl, alphabet |
| TSLA | tesla, tsla, elon musk |
| META | meta, facebook, zuckerberg |
| NVDA | nvidia, nvda, jensen |

Add your own with `--add-ticker SYMBOL:keyword1,keyword2`.

## Data

Articles are stored in `~/workspace/finviz/`:
- `finviz.db` — SQLite database (articles, tickers, settings)
- `articles/` — Full article text as Markdown files
- `finviz_settings.db` — Ticker and keyword configuration

## Background Service

The installer creates a service that runs the crawler continuously (restarts on failure):

```bash
# Linux
systemctl --user status finviz-crawler
systemctl --user start finviz-crawler
journalctl --user -u finviz-crawler -f

# macOS...