Scrapers

The Social Crawler Index uses a variety of scrapers to collect data from different social media platforms. Each scraper is tailored to the specific platform it targets.

Bluesky

The Bluesky scraper is responsible for collecting data from the Bluesky social network.

File: bluesky.ts

// Social-Crawler-Index/src/scrapers/bluesky.ts

Discord

The Discord scraper is used to collect messages and other data from Discord servers.

File: discord.ts

// Social-Crawler-Index/src/scrapers/discord.ts

Facebook

The Facebook scraper is designed to collect data from Facebook pages and groups.

File: facebook.ts

// Social-Crawler-Index/src/scrapers/facebook.ts

Instagram

The Instagram scraper is used to collect data from Instagram profiles and posts.

File: instagram.ts

// Social-Crawler-Index/src/scrapers/instagram.ts

LinkedIn

The LinkedIn scraper is used to collect data from LinkedIn profiles and company pages.

File: linkedin.ts

// Social-Crawler-Index/src/scrapers/linkedin.ts

Reddit

The Reddit scraper is used to collect data from subreddits and user profiles.

File: reddit.ts

// Social-Crawler-Index/src/scrapers/reddit.ts

Signal

The Signal scraper is used to collect messages from Signal groups.

File: signal.ts

// Social-Crawler-Index/src/scrapers/signal.ts

Telegram

The Telegram scraper is used to collect messages from Telegram channels and groups.

File: telegram.ts

// Social-Crawler-Index/src/scrapers/telegram.ts

TikTok

The TikTok scraper is used to collect data from TikTok videos and user profiles.

File: tiktok.ts

// Social-Crawler-Index/src/scrapers/tiktok.ts

Truth Social

The Truth Social scraper is used to collect data from Truth Social posts and user profiles.

File: truth-social.ts

// Social-Crawler-Index/src/scrapers/truth-social.ts

WhatsApp

The WhatsApp scraper is used to collect messages from WhatsApp groups.

File: whatsapp.ts

// Social-Crawler-Index/src/scrapers/whatsapp.ts

X (formerly Twitter)

The X scraper is used to collect data from X (formerly Twitter) timelines and user profiles.

File: x.ts

// Social-Crawler-Index/src/scrapers/x.ts