Three practical guides to turn your files, emails, and project data into queryable SQLite databases
If you've ever struggled with compliance audits (HIPAA, SOX, GDPR, SOC 2), security investigations, or e-discovery requests, you know the pain:
- Data is everywhere — Files on disks, emails in inboxes, issues in GitHub/Jira
- No unified view — Each system has its own limited search interface
- Manual exports — Hours spent copying data into spreadsheets for auditors
- No audit trail — Can't prove what data existed at a specific time
- Point-in-time only — No historical tracking of changes
What if you could turn all of this into queryable SQL databases and ask questions like:
-- Find all PDFs modified in the last 30 days
SELECT * FROM files WHERE extension = 'pdf' AND modified > date('now', '-30 days');
-- Show emails from vendors about invoices
SELECT * FROM emails WHERE sender LIKE '%vendor%' AND subject LIKE '%invoice%';
-- Track all code changes for SOC 2 audit
SELECT * FROM commits WHERE date > '2024-01-01' ORDER BY date DESC;
That's what surveilr enables.
What is surveilr?
surveilr is an edge-based, SQL-first surveillance platform that turns your files, emails, and external APIs into Resource Surveillance State Databases (RSSDs)—SQLite databases you can query with standard SQL.
Key Features
- 📂 File system scanning — Turn directories into queryable metadata
- 📧 Email ingestion — Connect to Gmail/Outlook via IMAP
- 🔌 Singer tap integration — Extract data from 600+ sources (GitHub, Jira, Salesforce, etc.)
- 🔍 Standard SQL — No custom query language to learn
- 🔒 Edge-based — All data stays on your machine (no cloud required)
- ⚖️ Compliance-ready — Perfect for HIPAA, SOX, GDPR, SOC 2
Why SQLite?
surveilr uses SQLite—the world's most deployed database:
- ✅ Zero dependencies — No servers, no installation, no configuration
- ✅ One file — Your entire database is a single
.dbfile - ✅ Portable — Works everywhere (macOS, Linux, Windows, mobile)
- ✅ No vendor lock-in — Standard SQLite works with 1000s of tools
Three Practical Guides
Guide 1: Audit Sensitive Files
Learn how to scan your file system for sensitive files and query them with SQL.
Perfect for: Security audits, GDPR/HIPAA compliance, incident response
Guide 2: Email Compliance Tracking
Learn how to ingest Gmail/Outlook emails via IMAP and query them for compliance reporting.
Perfect for: HIPAA compliance, SOX compliance, e-discovery, communication audits
Guide 3: GitHub/GitLab/Jira Tracking
Learn how to use Singer taps to extract project data and query it for SOC 2 and change management audits.
Perfect for: SOC 2 audits, DevOps metrics, security tracking, audit trails
Why surveilr vs. Other Tools?
vs. Commercial Compliance Platforms (Vanta, Drata, Secureframe)
| Feature | surveilr | Commercial Tools |
|---|---|---|
| Cost | Free, open source | $10,000-$50,000/year |
| Data location | Your machine (edge) | Their cloud |
| Query language | Standard SQL | Proprietary UIs |
| Extensibility | Unlimited (SQLite + Singer) | Limited integrations |
| Vendor lock-in | None (standard SQLite) | Complete lock-in |
vs. Custom Scripts
| Feature | surveilr | Custom Scripts |
|---|---|---|
| Setup | Minutes | Days/weeks |
| Maintenance | Automatic | Constant fixes |
| Audit trail | Built-in | You build it |
| Query language | SQL | grep/awk/jq |
| Historical tracking | Automatic | You build it |
Installation
macOS / Linux
brew tap surveilr/tap && brew install surveilr
Verify Installation
surveilr --version
Get Started
Pick one of the three guides above and start querying your data with SQL.
Each guide is self-contained and works independently—start with whichever solves your biggest problem.
Key Takeaways
✅ Turn files, emails, and APIs into queryable SQL databases
✅ Perfect for HIPAA, SOX, GDPR, SOC 2 compliance
✅ Standard SQLite = no vendor lock-in
✅ Edge-based = your data never leaves your machine