Step 1: Finding churches in public data 🔬
The IRS hosts a public dataset called the Exempt Organizations Business Master File. It's a quarterly released set of CSVs with basic profile information for millions of US nonprofits. With some filtering, that list can be honed to about 270,000 church-like entities. It's difficult to find a real total of churches in the US, as not every church files for nonprofit status for reasons like scale (maybe they're just some friends in a living room) or because they're anti-government zealots.
Step 2: Church-like nonprofits 🤝🏼 Web search results
Utilizing a paid dataset containing web search results for every church-like nonprofit in the IRS data, a small LLM hosted on a baseline M4 Mac mini performed over a million prompts to determine if a church's search results appeared to match the profile information from the IRS. For around 48% of churches in the IRS data, we'll get a match. Not every match is correct, or even a website at all (lots of small churches are Facebook only!).
Step 3: Acquire website 🧑🏼💻
A copy of the church website is acquired for further processing. Mostly drama-free. The web is quickly fronting itself with anti-bot armor, so retrieving updated information in the future will be more complicated. The goal is > 100,000 churches.
Step 4: Cleanup 🧹
The IRS data we talked about before wasn't especially clean. Many entities from entirely different faiths were tagged in with church metadata. Taking some passes to clean these out. More complicated is trying to find church-like orgs like schools or missions organizations which don't have strong flags indicating so.
Step 5: Enrich ✨
For each church, an enrichment LLM task is run to determine about 75 standardized fields.
Step 6: Release 🆓
The enriched church records are released to GitHub in real time as they are processed or updated.
Step 7: Derivative projects 🛠️
The bulk collection of websites contained several treasures worth diving deeper into. The biggest of which is sufficient sermon archives to help drive the /sermonsense project.
Additionally, several proof-of-concept projects will be created to help demonstrate and increase project visibility.
Step 8: Community ⛪️
A dataset of this scale is never truly completed. The long-term aspiration is to find a team of volunteers to increase the quantity of enriched churches, the quality of the data, and to help find new ways to increase the utility of the database so that the American church can be better understood from a distance.
The process is documented as a confidence measure for the dataset, but none of the tooling or intermediate data described below will be publicly released as part of this project.