scrapers-us-municipal

Source for municipal scrapers

To find out more about the ins and outs of these scrapers, as well as how to create your own, head on over to docs.opencivicdata.org's scraping page.

Issues?

Issues with the data coming from these scrapers should be filed in this repository.

Development

With Docker

Requires Docker and Docker Compose

Initialization

docker-compose run --rm scrapers pupa init YOUR_CITY_SCRAPER

Without Docker

Requires Python 3, PostGIS

Initialization

Assuming that you want to have your database be called opencivicdata on your local machine

pip install -r requirements.txt
createdb opencivicdata
export DATABASE_URL=postgresql:///opencivicdata
pupa dbinit us
pupa init YOUR_CITY_SCRAPER

At times, the release of ocd-django on PyPI differs from that of Github. This may cause problems if you need to create and run migrations. Specifically, you might encounter an ImproperlyConfigured error that instructs you to do the following:

You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.

Fix the problem by running:

export DJANGO_SETTINGS_MODULE=pupa.settings

Then, you should be able to successfully run:

django-admin makemigrations
django-admin migrate

Testing

Before submitting a PR, please run your scraper.

With Docker

docker-compose run --rm scrapers pupa update YOUR_CITY_SCRAPER

Without Docker

export DATABASE_URL=postgresql:///opencivicdata
pupa update YOUR_CITY_SCRAPER