Richmond Sunlight

This is the front-end of the website. See also: rs-machine, the collection of scrapers and parsers that provide the site's third-party data, rs-api, the API that powers (some of) the website, and rs-video-processor, the on-demand legislative-video-processing system.

History

Richmond Sunlight started in 2005 as a little RSS-based bill tracker, updating every few hours. In 2006 it was built out as Richmond Sunlight, launching publicly in January of 2007. It's remained a hobby site ever since. The code base hasn’t been overhauled in all that time, and it shows — the site’s tech stack shows the growth rings of being developed over the course of many years. But it continues to function, and has been modernized in some ways, e.g. by adding a CI/CD pipeline, moving to SOA, etc.

Branches

• master: The staging site.
• deploy: The production site.

Local development

The site can be run locally, in Docker:

1. Install Docker.
2. Clone this repository. Make sure you’re using the branch that you want.
3. Run ./docker-run.sh.
4. In your browser, open http://localhost:8000.

Tests can be run with ./docker-tests.sh.

Local tests

There are additional tests, in tests/local/, that run against a larger sample of the database than is stored in GitHub. They load every bill, legislator, committee, and minutes page in the sample, failing on any page that doesn't load or that raises a PHP warning, and check every legislator's video clips, every video's captions, and every bill's fiscal impact statements. This test data is not publicly available, and is provided only to trusted contributors.

1. Get the sample, deploy/mysql/local-test-records.sql (which is ignored by Git): aws s3 cp s3://deploy.richmondsunlight.com/test-data/local-test-records.sql.gz - | gunzip > deploy/mysql/local-test-records.sql. It's a few hundred bills from the two most recent sessions, chosen to cover every type and status of bill, plus a dozen videos. rs-machine exports it from the production database with deploy/database_export.sh (nightly during the session, and weekly otherwise).
2. Rebuild the database, which loads it.
3. Run ./docker-tests.sh --local, or just the local tests, with docker compose exec web /var/www/htdocs/includes/vendor/bin/phpunit --configuration /var/www/phpunit.xml --testsuite local.

The "Local Tests" GitHub Actions workflow runs these on the same schedule, against the latest sample, and reports failures to Slack. They don't block deployment, because what they find is mostly problems in the data.

Without the sample, the local tests are skipped, except for those in LocalDataTest, which fail to say that it's missing. In Docker, PHP logs errors to logs/php-errors.log, rather than to Apache's log, which is how the tests find them.

When you are done, run ./docker-stop.sh (or quit Docker).

Rebuilding the database

The database is initialized from SQL dumps when the MariaDB container is first created. If any SQL files under deploy/mysql/ change (including when you download a new local sample), those changes won't take effect until you delete the database and load it again:

./docker-run.sh --fresh

That deletes the db_data volume and rebuilds the database's image, which holds a copy of the SQL files, so that MariaDB loads them again.

Architecture

flowchart LR
  subgraph AWS
    CD[CodeDeploy]
    MACH["Machine (rs-machine)"]
    RDS[(RDS)]
    SQS[(SQS)]
    VP[Video Processor]
    S3[(Video S3)]
    FE["Front-End (this site)"]
    API["API (rs-api)"]
    CF[CloudFront]
  end

  Users["Users/Browsers"]

  CD --> MACH
  CD --> FE
  CD --> API
  CD --> VP

  MACH -->|reads/writes| RDS
  MACH -->|enqueue videos| SQS
  MACH -->|upload clips| S3

  SQS --> VP
  VP -->|transcoded video| S3

  FE <-->|cached| CF
  CF <-->|serves| Users

  FE <-->|app data| RDS
  FE <-->|calls| API
  API <-->|data| RDS

Loading