Congress.wiki

Does Congress represent you? In 2021, I found myself frustrated by the complexity of learning about Congress and our representatives. This led me to create Congress.wiki, a user-friendly platform designed to guide citizens through the intricacies of the U.S. Congress.

Existing congressional resources overwhelm users with dense documentation and complex interfaces. Sites like Congress.gov require extensive help centers with 50+ pages of instructions, creating significant barriers for citizens trying to engage with their government.

Congress.wiki aims to make congressional information intuitive and engaging through thoughtful design principles:

1. Show, Don't Tell: Design interfaces that are self-explanatory without lengthy documentation.
2. Visual Recognition: Use photos and icons instead of text-heavy navigation, helping users quickly identify representatives and build lasting recognition.
3. Geographic Context: Present congressional data in relation to constituents' locations, strengthening the connection between citizens and their representatives.
4. Focused Information: Display only the most relevant data upfront, with additional context available through progressive disclosure.
5. Respectful Presentation: Honor the importance of congressional data while making it approachable.
6. Meaningful Engagement: Reserve calls-to-action for appropriate moments to maintain user trust.

One of the most critical challenges in building Congress.wiki was data collection. While congressional data is publicly available, it's fragmented across numerous sources and requires significant effort to aggregate comprehensively.

Challenges

• Data fragmentation across multiple websites and APIs
• Strict API rate limits requiring careful request management
• Limited historical data availability in APIs
• Critical information locked in PDF documents
• Inconsistent and sometimes outdated data, especially for current Congress

Solution

To address these challenges, I needed to have a robust data collection system.

I needed the system to:

• Excel at JSON data processing
• Support flexible integration of multiple data sources and APIs
• Implement a clear, extensible class structure for future growth
• Cache progress during operations such as API crawls
• Manage rate limits with sophisticated exponential backoff
• Extract structured data from PDFs
• Process media files using tools like FFMPEG
• Provide a powerful CLI for granular task control
• Report collection progress to a central database
• Run efficiently on affordable hardware like Raspberry Pis

I chose TypeScript for implementation, leveraging Node.js's excellent single-threaded performance and native JSON handling capabilities. This aligned perfectly with the JSON-based APIs common in congressional data sources.

System Design

I designed the system with a user-first approach, starting with the command-line interface (CLI). This strategy ensured the data collector would be both intuitive and extensible, with a clean class structure that supports easy expansion of features and data sources.

The CLI interface was designed with user experience in mind, offering fine-grained control over:

• Data Selection: Choose specific datasets (e.g., member biographies, photos)
• Processing Steps: Control which data processing phases would run. (e.g. only collect and validate data)
  Data Processing Pipeline

  The data processing pipeline consists of distinct phases:

  • Collection: Gathering raw data from sources
  • Validation: Verifying data structure and content
  • Refinement: Parsing PDFs, correcting errors, merging sources
  • Secondary Validation: Ensuring refined data integrity
  • Storage: Distributing to various backends (database, S3, YouTube)
• Progress Persistence: Enable checkpoint-based recovery
• Congress Selection: Target specific congressional sessions
• Reporting: Configure email notifications for task completion/failure

Data Sources

The system pulls data from authoritative sources including, but not limited to:

• Official Government APIs (Congress.gov, GovInfo, OpenFEC)
• Legislative Websites (Senate.gov, House.gov)
• Public Information Portals (Wikipedia, C-SPAN)
• Government Data Repositories (US Census, United States GitHub)

Infrastructure

The collection processes run automatically via multiple CRON jobs on a Kubernetes cluster powered by a Raspberry Pi 5, ensuring reliable, up-to-date information while maintaining cost efficiency.

Automated Data Collection Monitoring

To ensure the data collection process is running smoothly, I've implemented a monitoring system that sends me an email if any of the CRON jobs fail.

Failed Data Collection Email Report

A snapshot of the automated email alert system, demonstrating the robust monitoring of the data collection processes.

One of the key principles of Congress.wiki was to present congressional data in a "Geographic Context". Congressional data is inherently geographic, and presenting it in a geographic context makes it more tangible and accessible. To accomplish this I knew I needed to have a map of not only the United States, but also its congressional districts.

Building the Maps

While getting an SVG of the United States map is rather easy and there are many freely available options on the internet, getting the same for congressional districts is not. I found a few sources that offered to sell the congressional district maps, however the cost was unreasonably high and updates were not included. Congressional districts change over time, so I needed a way to make sure my map was always up to date. I eventually came to terms that I not needed to build my own map.

To start I needed to find a source of congressional district data. I eventually found the United States Census Bureau's TIGER/Line Shapefile. These shapefiles contain detailed geographic data for the United States, including congressional districts. By using the Census's API I was able to extend my collection system to automatically gather the shapefiles and process them into usable formats such as SVGs. I accomplished this using:

• GeoJSON (a JSON format used to represent geographic data)
• topoJSON (an extension of GeoJSON used to stitch together multiple GeoJSON files).

To build the map I broke the process down into the following steps:

1. Gather shapefiles for each congressional district
2. Convert the shapefiles into GeoJSON
3. Combine the GeoJSON into topoJSON files
4. Convert the topoJSON into SVGs

The Interactive Map

I believe interactivity is a great way to improve user engagement. Having played many video games throughout my life, I had a lot of experience with interactive maps and i've come to appreciate the careful design choices that go into them. So I decided to take some of my favorite features from video game maps and build an interactive congressional district map.

To build the interactive map, I utilized the topoJSON produced by my collector program. I fed the topoJSON data into D3 and drew the SVG. D3's extensive API allowed me to easily add interactivity such as zooming, dynamically changing fill colors, and modifying shape rendering algorithms based on the current map zoom level.

To make legislative language more accessible, I've implemented an AI-powered summary generator that creates concise, easy-to-understand explanations of complex text such as legislation. The system works through an optimized pipeline:

1. First checks for existing summaries in the database for instant retrieval
2. For new summaries, sends the bill text to OpenAI's API with custom prompting to ensure consistent, high-quality output
3. Streams the AI response directly to the database.
4. During this time, the client will open a websocket connection to the database to stream the summary to the user interface in real-time.
5. Once finished the page is revalidated and cached.

This architecture provides several benefits:

• Instant access to existing summaries
• Real-time streaming feedback as summaries are generated
• Efficient resource usage by avoiding duplicate API calls
• Consistent summary quality through careful prompt engineering
• Scalable design that can handle high concurrent usage

The postcard service allows users to directly engage with their representatives:

1. Users compose messages and select recipients.
2. AI moderation ensures appropriate content.
3. Seamless integration with Stripe for secure payments.
4. Automated postcard ordering and delivery tracking.

Unique features:

• Constituent identification through postcard design. If the postcard is from a constituent, the postcard design will be red, otherwise it will be white.
• Response rate tracking via centralized return addressing.

Through user research and feedback from political staffers, I've refined my approach to ensure maximum impact and compliance with official communication protocols.

The majority of Congress.wiki's infrastructure is written in CDK (Cloud Development Kit, TypeScript) and deployed to AWS. The actual Next.js site is hosted on Vercel. Requests to specific routes are routed from Vercel to either the Next.js site or AWS infrastructure.

AWS infrastructure includes:

• RDS Postgres database
• Hasura GraphQL server on a auto scaling ECS cluster.
• S3 for cheap static file storage (images, videos, etc.).
• A step function orchestration to handle generating AI text and storing it in the database.

Congress.wiki has made significant strides in making congressional information more accessible, but there's still much to be done. While I'm currently focusing on other projects, I remain committed to maintaining and enhancing the platform. Key areas for future development include:

• Expanding educational resources for students and teachers
• Improving real-time updates for congressional activities
• Enhancing mobile experience and accessibility features
• Adding more interactive data visualizations
• Adding more AI tools to help users understand congressional information
• Adding more congressional data and historical context

You can track progress and upcoming features on our Project Roadmap on GitHub. I welcome community feedback to help shape the future of Congress.wiki.

You can support Congress.wiki by sharing the website with others and providing feedback on the website (via the Feedback button at the bottom of every page). Thank you to everyone who has supported the project so far!