What My Project Does?

I built an AI system that uses only official USA government sources with GPT4 to help us navigate the bureaucracy.

The result is pretty cool, you can play around at https://app.clerkly.co/ .

Target Audience

The target audience is everyone who has problems with dealing with USA government bureaucracy.

Comparison

• Google: I use only official government sources for answering your queries
• ChatGPT: I use a combination of agents and only government sources to provide answers with high-quality quotable sources .

How Did I Achieve This?

1. Locating Data

First, it's important to gather all the essential government data you need. This can involve a lot of time browsing various federal and local government websites. Your goal is to find the domains that hold the relevant information. Be patient and thorough during this phase because the quality of your data depends on it.

2. Scraping Data

To collect data from these sites, I used Apify (https://apify.com/). This platform allows you to set up web crawlers which can automatically extract data from web pages. However, setting up these crawlers can be tricky:

• You'll have to make sure to exclude irrelevant pages like random address books or archives.
• No single format works for all websites, so you'll need to customize your settings for each site.

For speedy data processing, Llama3 is a helpful to can make this part easier.

3. Processing Data

Once you have the raw data, the next step is to break it down into manageable chunks. This is crucial for later retrieval. Here’s how I did it:

• I looked into llamaIndex for inspiration but found it didn't meet all my requirements.
• Consequently, I developed my own solution to process the data into chunks, making it easier to store and retrieve.

4. Storing Data and Creating Links

For storage, I opted for No4j, which is a powerful tool for managing large datasets. Here’s how I made use of it:

• Using Llama3, I extracted key entities (like names, dates, events) from the data.
• These entities were then used to create linkages within GraphDB, connecting related pieces of data to enhance retrievability.

5. Retrieving Data

This is the most crucial step because the quality of your data retrieval directly impacts the answers generated by GPT-4. The retrieval process happens in two stages:

1. Initial Retrieval: Here you gather a broad set of potentially relevant data.
2. Refinement: Then you narrow down this broad set to the most relevant pieces of data.

This phase involves a lot of trial and error, so keep your target user in mind and refine your methods accordingly.

6. Generating Answers

Finally, after your query is processed and the appropriate context is obtained, you use the GPT-4 API to get your desired results. Here’s the simple process:

1. Formulate a Retrieval-Augmented Generation (RAG) prompt.
2. Call the GPT-4 API with this prompt.
3. Get the final, high-quality answer.

Key resources:

• llamaindex github
• Free graph RAG course

By following these steps, you can effectively locate, scrape, process, store, retrieve, and utilize data to generate high-quality answers using GPT-4. The process may seem daunting at first, but with time and practice, it becomes more manageable.