Working on improving the process of converting documents into Embeddings (for vectorStores) for LLM applications but i need some ideas and complaints from you!
I am CS undergrad interested in NLP, building LLM applications and uses of embeddings in professional settings.
I have been thinking about researching better ways to extract, transform and load (ETL pipelines) data from several formats into text embeddings for the aforementioned applications.
But it seens my initial ideas of contribution were already done...
First i thought about a better way to load CSV and tabular data files into embeddings, but PostGresVector DB was launched a month or so ago, so i guess i cant really do much that they havent done already lol
I have been thinking about other data types such as JSON or XML and how to treat them and load them into vectorDBs but i am not sure.
Do you guys have more ideas? Maybe one complaint you have when using such tools and data sources? I am curious and excited to hear these problems so maybe i could work on them
Posted Nov 1, 2023, 4:03 PM UTC