构建一个查询分析系统 #

1pip install -qU langchain langchain-community langchain-openai youtube-transcript-api pytube langchain-chroma

1pip install python-dotenv

1from dotenv import load_dotenv
2assert load_dotenv()
3
4import os
5MODEL_NAME = os.environ.get("MODEL_NAME")
6EMBEDDING_MODEL_NAME = os.environ.get("EMBEDDING_MODEL_NAME")

 1from langchain_community.document_loaders import YoutubeLoader
 2
 3urls = [
 4    "https://www.youtube.com/watch?v=HAn9vnJy6S4",
 5    "https://www.youtube.com/watch?v=dA1cHGACXCo",
 6    "https://www.youtube.com/watch?v=ZcEMLz27sL4",
 7    "https://www.youtube.com/watch?v=hvAPnpSfSGo",
 8    "https://www.youtube.com/watch?v=EhlPDL4QrWY",
 9    "https://www.youtube.com/watch?v=mmBo8nlu2j0",
10    "https://www.youtube.com/watch?v=rQdibOsL1ps",
11    "https://www.youtube.com/watch?v=28lC4fqukoc",
12    "https://www.youtube.com/watch?v=es-9MgxB-uc",
13    "https://www.youtube.com/watch?v=wLRHwKuKvOE",
14    "https://www.youtube.com/watch?v=ObIltMaRJvY",
15    "https://www.youtube.com/watch?v=DjuXACWYkkU",
16    "https://www.youtube.com/watch?v=o7C9ld6Ln-M",
17]
18docs = []
19for url in urls:
20    docs.extend(YoutubeLoader.from_youtube_url(url, add_video_info=True).load())

1import datetime
2
3# Add some additional metadata: what year the video was published
4for doc in docs:
5    doc.metadata["publish_year"] = int(
6        datetime.datetime.strptime(
7            doc.metadata["publish_date"], "%Y-%m-%d %H:%M:%S"
8        ).strftime("%Y")
9    )

1[doc.metadata["title"] for doc in docs]

 1['OpenGPTs',
 2 'Building a web RAG chatbot: using LangChain, Exa (prev. Metaphor), LangSmith, and Hosted Langserve',
 3 'Streaming Events: Introducing a new `stream_events` method',
 4 'LangGraph: Multi-Agent Workflows',
 5 'Build and Deploy a RAG app with Pinecone Serverless',
 6 'Auto-Prompt Builder (with Hosted LangServe)',
 7 'Build a Full Stack RAG App With TypeScript',
 8 'Getting Started with Multi-Modal LLMs',
 9 'SQL Research Assistant',
10 'Skeleton-of-Thought: Building a New Template from Scratch',
11 'Benchmarking RAG over LangChain Docs',
12 'Building a Research Assistant from Scratch',
13 'LangServe and LangChain Templates Webinar']

1docs[0].metadata

1{'source': 'HAn9vnJy6S4',
2 'title': 'OpenGPTs',
3 'description': 'Unknown',
4 'view_count': 7210,
5 'thumbnail_url': 'https://i.ytimg.com/vi/HAn9vnJy6S4/hq720.jpg',
6 'publish_date': '2024-01-31 00:00:00',
7 'length': 1530,
8 'author': 'LangChain',
9 'publish_year': 2024}

1docs[0].page_content[:500]

1"hello today I want to talk about open gpts open gpts is a project that we built here at linkchain uh that replicates the GPT store in a few ways so it creates uh end user-facing friendly interface to create different Bots and these Bots can have access to different tools and they can uh be given files to retrieve things over and basically it's a way to create a variety of bots and expose the configuration of these Bots to end users it's all open source um it can be used with open AI it can be us"

 1from langchain_chroma import Chroma
 2from langchain_openai import OpenAIEmbeddings
 3from langchain_text_splitters import RecursiveCharacterTextSplitter
 4
 5text_splitter = RecursiveCharacterTextSplitter(chunk_size=2000)
 6chunked_docs = text_splitter.split_documents(docs)
 7embeddings = OpenAIEmbeddings(model=EMBEDDING_MODEL_NAME)
 8vectorstore = Chroma.from_documents(
 9    chunked_docs,
10    embeddings,
11)

1search_results = vectorstore.similarity_search("how do I build a RAG agent")
2print(search_results[0].metadata["title"])
3print(search_results[0].page_content[:500])

1Build and Deploy a RAG app with Pinecone Serverless
2hi this is Lance from the Lang chain team and today we're going to be building and deploying a rag app using pine con serval list from scratch so we're going to kind of walk through all the code required to do this and I'll use these slides as kind of a guide to kind of lay the the ground work um so first what is rag so under capoy has this pretty nice visualization that shows LMS as a kernel of a new kind of operating system and of course one of the core components of our operating system is th

1search_results = vectorstore.similarity_search("videos on RAG published in 2023")
2print(search_results[0].metadata["title"])
3print(search_results[0].metadata["publish_date"])
4print(search_results[0].page_content[:500])

1OpenGPTs
22024-01-31
3hardcoded that it will always do a retrieval step here the assistant decides whether to do a retrieval step or not sometimes this is good sometimes this is bad sometimes it you don't need to do a retrieval step when I said hi it didn't need to call it tool um but other times you know the the llm might mess up and not realize that it needs to do a retrieval step and so the rag bot will always do a retrieval step so it's more focused there because this is also a simpler architecture so it's always