Website search tool #160
Merged
Website search tool #160
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
darielnoel
reviewed
Nov 26, 2024
| if (!this.content || this.content === '') { | ||
| throw new Error('Please provide content to process.'); | ||
| } | ||
| if (!query || query === '') { |
There was a problem hiding this comment.
Hi! I'd like to suggest updating the error messages in SimpleRAG to be more explicit for agent decision-making.
Currently, the tool throws errors, but since this tool is meant to be used by agents, we should return structured error messages that help agents make decisions. Here's the proposed change:
js
async call(input) {
const { content = this.content, query } = input;
if (!content) {
return "ERROR_MISSING_CONTENT: No text content was provided for analysis. Agent should provide content in the 'content' field.";
}
if (!query) {
return "ERROR_MISSING_QUERY: No question was provided. Agent should provide a question in the 'query' field.";
}
try {
const ragToolkit = this.ragToolkit;
await ragToolkit.addDocuments([{ source: content, type: 'string' }]);
const response = await ragToolkit.askQuestion(query);
return response;
} catch (error) {
return ERROR_RAG_PROCESSING: RAG processing failed. Details: ${error.message}. Agent should verify content format and query validity.;
}
}Key improvements:
- Returns errors as strings instead of throwing exceptions
- Adds clear
ERROR_prefixes for easy error type identification - Makes error messages explicit about what's missing or wrong
- Provides direct guidance on what the agent should do next
- Makes error states machine-parseable while remaining human-readable
Let me know if you'd like me to explain any part of these changes!
darielnoel
reviewed
Nov 26, 2024
| this.url = url; | ||
| } | ||
| if (!this.url || this.url === '') { | ||
| throw new Error('Please provide url to process.'); |
There was a problem hiding this comment.
darielnoel
reviewed
Nov 26, 2024
| this.chunkOptions = fields.chunkOptions; | ||
| this.embeddings = fields.embeddings; | ||
| this.vectorStore = fields.vectorStore; | ||
| this.llm = fields.llm; |
There was a problem hiding this comment.
Let's called it llmInstance ... To speak the same "language" of the KaibanJS framework
darielnoel
reviewed
Nov 26, 2024
| this.chunkOptions = fields.chunkOptions; | ||
| this.embeddings = fields.embeddings; | ||
| this.vectorStore = fields.vectorStore; | ||
| this.llm = fields.llm; |
There was a problem hiding this comment.
Let's called it llmInstance ... To speak the same "language" of the KaibanJS framework
darielnoel
reviewed
Nov 26, 2024
| this.llm = | ||
| options.llm || | ||
| new ChatOpenAI({ | ||
| model: 'gpt-4', |
There was a problem hiding this comment.
Can we use the 4o-mini as the default instead?
darielnoel
reviewed
Nov 26, 2024
|
|
||
| this.loaders = { | ||
| string: (source) => new TextInputLoader(source), | ||
| // text: source => new TextLoader(source), |
There was a problem hiding this comment.
Let's remove this from here if we are not using it yet
…te query in tool stories
…nd error handling
…F and textfile search exports
…nd restore replace plugin functionality
…ist for improved compatibility
…de.js and Browser environments refactor(pdf-search): rename PDF File Analyzer to PDF File Searcher for consistency refactor(textfile-search): rename Text File Analyzer to Text File Searcher for consistency refactor(textfile-search): update task description and expected output for semantic search fix(website-search): add installation instructions for cheerio dependency chore: add server.js to .gitignore
…ant exclusions for pdf-parse and pdfjs-dist
…in Rollup configuration
… with 'fs/promises'
…cies in README files
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
RAG-based Search Tools Release 🚀
This PR introduces four new powerful search tools leveraging Retrieval-Augmented Generation (RAG) technology.
What's New
SimpleRAG: Foundational RAG implementation with langchain componentsWebsiteSearch: Semantic search capabilities for web contentPDFSearch: Comprehensive PDF document analysis toolTextFileSearch: Optimized plain text document searchKey Features
Testing
Documentation
Dependencies
Added:
cheeriofor HTML parsingpdf-parsefor Node.js PDF processingpdfjs-distfor browser PDF processingRelated Issues
Closes #141