I implemented an AI agent training app where users upload custom data from training AI models. I need to write to disk before I can access the training data directory which is working perfectly. But when I moved to AWS S3 bucket, and pass the bucket URL to the function below as directoryPath, I get the error;
{"code":"ENOENT","errno":-2,"level":"info","message":"Error creating and updating pinecone database store... ENOENT: no such file or directory, scandir 's3://aiagent/1691381132249_i78q8fx3nw/'","path":"s3://aiagent/1691381132249_i78q8fx3nw/","stack":"Error: ENOENT: no such file or directory, scandir 's3://aiagent/1691381132249_i78q8fx3nw/'","syscall":"scandir"}
const loader = new DirectoryLoader(directoryPath, {
".txt": (path) => new TextLoader(path),
".md": (path) => new TextLoader(path),
".pdf": (path) => new PDFLoader(path),
});
How would I use training data stored on S3 bucket to create index at Pinecone? Thanks for any help including resources I could look into