问题
I am using Spring Integration Streaming Inbound Channel Adapter, to get stream from remote SFTP and parse every lines of content process.
I use :
IntegrationFlows.from(Sftp.inboundStreamingAdapter(template)
.filter(remoteFileFilter)
.remoteDirectory("test_dir"),
e -> e.id("sftpInboundAdapter")
.autoStartup(true)
.poller(Pollers.fixedDelay(fetchInt)))
.handle(Files.splitter(true, true))
....
And it can work now. But I can only get file from test_dir
directory, but I need to recursively get files from this dir and sub-directory and parse every line.
I noticed that the Inbound Channel Adapter
which is Sftp.inboundAdapter(sftpSessionFactory).scanner(...)
. It can scan sub-directory. But I didn't see anything for Streaming Inbound Channel Adapter
.
So, how can I implement the 'recursively get files from dir' in Streaming Inbound Channel Adapter
?
Thanks.
回答1:
You can use a two outbound gateways - the first doing ls -R
(recursive list); split the result and use a gateway configured with mget -stream
to get each file.
EDIT
@SpringBootApplication
public class So60987851Application {
public static void main(String[] args) {
SpringApplication.run(So60987851Application.class, args);
}
@Bean
IntegrationFlow flow(SessionFactory<LsEntry> csf) {
return IntegrationFlows.from(() -> "foo", e -> e.poller(Pollers.fixedDelay(5_000)))
.handle(Sftp.outboundGateway(csf, Command.LS, "payload")
.options(Option.RECURSIVE, Option.NAME_ONLY)
// need a more robust metadata store for persistence, unless the files are removed
.filter(new SftpPersistentAcceptOnceFileListFilter(new SimpleMetadataStore(), "test")))
.split()
.log()
.enrichHeaders(headers -> headers.headerExpression("fileToRemove", "'foo/' + payload"))
.handle(Sftp.outboundGateway(csf, Command.GET, "'foo/' + payload")
.options(Option.STREAM))
.split(new FileSplitter())
.log()
// instead of a filter, we can remove the remote file.
// but needs some logic to wait until all lines read
// .handle(Sftp.outboundGateway(csf, Command.RM, "headers['fileToRemove']"))
// .log()
.get();
}
@Bean
CachingSessionFactory<LsEntry> csf(DefaultSftpSessionFactory sf) {
return new CachingSessionFactory<>(sf);
}
@Bean
DefaultSftpSessionFactory sf() {
DefaultSftpSessionFactory sf = new DefaultSftpSessionFactory();
sf.setHost("10.0.0.8");
sf.setUser("gpr");
sf.setPrivateKey(new FileSystemResource(new File("/Users/grussell/.ssh/id_rsa")));
sf.setAllowUnknownKeys(true);
return sf;
}
}
来源:https://stackoverflow.com/questions/60987851/streaming-from-remote-sftp-directories-and-sub-directories-with-spring-integrati