How to use a dynamic URI in From()

Question

As mentioned in Apache Camel, it allows to write dynamic URI in To(), does it allows to write dynamic URI in From(). Cause I need to call the multiple FTP locations to download the files on the basis of configuration which I am going to store it in database.

(FTPHost, FTPUser, FTPPassword, FTPSourceDir, FTPDestDir)

I will read these configuration from the DB and will pass it to the Camel route dynamically at runtime.

Example: This is the camel route example that I have to write dynamically

<Route>
    <from uri="ftp://${ftpUser}@${ftpHost}:${ftpPort}/${FTPSourceDir}?password=${ftpPassword}&delete=true"/>
    <to uri="${ftpDestinationDir}"/>
</Route>

As you see in example, I need to pass these mentioned parameters dynamically. So how to use dynamic uri in From()

Answer 1

You can read it from property file as follows,

<bean id="bridgePropertyPlaceholder" class="org.apache.camel.spring.spi.BridgePropertyPlaceholderConfigurer">
    <property name="location" value="classpath:/config/Test.properties"/>
  </bean> 

<Route>
    <from uri="ftp://{{ftpUser})@${{ftpHost}}:{{ftpPort}}/${{FTPSourceDir}}?password={{ftpPassword}}&delete=true"/>
    <to uri="{{ftpDestinationDir}}"/>
</Route>

ftpUser, ftpHost.... - all are keys declared in Test.properties

If you want to get those variables from your exchange dynamically, you cannot do it in regular way as you mentioned in your example. You have to use consumer template as follows,

Exchange exchange = consumerTemplate.receive("ftp:"+url);
producerTemplate.send("direct:uploadFileFTP",exchange );

You have to do that from a spring bean or camel producer. Consumer template will consume from given component, and that producer template will invoke direct component declared in your camel-context.xml

Note: Consumer and Producer templates are bit costly. you can inject both in spring container and let the spring handle the life cycle.

Answer 2

From camel 2.16 on-wards, we can use pollenrich component to define polling consumer like file, ftp..etc with dynamic url/parameter value like below

<route>
  <from uri="direct:start"/>
  <pollEnrich>
    <simple>file:inbox?fileName=${body.fileName}</simple>
  </pollEnrich>
  <to uri="direct:result"/>
</route>

Its awesomeeee!!!

Refer: http://camel.apache.org/content-enricher.html

Answer 3

I help a team who operates a message broker switching about a million message per day. There are over 50 destinations from which we have to poll files over all file sharing brands (FTP, SFTP, NFS/file: ...). Maintaining up to 50 deployments that each listen to a different local/remote directory is indeed an overhead compared with a single FILE connector capable of polling files at the 50 places according to the specific schedule and security settings of each... Same story for getting e-mail from pop3 and IMAP mailboxes.

In Camel, the outline of a solution is as follows:

• you have no choice but use the java DSL to configure at least the from() part of your routes with an URI that you can indeed read/build from a database or get from an admin request to initiate a new route. The XML DSL only allows injecting properties that are resolved once when the Camel context is built and never again afterwards.
• the basic idea is to start routes, let them run (listen or poll a precise resource), and then shutdown & rebuild them on demand using the Camel context APIs to manage the state of RouteDefinitions, Routes, and possibly Endpoints
• personally, I like to implement such dynamic from() instantiation on minimalist routes with just the 'from' part of the route, i.e. from(uri).to("direct:inboundQueue").routeId("myRoute"), and then define - in java or XML - a common route chunk that handles the rest of the process: from("direct:inboundQueue").process(..).etc... .to(outUri)
• I'll advise strongly to combine Camel with the Spring framework, and in particular Spring MVC (or Spring Integration HttpGateway) so that you will enjoy the ability to quickly build REST, SOAP, HTTP/JSP, or JMX bean interfaces to administer route creation, destruction, and updates within a Spring + Camel container, both nicely integrated.
• You can then declare in the Spring application context a bean that extends SpringRouteBuilder, as usual when building Camel routes with the java DSL in Spring; in the compulsory @Override configure() method implementation, you shall save your routeDefinition object built by the from(uri) method, and assign it a known String route-id with the .routeId(route-id) method; you may for instance use the route-id as a key in a Map of your route definition objects already created and started, as well as a key in your DB of URI's.
• then you extend the SpringRouteBuilder bean you have declared with new methods createRoute(route-id), updateRoute(route-id), and removeRoute(route-id); The associated route-id parameters needed for create or update will be fetched from the database or another registry, and the relevant method, running within the RouteBuilder bean, will take advantage from the getContext() facility to retrieve the current ModelCamelContext, which in turn is used to stopRoute(route-id), removeRoute(route-id), and then addRouteDefinition(here is where you need the routeDefinition object), and finally startRoute(route-id) (Note: beware of possible ghost Endpoints that would not be removed, as explained in the removeRoute() javadoc)
• your administrative interface (which typically takes the form of a Spring @Controller component/bean that handles the HTTP/REST/SOAP traffic) will indeed have an easy job to get the previously created SpringRouteBuilder extension Bean injected by Spring in the controller bean, and thus access all the necessary createRoute(route-id), updateRoute(route-id), and removeRoute(route-id) methods that you have added to the SpringRouteBuilder extension Bean.

And that works nicely. The exact implementation with all the error handling and validation code that applies is a bit too much code to be posted here, but you have all the links to relevant "how to's" in the above.

Answer 4

I think you can implement your requirement within a Camel route.

Because you want to poll multiple FTP sites you'll have to somehow trigger this process. Maybe you could do this based on a Quartz2 timer. Once triggered you could read the configured FTP sites from your database.

In order to poll the given FTP sites you can use the Content Enricher pattern to poll (see: pollEnrich) a dynamically evaluated URI.

Your final basic route may look something like this (pseudocode):

from("quarz...")
to("sql...")
pollEnrich("ftp...")
...

Answer 5

Use Camel endpoint with spring spel expression.

Set up a Camel endpoint in the context so it can be accessed from any bean:

        <camelContext id="camel" xmlns="http://camel.apache.org/schema/spring">
            <endpoint id="inventoryQueue" uri="#{config.jms.inventoryQueueFromUri}"/>
        </camelContext>

Now you can reference the inventoryQueue endpoint within the `@Consume` annotation as follows:
        @org.apache.camel.Consume(ref = "inventoryQueue")
        public void updateInventory(Inventory inventory) {
            // update
        }

    Or:
    <route>
        <from ref="inventoryQueue"/>
        <to uri="jms:incomingOrders"/>
    </route>