Spring Batch - Reference Documentation

While open source software projects and associated communities have focused greater attention on web-based and SOA messaging-based architecture frameworks, there has been a notable lack of focus on reusable architecture frameworks to accommodate Java-based batch processing needs, despite continued needs to handle such processing within enterprise IT environments. The lack of a standard, reusable batch architecture has resulted in the proliferation of many one-off, in-house solutions developed within client enterprise IT functions.

SpringSource and Accenture have collaborated to change this. Accenture's hands-on industry and technical experience in implementing batch architectures, SpringSource's depth of technical experience, and Spring's proven programming model together mark a natural and powerful partnership to create high-quality, market relevant software aimed at filling an important gap in enterprise Java. Both companies are also currently working with a number of clients solving similar problems developing Spring-based batch architecture solutions. This has provided some useful additional detail and real-life constraints helping to ensure the solution can be applied to the real-world problems posed by clients. For these reasons and many more, SpringSource and Accenture have teamed to collaborate on the development of Spring Batch.

Accenture has contributed previously proprietary batch processing architecture frameworks, based upon decades worth of experience in building batch architectures with the last several generations of platforms, (i.e., COBOL/Mainframe, C++/Unix, and now Java/anywhere) to the Spring Batch project along with committer resources to drive support, enhancements, and the future roadmap.

The collaborative effort between Accenture and SpringSource aims to promote the standardization of software processing approaches, frameworks, and tools that can be consistently leveraged by enterprise users when creating batch applications. Companies and government agencies desiring to deliver standard, proven solutions to their enterprise IT environments will benefit from Spring Batch.

A typical batch program generally reads a large number of records from a database, file, or queue, processes the data in some fashion, and then writes back data in a modified form. Spring Batch automates this basic batch iteration, providing the capability to process similar transactions as a set, typically in an offline environment without any user interaction. Batch jobs are part of most IT projects and Spring Batch is the only open source framework that provides a robust, enterprise-scale solution.

Business Scenarios

Technical Objectives

Spring Batch is designed with extensibility and a diverse group of end users in mind. The figure below shows a sketch of the layered architecture that supports the extensibility and ease of use for end-user developers.

Figure 1.1: Spring Batch Layered Architecture

This layered architecture highlights three major high level components: Application, Core, and Infrastructure. The application contains all batch jobs and custom code written by developers using Spring Batch. The Batch Core contains the core runtime classes necessary to launch and control a batch job. It includes things such as a JobLauncher, Job, and Step implementations. Both Application and Core are built on top of a common infrastructure. This infrastructure contains common readers and writers, and services such as the RetryTemplate, which are used both by application developers(ItemReader and ItemWriter) and the core framework itself. (retry)

A job is represented by a Spring bean that implements the Job interface and contains all of the information necessary to define the operations performed by a job. A job configuration is typically contained within a Spring XML configuration file and the job's name is determined by the "id" attribute associated with the job configuration bean. The job configuration contains

A default simple implementation of the Job interface is provided by Spring Batch in the form of the SimpleJob class which creates some standard functionality on top of Job, namely a standard execution logic that all jobs should utilize. In general, all jobs should be defined using a bean of type SimpleJob:

  <bean id="footballJob"
        class="org.springframework.batch.core.job.SimpleJob">
    <property name="steps">
      <list>
        <!-- Step Bean details ommitted for clarity -->
        <bean id="playerload" parent="simpleStep" />
        <bean id="gameLoad" parent="simpleStep" />
        <bean id="playerSummarization" parent="simpleStep" />
      </list>
    </property>
    <property name="restartable" value="true" />
  </bean>

A JobInstance refers to the concept of a logical job run. Let's consider a batch job that should be run once at the end of the day, such as the 'EndOfDay' job from the diagram above. There is one 'EndOfDay' Job, but each individual run of the Job must be tracked separately. In the case of this job, there will be one logical JobInstance per day. For example, there will be a January 1st run, and a January 2nd run. If the January 1st run fails the first time and is run again the next day, it's still the January 1st run. (Usually this corresponds with the data its processing as well, meaning the January 1st run processes data for January 1st, etc) That is to say, each JobInstance can have multiple executions. (JobExecution is discussed in more detail below) and only one JobInstance corresponding to a particular Job can be running at a given time. The definition of a JobInstance has absolutely no bearing on the data the will be loaded. It is entirely up to the ItemReader implementation used to determine how data will be loaded. For example, in the EndOfDay scenario, there may be a column on the data that indicates the 'effective date' or 'schedule date' to which the data belongs. So, the January 1st run would only load data from the 1st, and the January 2nd run would only use data from the 2nd. Because this determination will likely be a business decision, it is left up to the ItemReader to decide. What using the same JobInstance will determine, however, is whether or not the 'state' (i.e. the ExecutionContext, which is discussed below) from previous executions will be used. Using a new JobInstance will mean 'start from the beginning' and using an existing instance will generally mean 'start from where you left off'.

Having discussed JobInstance and how it differs from Job, the natural question to ask is: "how is one JobInstance distinguished from another?" The answer is: JobParameters. JobParameters are any set of parameters used to start a batch job, which can be used for identification or even as reference data during the run. In the example above, where there are two instances, one for January 1st, and another for January 2nd, there is really only one Job, one that was started with a job parameter of 01-01-2008 and another that was started with a parameter of 01-02-2008. Thus, the contract can be defined as: JobInstance = Job + JobParameters. This allows you to effectively control how you define a JobInstance, since you control what parameters are passed in.

A JobExecution refers to the technical concept of a single attempt to run a Job. An execution may end in failure or success, but the JobInstance corresponding to a given execution will not be marked as complete unless the execution completes successfully. For instance, if we have a JobInstance of the EndOfDay job for 01-01-2008, as described above, that fails to successfully complete its work the first time it is run, when we attempt to run it again (with the same job parameters of 01-01-2008), a new job execution will be created.

A Job defines what a job is and defines how it is to be executed, and JobInstance is a purely organization object to group executions together, primarily to enable correct restart. A JobExecution, however, is the primary storage mechanism for what actually happened during a run, and as such contains many more properties that must be controlled and persisted:

These properties are important because they will be persisted and can be used to completely determine the status of an execution. For example, if the EndOfDay job for 01-01 is executed at 9:00 PM, and fails at 9:30, the following entries will be made in the batch meta data tables:

Now that the job has failed, let's assume that it took the entire course of the night for the problem to be determined, so that the 'batch window' is now closed. Assuming the window starts at 9:00 PM, the job will be kicked off again for 01-01, starting where it left off and completing successfully at 9:30. Because it's now the next day, the 01-02 job must be run as well, which is kicked off just afterwards at 9:31, and completes in it's normal one hour time at 10:30. There is no requirement that one JobInstance be kicked off after another, unless there is potential for the two jobs to attempt to access the same data, causing issues with locking at the database level. It is entirely up to the scheduler to determine when to run. Since they're separate JobInstances, Spring Batch will make no attempt to stop them from being run concurrently. (Attempting to run the same JobInstance while another is already running will result in a JobExecutionAlreadyRunningException being thrown) There should now be an extra entry in both the JobInstance and JobParameters tables, and two extra entries in the JobExecution table:

A Step contains all of the information necessary to define and control the actual batch processing. This is a necessarily vague description because the contents of any given Step are at the discretion of the developer writing a Job. A Step can be as simple or complex as the developer desires. A simple Step might load data from a file into the database, requiring little or no code. (depending upon the implementations used) A more complex Step may have complicated business rules that are applied as part of the processing.

Steps are defined by instantiating implementations of the Step interface. Two step implementation classes are available in the Spring Batch framework, and they are each discussed in detail in Chatper 4 of this guide. For most situations, the ItemOrientedStep implementation is sufficient, but for situations where only one call is needed, such as a stored procedure call or a wrapper around existing script, a TaskletStep may be a better option.

A StepExecution represents a single attempt to execute a Step. Using the example from JobExecution, if there is a JobInstance for the "EndOfDayJob", with JobParameters of "01-01-2008" that fails to successfully complete its work the first time it is run, when it is executed again, a new StepExecution will be created. Each of these step executions may represent a different invocation of the batch framework, but they will all correspond to the same JobInstance, just as multiple JobExecutions belong to the same JobInstance.

Step executions are represented by objects of the StepExecution class. Each execution contains a reference to its corresponding step and JobExecution, and transaction related data such as commit and rollback count and start and end times. Additionally, each step execution will contain an ExecutionContext, which contains any data a developer needs persisted across batch runs, such as statistics or state information needed to restart. The following is a listing of the properties for StepExecution:

An ExecutionContext represents a collection of key/value pairs that are persisted and controlled by the framework in order to allow developers a place to store persistent state that is scoped to a StepExecution. For those familiar with Quartz, it is very similar to JobDataMap. The best usage example is restart. Using flat file input as an example, while processing individual lines, the framework periodically persists the ExecutionContext at commit points. This allows the ItemReader to store its state in case a fatal error occurs during the run, or even if the power goes out. All that is needed is to put the current number of lines read into the context, and the framework will do the rest:

executionContext.putLong(getKey(LINES_READ_COUNT), reader.getPosition());

The call above will store the current number of lines read into the ExecutionContext. It should be made just before the framework commits. Being notified before a commit requires one of the various StepListeners, or an ItemStream, which are discussed in more detail later in this guide. When the ItemReader is opened, it can check to see if it has any stored state in the context, and initialize itself from there:

  if (executionContext.containsKey(getKey(LINES_READ_COUNT))) {
    log.debug("Initializing for restart. Restart data is: " + executionContext);

    long lineCount = executionContext.getLong(getKey(LINES_READ_COUNT));

    LineReader reader = getReader();

    Object record = "";
    while (reader.getPosition() < lineCount && record != null) {
       record = readLine();
    }
  }

The ExecutionContext can also be used for startistics that need to be persisted about the run itself. For example, if a flat file contains orders for processing that exist across multiple lines, it may be necessary to store how many orders have been processed (which is much different from than the number of lines read) so that an email can be sent at the end of the Step with the total orders processed in the body. The framework handles storing this for the developer, in order to correctly scope it with an individual JobInstance. It can be very difficult to know whether an existing ExecutionContext should be used or not. For example, using the 'EndOfDay' example from above, when the 01-01 run starts again for the second time, the framework recognizes that it is the same JobInstance and on an individual Step basis, pulls the ExecutionContext out of the database and hands it as part of the StepExecution to the Step itself. Conversely, for the 01-02 run the framework recognizes that it is a different instance, so an empty context must be handed to the Step. There are many of these types of determinations that the framework makes for the developer to ensure the state is given to them at the correct time. It is also important to note that exactly one ExecutionContext exists per StepExecution at any given time. Clients of the ExecutionContext should be careful because this creates a shared keyspace, so care should be taken when putting values in to ensure no data is overwritten, however, the Step stores absolutely no data in the context, so there is no way to adversely affect the framework.

When working with flat files in Spring Batch, regardless of whether it is for input or output, one of the most important classes is the FieldSet. Many architectures and libraries contain abstractions for helping you read in from a file, but they usually return a String or an array of Strings. This really only gets you halfway there. A FieldSet is Spring Batch’s abstraction for enabling the binding of fields from a file resource. It allows developers to work with file input in much the same way as they would work with database input. A FieldSet is conceptually very similar to a Jdbc ResultSet. FieldSets only require one argument, a String array of tokens. Optionally, you can also configure in the names of the fields so that the fields may be accessed either by index or name as patterned after ResultSet. In code it means it's as simple as:

  String[] tokens = new String[]{"foo", "1", "true"};
  FieldSet fs = new DefaultFieldSet(tokens);
  String name = fs.readString(0);
  int value = fs.readInt(1);
  boolean booleanValue = fs.readBoolean(2);

There are many more options on the FieldSet interface, such as Date, long, BigDecimal, etc. The biggest advantage of the FieldSet is that it provides consistent parsing of flat file input. Rather than each batch job parsing differently in potentially unexpected ways, it can be consistent, both when erroring out due to a format exception, or when doing simple data conversions.

A flat file is any type of file that contains at most two-dimensional (tabular) data. Reading flat files in the Spring Batch framework is facilitated by the class FlatFileItemReader, which provides basic functionality for reading and parsing flat files. FlatFileItemReader class has several properties. The three most important of these properties are Resource, FieldSetMapper and LineTokenizer. The FieldSetMapper and LineTokenizer interfaces will be explored more in the next sections. The resource property represents a Spring Core Resource. Documentation explaining how to create beans of this type can be found in Spring Framework, Chapter 4.Resources. Therefore, this guide will not go into the details of creating Resource objects. A resource is used to locate, open, and close resources. It can be as simple as:

        Resource resource = new FileSystemResource("resources/trades.csv");
        

In complex batch environments the directory structures are often managed by the EAI infrastructure where drop zones for external interfaces are established for moving files from ftp locations to batch processing locations and vice versa. File moving utilities are beyond the scope of the spring batch architecture but it is not unusual for batch job streams to include file moving utilities as steps in the job stream. It's sufficient to know that the batch architecture only needs to know how to locate the files to be processed. Spring Batch begins the process of feeding the data into the pipe from this starting point.

The other properties in FlatFileItemReader allow you to further specify how your data will be interpreted:

  public interface FieldSetMapper {
  
    public Object mapLine(FieldSet fs);

  }

  public interface LineTokenizer {
  
    FieldSet tokenize(String line);

  }

  ID,lastName,firstName,position,birthYear,debutYear
  "AbduKa00,Abdul-Jabbar,Karim,rb,1974,1996",
  "AbduRa00,Abdullah,Rabih,rb,1975,1999",
  "AberWa00,Abercrombie,Walter,rb,1959,1982",
  "AbraDa00,Abramowicz,Danny,wr,1945,1967",
  "AdamBo00,Adams,Bob,te,1946,1969",
  "AdamCh00,Adams,Charlie,wr,1979,2003"        

  public class Player implements Serializable {
        
  private String ID; 
  private String lastName; 
  private String firstName; 
  private String position; 
  private int birthYear; 
  private int debutYear;
        
    public String toString() {
                
      return "PLAYER:ID=" + ID + ",Last Name=" + lastName + 
        ",First Name=" + firstName + ",Position=" + position + 
        ",Birth Year=" + birthYear + ",DebutYear=" + 
        debutYear;
    }
   
    // setters and getters...
  }
          

  protected static class PlayerFieldSetMapper implements FieldSetMapper {
    public Object mapLine(FieldSet fieldSet) {
      Player player = new Player();

      player.setID(fieldSet.readString(0));
      player.setLastName(fieldSet.readString(1));
      player.setFirstName(fieldSet.readString(2)); 
      player.setPosition(fieldSet.readString(3));
      player.setBirthYear(fieldSet.readInt(4));
      player.setDebutYear(fieldSet.readInt(5));

      return player;
    }
  }    

  FlatFileItemReader itemReader = new FlatFileItemReader();
  itemReader.setResource = new FileSystemResource("resources/players.csv");
  //DelimitedLineTokenizer defaults to comma as it's delimiter
  itemReader.setLineTokenizer(new DelimitedLineTokenizer());
  itemReader.setFieldSetMapper(new PlayerFieldSetMapper());
  itemReader.read();

  tokenizer.setNames(new String[] {"ID", "lastName","firstName","position","birthYear","debutYear"}); 
          

    public class PlayerMapper implements FieldSetMapper {
        public Object mapLine(FieldSet fs) {
                        
           if(fs == null){
              return null;
           }
                        
           Player player = new Player();
           player.setID(fs.readString("ID"));
           player.setLastName(fs.readString("lastName"));
           player.setFirstName(fs.readString("firstName"));
           player.setPosition(fs.readString("position"));
           player.setDebutYear(fs.readInt("debutYear"));
           player.setBirthYear(fs.readInt("birthYear"));
                        
           return player;
        }

   }        

  <bean id="fieldSetMapper"
        class="org.springframework.batch.io.file.mapping.BeanWrapperFieldSetMapper">
    <property name="prototypeBeanName" value="player" />
  </bean>

  <bean id="person"
        class="org.springframework.batch.sample.domain.Player"
        scope="prototype" />

  UK21341EAH4121131.11customer1
  UK21341EAH4221232.11customer2
  UK21341EAH4321333.11customer3
  UK21341EAH4421434.11customer4
  UK21341EAH4521535.11customer5

  <bean id="fixedLengthLineTokenizer"
        class="org.springframework.batch.io.file.transform.FixedLengthTokenizer">
    <property name="names" value="ISIN, Quantity, Price, Customer" />
    <property name="columns" value="1-12, 13-15, 16-20, 21-29" />
  </bean>

  HEA;0013100345;2007-02-15
  NCU;Smith;Peter;;T;20014539;F
  BAD;;Oak Street 31/A;;Small Town;00235;IL;US
  SAD;Smith, Elizabeth;Elm Street 17;;Some City;30011;FL;United States
  BIN;VISA;VISA-12345678903
  LIT;1044391041;37.49;0;0;4.99;2.99;1;45.47
  LIT;2134776319;221.99;5;0;7.99;2.99;1;221.87
  SIN;UPS;EXP;DELIVER ONLY ON WEEKDAYS
  FOT;2;2;267.34

  <bean id="orderFileDescriptor"
        class="org.springframework.batch.io.file.transform.PrefixMatchingCompositeLineTokenizer">
    <property name="tokenizers">
     <map>
      <entry key="HEA" value-ref="headerRecordDescriptor" />
      <entry key="FOT" value-ref="footerRecordDescriptor" />
      <entry key="BCU" value-ref="businessCustomerLineDescriptor" />
      <entry key="NCU" value-ref="customerLineDescriptor" />
      <entry key="BAD" value-ref="billingAddressLineDescriptor" />
      <entry key="SAD" value-ref="shippingAddressLineDescriptor" />
      <entry key="BIN" value-ref="billingLineDescriptor" />
      <entry key="SIN" value-ref="shippingLineDescriptor" />
      <entry key="LIT" value-ref="itemLineDescriptor" />
      <entry key="" value-ref="defaultLineDescriptor" />
     </map>
    </property>
  </bean>

Writing out to flat files has the same problems and issues that reading in from a file must overcome. It must be able to write out in either delimited or fixed length formats in a transactional manner.

  public interface LineAggregator {

    public String aggregate(FieldSet fieldSet);

  }

  public interface FieldSetCreator {

    FieldSet mapItem(Object data);

  }

  public void write(Object data) throws Exception {
    FieldSet fieldSet = fieldSetCreator.mapItem(data);
    getOutputState().write(lineAggregator.aggregate(fieldSet) + LINE_SEPARATOR);
  }

  <bean id="itemWriter"
        class="org.springframework.batch.io.file.FlatFileItemWriter">
    <property name="resource"
              value="file:target/test-outputs/20070122.testStream.multilineStep.txt" />
    <property name="fieldSetCreator">
      <bean class="org.springframework.batch.io.file.mapping.PassThroughFieldSetMapper"/>
    </property>
  </bean>

The StaxEventItemReader configuration provides a typical setup for the processing of records from an XML input stream. First, lets examine a set of XML records that the StaxEventItemReader can process.

<?xml version="1.0" encoding="UTF-8"?>
<records>
  <trade xmlns="http://springframework.org/batch/sample/io/oxm/domain">
    <isin>XYZ0001</isin>
    <quantity>5</quantity>
    <price>11.39</price>
    <customer>Customer1</customer>
  </trade>
  <trade xmlns="http://springframework.org/batch/sample/io/oxm/domain">
    <isin>XYZ0002</isin>
    <quantity>2</quantity>
    <price>72.99</price>
    <customer>Customer2c</customer>
  </trade>
  <trade xmlns="http://springframework.org/batch/sample/io/oxm/domain">
    <isin>XYZ0003</isin>
    <quantity>9</quantity>
    <price>99.99</price>
    <customer>Customer3</customer>
  </trade>
</records>

To be able to process the XML records we need the following:

<property name="itemReader">
     <bean class="org.springframework.batch.io.xml.StaxEventItemReader">
         <property name="fragmentRootElementName"  value="trade" />
         <property name="resource" value="data/staxJob/input/20070918.testStream.xmlFileStep.xml" />
         <property name="fragmentDeserializer">
             <bean class="org.springframework.batch.io.xml.oxm.UnmarshallingEventReaderDeserializer">
                 <constructor-arg>
                     <bean class="org.springframework.oxm.xstream.XStreamMarshaller">
                         <property name="aliases" ref="aliases" />
                     </bean>
                 </constructor-arg>
             </bean>
         </property>
     </bean>
</property>
    

Notice that in this example we have chosen to use an XStreamMarshaller that requires an alias passed in as a map with the first key and value being the name of the fragment (i.e. root element) and the object type to bind. Then, similar to a FieldSet, the names of the other elements that map to fields within the object type are described as key/value pairs in the map. In the configuration file we can use a spring configuration utility to describe the required alias as follows:

        <util:map id="aliases">
                <entry key="trade"
                        value="org.springframework.batch.sample.domain.Trade" />
                <entry key="isin" value="java.lang.String" />
                <entry key="quantity" value="long" />
                <entry key="price" value="java.math.BigDecimal" />
                <entry key="customer" value="java.lang.String" />
        </util:map>
        

On input the reader reads the XML resource until it recognizes a new fragment is about to start (by matching the tag name by default). The reader creates a standalone XML document from the fragment (or at least makes it appear so) and passes the document to a deserializer (typically a wrapper around a Spring OXM Unmarshaller) to map the XML to a Java object.

In summary, if you were to see this in scripted code like Java the injection provided by the spring configuration would look something like the following:

      StaxEventItemReader xmlStaxEventItemReader = new StaxEventItemReader()
      Resource resource = new ByteArrayResource(xmlResource.getBytes()) 

      Map aliases = new HashMap();
      aliases.put("trade","org.springframework.batch.sample.domain.Trade");
      aliases.put("isin","java.lang.String");
      aliases.put("quantity","long");
      aliases.put("price","java.math.BigDecimal");
      aliases.put("customer","java.lang.String");
      Marshaller marshaller = new XStreamMarshaller();
      marshaller.setAliases(aliases);
      xmlStaxEventItemReader.setFragmentDeserializer(new UnmarshallingEventReaderDeserializer(marshaller));
      xmlStaxEventItemReader.setResource(resource);
      xmlStaxEventItemReader.setFragmentRootElementName("trade");
      xmlStaxEventItemReader.open(new ExecutionContext());

      boolean hasNext = true
      
      while (hasNext) {
        trade = xmlStaxEventItemReader.read();
        if (trade == null) {
                hasNext = false;
        } else {
                println trade;
        }
      }

Output works symmetrically to input. The StaxEventItemWriter needs a Resource, a serializer, and a rootTagName. A Java object is passed to a serializer (typically a wrapper around Spring OXM Marshaller) which writes to a Resource using a custom event writer that filters the StartDocument and EndDocument events produced for each fragment by the OXM tools. We'll show this in an example using the MarshallingEventWriterSerializer. The Spring configuration for this setup looks as follows:

<bean class="org.springframework.batch.item.xml.StaxEventItemWriter" id="tradeStaxWriter">
  <property name="resource"value="file:target/test-outputs/20070918.testStream.xmlFileStep.output.xml" />
  <property name="serializer" ref="tradeMarshallingSerializer" />
  <property name="rootTagName" value="trades" />
  <property name="overwriteOutput" value="true" />
</bean>

The configuration sets up the three required properties and optionally sets the overwriteOutput=true, mentioned earlier in the chapter for specifying whether an existing file can be overwritten. The TradeMarshallingSerializer is configured as follows:

<bean class="org.springframework.batch.item.xml.oxm.MarshallingEventWriterSerializer" id="tradeMarshallingSerializer">
  <constructor-arg>
   <bean class="org.springframework.oxm.xstream.XStreamMarshaller">
     <property name="aliases" ref="aliases" />
   </bean>
  </constructor-arg>
</bean>

To summarize with a Java example, the following code illustrates all of the points discussed, demonstrating the programmatic setup of the required properties.

     StaxEventItemWriter staxItemWriter = new StaxEventItemWriter()
     FileSystemResource resource = new FileSystemResource(File.createTempFile("StaxEventWriterOutputSourceTests", "xml"))

     Map aliases = new HashMap();
     aliases.put("trade","org.springframework.batch.sample.domain.Trade");
     aliases.put("isin","java.lang.String");
     aliases.put("quantity","long");
     aliases.put("price","java.math.BigDecimal");
     aliases.put("customer","java.lang.String");
     XStreamMarshaller marshaller = new XStreamMarshaller()
     marshaller.setAliases(aliases)

     MarshallingEventWriterSerializer tradeMarshallingSerializer = new MarshallingEventWriterSerializer(marshaller)

     staxItemWriter.setResource(resource)
     staxItemWriter.setSerializer(tradeMarshallingSerializer)
     staxItemWriter.setRootTagName("trades")
     staxItemWriter.setOverwriteOutput(true)

     ExecutionContext executionContext = new ExecutionContext()
     staxItemWriter.open(executionContext)
     Trade trade = new Trade()
     trade.isin = "XYZ0001"
     trade.quantity =5 
     trade.price = 11.39 
     trade.customer = "Customer1"
     println trade
     staxItemWriter.write(trade)
     staxItemWriter.flush()

For a complete example configuration of XML input and output and a corresponding Job see the sample xmlStaxJob.

Using a database cursor is generally the default approach of most batch developers. This is because it is the database's solution to the problem of 'streaming' relational data. The Java ResultSet class is essentially an object orientated mechanism for manipulating a cursor. A ResultSet maintains a cursor to the current row of data. Calling next on a ResultSet moves this cursor to the next row. Spring Batch cursor based ItemReaders open the a cursor on initialization, and move the cursor forward one row for every call to read, returning a mapped object that can be used for processing. The close method will then be called to ensure all resources are freed up. The Spring core JdbcTemplate gets around this problem by using the callback pattern to completely map all rows in a ResultSet and close before returning control back to the method caller. However, in batch this must wait until the step is complete. Below is a generic diagram of how a cursor based ItemReader works, and while a SQL statement is used as an example since it is so widely known, any technology could implement the basic approach:

The example illustrates the basic pattern. Given a 'FOO' table, which has three columns: ID, NAME, and BAR, select all rows with an ID greater than one but less than 7. This puts the beginning of the cursor (row 1) on ID 2. The result of this row should be a completely mapped Foo object, calling read() again, moves the cursor to the next row, which is the Foo with an ID of 3.

CREATE TABLE CUSTOMER (
 ID BIGINT IDENTITY PRIMARY KEY,  
 NAME VARCHAR(45),
 CREDIT FLOAT
);

public class CustomerCreditRowMapper implements RowMapper {

 public static final String ID_COLUMN = "id";
 public static final String NAME_COLUMN = "name";
 public static final String CREDIT_COLUMN = "credit";

 public Object mapRow(ResultSet rs, int rowNum) throws SQLException {
        CustomerCredit customerCredit = new CustomerCredit();

        customerCredit.setId(rs.getInt(ID_COLUMN));
        customerCredit.setName(rs.getString(NAME_COLUMN));
        customerCredit.setCredit(rs.getBigDecimal(CREDIT_COLUMN));

        return customerCredit;
 }

}

//For simplicity sake, assume a dataSource has already been obtained
JdbcTemplate jdbcTemplate = new JdbcTemplate(dataSource);
List customerCredits = jdbcTemplate.query("SELECT ID, NAME, CREDIT from CUSTOMER", new CustomerCreditRowMapper());

JdbcCursorItemReader itemReader = new JdbcCursorItemReader();
itemReader.setDataSource(dataSource);
itemReader.setSql("SELECT ID, NAME, CREDIT from CUSTOMER");
itemReader.setMapper(new CustomerCreditRowMapper());
int counter = 0;
ExecutionContext executionContext = new ExecutionContext();
itemReader.open(executionContext);
Object customerCredit = new Object();
while(customerCredit != null){
  customerCredit = itemReader.read();
  counter++;
}
itemReader.close(executionContext);

  HibernateCursorItemReader itemReader = new HibernateCursorItemReader();
  itemReader.setQueryString("from CustomerCredit");
  //For simplicity sake, assume sessionFactory already obtained.
  itemReader.setSessionFactory(sessionFactory);
  itemReader.setUseStatelessSession(true);
  int counter = 0;
  ExecutionContext executionContext = new ExecutionContext();
  itemReader.open(executionContext);
  Object customerCredit = new Object();
  while(customerCredit != null){
    customerCredit = itemReader.read();
    counter++;
  }
  itemReader.close(executionContext);

In the previous section, Cursor based database input was discussed. However, it isn't the only option. Many database vendors, such as DB2, have extremely pessimistic locking strategies that can cause issues if the table being read also needs to be used by other portions of the online application. Furthermore, opening cursors over extremely large datasets can cause issues on certain vendors. Therefore, many projects prefer to use a 'Driving Query' approach to reading in data. This approach works by iterating over keys, rather than the entire object that needs to be returned, as the following example illustrates:

As you can see, this example uses the same 'FOO' table as was used in the cursor based example. However, rather than selecting the entire row, only the ID's were selected in the SQL statement. So, rather than a FOO object being returned from read, an Integer will be returned. This number can then be used to query for the 'details', which is a complete Foo object:

As you can see, an existing DAO can be used to obtain a full 'Foo' object using the key obtained from the driving query. In Spring Batch, driving query style input is implemented with a DrivingQueryItemReader, which has only one dependency: a KeyCollector

  public interface KeyCollector {

   List retrieveKeys(ExecutionContext executionContext);

   void updateContext(Object key, ExecutionContext executionContext);
  }

  ExecutionContext executionContext = new ExecutionContext();
  List keys = keyStrategy.retrieveKeys(executionContext);
  //Assume keys contains 1 through 1,000
  keyStrategy.updateContext(new Long(500), executionContext);
  keys = keyStrategy.retrieveKeys(executionContext);
  //keys should now contains 500 through 1,000

  SingleColumnJdbcKeyCollector keyCollector = new SingleColumnJdbcKeyCollector(getJdbcTemplate(),
  "SELECT ID from T_FOOS order by ID");

  keyCollector.setRestartSql("SELECT ID from T_FOOS where ID > ? order by ID");

  ExecutionContext executionContext = new ExecutionContext();

  List keys = keyStrategy.retrieveKeys(new ExecutionContext());

  for (int i = 0; i < keys.size(); i++) {
    System.out.println(keys.get(i));
  }

1
2
3
4
5

  SingleColumnJdbcKeyCollector keyCollector = new SingleColumnJdbcKeyCollector(getJdbcTemplate(),
  "SELECT ID from T_FOOS order by ID");

  keyCollector.setRestartSql("SELECT ID from T_FOOS where ID > ? order by ID");

  ExecutionContext executionContext = new ExecutionContext();  

  keyStrategy.updateContext(new Long(3), executionContext);
  
  List keys = keyStrategy.retrieveKeys(executionContext);

  for (int i = 0; i < keys.size(); i++) {
    System.out.println(keys.get(i));
  }

4
5

  keyStrategy.updateContext(new Long(3), executionContext);