kmrshell_mpi − map-reduce by shell command pipeline
kmrshell_mpi -n m_num[:r_num] -m mapper [-k kvgenerator] [-r reducer] inputfile
kmrshell starts map-reduce shell processes (for "streaming"). It runs MPI program as mapper and reducer. The output files of the mapper are parsed by a key-value generator program (specified by ’-k’ option) to generate key-value pairs and then they are shuffled and written to files on each rank with ’key’-named as a text file whose line represents a key-value separated by a space. The file is passed to the reducer as the last parameter.
The following options are supported:
-n m_num[:r_num]
......
-m mapper
Specifies a mapper program. The program can have arguments, where they are separated by a whitespace.
Mapper specification: A mapper reads data from STDIN and outputs key-value data to STDOUT. Output data is a sequence of a line "key value\n", where the fields are separated by a whitespace.
-k kvgenerator
......
-r reducer
Specifies a reducer program. The program can have arguments, where they are separated by a whitespace.
Reducer Specification: A reducer reads key-value data from STDIN and outputs the result to STDOUT. Input data is a sequence of a line "key value\n", where the fields are separated by a whitespace. Lines with the same keys constitutes consecutive lines.