KMR
Classes | Macros | Functions
kmrshell_mpi.c File Reference

kmrshell_mpi is command line version of KMR and it runs a MapReduce program whose mapper and reducers are user specified MPI programs. More...

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <stdarg.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/param.h>
#include <dirent.h>
#include <unistd.h>
#include <mpi.h>
#include "kmr.h"

Go to the source code of this file.

Classes

struct  cmdinfo
 

Macros

#define ARGSIZ   8
 
#define ARGSTRLEN   (8 * 1024)
 
#define COMMANDLEN   1024
 
#define DEFAULT_MPI   2
 
#define LINELEN   32767
 

Functions

static int add_command_kv (KMR_KVS *, int, char **, char *, int)
 
static int delete_file (const struct kmr_kv_box, const KMR_KVS *, KMR_KVS *, void *, long)
 
static int generate_mapcmd_kvs (const struct kmr_kv_box, const KMR_KVS *, KMR_KVS *, void *, long)
 
static int generate_redcmd_kvs (const struct kmr_kv_box, const KMR_KVS *, KMR_KVS *, void *, long)
 
static void kmrsh_abort (int, const char *,...)
 
int main (int argc, char *argv[])
 
static void parse_args (char *, char *[])
 
static int run_kv_generator (const struct kmr_kv_box, const KMR_KVS *, KMR_KVS *, void *, long)
 
static int sleep_wait (const struct kmr_kv_box, const KMR_KVS *, KMR_KVS *, void *, long)
 
static int write_kvs (const struct kmr_kv_box[], const long, const KMR_KVS *, KMR_KVS *, void *)
 

Detailed Description

kmrshell_mpi is command line version of KMR and it runs a MapReduce program whose mapper and reducers are user specified MPI programs.

When kmrshell_mpi is used to run a MapReduce program, user should specify a simple program that generates key-value pairs from the output of mapper. The key-value generator program can be specified by '-k' option and can be implemented by reading outputs of mapper and then writing key-value pairs to the standard output. After shuffling the key-value paris, key-value pairs are written to files on each rank with 'key'-named text files whose line represents a key-value separated by a space. The file is passed to the reducer as the last parameter.

kmrshell_mpi can run Map-only MapReduce where no reducer is run. This is very useful if you want to run multiple tasks as a single job.

Options

Usage

$ mpiexec -machinefile machines -n 4 \
./kmrshell_mpi -n m_num[:r_num] -m mapper [-k kvgenerator] [-r reducer] \
inputfile

Examples

e.g.1) Run MPI mapper and MPI reducer with 2 MPI processes each.
$ mpiexec -np 2 ./kmrshell_mpi -n 2 -m "./mpi_pi.mapper" -k "./mpi_pi.kvgen.sh" -r "./mpi_pi.reducer" ./work
e.g.2) Run MPI mapper with 2 MPI processes and MPI reducer with 4 processes.
$ mpiexec -np 2 ./kmrshell_mpi -n 2:4 -m "./mpi_pi.mapper" -k "./mpi_pi.kvgen.sh" -r "./mpi_pi.reducer" ./work
e.g.3) Only run MPI mapper with 2 MPI processes
$ mpiexec -np 2 ./kmrshell2 -n 2 -m "./mpi_pi.mapper" ./work

Definition in file kmrshell_mpi.c.