kmrshell.c File Reference

KMR-Shell for Streaming. More...

#include <stdio.h>
#include <stdlib.h>
#include <stddef.h>
#include <string.h>
#include <unistd.h>
#include <fcntl.h>
#include <signal.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/wait.h>
#include <sys/param.h>
#include <dirent.h>
#include <assert.h>

Go to the source code of this file.

Macros
#define	ARGSIZ   8
	Maximum number of arguments to mapper and reducer programs. More...
#define	ARGSTRLEN   (8 * 1024)
	Buffer string size of arguments to mapper and reducer programs. More...
#define	LINELEN   32767
	Maximum length of a line of data. More...

Enumerations
enum	{ PIPE_IN = 1, PIPE_OUT = 2 }
	Parameter for pipeops(). More...

Functions
static int	execute (char *prog, char *const args[], pid_t *cldpid)
	Forks and execs a process. More...
static int	filereader (char *path)
	Reads files (file-reader). More...
int	main (int argc, char *argv[])
	Starts map-reduce shell processes (for "streaming"). More...
static void	parse_args (char *argstr, char *argary[])
	Parses command parameters given for mapper and reducer arguments. More...
static int	pipeops (int *pipefd, int direc)
	Sets up pipe states. More...
static void	putfilecontents (char *)
static void	reapchild (int exitstat)
	Handles SIGCHLD signal. More...

Variables
static char *	shuffler = "kmrshuffler"

Detailed Description

KMR-Shell for Streaming.

It forks processes of a mapper, a shuffler, and a reducer, then, it reads a number of specified input files and passes their data to a mapper via a pipe. A mapper, a shuffler, and a reducer are shell executables.

Definition in file kmrshell.c.

Macro Definition Documentation

#define LINELEN   32767

Maximum length of a line of data.

Definition at line 25 of file kmrshell.c.

#define ARGSIZ   8

Maximum number of arguments to mapper and reducer programs.

Definition at line 28 of file kmrshell.c.

#define ARGSTRLEN   (8 * 1024)

Buffer string size of arguments to mapper and reducer programs.

Definition at line 31 of file kmrshell.c.

Enumeration Type Documentation

anonymous enum

Parameter for pipeops().

Enumerator
PIPE_IN	Attach stdin to pipe.
PIPE_OUT	Attach stdout to pipe.

Definition at line 34 of file kmrshell.c.

Function Documentation

static int pipeops ( int *  pipefd, int  direc  )

static

Sets up pipe states.

It attaches pipes to stdin or stdout.

Definition at line 61 of file kmrshell.c.

static int execute ( char *  prog, char *const  args[], pid_t *  cldpid  )

static

Forks and execs a process.

Parameters

prog	program path.
args	argments to program.
cldpid	pid of child (out).

Definition at line 102 of file kmrshell.c.

static void reapchild ( int  exitstat )

static

Handles SIGCHLD signal.

Definition at line 51 of file kmrshell.c.

static int filereader ( char *  path )

static

Reads files (file-reader).

It reads possibly multiple files and writes their contents to stdout. If "path" is a directory, it enumerates the files under the directory and reads each file.

Parameters

path	path to file (or directory).

Definition at line 186 of file kmrshell.c.

static void parse_args ( char *  argstr, char *  argary[]  )

static

Parses command parameters given for mapper and reducer arguments.

It scans an argument string like "mapper arg0 arg1" for the -m and -r options, and generates an argv array {"mapper", "arg0", "arg1", 0}. The separator is a whitespace.

Parameters

argstr	string given for -m or -r options.
argary	array to be filled by argument strings.

Definition at line 257 of file kmrshell.c.

int main	(	int	argc,
		char *	argv[]
	)

Starts map-reduce shell processes (for "streaming").

It forks and execs a mapper, a shuffler, and a reducer, which are connected via pipes.

• -m mapper program.
• -r reducer program.
• input file or directory.

Definition at line 293 of file kmrshell.c.