kmrshuffler.c File Reference

KMR-Shell Shuffler. More...

#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include "kmr.h"

Go to the source code of this file.

Macros
#define	LINELEN   32767
	Maximum length of a line of data. More...

Functions
int	main (int argc, char *argv[])
	Runs KMR shuffler for streaming map-reduce. More...
static int	streaminputfn (const struct kmr_kv_box kv0, const KMR_KVS *kvs0, KMR_KVS *kvo, void *p, const long i_)
	Reads-in key-value lines from stdin into KVS. More...
static int	streamoutputfn (const struct kmr_kv_box kv[], const long n, const KMR_KVS *kvs, KMR_KVS *kvo, void *p)
	Writes-out key-value pairs in KVS to stdout. More...

Detailed Description

KMR-Shell Shuffler.

It is a processor for map-reduce by shell command pipelining ("streaming"). It works on stdin and stdout, with one line for one key-value pair. It reads lines of key-value pairs from stdin, shuffles the pairs, and writes lines of key-value pairs to stdout. The fields of a key-value pair are separated by a whitespace. Lines with the same keys constitutes consecutive lines in the output for reduction. It is a simple application. Lines are limited to 32K bytes and have no escaping of whitespaces in keys.

Definition in file kmrshuffler.c.

Macro Definition Documentation

#define LINELEN   32767

Maximum length of a line of data.

Definition at line 21 of file kmrshuffler.c.

Function Documentation

static int streaminputfn ( const struct kmr_kv_box  kv0, const KMR_KVS *  kvs0, KMR_KVS *  kvo, void *  p, const long  i_  )

static

Reads-in key-value lines from stdin into KVS.

It reads a line from mapper output, and sets it as a key-value pair in KVS.

Definition at line 27 of file kmrshuffler.c.

static int streamoutputfn ( const struct kmr_kv_box  kv[], const long  n, const KMR_KVS *  kvs, KMR_KVS *  kvo, void *  p  )

static

Writes-out key-value pairs in KVS to stdout.

Definition at line 75 of file kmrshuffler.c.

int main	(	int	argc,
		char *	argv[]
	)

Runs KMR shuffler for streaming map-reduce.

It reads maped data from stdin, shuffles, and prints shuffled data to stdout.

Definition at line 93 of file kmrshuffler.c.