kmriolb.c File Reference

load balanced MapReduce based on data locality More...

#include <mpi.h>
#include <assert.h>
#include <math.h>
#include "../config.h"
#include "kmr.h"
#include "kmrimpl.h"
#include "kmrfefs.h"
#include <mpi-ext.h>

Go to the source code of this file.

Classes
struct	io_grp

Macros
#define	K_MAX_X   24
#define	K_MAX_Y   18
#define	MAX_IO_GRPS   432 /* = 82944/192, 24x18 */

Functions
int	kmr_assign_file (KMR_KVS *kvi, KMR_KVS *kvo, struct kmr_option opt)
	Assigns files to ranks based on data locality. More...
static int	kmr_iolb_calc_xy_of_rank (int rank)
static int	kmr_iolb_find_file_location_k (const struct kmr_kv_box kv, const KMR_KVS *kvi, KMR_KVS *kvo, void *p, long i_)
static void	kmr_iolb_load_stripe (const char *filename, struct io_grp *grps, int *grps_cnt)
static int	kmr_iolb_print_assigned_files (const struct kmr_kv_box kv, const KMR_KVS *kvi, KMR_KVS *kvo, void *p, long i_)
static void	kmr_iolb_shift_grps (struct io_grp *grps, int grps_cnt, int *shift_x, int *shift_y)
static void	kmr_iolb_unshift_xy (int *x, int *y, int shift_x, int shift_y)
static _Bool	kmr_iolb_value_is_equal (const struct kmr_kv_box kv1, const struct kmr_kv_box kv2)

Detailed Description

load balanced MapReduce based on data locality

Definition in file kmriolb.c.

Function Documentation

int kmr_assign_file	(	KMR_KVS *	kvi,
		KMR_KVS *	kvo,
		struct kmr_option	opt
	)

Assigns files to ranks based on data locality.

It assumes that values of key-value pairs in the input KVS are file paths and it shuffles the key-value pairs and writes results to the output KVS so that the files are assigned to near ranks. If the value of a key-value pair is file paths separated by '\0', it will find a rank near from all the files specified in the value. Currently, it only works on the K computer. On the other systems, it just performs kmr_shuffle(). Effective-options: INSPECT, TAKE_CKPT. See struct kmr_option.

Definition at line 257 of file kmriolb.c.