Thursday, 22 December 2011

mincemeat.py: MapReduce on Python

Fascinating!
http://remembersaurus.com/mincemeatpy/

Introduction

mincemeat.py is a Python implementation of the MapReduce distributed computing framework.

mincemeat.py is:
  • Lightweight - All of the code is contained in a single Python file (currently weighing in at <13kB) that depends only on the Python Standard Library. Any computer with Python and mincemeat.py can be a part of your cluster.
  • Fault tolerant - Workers (clients) can join and leave the cluster at any time without affecting the entire process. (Master checkpointing coming in future versions)
  • Secure - mincemeat.py authenticates both ends of every connection, ensuring that only authorized code is executed. (TLS support coming in future versions)
  • Open source - mincemeat.py is distributed under the MIT License, and consequently is free for all use, including commercial, personal, and academic, and can be modified and redistributed without restriction.

No comments:

Post a Comment

Datanami, Woe be me