OLAP Cube Construction with Erlang

I've managed to build the parallel OLAP cube constructor in Erlang. This program achieves parallelization through creating a process for every dimension in the OLAP cube. Each process manages the file that holds the dimension data. Messages are passed from the first dimension all the way down to the last dimension which stores the measures themselves. To further parallelize things, you can partition any dimension using a modulus, which creates another file and process. This helps get around the 2 GB limit for dets tables.

I also have basic path based querying working, which is also parallelized through sending the query message through each dimension. While the querying itself isn't parallel for a particular client, it will theoretically scale to handle many clients.

When I move to move traditional querying to generate a traditional tabular result set, I will be able to parallelize the query for a single client.

I'll post the working code once I can choose a suitable license. I'm very interested to hear feedback, as I'm very much still an Erlang n00b.

Next up I'll generate some performance numbers to see if this thing will actually perform in the real world.

I have to say, functional programming is great when your solution is algorithmic. Previous implementations of mine were done in Java or Ruby, which are object oriented. The classes and object obscured the algorithm, which in the case of OLAP cubes is the primary focus.

Popular posts from this blog

The 29 Healthiest Foods on the Planet

The 29 Healthiest Foods on the Planet > The following is a "healthy food hot list" consisting of the 29 food that will give you the biggest nutritional bang for you caloric buck, as well as decrease your risk for deadly illnesses like cancer, diabetes and heart disease.
Read more

Lists and arrays in Dart

(This is Part 3 in a series about Dart. Check out Part 2, Function in Dart .) Warning : We expect the Dart libraries to undergo potentially sweeping changes before Dart goes to alpha. This document is relevant as of 2011-12-22. Intro Dart is a "batteries included" effort to help app developers build modern web apps. An important "battery" is the bundled core Dart libraries, providing common and rich functionality. Dart is building a solution for large, complex web apps, and providing well tested, integrated, and common libraries is key to helping a web app developer be more productive out of the box. The Collection libraries are a crucial set of APIs that Dart developers get for free. Much more than simple arrays and maps, the Collection library includes standard ways to filter, iterate, inspect, compose, and sort your data. This post specifically looks at List<E> , Dart's ordered, indexable collection of objects. Aside: The Dart project is lucky...
Read more

Converting Array to List in Scala

Now, this has to have a built-in somewhere in Scala , because it just seems too common. So, how to convert an Array to a List in Scala? Why do I need this? I needed to drop to Java for some functionality, which in this case returns an Array. I wanted to get that Array into a List to practice my functional programming skillz. **Update**: I figured out how to convert Arrays to Lists the Scala way. Turns out it's a piece of cake. val myList = List.fromArray(Array("one", "two", "three")) or val myList = Array("one","two","three").elements.toList The call to elements returns an Iterator , and from there you can convert to a List via toList . Nice. Because my first version wasn't actually tail recursive, what follows is a true tail recursive solution, if I were to implement this by hand. The above, built in mechanism is much better, though. object ArrayUtil { def toList[a](array: Array[a]): List[a] = { d...
Read more