Choose a new LMS using integrated multi-criteria decision making approach

Imagine you're tasked with selecting a new course/learning management system for your institution. Obviously you will want to involve the various stakeholders (a group that is growing in diversity each year, by the way) and identify a laundry list of criteria. But how do you develop an objective method for evaluating various systems?

I just came across the following paper that describes "an integrated multiple criteria decision making approach, which combines the analytic hierarchy process (AHP) and quality function deployment (QFD), to evaluate and select the best system. The evaluating criteria are derived from the requirements of those who use the system."

Measuring performance of virtual learning environment system in higher education
William Ho, Helen E. Higson, Prasanta K. Dey, Xiaowei Xu, Rami Bahsoon
Quality Assurance in Education, Vol. 17 No. 1, 2009 pp. 6-29

Article URL for subscribers:
http://www.emeraldinsight.com/10.1108/09684880910929908

I admit that I had no idea what AHP meant. (According to the QFD Institute: The Analytic Hierarchy Process (AHP) is a mathematically rigorous, proven process for prioritization and decision-making. By reducing complex decisions to a series of pair-wise comparisons, then synthesizing the results, decision-makers arrive at the best decision with a clear rationale for that decision.) OK...that sounds useful and relevant to the process of selecting a C/LMS in my mind.

Oh, "QFD is a comprehensive quality system that systematically links the needs of the customer with various business functions and organizational processes", according to the QFD Institute.

The article elaborates on this analytical process. Skimming to the conclusion, I find that the authors explain the value in this approach for institutions with existing VLEs:

"...the proposed approach can support the decision makers of universities in reviewing their existing VLE systems and determining whether it is necessary to replace the existing systems by a better one...[T]he proposed approach can [also] support the VLE system developers in analyzing their strengths and weaknesses, and also identifying the opportunities and threats against the competing systems."

Unfortunately, I don't have time now to dive deeper into this topic or mentally apply it to my own institution's environment. (It does make me wonder what analytical process was used 6+ years ago when UW switched from WebCT to Desire2Learn.)

I will, however, be saving this article for future reference.