Data analytics are increasingly becoming important for organizational tactical and strategic decision making. This presentation describes how MERLOT uses data analytics to understand usage trends of its system, in order to address the needs of its users, to improve the currency of the system, and to develop new methodologies and functions to ensure MERLOT's continued relevance for teaching and learning in the 21st Century.
MERLOT (www.merlot.org) is 20 years old and it is no accident we’ve survived this long because we have been very attentive to the use of different kinds of data for tactical and strategic decision making that has allowed us to stay in business this long. In general we collect a variety of data types using different kinds of data collection tools to provide us with input for analytic-based decision making
Google Analytics (Global Community Pulse)
MERLOT uses Google Analytics (GA) to track/report web traffic to our site. GA provides us with a perspective related to MERLOT access from the outside world - our “Global Community Pulse.” GA provides a host of data related to site visits, such as number of pages visited, time spent on the site, devices used to visit MERLOT, bounce rate, geographical location, new/returning visitors, etc.
Understanding where our users are geographically located was particularly useful when it came to servicing colleagues in the Peoples Republic of China. From China, it is not possible to access sites “controlled by Google.” Since MERLOT uses Google tools – GA and Google Translate - www.merlot.org was blocked in China. By detecting accesses from IP addresses sourced from China, we now turn off non-essential Google tools and software allowing the MERLOT to be accessed from China.
Other examples include the use of GA to help us allocate develop resources for projects with the most payoff to us. For example, recently it has become important to know the percentage of users coming to our site who use a mobile device such as a smart phone. GA allows us to determine how much effort to expend on development of the site for such use by knowing how many users that affects. A few years ago, we used data on bounce rates to verify user comments and our own concerns that our site was very convoluted and had to be redesigned. Digging in GA data allowed us to understand how people were using the site (where they were going, where they were and weren’t clicking) so we could rebalance and relocated most-needed versus less-needed functions on our redesigned website.
MERLOT Internal Analytics (System Pulse)
In addition to external access data, we also gather internal repository usage data – something not possible from GA – our MERLOT “System Pulse”. Our Internal Analytics (IA) are quite extensive and not only used by the development team but also Management and our volunteer Editorial Boards. These data include, for example, the number of materials and members added to the repository on a monthly basis, as well as the number of Comments, Learning exercises, Bookmark Collections, Course ePortfolios, Reviews, Web Service usage, and QR code and Mobile sharing usage.
Our IA has allowed us to deploy our “15% Solution” program – a badging reward system for accomplishments regarding various aspects of the use of our system. The overall goal of the program is to increase the number of users who use academic technology in higher education. We provide our registered Members ways to participate and gain “points” which translate into gold, silver and bronze merit “ribbons” displayed on their profile for other users to see.
We also use our Internal Analytics to track the growth of our collection. For example, when GA indicated a drop in visits during vacation breaks, we were less alarmed than had that happened at the start of a semester because our IA did indicate a growth in the collection. It seems that a handful of members were contributing ‘heartily’ to the collection at that time. On the other hand, when our visits were up and our repository growth down, we took steps to understand the data, and took action to develop repository growth strategies.
Another important use of our IA has been by our discipline Editors. MERLOT has approximately 25 academic discipline boards; almost all Learning Objects (LO) contributed to the collection are “triaged” by the Editors of those boards. Not surprisingly, it’s important for the Editors responsible for the peer review of new LOs in their discipline, to be overwhelmed by submissions and wonder which submissions should be prioritized for review by their board. Our IA are used to provide each Editor with a list - a few times a year - of the “Top 100” materials MERLOT users go to in their discipline. This allows each board to determine priority of reviewing materials to keep the collection current.
Social Media (Social Pulse)
Like other companies and organizations, MERLOT is active on Twitter, Facebook, and LinkedIn. Social media are necessary outlet for us to operate a contemporary 21st Century website community. We have found social media to be valuable tools to promote news, features, and other higher education information related to MERLOT. Another benefit of this “Social Pulse” is that it drives more users to the site, providing a greater measure of community engagement with MERLOT.
MERLOT captures data related to “reaches,” such as ‘likes’ ‘retweets’. A benefit of all this is that when we launch a new program or feature, we can capitalize on these media to promote news of the new programs. We can then differentially assess the utility of each social medium in helping us get our messages out.
Also, using different social media and following other higher education institutions and companies enables us to pass along trusted information to our users about what is going on with online learning, technology, and education. MERLOT is a reliable and trusted voice and that allows us to attract a community of users, some of which were not aware of MERLOT before our cross pollination with other social media outlets.
Dashboard Analytics (Partners Pulse)
MERLOT has a unique set of data called the Data Dashboard (DD), our “Community Pulse.” Each MERLOT Partner has access to their own institution’s real time, online DD that shows the Partner Project Director the detailed activity of every registered MERLOT Member from their institution. The DD shows, at the individual user level what each is doing (contributions such as materials, comments, etc...), when (last log in), etc. These data help our Partners justify their place in the consortium by confirming they are getting the most out of their partnership.
The DD can be used to help identify gaps in an institution’s faculty development program(s) related to their use of MERLOT and OERs. For example, we are able to provide our Partners with tailored programs such Affordable Learning Solutions connections, customized teaching commons portals, onsite visits, webinars, etc., all to promote their participation and use of OERs in MERLOT.
Automated Data-Driven Recommender Functions – Present and Future
Present: The data-driven analytics described above enable humans to examine and analyze collected data in order to make deliberate and optimum tactical and strategic decisions related to their own spheres of influence. However, MERLOT also collects data that are automatically processed by internal logic to guide users towards accomplishing their teaching/learning goals. Our new “recommender” function uses internally collected data to make recommendations to users regarding which LOs to consider examining, based on their current search. In general, we log every LO viewed (if possible), who viewed it, and the IP address from which it originated. Based on such data types, complex coded algorithms make best guess recommendations regarding which other related LOs a user might wish to examine.
Future: MERLOT is currently undertaking a new project involving the use of Big Data, artificial intelligence-based, analytical heuristic methodologies. Specifically, we are trying to help our users to more efficiently discover useful and appropriate OERs to meet their general instructional needs. We intend to identify OERs related to specific course outcomes (learning objectives) as documented in course syllabi. We hope to deploy semantic parsing methodologies of syllabi to analyze learning outcomes/objectives to recommend OERs that directly relate to those course outcomes.
To accomplish this goal requires significant detailed analyses of all our internal, historical user logs to determine extant relationships among the LO metadata in the collection, the parsed learning objects, and related historical user accesses. If (when?) we are able to accomplish this, we will be able to provide users with a more direct solution to their instructional needs than currently, when users must search repositories for what they intuit will be useful for their teaching/learning objectives.
Conclusion: We are constantly watching what is going on inside and out of MERLOT to help us make strategic decisions to improve a users’ experience. Analytic methodologies have evolved over the past few decades and we too are evolving our use of them. By employing a variety of methods, we can responsibly report analytics related to MERLOT to improve a user’s experience and make sensible business decisions. User participation and engagement is vital to another 20 years of survival for MERLOT and with diligence to our analytics, we will continue to make good business decisions.