Quality of Experience Evaluation Methodology via Crowdsourcing

Provisioning of digital video services is a difficult task as it is hard to estimate optimal settings of video parameters, given transmission constraints, while maximizing the overall end-user quality. With Internet streaming services becoming part of our everyday life, end-to-end optimization of such systems is important. On one hand, huge effort is given into subjective or objective evaluation of the end-user perception. High quality audiovisual perception with respect to the minimized costs of the provided service is one of the main interests for the network providers. On the other hand, subjective evaluations to determine best video and audio configurations are often evaluated in controlled test laboratory environments, which have little to do with the real environments in which consumers enjoy such content. Unfortunately, no serious attempts have been made to take into account interactions between quality of the content and its perception at the end-user side when provisioning end-to-end video transmission services. In contrary to the traditional laboratory assessments, new methodologies for conducting subjective assessments in the end-user?s natural environment are being developed. In this context, crowdsourcing emerges as an interesting concept for conducting subjective user tests in a real world environment. The crowdsourcing users conduct the tests remotely at their own computers in a familiar environment. In particular, they launch a web-based application in their browser and click through the subjective test. The basic idea of crowdsourcing is to utilize a huge number of Internet users, such that tasks are completed within short time and at low costs. However, the remote characteristics of the proposed methodologies bring also several important aspects, which need to be coped with and which are not present in the controlled laboratory environment. Aspects such as reliability of the users, efficiency of the proposed methodology, as well as the economic efficiency of the whole assessment will be discussed in presented work. This thesis will describe techniques and recommendations for performing crowdsourcing-based subjective quality assessments and it proposes a novel methodology for Quality of Experience (QoE) evaluation at the end-user side exploiting crowdsourcing possibilities. For this purpose, the new web-browser-based application implemented into Facebook.com GRAPH API has been developed.