This study aims to compare the effectiveness and efficiency of a web page content delivery that is displayed on a computer, tablet, and smartphone screen. This research presents on how the effectiveness and efficiency of content delivery in a web page content layout, which is designed using Google Material Design Guidelines (MDG) and User-Centered Design (UCD) approach, might be improved on smaller screens.

A. Specifying Target

Several responsive website layouts come in one column, two columns, or three columns which are targeting typical computer screen resolution by default. However, due to the smaller screen size of tablet and smartphone, it is uncommon that a web page is displayed in three columns layout. This research specifies three web pages as the evaluation target, i.e., Jurnal Pengembangan TIIK website [15] as one column web page, Jurnal UB website [16] as two columns web page, and VLM UB website [17] as three columns web page. The web page evaluation process involves several particular users. They will run several tasks to measure the effectiveness and efficiency of the evaluated web page layout. The measurement result of the initial page layout and the modeled page layout were compared. The modeling layout was designed based on UCD and MDG by gathering user feedback or as specified in the guidelines. Target users are determined based on the environmental analysis, which identifies the environment and the target user. Also, user characteristics are identified to assess user’s eligibility to be involved in the design, evaluation, and analysis phase. This research specifies criteria that the users are familiar with browsing a website and ever been using the target websites. They were also never accessing the target websites through devices other than desktop and laptop computers. Also, the requirements elicitation process is performed as the primary task in evaluating the prototype‚Äôs layout resulting from the UCD and MDG approach. User test scenarios are designed based on task analysis. The tasks cover three difficulties level: easy, medium, and hard. They were specified based on the user interview process towards their activities within the website. The tasks that fall into the easy level are determined based on the most frequently used features on the web page. The medium level tasks are determined based on features that are less often used by most users. The hard level tasks are determined based on features that exist on the web page, but they are unnoticeable by most users.

B. Preliminary Measurement

Preliminary measurements were performed to assess current web page layouts towards the effectiveness and efficiency of content delivery in desktop, tablet, and smartphone screen. The preliminary assessment will serve as a basis for describing how the implementation of MDG and UCD in the transformation process of a web page layout design will affect the usability regarding effectiveness and efficiency. The measurements were performed based on the tasks as described in Table 1. The measurements involve 11 to 15 users, and their performance was measured in a two-week interval to avoid bias in the measurement as users may become more proficient in learning and remembering the layout. Users are requested to perform the task on a tablet computer in the first measurement and perform the same tasks on a smartphone in the second measurement process. The evaluation and measurement process of the prototype utilizes several instruments, such as desktop computer or laptop, tablet computers, and smartphones. Stopwatches and a printed version of the task description were also prepared. The users were requested to perform a specific task using the prototypes that were installed on the specified devices. When a user is performing a task, the facilitator records every step that a user performs while measuring the time required to complete.

C. Effectiveness Measurement

Effectiveness is defined as the accuracy of task completeness [18]. The task completeness value will be set as “1” if the user can complete the task or “0” if the user has failed to complete the task [19] [20]. The overall integral product effectiveness is defined as the percentage of the number users that completed the tasks divided by the multiplication of the number of respondents and the total number of scenarios [20]. The effectiveness calculation follows the formula as in [20].

D. Efficiency Measurement

Efficiency is defined as time spent by a user to complete the tasks [18]. This research measures the time base efficiency and overall relative efficiency. Time-based efficiency can be represented as the average time that user spent in completing a task as formula written on [20]. If there are any incomplete scenarios, then the time measurement will continue to proceed until the user is giving up. Overall relative efficiency provides the time ratio between the time spent of satisfying users and the total time spent by users regardless the success status. The overall relative efficiency is formulated as in [20]. By calculating the value of user performance as recorded in Table 2, the overall integral product effectiveness, time-based efficiency, and overall relative efficiency of the current web page layout design are described in Table 3. By entering the values obtained from user performance records as in Table 2 into (1), (2), and (3), The overall integral product effectiveness, time-based efficiency, and overall relative efficiency of the current web page layout design are described in Table 3.

Referring to Fig. 1 and Table 3, the effectiveness of content and information delivery of a web page in one column layout, two columns layout, and three columns layout are considered as good, normal, and normal-to-bad category respectively. The test result as shown in Table 3 indicates that there are slightly drops of all evaluated web layout concerning effectivity and efficiency of information and content delivery. The effectivity and efficiency are lower on web page layouts that use more than one column. Three columns web page layout scores the worst in both effectivity and efficiency. The time-based efficiency for a user to accomplish a task tends to decline when one has to complete on devices with smaller screen size. On average, the conversion of a web page layout from desktop resolution to a tablet or smartphone screen will reduce the effectiveness of contents and information delivery by 2%. The transformation of a web page desktop layout to tablet and smartphone will also lessen the time-based efficiency for about 21.3% on average. However, the conversion only drops approximately 3.3% on average regarding the overall relative efficiency.

E. MDG and UCD Prototyping

In both UCD and MDG approach, the design of a web page layout that is presented in one column layout for the desktop screen tend to not have much differences in the designed layout for the tablet and smartphone screen. This research adopts a UCD checklist, which is developed by [21], and used as an instrument to obtain user recommendations. In UCD, most users recommend adding more frequently used icons and shortcuts somewhere near the page header and moving the menu to the top-right side of the screen. On the other hand, MDG recommends to put the menu on the top-left side of the screen and represents several functionalities with icons. In two columns web page layout, most users recommend to just reposition the page elements without changing the visual representation. The content layout that previously represented in two columns transformed into one column layout for both tablet and smartphone. In MDG, there is no change in the default tablet and smartphone screen layout transformation.

However, the menu is moved to the top-left portion of the device’s screen. When a web page was presented in three columns layout, the page tends to have much information and functionalities to serve on one screen. In three columns layout, there is a significant overhaul regarding page layout design using MDG. Most page elements and functionalities were simplified into icons and menus. In UCD, users are recommending to remove several features that are considered inappropriate or useless. The design resulting from the MDG and UCD approach are then manifested in prototypes using high-fidelity prototyping software. The software is used to speed up the implementation process without having to hard-code the whole web page for testing.

F. Prototype Evaluation

The prototypes were evaluated using the same procedure as the previous measurement of the target web page. This time, the evaluation and measurement process was performed using the resulting prototypes that target the tablet and smartphone screen using the MDG and UCD approach. All the tests were evaluated using the latest Google Chrome web browser on all platforms being assessed to eliminate browser factors from the evaluation process. Table 4 describes the evaluation and measurement results that were performed to the prototypes. It shows that MDG and UCD show some slight improvement regarding information and content delivery effectiveness. However, both method also shows noticeable improvements in the efficiency of information and content delivery of a web page in smartphone and tablet screen.