The Final Experiment For Project:
This presentation is the property of its rightful owner.
Sponsored Links
1 / 41

Experiments Set Up (Hardware) PowerPoint PPT Presentation


  • 60 Views
  • Uploaded on
  • Presentation posted in: General

The Final Experiment For Project: “Measuring User Preferences On Design Variations Through VR” Is Completed !!!!. Experiments Set Up (Hardware). MuseV3. Traditional. Desktop CAVE VR special construction frame 2 projectors (front and top) front screen tablet mouse, keyboard

Download Presentation

Experiments Set Up (Hardware)

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Experiments set up hardware

The Final Experiment For Project:“Measuring User Preferences On Design Variations Through VR”Is Completed !!!!


Experiments set up hardware

Experiments Set Up (Hardware)

MuseV3

Traditional

  • Desktop CAVE VR

  • special construction frame

  • 2 projectors (front and top)

  • front screen

  • tablet

  • mouse, keyboard

  • navigation joystick

  • set of speakers

  • PC station

  • standard PC

  • navigation joystick

  • set of speakers


Experiments set up hardware

VR Experiment Set Up

MuseV3


Experiments set up hardware

CA Experiment Set Up

Traditional


Users experiment

Users & Experiment

  • Steps through the experiment:

  • Pick up a “ticket” with description and order of tasks

  • Introduction movie.

  • Task Explanation.

  • Tutorial

  • Experiment

  • Change of Stations (for VR or CA)

  • Introduction movie.

  • Task Explanation.

  • Tutorial.

  • Experiment.

  • Evaluation Questionnaire.

FIRST TASK

SECOND TASK


User tasks

User Tasks

There are in total four experiment types

(FMVR, OEVR, MECA, VECA).

Two in each of two groups (VR and CA).

Each respondent had to complete two random tasks (one from each group), however each combination of tasks should be presented approximately equal number of times.


Experiment types

Experiment Types


Sc experiment attributes

SC Experiment, attributes


Sc experiment attributes1

SC Experiment, attributes

Design Description (Multimedia and Verbal)

The FFD consists of 32 profiles (to estimate main effect).

The design includes 9 HOLDOUTS.

In total 41 profiles were presented to each respondent.

Consequently:

Each respondent viewed 20 choice sets.

Each choice set contained 3 profiles (2 at random + BLD).


Vr experiment attributes

VR Experiment, attributes

Regardless of the form of presentation we were looking for following changes of the design:


Short explanation of bn

Short explanation of BN

  • What it is?

  • Belief network (BN) also known as a Bayesian network or probabilistic causal network

  • BN captures believed relations (which may be uncertain, stochastic, or imprecise) between a set of variables which are relevant to some problem (e.g. coefficients and choices).

How does it work?

After the belief network is constructed, it may be applied to a particular case. For each variable you know the value of, you enter that value into its node as a finding (also known as “evidence”). Then Netica does probabilistic inference to find beliefs for all the other variables.

Incremental learning.

After the beliefs are found (post priori) MuseV updates the network, so they become a’ priori for the next respondent.


Experiments set up hardware

Step 0

Step 1

Step 5

Step 15

Step 64


Vr experiment network description

VR Experiment - Network Description

Personal Information

Coefficients 

Probability for choosing design element

User Choices


Experiments set up hardware

ANALYSES


Analyses

ANALYSES

The truth about the respondents:

We sent 1,600 letters in total !!!!

The preparations took 2,5 days for two people (Vincent and Maciek)

Within 2 weeks we received 96 positive conformations.

At the end of the experiment we end up with solid number of 64

respondents that have completed the both appointed to them tasks!

5 of the 64 respondents would not buy the house that they have designed.

4 respondents did not completed second task

(as the design was not relevant to them)

2 respondents did not started the experiment for the same reason!


Analyses1

ANALYSES

The truth about the respondents:

First VR Experiment:

First CA Experiment:


Final questionnaire the most preferred system

Final QuestionnaireThe most preferred system


Final questionnaire the difficultness and pleasure of the tasks

Final QuestionnaireThe difficultness and pleasure of the tasks.


Final questionnaire house situation

Final QuestionnaireHouse situation


Final questionnaire hours

Final QuestionnaireHours


Analyses bn

ANALYSES - BN


Analyses bn1

ANALYSES - BN

We have analyzed the data collected via BN as follows:


Analyses bn goodness of fit measures for choice prediction

ANALYSES – BNGoodness-of-fit measures for choice prediction

(1) GOF - overall : comparison between observed and predicted across 6x32 cases

(2) GOF - by choice i : as before for attribute i only (across 32 cases)

32*6 pred obsv pred obsv

logLL= Sum ( ln (P * P + P * P ) )

k=1 k1 k1 k2 k2

32*6 obsv obsv

logLL(0)= Sum ( ln ( 0.5 * P + 0.5 * P ))

k=1 k1 k2

32*6 InitStat obsv InitState obsv

LL(InitState)= Sum ( ln ( P * P + P * P ))

k=1 k1 k1 k2 k2

r1 = ( LL - LL(0) ) / LL(0)

r2 = ( LL - LL(InitState) ) / LL(InitState)


Analyses bn2

ANALYSES - BN

Example of GOF for FMVR

BN with personal information

R1

R2


Analyses bn goodness of fit between networks fm and oe for estimation of betas

ANALYSES – BN Goodness-of-fit between networks (FM and OE) for estimation of betas

  • OVERALL G-O-F

  • G-O-F for each BETA

  • G-O-F for each PROFILE

  • G-O-F for each profile and beta

  • 1 2

  • The GOF is based on the expected value of the attribute: GOF(N1:N2) = | E - E |

  • where: j j

  • N1 k N1

  • E = SUM (P * B )

  • j i ij ij


Analyses bn3

ANALYSES - BN

The following graphs show the differences between parameter estimation

for Network 1 (FMVR) and Network 2 (OEVR):

G-O-F = 2.657


Analyses bn4

ANALYSES - BN

G-O-F = 1.72

G-O-F = 1.677


Analyses bn5

ANALYSES - BN

G-O-F = 0.089

G-O-F = 0.292


Analyses bn6

ANALYSES - BN

G-O-F = 4.378

G-O-F = 0.111


Analyses bn convergence of betas

ANALYSES – BNConvergence of BETAS

n 2

LL = Sum (P )

j k=1 jk

BN with NO personal information (COMBINED)


Analyses ca

ANALYSES - CA


Analyses ca checking for order effect checking on the rho 2

ANALYSES – CACHECKING FOR ORDER EFFECTCHECKING ON THE Rho2


Analyses ca checking for order effect in all cases the ca was done first scale effect

ANALYSES – CA CHECKING FOR ORDER EFFECT (in all cases the CA was done first)SCALE EFFECT


Analyses ca checking for order effect conclusions

ANALYSES – CA CHECKING FOR ORDER EFFECT CONCLUSIONS

Based on the tests we can conclude that order effect has no significant influence on the estimated models for following combination of experiments:

Multimedia(CA) with Preset Options(VR) Verbal(CA) with Free Modification(VR)

There is an order effect for following combination of experiments:

Multimedia(CA) with Free Modification(VR) Verbal(CA) with Preset Options(VR)

In this case the fact that VR that was done as the first task, improves the model estimation.


Analyses ca model estimation checking on the rho 2

ANALYSES – CAMODEL ESTIMATIONCHECKING ON THE Rho2


Analyses ca model estimation parameters values and their significance

ANALYSES – CAMODEL ESTIMATIONPARAMETERS VALUES AND THEIR SIGNIFICANCE


Analyses ca comparing preferences of different groups scale effect

ANALYSES – CACOMPARING PREFERENCES OF DIFFERENT GROUPS(SCALE EFFECT)


Analyses ca predicting holdouts counting correct and wrong predictions based on highest utility

ANALYSES – CAPREDICTING HOLDOUTS(counting correct and wrong predictions based on highest utility)

The table illustrates the numbers of correct predictions against wrong.


Analyses ca bn g o f of holdouts prediction rho 2 calculation based on log likelihood

ANALYSES – CA & BNG-O-F of HOLDOUTS PREDICTION (Rho2 calculation based on log likelihood)

  • Based on Estimated parameters for each listed model model, we did:

  • calculating utility for each profile, as Vj=SUM(Bi*Xi)

  • calculating market shares for each profile k in each choice set as Pk=expVk/Sum(exp(Vj)) j=1 to 3

  • choice in a choice set is made based on the highest utility or highest market share

  • calculating logLL(B)= SUM (ln(Pi)); where Pi - probability for max utility value in the choice set j

  • Based on null model:

  • calculating logLL(null model), calculations as above, but the utility Vj=0 for each profile

  • calculating Rho2 = 1 - logLL(B)/logLL(0)


Benchmark rl against ca and bn

Benchmark (RL) against CA and BN

G-O-F of REAL LIFE PREDICTION (Rho2 calculation based on log likelihood)Calculations based on BETAS

GOF (CA) = 0.417

GOF (BN) = 0.502

  • Creating choice sets:

  • All possible combinations of attributes are combined into one choice set for one respondent. In total we have 15 choice sets (the same as the number of respondents or households), in each choice set there are 32 profiles. In case of BN each profile is translated into choices available in the network. Choices for the most preferred profile are made based on the real life data (actual decisions that the respondents did while buying a house).

  • Steps In Calculations:

  • utility for each profile Vj=SUM(Bi*Xi)

  • market shares for each profile k in each choice set as Pk=expVk/Sum(exp(Vj)), j = 1 to 32

  • log likelihood logLL(B)= SUM (ln(Pj)); where Pi - probability value for the profile that the respondent have chosen

  • calculating logLL(null model), calculations as above, but the utility Vj=0 for each profile

  • calculating Rho2 = 1 - logLL(B)/logLL(0)


Benchmark rl against ca and bn1

Benchmark (RL) against CA and BN

The table illustrates ratio (percentage) of choosing certain design option.

In case of real life - based on numbers of subjects buying certain option.

In case of BN based on beliefs.


  • Login