Background
Rater B | ||||
---|---|---|---|---|
Yes | No | Total | ||
Rater A
|
Yes
| x11
| x12
| g1 |
No
| x21
| x22
| g2 | |
Total
| f1 | f2 | N |
Methods
Different agreement indices
Scenario | Type of table | x11
| x12
| x21
| x22
| f1 | f2 | g1 | g2 | PO
| BI | PI |
---|---|---|---|---|---|---|---|---|---|---|---|---|
Paradox 1:
| ||||||||||||
1 | Symmetrical balance | 40 | 9 | 6 | 45 | 49 | 51 | 46 | 54 | .85 | .03 | -.05 |
2 | Symmetrical imbalance | 80 | 10 | 5 | 5 | 90 | 10 | 85 | 15 | .85 | .05 | .75 |
3 | Perfect symmetrical imbalance | 90 | 5 | 5 | 0 | 95 | 5 | 95 | 5 | .90 | 0 | .90 |
Paradox 2:
|
P
O
set at 0.60
| |||||||||||
4 | Symmetrical imbalance | 45 | 15 | 25 | 15 | 60 | 40 | 70 | 30 | .60 | -.10 | .30 |
5 | Asymmetrical imbalance | 25 | 35 | 5 | 35 | 60 | 40 | 30 | 70 | .60 | .30 | .10 |
6 | Perfect symmetrical imbalance | 40 | 20 | 20 | 20 | 60 | 40 | 60 | 40 | .60 | 0 | .20 |
7 | Asymmetrical imbalance | 40 | 35 | 5 | 20 | 75 | 25 | 45 | 55 | .60 | .30 | .20 |
8 | Asymmetrical imbalance | 30 | 30 | 10 | 30 | 40 | 60 | 60 | 40 | .60 | .20 | 0 |
P
O
set at 0.90
| ||||||||||||
9 | Perfect symmetrical imbalance | 85 | 5 | 5 | 5 | 90 | 10 | 90 | 10 | .90 | 0 | .80 |
10 | Symmetrical imbalance | 70 | 10 | 0 | 20 | 80 | 20 | 70 | 30 | .90 | .10 | .50 |
P
O
low (≤50%)
| ||||||||||||
11 | Perfect symmetrical balance | 25 | 25 | 25 | 25 | 50 | 50 | 50 | 50 | .50 | 0 | 0 |
12 | Asymmetrical imbalance | 30 | 30 | 20 | 20 | 60 | 40 | 50 | 50 | .50 | .10 | .10 |
13 | Perfect symmetrical balance | 20 | 30 | 30 | 20 | 50 | 50 | 50 | 50 | .40 | 0 | 0 |
14 | Perfect symmetrical balance | 5 | 45 | 45 | 5 | 50 | 50 | 50 | 50 | .10 | 0 | 0 |
Results
Paradox 1
Scenario | Type of table | Po
| Pe
|
PABAK
| ||||||
---|---|---|---|---|---|---|---|---|---|---|
Paradox 1:
| ||||||||||
1 | Symmetrical balance | .85 | .50 | .70 | .72 | .70 | .70 | .70 | .68 | .68 |
2 | Symmetrical imbalance | .85 | .78 | .32 | .82 | .70 | .81 | .55 | .69 | .68 |
3 | Perfect symmetrical imbalance | .90 | .905 | -.05 | .895 | .80 | .89 | - | .78 | .77 |
Paradox 2:
|
P
O
set at 0.60
| |||||||||
4 | Symmetrical imbalance | .60 | .54 | .13 | .41 | .20 | .27 | .15 | .21 | .20 |
5 | Asymmetrical imbalance | .60 | .46 | .26 | .4 | .20 | .21 | .33 | .32 | .31 |
6 | Perfect symmetrical imbalance | .60 | .52 | .17 | .38 | .20 | .23 | .18 | .19 | .19 |
7 | Asymmetrical imbalance | .60 | .475 | .24 | .42 | .20 | .23 | .32 | .32 | .31 |
8 | Asymmetrical imbalance | .60 | .48 | .23 | .38 | .20 | .20 | .25 | .24 | .24 |
P
O
set at 0.90
| ||||||||||
9 | Perfect symmetrical imbalance | .90 | .82 | .44 | .88 | .80 | .88 | .68 | .78 | .77 |
10 | Symmetrical imbalance | .90 | .62 | .74 | .85 | .80 | .84 | - | .83 | .82 |
P
O
low (≤50%)
| ||||||||||
11 | Perfect symmetrical balance | .50 | .50 | 0 | .25 | 0 | 0 | 0 | 0 | 0 |
12 | Asymmetrical imbalance | .50 | .50 | 0 | .26 | 0 | -.11 | 0 | .01 | .01 |
13 | Perfect symmetrical balance | .40 | .50 | -.20 | .16 | -.20 | -.20 | - | -.19 | -.19 |
14 | Perfect symmetrical balance | .10 | .50 | -.80 | .01 | -.80 | -.80 | -.18 | -.77 | -.77 |