MTR Psi Testing
This is a "lab notes" page for my (MikeTheRed) Psi Testing, if anybody wants all the gory details.
Conventions: Numbers such as 95/16 always mean, e.g., psi strength 95, psi skill 16. MC = psionic mind control.
Background
The equations governing psionic success have long been a subject of mystery and debate. If they were better understood, players could compare versus alien stats and know exactly how much Psi Strength and Psi Skill is needed, both as a minimum, and where one is maxxed out versus all aliens.
The Official Strategy Guide supplies psionic equations, but while they are very intriguing, they make no mathematical sense, as if the math operators are typo'd or something.
As part of my previous Experience testing ca. October 2005, I found that a 95/16 soldier directly next to a 25/0 muton had a MC success rate of 49% (1192/2420=49.26%), and that a 95/44 never failed. But I soon began to directly edit UNITREF.DAT experience counters for my tests, and thus didn't pursue in-game psi tests any more.
Ethereal Cereal performed the first true testing in May 2006, arriving at the equations:
Attack Strength (AS) = psi str * psi skill / 50 Defense Strength (DS) = psi str + (psi skill / 5)
Panic Attack chance = 44% + AS - DS Mind Control Attack chance = 24% + AS - DS
For these equations, one is using the attack strength of one party versus the defense strength of the target.
He had a good number of data points, and the results appeared solid. However, they didn't agree with my one highly-tested point. His equation equals 29% for my 95/16 versus 25/0 situation, whereas I got 49% success.
One potentially important difference is that he had an alien MCing his soldiers, whereas I have my soldiers MCing aliens. Anyway, the results remained puzzling to me, but I didn't have time to do psi testing then. Another potential difference is that he's using the WinCE Gold version, while I'm using the DOS 1.4 version (in DosBox). I'd consider the alien-vs-human target-type proposition to be more likely, because as far as folks can tell, the programmers tried to make an exact replica of the DOS version for Windows (a few bugs notwithstanding). And the target-type proposition is testable, as well (if we have the time, fingers crossed!)
Another reason it'd be good to know psi equations, is because effectiveness clearly decreases with distance. Again, how much strength and skill is needed to "totally rule", even across a large map? No one can say until the equations are deduced.
Test 1: Basics
For my test setup, I made a map with 16 mutons, and had 16 soldiers with psi amps that were directly next to and facing the mutons (zip file of it here).
For this first test, I wanted to keep it simple. I made all aliens 25/0, to compare with my earlier finding. Based on Ethereal Cereal's equations (see inset), if one attribute (Strength or Skill) is 100, the other can be up to 50% before the MC success rate is clipped at 100%. To simplify, half my 16 guys had Strength=100, half had Skill=100, then I chose eight equally spaced points within the range that would be applied to Strength or Skill (whichever was not 100 for those 8 soldiers). I chose this "alternating 100" approach because it would test the symmetry and, if it did look symmetrical, the results could be combined to give them more power (a better correlation coefficient on a regression line).
When choosing the eight points from 0 to 50, I also took into consideration how my finding was different from EC's, by 20 points. But I screwed up here and aimed for 0 to 70+... actually my finding was lower than his, so 0 to 50 would've encompassed all concerns. Anyway, I made my soldiers be 9 to 72 in the non-100 attribute.
Very quickly I saw that the higher half of each stick of eight was MCing 100%. So I limited my testing to the lower half of each stick (one attribute = 9, 18, 27, or 36; other attribute 100). I aimed for 50 trials, but went over by one, so N=51 per soldier. Each soldier only made one MC attempt per game turn (otherwise, it's problematical to have the target right next to them). I used the psi experience counter UNITREF.DAT[84] to determine success, because it's much less error prone than counting manually (this work is SO tedious!). But it counts by 1 for a failed attempt and 3 for a successful attempt, so this shows the success rate:
Successes = (UR[84] - Attempts)/2 = ([84]-51)/2 Success rate = Successes/Attempts = Successes/51
Given 51 attempts per soldier, the results were:
UNITREF.DAT Pooled
[57] [37] MC success Pooled Pooled success
Soldier Str Skl success rate attrib. success rate
01 9 100 20 39.2% 9 39 38.2%
02 18 100 29 56.9% 18 62 60.8%
03 27 100 48 94.1% 27 96 94.1%
04 36 100 51 100.0% 36 102 100.0%
09 100 9 19 37.3%
10 100 18 33 64.7%
11 100 27 48 94.1%
12 100 36 51 100.0%
Although I suspected that the soldiers with an attribute level of 36 were maxed, I kept testing them, both to be entirely sure they were maxed, and as a check that I counted the number of attempts correctly, my UNITREF approach was working, etc.
The results do appear to be symmetrical, so they can be combined as shown in the pooled results above. N=102 for each of the four points, but the attribute=36 point is probably clipped (above the maximum) so it should be dropped from regression analysis. A regression line then drawn for success rate versus attribute level for the three points above (with the other attribute fixed at 100) gives the following for y=mx+b equations, where y is percent MC success, x is attribute level, and m is slope of line:
Int=0? m b R2 100% Intercept no 3.10% 8.50% 0.9877 29.5 yes 3.51% 0.00% 0.9682 28.5
In other words, when one attribute (psi strength or skill) is held to 100, every point of increase of the other skill increases MC success by 3.1% (if the regression line is not forced to go through success=0 at attribute=0) or 3.5% if the y intercept is set to 0. Either approach (intercept=8.5% or forcing it to zero) is problematic, however, because the game is liable to be clipping if values fall below zero, and Attack Strength minus Defense Strength has the potential to result in negative numbers which get clipped.
The 100% success ceiling is reached when the non-100 attribute equals ~29. But that's only an estimate, because it's an estimate of a line that's not perfectly correlated. The finding of psi 100/36 giving 100% success (N=102) is in agreement, though... it's probably somewhere around 29.
The coefficient of variability (R squared) is high.
Clearly, MC is more effective for me (i.e., lower attributes needed) than predicted by EC's equations. Until such time as it can be further understood, then, it appears that aliens are different somehow versus X-COM soldiers, such that they are less effective for a given psi strength and skill.
Test 2: Constants and Multiplication
For the next test, I wanted to make sure that the underlying assumptions are as we expect them to be:
- What is the constant in the MC equation? (EC believe it's 24%)
- Is multiplication correct for Attack Strength? (Or might psi skill or strength contribute in an additive or subtractive way?)
These were addressed by:
- Setting the muton targets to psi strength of 0. (They're already skill 0.) This has the effect of making any MC attack constant "stand clear" of any potential obfuscation caused by misunderstanding the Defense Strength.
- Setting soldiers' psi strength to 0, and varying their skill. If possible, I would have interlaced strength 0 and skill 0 for soldiers, but a skill of 0 disables psi capability. Anyway, setting one of the two variables in the Attack Strength equation (see Background above) to 0 should make it all equal 0, if it only uses multiplcation or division (but not if it uses Skill with addition or subtraction somehow).
The soldiers were put into groups of four based on psi skill, to increase sampling and allow pooling at those points. Psi skills and results were:
Percent Soldier Skill Success Success (30 attempts each) 01 1 12 40.0% 02 1 10 33.3% 03 1 10 33.3% 04 1 11 36.7%_ 05 50 13 43.3% 06 50 12 40.0% 07 50 13 43.3% 08 50 11 36.7%_ 09 100 12 40.0% 10 100 16 53.3% 11 100 15 50.0% 12 100 17 56.7%_ 13 255 11 36.7% 14 255 9 30.0% 15 255 14 46.7% 16 255 15 50.0%_
This can be summarized:
Psi Skl Successes N Min Ave+/-SDs Max
1 43 120 33.3% 35.8% 3.2% 40.0%
50 49 120 36.7% 40.8% 3.2% 43.3%
100 60 120 40.0% 50.0% 7.2% 56.7%
255 49 120 30.0% 40.8% 9.2% 50.0% _
Overall 201 480 30.0% 41.9% 7.7% 56.7%
There is considerable variation; more samples sure would help. But this work is SO tedious!
Although the success rate for the 1-50-100 skill progression hints at a trend, Skill=255 is back down to the level of Skill=50 (and the single lowest soldier success rate, a 9, is in this group). I see no clear difference between the groups, especially if you consider that Skill=255 should be MUCH higher than the lower groups, if Skill were indeed influencing the groups. So, while the data is messy, I buy that it's a multiplicative equation, at least insofar as Skill is concerned.
The best guess at the constant is 41.875%, although it could easily be 40% or something else, with so much variability present. Notice how Test 1 could be taken to hint that the constant is 33.5 (alien psi strength of 25 plus b intercept of 8.5). However, there is considerably more error in an extended regression line... there's a fair amount of variability in the slope, which gets compounded by extending it down to zero. In any event, it also hints at a fairly high constant.
As things become clearer, the data from Test 1 may be able to be pooled with other data in order to pin down the constant better, but for now, 41.875 is the best guess.
More to come... got any particular ideas on what to do next, EC? Looks like it just needs more and more data... Man, this stuff is mind numbing! --MikeTheRed 10:24, 2 September 2006 (PDT)