## ŒCT Comparison Tolerance

The setting of the system variable ŒCT (comparison tolerance) determines the accuracy of the comparative and logical functions. Comparison Tolerance will only matter, in practice, if either of the arguments of an affected function is represented internally as a floating-point number.

The following primitive functions are affected by ŒCT:

                 — ˜ < ˆ = ‰ > ¬  ¹ ¼  |  º  ¦  »  ~

For equality tests, two numbers are judged to be equal if the magnitude of their difference does not exceed the value of ŒCT multiplied by the larger of their magnitudes. X‰Y is true if X-Y is greater than or equal to ŒCT multiplied by larger of the magnitudes of X or Y. X>Y is true if X‰Y is true and X=Y is not.

The effect on — and ˜ is similar. Both of these functions will have no effect on an integer. A value which is close to an integer by proportionately less than ŒCT returns that integer irrespective of the direction in which it differs from that integer. All other values behave as expected. The residue function | is fuzzy, and thus A|B will return 0 if B÷A is within ŒCT of an integer value.

The default in 32-bit implementations of APLX is 1E¯13, and in 64-bit implementation is 3E¯15. It can be reset by assignment to a value between 0 and just less than 1.

                 ŒCT                     (ŒCT at normal setting)
           1E¯13
                 4=3.9
           0

Compare it with the result produced by the same expression after ŒCT has been changed:

                 ŒCT„ .026               (4×ŒCT is greater than the
                 4=3.9                    difference between the two numbers)
           1
                 ŒPP„15                  (X is less than 1 by
                 Œ„X„1-.9×ŒCT             proportionately less than ŒCT)
           0.9766
                 ˜X
           1
                 —X
           1

### Special considerations for 64-bit versions of APLX

In APLX64, integers are represented as 64-bit numbers, and floating-point numbers are represented in 64-bit IEEE floating-point format, with 53 bits of precision. The default value of ŒCT is 3E¯15. This means that, if you are dealing with numbers larger than around 2*48 (approx 2.8E14), you may get different answers for operations which depend on ŒCT, according on whether the number is represented internally as an integer or a floating-point number. This is because operations which act on integers are carried out using exact arithmetic, without reference to ŒCT, whereas operations which involve floating-point numbers do take account of ŒCT.

Consider this sequence:

          X„2*50
          X
    1125899906842624
          ŒDR X
    2
          X=X+1
    0

          Y„1.0×X
          Y
    1125899906842624
          ŒDR Y
    3
          Y=Y+1
    1

In this example, X is represented internally as a 64-bit integer. It is distinct from the 64-bit integer X+1, because ŒCT is ignored when APLX compares numbers represented as integers.

In contrast, Y (which has the same value as X) is held internally as a floating-point number. When the comparison with Y+1 is made, APLX reports that they are equal because the relative difference is less than ŒCT.

Note that, in 32-bit versions of APLX, the behavior is the same, but you do not normally notice it because, with the default value of ŒCT, no number which could be represented as a 32-bit integer is within comparison tolerance of the adjacent integer values.

### Practical implications for 64-bit APLX applications

If the magnitude of the numbers you are dealing with is less than 2*48, then with the default value of ŒCT there should be no problems. The results of arithmetic and comparison operations will not depend on whether the numbers are represented internally as floating-point or integer.

If the magnitude of the numbers you are dealing with is between 2*48 and 2*53, you might need to reduce ŒCT to ensure consistent results, or alternatively force the numbers to be represented as integers or as floats before making the comparisons.

If the magnitude of the numbers you are dealing with is greater than 2*53, then numbers represented internally as floating-point cannot be converted to integer because there is not enough precision in the floating-point representation to know which integer is the correct one. If you want exact comparisons, you need to ensure that the numbers remain in integer form (for example, do not place them in the same array as non-integral values).

Topic: APLX Help : Help on APL language : System Functions & Variables : ŒCT Comparison Tolerance

[ Previous | Next | Contents | Index ]