Comparison Tolerance: OCT
124
0
The value of this variable determines the maximum tolerance (how different the two
numbers must be to be considered unequal) when using any relational function and
at least one argument is a noninteger. For example, two numbers are considered un-
equal if the relative difference between the two numbers exceeds the comparison
tolerance value. The following illustration shows how the comparison tolerance
works with the relational functions:
Value of argument A
Real number line
The relationship of
any value (argument 6)
to argument A - ArB
- A>B
Note: The OCT function considers any number in decimal form a noninteger. For
example, 1000 is an integer and 1000. is a noninteger.
The value of the comparison tolerance variable also affects the floor and ceiling
functions. If an integer is in the range of the right argument plus or minus the
comparison tolerance, the integer is the result. For example:
[IC1’+. , 03
1-2 , 98
I.. 2 , 9 6
I‘ 3 1 0 3
2.98 + .03 = 3.01 (The integer 3 is in the range of
2.96 + .03 = 2.99
3.03 - -03 = 3 (The integer 3 is in the range of
3 2.98 ! .03.)
3
3 3.03 ? .03.)
3
rz. 04. 3.04 - .03 3.01
In a clear workspace, the comparison tolerance value is set to 1 E-13 (see
Chaptey 3 for an explanation of scaled representation).
124
0
The value of this variable determines the maximum tolerance (how different the two
numbers must be to be considered unequal) when using any relational function and
at least one argument is a noninteger. For example, two numbers are considered un-
equal if the relative difference between the two numbers exceeds the comparison
tolerance value. The following illustration shows how the comparison tolerance
works with the relational functions:
Value of argument A
Real number line
The relationship of
any value (argument 6)
to argument A - ArB
- A>B
Note: The OCT function considers any number in decimal form a noninteger. For
example, 1000 is an integer and 1000. is a noninteger.
The value of the comparison tolerance variable also affects the floor and ceiling
functions. If an integer is in the range of the right argument plus or minus the
comparison tolerance, the integer is the result. For example:
[IC1’+. , 03
1-2 , 98
I.. 2 , 9 6
I‘ 3 1 0 3
2.98 + .03 = 3.01 (The integer 3 is in the range of
2.96 + .03 = 2.99
3.03 - -03 = 3 (The integer 3 is in the range of
3 2.98 ! .03.)
3
3 3.03 ? .03.)
3
rz. 04. 3.04 - .03 3.01
In a clear workspace, the comparison tolerance value is set to 1 E-13 (see
Chaptey 3 for an explanation of scaled representation).