Jump to content
  • 0

Non-accurate results of integers division


CmP

Question

Posted

Greetings, I have experienced strange problem with integers division while writing a script.

Dividing one integer by another always resulted in a number without fractional part, even when it is supposed to be present.
Let me illustrate this with an example. Here is the code:

local test = {}
test = gg.prompt({'Input number:'}, {}, {'number'})
print(test[1])
print(tonumber(test[1]))
print(tonumber(test[1]) / 7)

And the result of it's execution:

 image.thumb.png.9e240e54108b790c2c4b60c689c88140.png

As you see, fractional part is not there, athough it should have been.

I have figured out a solution to fix this. Adding ".0" to an integer and passing this to "tonumber" function seems to produce a floating-point number, I guess.
Replacing last line from code above with this one:

print(tonumber(test[1] .. '.0') / 7)

results in next output (division result with fractional part):

image.thumb.png.687a335b5dcfb8e64933ab11a2fa4d0f.png


So my questions are next.
Why is this happening?
Are there other ways to deal with this problem?

Thank you in advance!

P.S. This problem does not appear when executing practically same code as above with Lua interpreter.
Code:

local test = {}
test[1] = 100
print(test[1])
print(tonumber(test[1]))
print(tonumber(test[1]) / 7)

Output:
image.png.3d53a4cb19c1098cac383ae38ef608b4.png

1 answer to this question

Recommended Posts

  • Administrators
Posted

Integer division if both is integer. If you need float division, then make first argument float.

Float can lose precision of big numbers.

Archived

This topic is now archived and is closed to further replies.

×
×
  • Create New...

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.