Topic: simple subtraction

Hello guys,

Does anyone know why

10.8 - 10 gives as a result 0.800000000000001 instead simply 0.8

and 10.8 - 9 gives correctly 1.8 ?


Thanks

Sig

Re: simple subtraction

Its due to the way floats are rounded, in this case eight tenths can't be accurately represented as a binary fraction, so it rounds it.

Google 'representation error' for some pretty heavy reading about this smile

Also take a look at Ruby's bigdecimal class as this does fixed point rather than floating point which can help.

Rob Anderton
TheWebFellas

Re: simple subtraction

Thanks for your reply.

Is there a way to fix it?

Re: simple subtraction

Depends what you're trying to acheive, and what the numbers actually represent (prices, distances, etc).

As I say you could use BigDecimal:

require 'bigdecimal'
puts BigDecimal.new("10.8") - BigDecimal("10")
=> 0.8E0

Docs are here: http://ruby-doc.org/stdlib/libdoc/bigde … index.html

Rob Anderton
TheWebFellas

Re: simple subtraction

I'm  trying to get the right result when I subtract floats. The number represents analytical tests, for instance the pH of different samples.

I take a look of BigDecimal.

Thanks

Sig

Re: simple subtraction

ya i think  rob-twf give u ri8 ans , u can follow it