September 25, 2006

Imprecision

Posted in Uncategorized at 4:14 am by lugnuthorsefly

Reminder to self: Even the most mundane floating point operations can be inaccurate


irb(main):030:0> str = '$2.01'
=> "$2.01"
irb(main):031:0> str =~ /\$(\d+\.\d+)/
=> 0
irb(main):032:0> $1
=> "2.01"
irb(main):033:0> ($1.to_f * 1000).to_i
=> 2009
irb(main):034:0> (($1.to_f * 1000.0) + 0.5).floor
=> 2010

Advertisements