September 25, 2006

Imprecision

Posted in Uncategorized at 4:14 am by lugnuthorsefly

Reminder to self: Even the most mundane floating point operations can be inaccurate


irb(main):030:0> str = '$2.01'
=> "$2.01"
irb(main):031:0> str =~ /\$(\d+\.\d+)/
=> 0
irb(main):032:0> $1
=> "2.01"
irb(main):033:0> ($1.to_f * 1000).to_i
=> 2009
irb(main):034:0> (($1.to_f * 1000.0) + 0.5).floor
=> 2010

Advertisements

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: