Performance of importData(9)

Performance of importData(9) - Messages

#1 Posted: 8/19/2013 12:16:38 PM
Martin Kraska

Martin Kraska

1222 likes in 2150 posts.

Group: Moderator

Perhaps not a bug, but a problem. The importdata(9) function has a quadratic dependence of time consumption versus datasets read. For large files this becomes prohibitive. Pre-allocation of the matrix does not help.

If you read the file piecewise into separate matrices, then the time consumption is
linear. However, combining them using stack() turns out to be even more ineffective. Other operations like column extraction or plotting augmented columns fast in comparison to read operation. Therefore, an efficient text file reading routine would be highly welcome.

If you want to try the example, put data and .sm file in the same directory and launch the .sm file there.
readtest.PNG
Martin Kraska Pre-configured portable distribution of SMath Studio: https://en.smath.info/wiki/SMath%20with%20Plugins.ashx
1 users liked this post
Radovan Omorjan 8/19/2013 12:46:00 PM
#2 Posted: 9/8/2013 6:17:20 AM
Martin Kraska

Martin Kraska

1222 likes in 2150 posts.

Group: Moderator

Problem solved. The quadratic time dependence holds for optimization> symbolic. For optimization> numeric, the time dependence is approximately linear.
DataExchange.zip (381 KiB) downloaded 53 time(s).
Martin Kraska Pre-configured portable distribution of SMath Studio: https://en.smath.info/wiki/SMath%20with%20Plugins.ashx
  • New Posts New Posts
  • No New Posts No New Posts