Time for calculating the variance() & stddev() from csv file

Time for calculating the variance() & stddev() from csv file - Сообщения

#1 Опубликовано: 03.04.2024 20:10:07
MN283

MN283

0 сообщений из 2 понравились пользователям.

Группа: User

Hi SMath community,

I would like to calculate the variance() and StdDev() of the second column "V(3v3)" of the attached file with 50k datapoints. Unfortunately the calculation takes forever (aborted it after 30min).

The file is an export of a circuit simulation result (gaussian tolerance distribution in LTSpice). Originally it is in .txt-Format and it seems that I have to convert it in excel to .csv-Format before I can use is. I attached both, the original LTSpice export (.txt) and the converted one from Excel (.csv).

How can I handle this in sMath?

Archiv.zip (585 КиБ) скачан 31 раз(а).

Thanks in advance,
Marc
#2 Опубликовано: 03.04.2024 22:34:03
Davide Carpi

Davide Carpi

1415 сообщений из 2872 понравились пользователям.

Группа: Moderator

With import file functions you might need to set the optimization to numeric to speed up the loading.
If you like my plugins please consider to support the program buying a license; for personal contributions to me: paypal.me/dcprojects
#3 Опубликовано: 03.04.2024 23:32:05
MN283

MN283

0 сообщений из 2 понравились пользователям.

Группа: User

Optimizations were all set to numeric - did not help. What helped was that I substituted the statistic function variance() and StdDev with the original/native formula for the variance and standard deviation.

This speeds it up to around 1s.

Don´t know what happens inside the functions of the statistic plugin but it is very slow.

calculation.png

Thanks
#4 Опубликовано: 11.04.2024 06:24:17
Fikret Atalay

Fikret Atalay

2 сообщений из 24 понравились пользователям.

Группа: User

I have noticed the same thing; the built-in function for standard deviation takes forever but the calculation runs pretty quickly if I just code in the formula in the worksheet.
#5 Опубликовано: 12.04.2024 16:53:41
Davide Carpi

Davide Carpi

1415 сообщений из 2872 понравились пользователям.

Группа: Moderator

Confirmed, the bottleneck seems to be in the final ouput processing to return the result from the equation [f.e. 1/{n-1}*sum(((el(X,i)-mean)^2),i,1,n)) for StdDev]; no idea not sure what could be the cause...
If you like my plugins please consider to support the program buying a license; for personal contributions to me: paypal.me/dcprojects
  • Новые сообщения Новые сообщения
  • Нет новых сообщений Нет новых сообщений