SQLite Interface Plugin - A plugin that allows SMath Studio to interface the SQLite relational database engine - Messages






Description:
This plugin allows SMath Studio to interface the SQLite relational database engine. It has been developed for .NET 2.0 and SMath 0.98.5935 or newer
Download & Installation:
For SMath versions 0.98.5935 or newer the plugin can be conveniently downloaded and installed using the extension manager.
Example and source code are now available in the SVN as well: https://smath.info/svn/public/plugins/SQLitePlugin/
To manually install the plugin, copy the folder (and all its contents) from the "SQLitePluging-[version].zip" archive into "%appdata%\SMath\extensions\plugins\".
To run the example extract the contents of the "SQLiteExample-[version].zip" to any directory, make sure database and SMath sheet are located in the same directory and open the example sheet. Some older versions of SMath seem to require an additional "eval" in some of the statements, so two versions of the example sheet are provided.
Usage
Usage of the plugin is very simple:
SQLiteQuery("path", "query"
All used database files are automatically closed when SMath Studio is closed.
Example
Limitations, Known Problems and Remarks
- Beta version, use with caution
- I have not yet checked whether different culture settings lead to conversion errors
Only SELECT statements have been tested so far
Limited testing on write access queries seems promising, so farTwo DLL files need to be copied to the plugins folder, I intend to merge the assemblies in futureAssemblies merged, plugin is now a single dll
Complete folder provided, Extension Manager upload in progressOnly 64 bit is currently supported, sorry. Working on 32 bit support
Both 64 and 32 bit now supported (tested in 5935, 6081 and 6083 using WoW64)Since it is not possible to use "=" within a SMath string this has to be considered when designing a query. "IS" and is usually a suitable replacement. "IS NOT" serves as a replacement for "!="
The boolean 'equal' (Ctrl + =) works very well (tested). Thanks Martin Kraska and Mike Kaganski for pointing out my error
Story
I started using SMath Studio about 6 months ago, when I got sufficiently annoyed with the free version of MathCAD (no linterp? seriously?) and have since grown quite fond of it. I had a look around the Forum and the sheer amount of time, effort and dedication some of the people involved seem to invest is rather inspiring.
Yesterday I found myself in need of a way to conveniently load a whole set of material properties and coefficients for curve fits by only specifying a part number - a task that a relational database seemed well suited for. I could not find an existing database interface and was quite interested in learning more about SMath plugin development, so I wrote the plugin myself (having previous experience with SQLite and C#).
The performance seems pretty good, compared with the Excel I/O plugin. Comparing the corresponding examples the SQLite version executes on my machine initially in 0.7s, then in about 0.03s once the DB is opened. The Excel version takes 3.2s initially, 0.18s once the excel sheet is open.
Since I enjoy the privilege to freely use SMath Studio, I gladly provide my plugin here as well, hoping you may find it to be useful.
Please feel free to have a look and let me know what you think. I would very much welcome any problem reports, suggestions for improvements, requests for further explanations etc.
Acknowledgements
- Alex.M - I have taken the liberty here to adapt the example and database form Alex' Excel I/O plugin to demonstrate the SQLite interface within a context that might be familiar to some users. I had to change some property names since SQLite ignores case, b and B have become b1 and b2 etc.
- Davide Carpi - The XlsxImportExport plugin demonstrated to me how to properly return results in form of a Matrix. Also thanks for the useful feedback
- Martin Kraska - I found the "SMath Studio Handbuch" to be a great starting point and reference
- Martin Kraska & Kay Graubmann - The Maxima plugin source code provided some valuable insights into SMath plugin development
- Mike Kaganski - His quick testing, excellent feedback and good ideas have substancially accellerated and improved the development of this plugin
Changelog
- 08/09/16 - Made plugin available to download via extension manager, uploaded source code and example to SVN, fixed relative path as suggested by Davide, changed loading of linked SQLite assemblies for x86 and x64 to allow installation from extension manager for SMath below 8069
- 31/08/16 - Removed single argument SQLiteQuery due to inconsistencies with partial recalculation. Null entries in database return custom unit 'NULL and are now distinguishable from empty strings, simplified code. Fixed and improved the example file, added typetest table to example database
- 29/08/16 - Updated SQLiteQuery to use path as handle, removed SQLiteOpen and SQLite close, added automatic closing of database connections at SMath exit, typecasting bugfixes, refactoring and minor improvements, repackaged for proper plugin installation
- 27/08/16 - Merged assemblies, now the plugin only consists of a single dll. Correct SQLite interop is automatically loaded, 64bit and 32bit are now supported. (Tested with SMath 5935, 6081 and 6083 )
- 26/08/16 - Added support for multiple simultaneous database connections, returning results consistently as matrix now
- 25/08/16 - First release
SQLite resources:
- https://sqlite.org/about.html - SQLite overview
- https://www.sqlite.org/datatype3.html - More about SQLite type affinities
- http://www.tutorialspoint.com/sqlite/index.htm - Useful SQLite tutorial
- http://portableapps.com/apps/development/sqlite_database_browser_portable - Portable SQLite database file editor
- Could you consider using a identifier (number or string) as a "handle" to open connection? For example,Or the SQLiteOpen could return the numeric handle on succcess, or an error with a description on failure.
SQLiteOpen("connection1","path1"
="open" SQLiteOpen("connection2","path2"
="open" m:=SQLiteQuery("connection1","query"
query2:=<some processing involving m> SQLiteQuery("connection2",query2) <some other processing with both connections 1 and 2>
If no fields are returned by the database engine, the function will return the string "No Results"
Imo, it would be better to return matrix(0,0) in this case. And please return a matrix in all cases, even on single return, thus eliminating the need for conditionals in the calculations. If one knows that only single return will come, this may be handled by simply using el(result,1).- It is possible to use "=" in SMath strings. You just need to prepate the string elsewhere.
For some reason, SMath Studio 0.98.6081.26558 doesn't start with the plugin, so I cannot test it. Does it handle partial recalculations? Doesn't closing in the end prevent partial recalcs? Is it possible to skip closing the connection altogether and thus skip overhead opening new connection on recalc? I.e. does it internally checks in SQLiteOpen if the connection is already open?
Crappy PTC vs so nice Mathsoft !
In case you need linterp in what works from your free version:
=> here is the code, except for the Mathcad index 0 ORIGIN.
In supplement, I have the code l_p_cspline [not attached].
Jean
Interpolate [Linterp, Table].sm (39 KiB) downloaded 131 time(s).
Spline Matrix Tutorial.sm (54 KiB) downloaded 134 time(s).
Wrote[size=8][b]Since it is not possible to use "=" within a SMath string this has to be considered when designing a query. "IS" and is usually a suitable replacement. "IS NOT" serves as a replacement for "!=" "a <= b" can be replaced with "(a < b ) or (a is b )"
Actually you can have = in strings. Just use the = symbol from the boolean palette.
I have edited the the inital post with an updated version of the plugin.
WroteCould you consider using a identifier (number or string) as a "handle" to open connection?
Yes, thank you. This is exactly what I have done, SQLiteOpen will now return the index of the database connection that has just been opened, starting with 0. To prevent memory leaks associated with undesired reopening of database files and to speed up sheet recalculation the SQLite plugin will not reopen the same file again and instead return the handle that has been generated when the file was opened the first time. If the file has since been closed it will be reopened but the handle will not change.
WroteImo, it would be better to return matrix(0,0) in this case. And please return a matrix in all cases, even on single return, thus eliminating the need for conditionals
A very good suggestion, implemented straight away.
WroteIt is possible to use "=" in SMath strings.
WroteActually you can have = in strings.
Thank you for pointing out my mistake, tested and found to be working nicely, updated in original post.
WroteFor some reason, SMath Studio 0.98.6081.26558 doesn't start with the plugin, so I cannot test it. Does it handle partial recalculations? Doesn't closing in the end prevent partial recalcs? Is it possible to skip closing the connection altogether and thus skip overhead opening new connection on recalc? I.e. does it internally checks in SQLiteOpen if the connection is already open?
I'll download 0.98.6081.26558 and try to solve the issue as soon as possible.
Partial recalculation seems to work as far as I can tell.
You are quite right, closing the database connection in the end makes a full recalculation necessary.
Closing the database file is not necessary if the database is only read. At the latest when SMath Studio is closed the database connections are terminated. I am however not sure whether "SQLiteClose" can be safely ignored if write operations to the database have been performed.
Opening a file will not impose additional overhead if the file is already open. The plugin will now recognise this and simply report the handle.
Please let me know if you have further ideas or requests.

WroteYou are quite right, closing the database connection in the end makes a full recalculation necessary.
Closing the database file is not necessary if the database is only read. At the latest when SMath Studio is closed the database connections are terminated. I am however not sure whether "SQLiteClose" can be safely ignored if write operations to the database have been performed.
You can always collect any handle opened and close all them internally when SMath closes (Dispose() method)

1. SQLiteOpen() is unnecessary.
Rationale: currently, you maintain internally a structure that maps a DB path, "handle" that is returned by SQLiteOpen() and (presumably) an internal session handle. Each SQLiteOpen() invocation checks if the DB denoted by path is already open, and if so, it returns already assigned "handle", otherwise it creates a new session.
This may be implemented instead in any functional SQLite function (like SQLiteQuery()), so that the only SMath database "handle" would be the DB path itself.
The proposal is the following: drop SQLiteOpen(); in all other SQLite functions, make a mandatory first argument "path". At each point in the code where SQLite DB is accessed, use the map lookup to find out if the DB is already open, and open it if necessary.
I believe that the overhead of string comparison is negligible compared to SQL parsing etc., and this would simplify the interface and make it more functional (avoid purely auxiliary function that doesn't return result that is directly useful for user).
2. SQLiteClose is unnecessary.
Rationale: the function's intention is apparently to (1) flush the possible changes made by modifying SQL, and (2) to release the DB file (e.g., to allow its renaming/removing/other file management). But the function isn't guaranteed to be executed. E.g.: if the DB is open (and possibly modified), then goes a lengthy block of calculations, and after that, the SQLiteClose is invoked. A user can interrupt the calculations, thus disrupting the intended resource flush/cleanup. Also, see no.4 below for a concurrent access discussion.
So, to have a robust DB handling, the following is proposed: drop SQLiteClose() entirely as purely auxiliary; instead, on plugin cleanup*, close all open DB connections properly**.
* The cleanup is either at plugin unload (at SMath unload), or at sheet close - see no.4 below.
** The proposed method of closing properly is Commit - see no.3 below.
3. Two new functions are needed: SQLiteCommit(path) and SQLiteRoolback(path).
Rationale: these are functions that allow user to control if and when the data is actually committed to DB. It mat be used if necessary; and may be omitted, thus defaulting to commit at DB closing (see no.2 above).
4. Concurrency should be considered.
It is possible that two sheets in one SMath process try to access the same DB simultaneously; or that two SMath instances try to access same DB; or that SMath and another app try to use the same DB; or two different computers try to use a DB on a network share.
I am not familiar with how SQLite solves these situations itself; it is a sane assumption that accessing a DB from different processes could be prohibited (file lock).
Opening it from the same process / different sheets is required to be thread-safe. Also, it is desirable that closing any single SMath sheet would commit data and close its connections (as discussed in no.2 above).
Alternatively, if there isn't a way to catch the "sheet closing" event, then it could be done at SMath exit, but it's an inferior solution, because it will keep DB files locked after using sheets are closed.
But there is a problem WRT commit/rollback logic. If it's possible to keep different transactions for each sheet, then there is a question how the resulting conflicting commits will be handled. Consider this: sheet A inserts a record to a table into a DB; sheet 2 inserts another record into the same DB. Sheet 1 is closed. Sheet 2 is closed. If they maintained different transactions, the two records may happen to have the same primary key -> ...? If they share the same transaction, then the problem is different: if sheet 1 commits, and after that sheet 2 rollbacks, then the rollback of sheet 2 would be unsuccessful. I suppose that if these problems are not solved in a consistent way, then concurrent use of DBs should be prohibited.
I have updated the plugin
with the following changes (link in initial post):
- 32 Bit and 64 Bit are now supported
- The assemblies are now merged, only a single dll file needs to be copied to the plugins folder
Unfortunately I could not reproduce this problem:
WroteFor some reason, SMath Studio 0.98.6081.26558 doesn't start with the plugin
I have successfully tested the updated version with SMath Studio versions 0.98.5935, 0.98.6081 and 0.98.6083, both in native 64 bit mode as well as in 32 bit mode using WoW64.
The behaviour of concat seems to be different in 0.98.6081, 0.98.6083 which seems to break the example file (will update soon) for these versions. The plugin itself works for me, however. Any feedback from other users would be much appreciated.
I would be especially grateful if a user with 32 bit system could report whether the plugin works as intended. If this is the case I will upload it to the extension manager as a beta version.
On the other hand, if I put it into %appdata%\SMath\extensions\plugins\SQLite\1.0.6083.3, and put a config.ini in parent folder with "1.0.6083.3", then SMath crashes on start.
Next question/feature request: a UI for creating DB and tables. That would be wonderful, esp. for those who don't have other utilities for that. I don't think it's best done with custom functions, rather some dialog UI would be perfect... or a menu item that starts a free external utility for that, that is distributed in the directory with the plugin. Well, I propose dialog UI just because I don't see a good way to do it via functions - but I may well be wrong...
Also:
SQLiteQuery(0,"select * from Shapes"
SQLiteQuery(0,"select * from Units"
I suppose it may have something to do with decimal separator? Or a problem when returning an empty value?
Also2:
SQLiteQuery(0,"select 'h/tw' from Shapes where AISC_Manual_Label is 'W44X335'"
=["TYPE ERROR"]
with the following changes (link in initial post):
- SQLiteOpen is not necessary anymore, the handle in SQLiteQuery is now the filename as suggested by Mike Kaganski
- SQLiteClose is not necessary anymore. Open database connections are closed on Dispose as suggested by Davide Carpi
- Fixed illegal cast and type error bugs as kindly reported by by Mike Kaganski (both data type conversion issues)
- Since deployment as single dll does not seem to be necessary I split the assembly back up to reduce size and increase loading speed
- Refactoring and minor improvements
- NULL values are now returned as "" instead of "NULL"
Thanks to Mike Kaganski for testing the plugin so quickly and reporting problems.
WroteOk, the plugin opens *if* it is placed into %ProgramFiles(x86)%\SMath Studio\Plugins. That's a wrong place for custom plugins.
On the other hand, if I put it into %appdata%\SMath\extensions\plugins\SQLite\1.0.6083.3, and put a config.ini in parent folder with "1.0.6083.3", then SMath crashes on start.
I was not aware of the correct way to install plugins before, thanks for this information. The crash can be avoided by using a folder name which is any valid GUID,
e.g: "%appdata%\SMath\extensions\plugins\a79333d4-bf34-4aa6-93e5-5a06b6548d24\1.0.6083.4".
The updated zip file in the initial post now contains this folder. (GUID generated here)
WroteNext question/feature request: a UI for creating DB and tables.
Good idea! I guess convenience is much more important than HD memory usage. As soon as I find out how to add menu items I will bundle the SQLite Plugin with SQLite Studio (GPL)
I will look into points 3 and 4 over the next couple of days but from what I understand SQLite seems to already handle these automatically to some degree. I'm fairly confident that write access queries would be immediately committed as the plugin is currently implemented, but I will double check and confirm this. Another problem that arises from this is insert speed. Since every query individually implicitly creates a transaction, allocates memory, prepares and execures a statement etc. write access is slow at the moment. The use of preprepared statements and transactions could improve this tremendously, but I don't yet see how to integrate this with SMath in a sensible way... will spend some more thought on this.
Will test ASAP.
Is isolated SQLiteQuery(sql) still available (as updated top post suggests)? (I doubt it's useful. A variable with DB path is short enough, and won't allow for difficult-to-find errors where a user inserts a SQLite query to another DB before the simplified query, and will see strange results).
Also: an idea how to return NULLS: you may return instead SMath custom unit 'NULL. You don't have to define units beforehand; it won't cause "undefined" errors, and will allow testing for "if return='NULL then ...". Also it will help differentiate between returned real empty strings and nulls.
A side question. I see some fancy column names in the sample DB (like "h/tw" ); how do I query for them (aside from using "*" syntax)?
WroteBut I see a problem with returning empty result sets:
Something's wrong with the approach I suggested (using matrix(0,n)). Maybe Andrey could suggest something here?
I think the approach "matrix as result" is good, maybe there is something to improve in the SS logic. If you try to assign matrix(0,n) to a variable, you can show mat(0,n) in a standalone display of the result with numeric evaluation if at definition is stored symbolically, but not if it is stored numerically (that is the same error while you are trying to display the inline result); this affects also rows()/cols()/length() (unable to process the variable when it is stored numerically). I'll talk with Andrey about this.
BTW 'NULL as result is very good idea, +1 for this.
Jason
I must yet get used to your new status as SMath co-developer. I haven't yet had a chance to congratulate you with that


WRT empty matrices. There used to be yet another application for them: using as initializers for loops, to assign an empty n-col 0-row matrix to a variable, and then use stack() uniformly in a loop to fill it (of course, if the number of rows is known in advance, that may be done differeently and more efficient). This functionality was definitely available at some point (unfortunately, I cannot find a link to it atm, because I explained this technique in Russian forum, that is unavailable now).

I know about this useful feature, that's why I think this particular form should be enforced
BTW the russian side of the forum is still available for reference

with the following changes (link in initial post):
- Removed single argument SQLiteQuery due to inconsistencies with partial recalculation.
- Null entries in database return custom unit 'NULL and are now distinguishable from empty strings, simplified code.
- Fixed and improved the example file
- Added typetest table to example database and tested every case successfully
Thanks yet again for all the useful feedback I've received.
Wrote
I completely oversaw this issue, to preserve consistency and repeatability even with partial recalculation I have removed the single argument version of SQLiteQuery
Wroteyou may return instead SMath custom unit 'NULL.
A very good idea again, implemented.
WroteI see some fancy column names in the sample DB (like "h/tw" ); how do I query for them
Either of the following will work:
Select `twdet/2` from Shapes;
Select 'twdet/2' from Shapes;
WroteI'm not very familiar with SQL, but this looks like it would be very useful! (...) AISC does make their tables available in spreadsheet format
Thanks for your interest Jason. I'd quite happily convert that spreadsheet into an SQLite database if it helps your endeavours

Wrote
Mike Kaganski - His quick testing, excellent feedback and good ideas have substancially accellerated and improved the development of this plugin
Oh, thank you, but you know, generating ideas is light years away easier than all that coding heavy-lifting.
Wrote
Either of the following will work:... Select 'twdet/2' from Shapes;
Well, this specific second form doesn't work for me; (actually I tried it before) but the first one you've suggested works like a charm! Thanks.
Wrote
WroteI'm not very familiar with SQL, but this looks like it would be very useful! (...) AISC does make their tables available in spreadsheet format
Thanks for your interest Jason. I'd quite happily convert that spreadsheet into an SQLite database if it helps your endeavours
I appreciate the offer, but it looks like you already have.

This really opens up some exciting possibilities for me on some of my worksheets once I learn how to use this database stuff

Jason
http://smath.info/upload/Extensions.aspx redirects me to http://smath.info/ReleaseManager/ which unfortunately results in a Server Error for me ExtensionManagerError.zip (1 KiB) downloaded 100 time(s).
Would anyone happen to know how to get around this?
-
New Posts
-
No New Posts