Paul's Programming Notes PostsRSSGithub

SQLAlchemy - Is it necessary to commit after session.execute()?

Updated 2026-07-16: updated the answer and links for SQLAlchemy 2.0.

Session.execute runs its statement within the current transaction, and SQLAlchemy’s autobegin starts that transaction on first use.

So if the statement writes (INSERT, UPDATE, DELETE), yes, you need to commit(), or the transaction rolls back when the session closes. A read-only SELECT changes nothing, so there is nothing to commit.

Here’s a code example along with the output from echo=True, showing the session begins a transaction but doesn’t end it until you run session.commit():

SQLAlchemy 1.1.4's New "server_side_cursors" Option

Before SQLAlchemy 1.1.4, if you wanted to stream your MySQL query using server side cursors, you would need to run your query using execution_options(stream_results=True) and pass SSCursor into create_engine’s connect_args. Now, all you need to do is pass server_side_cursors=True into create_engine and it will automatically stream the results for your select queries.

http://docs.sqlalchemy.org/en/latest/dialects/mysql.html#server-side-cursors

Do I need to use SQLAlchemy's scoped_session?

If your application has the potential to run in multiple threads, then you absolutely should use scoped_session. A Session isn’t safe to share across threads, and scoped_session makes sure each thread gets its own.

from sqlalchemy.orm import scoped_session, sessionmaker

Session = scoped_session(sessionmaker(bind=engine))

session = Session()  # the same session everywhere in this thread

It’s a registry keyed on the current thread by default. Every call to Session() from the same thread hands back the same object, so you can get at the session deep inside a function without threading one down through every caller, and another thread gets its own instead of stepping on yours.

You do have to call Session.remove() when the unit of work is finished, otherwise the session stays in the registry along with every object in its identity map. Web frameworks usually hook that into the end of a request. Flask-SQLAlchemy sets all of this up for you, so if you’re using that you already have a scoped session.

If your application is single threaded and you’re already passing one session around explicitly, you don’t need it.

Is it necessary to run SQLAlchemy's session.remove()?

If you’re using flask-sqlalchemy, then the answer is no - it’s already doing it for you when the request finishes: https://github.com/mitsuhiko/flask-sqlalchemy/blob/2.1/flask_sqlalchemy/init.py#L823

If you’re using scoped_session and not explicitly running session.remove(), then your connections will only be returned to the pool after your thread finishes and garbage collection occurs. It’s not a good idea to leave this to garbage collection, because you can’t guarantee your connections will be returned to the pool when your application is busy. This will often lead to errors like this: “TimeoutError: QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30”

The solution is to make sure session.remove() runs when the work finishes (like flask-sqlalchemy does), as described here: http://docs.sqlalchemy.org/en/latest/orm/contextual.html#using-thread-local-scope-with-web-applications

Another alternative is not making the session global and using a context manager, as described at the bottom of this section: http://docs.sqlalchemy.org/en/latest/orm/session_basics.html#when-do-i-construct-a-session-when-do-i-commit-it-and-when-do-i-close-it However, this can make things difficult when you need to access the query results outside of the context manager. If you access the query results outside of the context manager without running session.expunge_all(), you’ll see all kinds of errors like this: “DetachedInstanceError: Instance <ReportingJob at 0xa41cd8c> is not bound to a Session; attribute refresh operation cannot proceed”

Another option is setting autocommit to True, but that has quite a few gotchas. Turning on autocommit will acquire connections from the engine on an as-needed basis and return them immediately after their use. You will have to explicitly start transactions with session.begin() if you enable it. It seems like this setting should be called “autotransaction” instead. It’s also not very efficient, because it needs to get and return a connection each time you run a query.

PHP - max_execution_time doesn't work for socket operations

In new relic transaction logs, I was seeing some PHP requests last as long as an hour. It turns out that time spent waiting on sockets doesn’t apply toward max_execution_time, and loops that involved waiting on sockets could end up taking a really long time without timing out.

PHP docs have this to say about max_execution_time:

“The set_time_limit() function and the configuration directive max_execution_time only affect the execution time of the script itself. Any time spent on activity that happens outside the execution of the script such as system calls using system(), stream operations, database queries, etc. is not included when determining the maximum time that the script has been running. This is not true on Windows where the measured time is real.”

The solution ended up being setting request_terminate_timeout in php-fpm.

Python - Cachetools LRUCache KeyError

If your application is threaded and you’re getting a “KeyError” while using the non-decorator version of cachetool’s LRUCache, then you need to put whatever is manipulating the cache object inside of a lock. Also, since LRUCache is modified when values are gotten from it, you will also need to make sure you’re locking when you get values from cache too. If you can use the decorator version of LRUCache, that’s preferred since it has built-in locking.

Here’s an example of the error:

And an example of the fix: https://bitbucket.org/zzzeek/dogpile.cache/pull-requests/32/add-a-cachetools-lru-lfu-in-memory-backend/diff#comment-22242704

Javascript's Round vs PHP's Round

Today I learned that by default PHP rounds differently than javascript.

PHP (using the default PHP_ROUND_HALF_UP)

php > echo round(-1.5);
-2

“Round val up to precision decimal places away from zero, when it is half way there. Making 1.5 into 2 and -1.5 into -2.”

https://secure.php.net/manual/en/function.round.php

Javascript

Math.round(-1.5);
-1

“For negative numbers, if the decimal portion is exactly -0.5, the return value is the smallest integer that is greater than the number.”

https://msdn.microsoft.com/en-us/library/5cza0web(v=vs.94).aspx