Sub-Interpreters

RedisGears provides a complete embedded Python interpreter, which makes it possible to run Python code inside Redis using the RG.PYEXECUTE command. Because the interpreter is a singleton that's shared among all calls to RG.PYEXECUTE, there's the risk of different executions using the same identifiers (e.g. for global variables, function or class names).

One possible way to address this problem by restarting the interpreter before each execution. However, an interpreter restart is a costly operation, time-wise, which this approach less-than-desirable. In addition, two executions may be running simultaneously, either in two different threads or even in the same thread (while the execution of the Python code itself is always non-parallelized, the rest of the execution's lifecycle may be). Obviously, when more than one execution is running, restarting the interepreter is no longer an option. To address the issue correctly RedisGears make use of Sub-Interpreters.

A sub-interpreter is a (almost) totally separate environment for the execution of Python code. The Python C API makes it possible to create a new sub-interpreter using Py_NewInterpreter, destroy it using Py_EndInterpreter and switch between sub-interpreters using PyThreadState_Swap. RedisGears invokes these internally and maintains the association between the user's call to RG.PYEXECUTE and its respective sub-interpreter.

When RG.PYEXECUTE is called, a new sub-interpreter is created to execute the provided script. That sub-interpreter is also "inherited" by all subsequent operations - i.e. executions, registrations and timeEvents, that the script creates. Because there may be multiple owner for the sub-interpreter, RedisGears keeps an inernal reference count for each one so it can be safely freed.

Notice that the isolation between Sub-Interpreters isn’t absolute. For example, when using low-level file operations like os.close() they can (accidentally or maliciously) affect each other’s open files because of the way extensions are shared between sub-interpreters. For more information, read sub-interpreters documentation.