9.6 使用Python CFFI混合C,C++,Fortran和Python

NOTE:此示例代码可以在 https://github.com/dev-cafe/cmake-cookbook/tree/v1.0/chapter-9/recipe-06 中找到,其中有一个C++示例和一个Fortran示例。该示例在CMake 3.11版(或更高版本)中是有效的,并且已经在GNU/Linux、macOS和Windows上进行过测试。

前面的三个示例中,我们使用Cython、Boost.Python和pybind11作为连接Python和C++的工具。之前的示例中,主要连接的是C++接口。然而,可能会遇到这样的情况:将Python与Fortran或其他语言进行接口。

本示例中,我们将使用Python C的外部函数接口(CFFI,参见https://cffi.readthedocs.io)。由于C是通用语言,大多数编程语言(包括Fortran)都能够与C接口进行通信,所以Python CFFI是将Python与大量语言结合在一起的工具。Python CFFI的特性是,生成简单且非侵入性的C接口,这意味着它既不限制语言特性中的Python层,也不会对C层以下的代码有任何限制。

本示例中,将使用前面示例的银行帐户示例,通过C接口将Python CFFI应用于Python和C++。我们的目标是实现一个上下文感知的接口。接口中,我们可以实例化几个银行帐户,每个帐户都带有其内部状态。我们将通过讨论如何使用Python CFFI来连接Python和Fortran来结束本教程。

第11章第3节中,通过PyPI分发一个用CMake/CFFI构建的C/Fortran/Python项目,届时我们将重新讨论这个例子,并展示如何打包它,使它可以用pip安装。

准备工作

我们从C++实现和接口开始,把它们放在名为account/implementation的子目录中。实现文件(cpp_implementation.cpp)类似于之前的示例,但是包含有断言,因为我们将对象的状态保持在一个不透明的句柄中,所以必须确保对象在访问时已经创建:

  1. #include "cpp_implementation.hpp"
  2. #include <cassert>
  3. Account::Account()
  4. {
  5. balance = 0.0;
  6. is_initialized = true;
  7. }
  8. Account::~Account()
  9. {
  10. assert(is_initialized);
  11. is_initialized = false;
  12. }
  13. void Account::deposit(const double amount)
  14. {
  15. assert(is_initialized);
  16. balance += amount;
  17. }
  18. void Account::withdraw(const double amount)
  19. {
  20. assert(is_initialized);
  21. balance -= amount;
  22. }
  23. double Account::get_balance() const
  24. {
  25. assert(is_initialized);
  26. return balance;
  27. }

接口文件(cpp_implementation.hpp)包含如下内容:

  1. #pragma once
  2. class Account
  3. {
  4. public:
  5. Account();
  6. ~Account();
  7. void deposit(const double amount);
  8. void withdraw(const double amount);
  9. double get_balance() const;
  10. private:
  11. double balance;
  12. bool is_initialized;
  13. };

此外,我们隔离了C-C++接口(c_cpp_interface.cpp)。这将是我们与Python CFFI连接的接口:

  1. #include "account.h"
  2. #include "cpp_implementation.hpp"
  3. #define AS_TYPE(Type, Obj) reinterpret_cast<Type *>(Obj)
  4. #define AS_CTYPE(Type, Obj) reinterpret_cast<const Type *>(Obj)
  5. account_context_t *account_new()
  6. {
  7. return AS_TYPE(account_context_t, new Account());
  8. }
  9. void account_free(account_context_t *context) { delete AS_TYPE(Account, context); }
  10. void account_deposit(account_context_t *context, const double amount)
  11. {
  12. return AS_TYPE(Account, context)->deposit(amount);
  13. }
  14. void account_withdraw(account_context_t *context, const double amount)
  15. {
  16. return AS_TYPE(Account, context)->withdraw(amount);
  17. }
  18. double account_get_balance(const account_context_t *context)
  19. {
  20. return AS_CTYPE(Account, context)->get_balance();
  21. }

account目录下,我们声明了C接口(account.h):

  1. #ifndef ACCOUNT_API
  2. #include "account_export.h"
  3. #define ACCOUNT_API ACCOUNT_EXPORT
  4. #endif
  5. #ifdef __cplusplus
  6. extern "C"
  7. {
  8. #endif
  9. struct account_context;
  10. typedef struct account_context account_context_t;
  11. ACCOUNT_API
  12. account_context_t *account_new();
  13. ACCOUNT_API
  14. void account_free(account_context_t *context);
  15. ACCOUNT_API
  16. void account_deposit(account_context_t *context, const double amount);
  17. ACCOUNT_API
  18. void account_withdraw(account_context_t *context, const double amount);
  19. ACCOUNT_API
  20. double account_get_balance(const account_context_t *context);
  21. #ifdef __cplusplus
  22. }
  23. #endif
  24. #endif /* ACCOUNT_H_INCLUDED */

我们还描述了Python接口,将在稍后对此进行讨论(__init_ _.py):

  1. from subprocess import check_output
  2. from cffi import FFI
  3. import os
  4. import sys
  5. from configparser import ConfigParser
  6. from pathlib import Path
  7. def get_lib_handle(definitions, header_file, library_file):
  8. ffi = FFI()
  9. command = ['cc', '-E'] + definitions + [header_file]
  10. interface = check_output(command).decode('utf-8')
  11. # remove possible \r characters on windows which
  12. # would confuse cdef
  13. _interface = [l.strip('\r') for l in interface.split('\n')]
  14. ffi.cdef('\n'.join(_interface))
  15. lib = ffi.dlopen(library_file)
  16. return lib
  17. # this interface requires the header file and library file
  18. # and these can be either provided by interface_file_names.cfg
  19. # in the same path as this file
  20. # or if this is not found then using environment variables
  21. _this_path = Path(os.path.dirname(os.path.realpath(__file__)))
  22. _cfg_file = _this_path / 'interface_file_names.cfg'
  23. if _cfg_file.exists():
  24. config = ConfigParser()
  25. config.read(_cfg_file)
  26. header_file_name = config.get('configuration', 'header_file_name')
  27. _header_file = _this_path / 'include' / header_file_name
  28. _header_file = str(_header_file)
  29. library_file_name = config.get('configuration', 'library_file_name')
  30. _library_file = _this_path / 'lib' / library_file_name
  31. _library_file = str(_library_file)
  32. else:
  33. _header_file = os.getenv('ACCOUNT_HEADER_FILE')
  34. assert _header_file is not None
  35. _library_file = os.getenv('ACCOUNT_LIBRARY_FILE')
  36. assert _library_file is not None
  37. _lib = get_lib_handle(definitions=['-DACCOUNT_API=', '-DACCOUNT_NOINCLUDE'],
  38. header_file=_header_file,
  39. library_file=_library_file)
  40. # we change names to obtain a more pythonic API
  41. new = _lib.account_new
  42. free = _lib.account_free
  43. deposit = _lib.account_deposit
  44. withdraw = _lib.account_withdraw
  45. get_balance = _lib.account_get_balance
  46. __all__ = [
  47. '__version__',
  48. 'new',
  49. 'free',
  50. 'deposit',
  51. 'withdraw',
  52. 'get_balance',
  53. ]

我们看到,这个接口的大部分工作是通用的和可重用的,实际的接口相当薄。

项目的布局为:

  1. .
  2. ├── account
  3. ├── account.h
  4. ├── CMakeLists.txt
  5. ├── implementation
  6. ├── c_cpp_interface.cpp
  7. ├── cpp_implementation.cpp
  8. └── cpp_implementation.hpp
  9. ├── __init__.py
  10. └── test.py
  11. └── CMakeLists.txt

具体实施

现在使用CMake来组合这些文件,形成一个Python模块:

  1. CMakeLists.txt文件包含一个头文件。此外,根据GNU标准,设置编译库的位置:

    1. # define minimum cmake version
    2. cmake_minimum_required(VERSION 3.5 FATAL_ERROR)
    3. # project name and supported language
    4. project(recipe-06 LANGUAGES CXX)
    5. # require C++11
    6. set(CMAKE_CXX_STANDARD 11)
    7. set(CMAKE_CXX_EXTENSIONS OFF)
    8. set(CMAKE_CXX_STANDARD_REQUIRED ON)
    9. # specify where to place libraries
    10. include(GNUInstallDirs)
    11. set(CMAKE_LIBRARY_OUTPUT_DIRECTORY
    12. ${CMAKE_BINARY_DIR}/${CMAKE_INSTALL_LIBDIR})
  2. 第二步,是在account子目录下包含接口和实现的定义:

    1. # interface and sources
    2. add_subdirectory(account)
  3. CMakeLists.txt文件以测试定义(需要Python解释器)结束:

    1. # turn on testing
    2. enable_testing()
    3. # require python
    4. find_package(PythonInterp REQUIRED)
    5. # define test
    6. add_test(
    7. NAME
    8. python_test
    9. COMMAND
    10. ${CMAKE_COMMAND} -E env ACCOUNT_MODULE_PATH=${CMAKE_CURRENT_SOURCE_DIR}
    11. ACCOUNT_HEADER_FILE=${CMAKE_CURRENT_SOURCE_DIR}/account/account.h
    12. ACCOUNT_LIBRARY_FILE=$<TARGET_FILE:account>
    13. ${PYTHON_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/account/test.py
    14. )
  4. account/CMakeLists.txt中定义了动态库目标:

    1. add_library(account
    2. SHARED
    3. plementation/c_cpp_interface.cpp
    4. implementation/cpp_implementation.cpp
    5. )
    6. target_include_directories(account
    7. PRIVATE
    8. ${CMAKE_CURRENT_SOURCE_DIR}
    9. ${CMAKE_CURRENT_BINARY_DIR}
    10. )
  5. 导出一个可移植的头文件:

    1. include(GenerateExportHeader)
    2. generate_export_header(account
    3. BASE_NAME account
    4. )
  6. 使用Python-C接口进行对接:

    1. $ mkdir -p build
    2. $ cd build
    3. $ cmake ..
    4. $ cmake --build .
    5. $ ctest
    6. Start 1: python_test
    7. 1/1 Test #1: python_test ...................... Passed 0.14 sec
    8. 100% tests passed, 0 tests failed out of 1

工作原理

虽然,之前的示例要求我们显式地声明Python-C接口,并将Python名称映射到C(++)符号,但Python CFFI从C头文件(示例中是account.h)推断出这种映射。我们只需要向Python CFFI层提供描述C接口的头文件和包含符号的动态库。在主CMakeLists.txt文件中使用了环境变量集来实现这一点,这些环境变量可以在__init__.py中找到:

  1. # ...
  2. def get_lib_handle(definitions, header_file, library_file):
  3. ffi = FFI()
  4. command = ['cc', '-E'] + definitions + [header_file]
  5. interface = check_output(command).decode('utf-8')
  6. # remove possible \r characters on windows which
  7. # would confuse cdef
  8. _interface = [l.strip('\r') for l in interface.split('\n')]
  9. ffi.cdef('\n'.join(_interface))
  10. lib = ffi.dlopen(library_file)
  11. return lib
  12. # ...
  13. _this_path = Path(os.path.dirname(os.path.realpath(__file__)))
  14. _cfg_file = _this_path / 'interface_file_names.cfg'
  15. if _cfg_file.exists():
  16. # we will discuss this section in chapter 11, recipe 3
  17. else:
  18. _header_file = os.getenv('ACCOUNT_HEADER_FILE')
  19. assert _header_file is not None
  20. _library_file = os.getenv('ACCOUNT_LIBRARY_FILE')
  21. assert _library_file is not None
  22. _lib = get_lib_handle(definitions=['-DACCOUNT_API=', '-DACCOUNT_NOINCLUDE'],
  23. header_file=_header_file,
  24. library_file=_library_file)
  25. # ...

get_lib_handle函数打开头文件(使用ffi.cdef)并解析加载库(使用ffi.dlopen)。并返回库对象。前面的文件是通用的,可以在不进行修改的情况下重用,用于与Python和C或使用Python CFFI的其他语言进行接口的其他项目。

_lib库对象可以直接导出,这里有一个额外的步骤,使Python接口在使用时,感觉更像Python:

  1. # we change names to obtain a more pythonic API
  2. new = _lib.account_new
  3. free = _lib.account_free
  4. deposit = _lib.account_deposit
  5. withdraw = _lib.account_withdraw
  6. get_balance = _lib.account_get_balance
  7. __all__ = [
  8. '__version__',
  9. 'new',
  10. 'free',
  11. 'deposit',
  12. 'withdraw',
  13. 'get_balance',
  14. ]

有了这个变化,可以将例子写成下面的方式:

  1. import account
  2. account1 = account.new()
  3. account.deposit(account1, 100.0)

另一种选择则不那么直观:

  1. from account import lib
  2. account1 = lib.account_new()
  3. lib.account_deposit(account1, 100.0)

需要注意的是,如何使用API来实例化和跟踪上下文:

  1. account1 = account.new()
  2. account.deposit(account1, 10.0)
  3. account2 = account.new()
  4. account.withdraw(account1, 5.0)
  5. account.deposit(account2, 5.0)

为了导入account的Python模块,需要提供ACCOUNT_HEADER_FILEACCOUNT_LIBRARY_FILE环境变量,就像测试中那样:

  1. add_test(
  2. NAME
  3. python_test
  4. COMMAND
  5. ${CMAKE_COMMAND} -E env ACCOUNT_MODULE_PATH=${CMAKE_CURRENT_SOURCE_DIR}
  6. ACCOUNT_HEADER_FILE=${CMAKE_CURRENT_SOURCE_DIR}/account/account.h
  7. ACCOUNT_LIBRARY_FILE=$<TARGET_FILE:account>
  8. ${PYTHON_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/account/test.py
  9. )

第11章中,将讨论如何创建一个可以用pip安装的Python包,其中头文件和库文件将安装在定义良好的位置,这样就不必定义任何使用Python模块的环境变量。

讨论了Python方面的接口之后,现在看下C的接口。account.h内容为:

  1. struct account_context;
  2. typedef struct account_context account_context_t;
  3. ACCOUNT_API
  4. account_context_t *account_new();
  5. ACCOUNT_API
  6. void account_free(account_context_t *context);
  7. ACCOUNT_API
  8. void account_deposit(account_context_t *context, const double amount);
  9. ACCOUNT_API
  10. void account_withdraw(account_context_t *context, const double amount);
  11. ACCOUNT_API
  12. double account_get_balance(const account_context_t *context);

黑盒句柄account_context会保存对象的状态。ACCOUNT_API定义在account_export.h中,由account/interface/CMakeLists.txt生成:

  1. include(GenerateExportHeader)
  2. generate_export_header(account
  3. BASE_NAME account
  4. )

account_export.h头文件定义了接口函数的可见性,并确保这是以一种可移植的方式完成的,实现可以在cpp_implementation.cpp中找到。它包含is_initialized布尔变量,可以检查这个布尔值确保API函数按照预期的顺序调用:上下文在创建之前或释放之后都不应该被访问。

更多信息

设计Python-C接口时,必须仔细考虑在哪一端分配数组:数组可以在Python端分配并传递给C(++)实现,也可以在返回指针的C(++)实现上分配。后一种方法适用于缓冲区大小事先未知的情况。但返回到分配给C(++)端的数组指针可能会有问题,因为这可能导致Python垃圾收集导致内存泄漏,而Python垃圾收集不会“查看”分配给它的数组。我们建议设计C API,使数组可以在外部分配并传递给C实现。然后,可以在__init__.py中分配这些数组,如下例所示:

  1. from cffi import FFI
  2. import numpy as np
  3. _ffi = FFI()
  4. def return_array(context, array_len):
  5. # create numpy array
  6. array_np = np.zeros(array_len, dtype=np.float64)
  7. # cast a pointer to its data
  8. array_p = _ffi.cast("double *", array_np.ctypes.data)
  9. # pass the pointer
  10. _lib.mylib_myfunction(context, array_len, array_p)
  11. # return the array as a list
  12. return array_np.tolist()

return_array函数返回一个Python列表。因为在Python端完成了所有的分配工作,所以不必担心内存泄漏,可以将清理工作留给垃圾收集。

对于Fortran示例,读者可以参考以下Git库:https://github.com/dev-cafe/cmake-cookbook/tree/v1.0/chapter09/recipe06/Fortran-example 。与C++实现的主要区别在于,account库是由Fortran 90源文件编译而成的,我们在account/CMakeLists.txt中使用了Fortran 90源文件:

  1. add_library(account
  2. SHARED
  3. implementation/fortran_implementation.f90
  4. )

上下文保存在用户定义的类型中:

  1. type :: account
  2. private
  3. real(c_double) :: balance
  4. logical :: is_initialized = .false.
  5. end type

Fortran实现可以使用iso_c_binding模块解析account.h中定义的符号和方法:

  1. module account_implementation
  2. use, intrinsic :: iso_c_binding, only: c_double, c_ptr
  3. implicit none
  4. private
  5. public account_new
  6. public account_free
  7. public account_deposit
  8. public account_withdraw
  9. public account_get_balance
  10. type :: account
  11. private
  12. real(c_double) :: balance
  13. logical :: is_initialized = .false.
  14. end type
  15. contains
  16. type(c_ptr) function account_new() bind (c)
  17. use, intrinsic :: iso_c_binding, only: c_loc
  18. type(account), pointer :: f_context
  19. type(c_ptr) :: context
  20. allocate(f_context)
  21. context = c_loc(f_context)
  22. account_new = context
  23. f_context%balance = 0.0d0
  24. f_context%is_initialized = .true.
  25. end function
  26. subroutine account_free(context) bind (c)
  27. use, intrinsic :: iso_c_binding, only: c_f_pointer
  28. type(c_ptr), value :: context
  29. type(account), pointer :: f_context
  30. call c_f_pointer(context, f_context)
  31. call check_valid_context(f_context)
  32. f_context%balance = 0.0d0
  33. f_context%is_initialized = .false.
  34. deallocate(f_context)
  35. end subroutine
  36. subroutine check_valid_context(f_context)
  37. type(account), pointer, intent(in) :: f_context
  38. if (.not. associated(f_context)) then
  39. print *, 'ERROR: context is not associated'
  40. stop 1
  41. end if
  42. if (.not. f_context%is_initialized) then
  43. print *, 'ERROR: context is not initialized'
  44. stop 1
  45. end if
  46. end subroutine
  47. subroutine account_withdraw(context, amount) bind (c)
  48. use, intrinsic :: iso_c_binding, only: c_f_pointer
  49. type(c_ptr), value :: context
  50. real(c_double), value :: amount
  51. type(account), pointer :: f_context
  52. call c_f_pointer(context, f_context)
  53. call check_valid_context(f_context)
  54. f_context%balance = f_context%balance - amount
  55. end subroutine
  56. subroutine account_deposit(context, amount) bind (c)
  57. use, intrinsic :: iso_c_binding, only: c_f_pointer
  58. type(c_ptr), value :: context
  59. real(c_double), value :: amount
  60. type(account), pointer :: f_context
  61. call c_f_pointer(context, f_context)
  62. call check_valid_context(f_context)
  63. f_context%balance = f_context%balance + amount
  64. end subroutine
  65. real(c_double) function account_get_balance(context) bind (c)
  66. use, intrinsic :: iso_c_binding, only: c_f_pointer
  67. type(c_ptr), value, intent(in) :: context
  68. type(account), pointer :: f_context
  69. call c_f_pointer(context, f_context)
  70. call check_valid_context(f_context)
  71. account_get_balance = f_context%balance
  72. end function
  73. end module

这个示例和解决方案的灵感来自Armin Ronacher的帖子“BeautifulNative Libraries”: http://lucumr.pocoo.org/2013/8/18/beautiful-native-libraries/