Serving Flow

If you come to this page, most likely you have already built some cool stuff with Jina and now you want to share it with the world. This cookbook will guide you from basic serving for demo purpose to advanced serving in production.

Minimum working example

Server

from jina import Flow
f = Flow(protocol='grpc', port_expose=12345)
with f:
    f.block()

Client

from jina import Client, Document
c = Client(protocol='grpc', port=12345)
c.post('/', Document())

Flow-as-a-Service

A Flow is a service by nature. Though implicitly, you are already using it as a service.

When you start a Flow and call .post() inside the context, a jina.Client object is created and used for communication.

Many times we need to use Flow & Client in a more explicit way, often due to one of the following reasons:

• Flow and Client are on different machines: one on GPU, one on CPU;

• Flow and Client have different lifetime: one lives longer, one lives shorter;

• Multiple Clients want to access one Flow;

• One Client want to interleave its access to multiple Flow;

• Client is browser/curl/Postman.

Before this cookbook, you are mostly using Flow as an implicit service. In the sequel, we will show you how to serve Flow in an explicit C/S style.

Supported communication protocols

Jina supports the three communication protocols grpc, websocket, and http between Flow and Client.

The protocol is controlled by the protocol= argument in Flow/Client’s constructor.

via gRPC

On the server-side, create an empty Flow and use .block to prevent the process from exiting.

from jina import Flow
with Flow(port_expose=12345) as f:
    f.block()

        [email protected][L]:ready and listening
           [email protected][I]:🎉 Flow is ready to use!
    🔗 Protocol:         GRPC
    🏠 Local access:    0.0.0.0:12345
    🔒 Private network:    192.168.1.15:12345
    🌐 Public address:    197.26.36.43:12345

Note that the host address is 192.168.1.15 and port_expose is 12345.

While keeping this server open, let’s create a client on a different machine:

from jina import Client
c = Client(host='192.168.1.15', port=12345)
c.post('/')

[email protected][S]:connected to the gateway at 0.0.0.0:12345!

Warning

Multiple gRPC Client cannot be spawned using Threads because of an upstream issue. Use multiprocessing instead.

via WebSocket

from jina import Flow
f = Flow(protocol='websocket', port_expose=12345)
with f:
    f.block()

        [email protected][L]:ready and listening
           [email protected][I]:🎉 Flow is ready to use!
    🔗 Protocol:         WEBSOCKET
    🏠 Local access:    0.0.0.0:12345
    🔒 Private network:    192.168.1.15:12345
    🌐 Public address:    197.26.36.43:12345

This will serve the Flow with WebSocket, so any Client connecting to it should follow the WebSocket protocol as well.

from jina import Client
c = Client(protocol='websocket', host='192.168.1.15', port=12345)
c.post('/')

[email protected][S]:connected to the gateway at 0.0.0.0:12345!

via HTTP

To enable a Flow to receive HTTP requests, you can add protocol='http' in the Flow constructor.

from jina import Flow
f = Flow(protocol='http', port_expose=12345)
with f:
    f.block()

        [email protected][L]:ready and listening
           [email protected][I]:🎉 Flow is ready to use!
    🔗 Protocol:         HTTP
    🏠 Local access:    0.0.0.0:12345
    🔒 Private network:    192.168.1.15:12345
    🌐 Public address:    197.26.36.43:12345
    💬 Swagger UI:        http://localhost:12345/docs
    📚 Redoc:        http://localhost:12345/redoc

Flow with HTTP protocol

Enable cross-origin-resources-sharing (CORS)

CORS is by default disabled for security. That means you can not access the service from a webpage with a different domain. To override this, simply do:

from jina import Flow
f = Flow(cors=True, protocol='http', port_expose=12345)
with f:
    f.block()

Use swagger UI to send HTTP request

You can navigate to the Swagger docs UI via http://localhost:12345/docs:

Use curl to send HTTP request

Now you can send data request via curl/Postman:

$ curl --request POST 'http://localhost:12345/post' --header 'Content-Type: application/json' -d '{"data": [{"text": "hello world"}],"execEndpoint": "/index"}'
{
   "header":{
      "requestId":"3a10f4a3711b441982ea17a162dec176",
      "status":null,
      "execEndpoint":"/index"
   },
   "parameters":null,
   "routes":[
      {
         "executor":"gateway",
         "startTime":"2022-01-26T10:40:43.988564+00:00",
         "endTime":"2022-01-26T10:40:43.989243+00:00",
         "status":null
      }
   ],
   "data":[
      {
         "id":"69f99b5a7e9411ec91f1e86a64801cb1",
         "parent_id":null,
         "granularity":null,
         "adjacency":null,
         "blob":null,
         "tensor":null,
         "mime_type":"text/plain",
         "text":"hello world",
         "weight":null,
         "uri":null,
         "tags":null,
         "offset":null,
         "location":null,
         "embedding":null,
         "modality":null,
         "evaluations":null,
         "scores":null,
         "chunks":null,
         "matches":null
      }
   ]
}

Use Python to send HTTP request

One can also use Python Client to send HTTP request, simply:

from jina import Client
c = Client(protocol='http', port=12345)
c.post('/', ...)

Warning

This HTTP client is less-performant on large data, it does not stream. Hence, it should be only used for debugging & testing.

Extend HTTP Interface

By default the following endpoints are exposed to the public:

Hide CRUD and debug endpoints from HTTP interface

User can decide to hide CRUD and debug endpoints in production, or when the context is not applicable. For example, in the code snippet below, we didn’t implement any CRUD endpoints for the executor, hence it does not make sense to expose them to public.

from jina import Flow
f = Flow(protocol='http',
         no_debug_endpoints=True,
         no_crud_endpoints=True)

Expose customized endpoints to HTTP interface

Flow.expose_endpoint can be used to expose executor’s endpoint to HTTP interface, e.g.

from jina import Executor, requests, Flow
class MyExec(Executor):
    @requests(on='/foo')
    def foo(self, docs, **kwargs):
        pass
f = Flow(protocol='http').add(uses=MyExec)
f.expose_endpoint('/foo', summary='my endpoint')
with f:
    f.block()

Now, sending HTTP data request to /foo is equivalent as calling f.post('/foo', ...) in Python.

You can add more kwargs to build richer semantics on your HTTP endpoint. Those meta information will be rendered by Swagger UI and be forwarded to the OpenAPI schema.

f.expose_endpoint('/bar',
                  summary='my endpoint',
                  tags=['fine-tuning'],
                  methods=['PUT']
                  )

You can enable custom endpoints in a Flow using yaml syntax as well.

jtype: Flow
with:
  protocol: http
  expose_endpoints:
    /foo:
      methods: ["GET"]
    /bar:
      methods: ["PUT"]
      summary: my endpoint
      tags:
        - fine-tuning
    /foobar: {}
executors:
  - name: indexer

Add non-Jina related routes

If you want to add more customized routes, configs, options to HTTP interface, you can simply override jina.helper.extend_rest_interface function as follows:

import jina.helper
from jina import Flow
def extend_rest_function(app):
    @app.get('/hello', tags=['My Extended APIs'])
    async def foo():
        return 'hello'
    return app
jina.helper.extend_rest_interface = extend_rest_function
f = Flow(protocol='http')
with f:
    f.block()

And you will see /hello is now available: