Is the gRPC in BentoML stable and actively maintained?

[ryan-3m]

As far as I know, gRPC is a new feature added after version 1.x of BentoML. Is it still considered experimental?

I have some high-traffic tasks, such as processing videos, for which REST would incur too much overhead. Therefore, I have to choose gRPC.

[parano]

Hi @ryan-3m, yes the gRPC support is actively being developed and maintained, however still considered a beta feature. It would be great to hear your feedback on this.

[aarnphm]

Additionally, I don't think communication overhead is a big concern even for HTTP, given we are just sending bytes over the wire (I might be wrong).

we might want to investigate with Zmq and probably do another internal benchmark if migrating to a language other than Python makes sense. This has been floating around the office for a while now 😄

[optiluca]

The desire to use gRPC is driven by the fact that sending (binary) protobufs is much more efficient than sending (text) JSONs. Plus using proto means I can automatically generate clients for multiple programming languages, as you say.

But you're right, I guess the fact I associate HTTP REST with JSONs is just because that's what's customary, and I could probably send binary data as a message and get performance that should be somewhere in between.

[oogou11]

In my project based on the Python language, I adopted gRPC communication, primarily to construct multiple LLM microservices. Previously, I used HTTP communication and directly transmitted tensors. When testing communication latency (communication only) on my server, the results were: gRPC - 1.308 seconds, HTTP - 4.53 seconds, with tensor.shape = 1*4.19 million. However, when using gRPC communication, I first applied the tolist operation to the tensor.

environment:

• bentoml: 1.3.3
• GPU: NVIDIA GeForce RTX 4090
• CUDA: 12.4
• Operating System: Ubuntu 20.04.6

[Sytronik]

With the current bentoml v1.4.x, does the >v1.2 API not support gRPC yet?

[aarnphm]

not yet.

[MrBeeMovie]

Hi @ryan-3m, yes the gRPC support is actively being developed and maintained, however still considered a beta feature. It would be great to hear your feedback on this.

I have been trying to run the sentence-transformers example repo you guys have but serve-grpc always fails with:

bentoml serve-grpc
2024-03-19T12:23:19-0400 [WARNING] [cli] Deprecated build option: 'docker.env' is used, please use 'envs' instead.
Error: [serve] `serve-grpc` failed: <class '_bentoml_sdk.service.factory.Service'> type doesn't support gRPC serving

I have made sure to install bentoml[grpc]. Any ideas?

[EraYaN]

According to a conversation in slack, the new 1.2 API is not yet supported by gRPC. So you would need to use the one 1.1 style bento service for gRPC to work (not a class but functions).