Carton is a machine learning framework that aims to decouple the inference code from specific frameworks like Torch or TF, allowing users to easily keep up with the cutting-edge. It boasts optimized async Rust code, resulting in an overhead of less than 100 microseconds per inference call for small inputs. The platform support includes x86_64 Linux and macOS, aarch64 Linux and macOS, and WebAssembly. Carton uses a zip file called a carton as the output of the packing step, which contains the original model and metadata without modifying the model itself. Unlike ONNX, Carton wraps models instead of converting them, making it easier to use custom ops and TensorRT without changes. However, Carton plans to support ONNX models in the future.
https://carton.run