Amazon SageMaker AI 双向流式推理推出按用量计费

Usage-based metering for bidirectional streaming solves a software billing problem for real-time speech AI. Customers deploying streaming models on Amazon SageMaker AI bidirectional streaming pay for the endpoint infrastructure that hosts the model. Separately, model providers listing on AWS Marketplace need a way to charge for what customers consume during a live stream. Today, we’re announcing usage-based metering for Amazon SageMaker bidirectional (bidi) streaming inference, so model providers can report actual usage for the AWS Marketplace software charge. Amazon SageMaker AI endpoint infrastructure charges still apply.

Deepgram, the real-time speech AI company, is the launch partner for this capability, bringing usage-based Speech-to-Text (STT) and Text-to-Speech (TTS) billing to AWS Marketplace. This post builds on a series of innovations between Deepgram and AWS. First, we introduced bidirectional streaming for real-time inference on Amazon SageMaker AI, which supports long-lived bidirectional streams between clients and model servers. Then, Deepgram enhanced Amazon SageMaker AI support with AWS Identity and Access Management (AWS IAM) temporary delegation, simplifying secure authentication for enterprise deployments, and expanded observability with enhanced metrics. Now, with usage-based metering, we further strengthen the foundation for production real-time AI, aligning billing with how customers consume streaming and real-time AI workloads.

Why streaming broke the existing billing model

With AWS Marketplace software metering for SageMaker AI real-time request-response inference, model providers can report a unit count in an HTTP response header. AWS Marketplace uses that quantity for the software charge. This works for request-response APIs where the billable quantity is known when the response is sent.

Bidirectional streaming is different. A client streams audio and receives transcriptions through the SageMaker AI API using an HTTP/2 event stream. SageMaker AI connects to the model server over WebSockets. A session can last 30 seconds or 30 minutes. Usage accumulates as the session runs, and the final total isn’t known until it ends. A response header alone can’t report periodic usage during that live session.

Hourly software pricing can charge customers while a model is available, including quiet periods. With usage-based metering, providers can align the AWS Marketplace software charge with reported usage. SageMaker AI endpoint infrastructure charges remain separate and continue to apply while the endpoint is provisioned.

How companion metadata streaming works

The solution introduces a second WebSocket connection alongside the existing inference stream. When a bidirectional streaming session starts, SageMaker AI establishes two connections to the model server:

• Data connection: carries inference request and response data, the existing bidirectional stream.
• Metadata connection: a companion WebSocket dedicated to metering data.

Model servers opt in by returning one header on the WebSocket upgrade response:

X-Amzn-SageMaker-Metadata-Stream-Supported: true

SageMaker AI then opens a companion WebSocket to the model server at /invocations-bidirectional-stream-metadata. Both connections carry the same X-Amzn-SageMaker-Request-Id header, which the model server uses to correlate metering records with the active inference session.

The model server sends metering data as JSON text frames on the metadata connection. Send records at a frequency that matches the billing model: incrementally as usage accrues (recommended), either as each unit is consumed or as periodic batches of units consumed since the previous record. Send any final record as soon as the session ends. Each frame contains exactly one metering record, rather than an array of records. The metadata stream has a rate quota of one message per second per connection, with a maximum message size of 512 bytes.

We recommend sending metering records incrementally as usage accrues. This reduces the amount of usage left unreported if a session ends unexpectedly. Transport delivery alone does not establish that usage has been billed. Validate reported units against AWS Marketplace reports and buyer charges.

{
  "Metering": {
    "Dimension": "inference.count",
    "ConsumedUnits": 720,
    "ClientToken": "a3f2b1c0-unique-uuid"
  }
}

Metering messages use the inference.count dimension, and ConsumedUnits supplies the quantity for that dimension. If the product charges per second of audio, a record reporting 12 minutes of metered audio carries 720 seconds as ConsumedUnits: 720. The dimension name doesn’t mean one unit per request. The provider defines which audio counts as metered usage, and the AWS Marketplace offer maps the unit to its price. The listing shows one price per unit, labeled per request, with up to three decimal places. The ClientToken field supports idempotency. If a duplicate token arrives on the same connection, SageMaker AI drops the duplicate message silently.

The metadata connection runs over TCP, which provides ordered, reliable delivery at the transport layer. If it fails to establish or drops mid-session, inference continues by default because X-Amzn-SageMaker-Metadata-Stream-Required defaults to false. Set this header to true when the provider must stop inference if the metadata stream becomes unavailable. This choice trades inference availability for metering continuity. It doesn’t guarantee that malformed or dropped records are billed.

What this means for customers

If you deploy real-time AI models for transcription, contact centers, voice agents, or captioning, the new metering model changes how the AWS Marketplace software charge is calculated. SageMaker AI endpoint infrastructure charges still apply.

Transparent, per-unit billing through AWS Marketplace means usage flows directly into your AWS bill. You get consolidated AWS billing, negotiated rates through private offers, and no separate billing relationship with the model provider to manage.

Pay for audio processed according to the product’s unit definition. If the product charges per second of audio, 12 minutes of metered audio equals 720 billable seconds and ConsumedUnits: 720. An open connection lasting 12 minutes doesn’t, by itself, define the billable audio quantity. The provider’s counting rules determine the metered usage, whereas endpoint infrastructure charges remain separate.

Model providers can offer negotiated per-unit rates or fixed-price contracts through AWS Marketplace private offers.

We recommend discussing directly with your AWS account representative or partner representative.

What this means for model providers

If you’re a model provider listing on AWS Marketplace, you can configure your model server to report usage during streaming inference sessions through companion metadata streaming.

The SageMaker Marketplace Onboarding Agent Skill accelerates onboarding with an opt-in inference metering path. Its starter templates advertise metadata-stream support and build validated, incremental ConsumedUnits frames for the companion WebSocket. Review and production-harden the code before submission.

Align billing with your pricing model by defining what one unit represents and reporting units as the session progresses. AWS Marketplace charges one per-unit price for real-time inference.

The following three-step walkthrough describes how to integrate your model server to report metering data during bidirectional streaming inference sessions.

Step 1: Opt in using data WebSocket response headers

When SageMaker AI opens the data WebSocket connection to your model server, include these headers in your WebSocket upgrade (101) response:

Step 2: Accept the metadata WebSocket connection

After your model server returns X-Amzn-SageMaker-Metadata-Stream-Supported: true, SageMaker AI opens a second WebSocket connection to your model server at the default path /invocations-bidirectional-stream-metadata (or the custom path you specified by using X-Amzn-SageMaker-Metadata-Invocation-Path).

SageMaker AI includes the header X-Amzn-SageMaker-Request-Id on both the data and metadata WebSocket handshakes. Your model server must do the following:

• Accept the WebSocket upgrade on the metadata path.
• Use X-Amzn-SageMaker-Request-Id to correlate the metadata connection with the corresponding data connection for the same inference session.

Step 3: Send metering messages

Send metering data as a JSON object in a WebSocket text frame on the metadata connection. Each frame must contain exactly one metering record. In this example, the product charges per second of audio, and the record reports 12 minutes of metered audio as 720 units:

{
  "Metering": {
    "Dimension": "inference.count",
    "ConsumedUnits": 720,
    "ClientToken": "a3f2b1c0-unique-uuid"
  }
}

Message fields:

Send metering records incrementally as usage accrues to reduce unreported usage if a session ends unexpectedly. Reconcile the reported quantities with AWS Marketplace reports and buyer charges to verify billing, including interrupted-session tests.

Constraints and limits

Error handling and failure modes

The Required header controls what happens when the metadata connection becomes unavailable. Set it to true to stop inference, or use the default false to let inference continue in data-only mode. This is a choice between metering continuity and inference availability. Malformed messages can still be dropped with either setting.

List your model on AWS Marketplace

1. Register your AWS account as an AWS Marketplace seller. For paid products, complete the tax and banking information.
2. Create a SageMaker AI model package from your container in a supported publishing Region, and an AWS Identity and Access Management (IAM) role that AWS Marketplace can assume to read it.
3. In AWS Partner Central, choose Build, Machine learning, then Create machine learning product, and add the model package as the first version.
4. Choose per-inference pricing. You set one per-unit price for real-time inference across all instance types, shown per request with up to three decimal places. You also set an hourly price for each batch transform instance type. Choose the pricing model when you create the listing because you can’t switch a released listing between hourly and per-inference pricing.
5. After validation, request Public visibility. For enterprise buyers, extend private offers with a negotiated per-unit rate or a fixed-price contract.

Validate billing before release

Validate the integration with a Limited listing and an allowlisted test buyer. Use an appropriate offer, including a private offer where needed, so the test exercises the intended unit and price mapping.

1. Run a known-quantity test. For a product priced per second of audio, send 12 minutes of metered audio and check that the reported records total 720 units.
2. Reconcile the model-server records with the seller’s usage reports and the buyer’s charges. Allow time for reports to populate and verify the usage type <region code>-SMMP:inference.count.m.i.c (for example, USE1-SMMP:inference.count.m.i.c in us-east-1). Confirm the offer’s unit and price mapping rather than assuming a fixed reporting delay.
3. Test normal completion, interrupted sessions, metadata-stream loss, and malformed messages. Compare reported and billed quantities for each case with the SageMaker AI and AWS Marketplace teams before release.
4. Delete test endpoints when testing is complete to avoid ongoing infrastructure charges.

Getting started with Deepgram on AWS Marketplace

To deploy Deepgram with usage-based metering on SageMaker AI, follow these steps.

• Subscribe to the Deepgram Voice AI streaming model you need on AWS Marketplace, such as Deepgram Nova-3 Monolingual Speech-to-Text (STT) Streaming.
• Deploy a SageMaker AI real-time endpoint using the Deepgram model package from your AWS Marketplace subscription.
• Connect to the endpoint using the SageMaker AI bidirectional streaming API and stream audio through its HTTP/2 event stream.

Deepgram’s model server calculates metering automatically, and Amazon SageMaker AI and AWS Marketplace process it. You don’t need to configure anything on the client side.

Clean up

When you finish testing, delete the SageMaker AI real-time endpoint and model resource to stop infrastructure charges. If you no longer need the product, remove the AWS Marketplace subscription.

Conclusion

With usage-based metering for bidirectional streaming, model providers can align AWS Marketplace software pricing with reported usage during a live session. Customers pay the software charge for the metered units defined by the product and offer. Amazon SageMaker AI endpoint infrastructure charges remain separate and still apply. Learn more about Amazon SageMaker AI.

To get started:

• For customers, subscribe to a Deepgram Voice AI streaming model on AWS Marketplace, such as Deepgram Nova-3 Monolingual Speech-to-Text (STT) Streaming
• For model providers, follow the integration walkthrough in this post to add usage-based metering to your own model server.

Acknowledgements

Special thanks to Kareem Syed-Mohammed (Principal Product Manager), Susmitha Marupaka (Product Marketing Manager), Zhengfei Ji (Software Development Engineer for Amazon SageMaker AI), and EJ Ferrell for reviewing this post and leading the development of this capability.

About the authors

Victor Wang

Victor is a Staff Software Engineer at Deepgram and Technical Advisor to the VP of Engineering based in San Francisco, CA. Before joining Deepgram, he was a Sr. Solutions Architect at AWS working with generative AI startups and held engineering, consulting, and program management roles across AWS. He enjoys learning new technologies and traveling, having flown over two million miles.

Lingran Xia

Lingran is a software development engineer at AWS. He currently focuses on improving inference performance of machine learning models. In his free time, he enjoys traveling and skiing.

Andre Gomes

Andre is a Frontier AI Solutions Architect at AWS, working with AI startups on large-scale training, real-time inference, and Amazon SageMaker AI. Before joining AWS, Andre earned his PhD at Universidade Federal de Minas Gerais, with research conducted at the Institute for Manufacturing, University of Cambridge, applying large language models and neural topic modeling to strategic roadmapping.

Daniel Wirjo

Daniel is a Solutions Architect at AWS, focused on frontier AI startups. As a former startup CTO, he enjoys collaborating with founders and engineering leaders to drive growth and innovation on AWS. Outside of work, Daniel enjoys taking walks with a coffee in hand, appreciating nature, and learning new ideas.