feat(telegram): split connector into home validator + remote bot
CI / changes (pull_request) Successful in 2s
CI / unit (pull_request) Successful in 10s
CI / integration (pull_request) Successful in 15s
CI / ui (pull_request) Successful in 53s
CI / gate (pull_request) Successful in 1s
CI / deploy (pull_request) Failing after 2m6s

Move all Telegram egress off the main host. The single connector held the
bot token, long-polled Telegram and answered the gateway/backend over the
trusted internal network, so the whole component (including login validation)
shared fate with its VPN sidecar. Split it into two binaries that share the
token:

- cmd/validator (home, no VPN): Mini App initData + Login Widget HMAC only,
  never calls the Bot API. The gateway dials it for Telegram auth, so game
  login is now independent of Telegram reachability.
- cmd/bot (remote): Bot API long-poll + sendMessage, the only component
  reaching Telegram. It holds no inbound port — it dials the gateway over a
  new reverse mTLS bot-link (pkg/proto/botlink/v1) and executes the send
  commands the gateway pushes.

The gateway funnels sends to the bot-link: out-of-app push is fire-and-forget
(at-most-once, dropped if no bot is connected); the backend admin broadcasts
reach a gateway-served relay that forwards them and awaits the bot's ack
(SendToUser/SendToGameChannel contract preserved). mTLS (pkg/mtls) is the one
inter-service link that leaves the trusted segment; validator<->gateway and
the relay stay plaintext internal. The bot is Telegram-rate-limited.

One bot now; the gateway bot registry, an owns_updates flag and per-command
ids leave seams for N later. Webhook rejected (one URL per token, adds inbound
+ a static address).

The unified test contour runs the split (the bot keeps its VPN sidecar and
dials the gateway by its internal name; bot-link certs from deploy/gen-certs.sh,
generated in CI). The prod wiring — the bot on a separate host (no VPN), the
gateway bot-link port published, PROD_ certs with scheduled rotation, an SSH
deploy of both hosts together — is the deferred final stage (PRERELEASE.md TX,
Stage 18).

Docs: ARCHITECTURE, PRERELEASE (phase TX), platform/telegram + gateway +
backend + deploy READMEs, FUNCTIONAL(+ru), CLAUDE.md, .env.example.
This commit is contained in:
Ilia Denisov
2026-06-21 00:19:07 +02:00
parent 2a8717c930
commit 6aeb529f13
42 changed files with 3073 additions and 714 deletions
+14 -6
View File
@@ -45,10 +45,14 @@ operations are unauthenticated and return the minted token. A unary domain
outcome rides back in `ExecuteResponse.result_code` (HTTP 200); only edge
failures become Connect error codes.
`auth.telegram` validates the Mini App `initData` by calling the **Telegram connector**
(`GATEWAY_CONNECTOR_ADDR`), which holds the bot token; the gateway also routes
out-of-app push to that connector for recipients with no live in-app stream
(ARCHITECTURE.md §10). When `GATEWAY_CONNECTOR_ADDR` is unset, both are disabled.
`auth.telegram` validates the Mini App `initData` by calling the **Telegram validator**
(`GATEWAY_VALIDATOR_ADDR`), which holds the bot token (HMAC); out-of-app push for
recipients with no live in-app stream goes to the remote **bot** over the reverse **mTLS
bot-link** (`GATEWAY_BOTLINK_ADDR`, fire-and-forget), and the backend admin broadcasts
arrive on the gateway's plaintext **relay** (`GATEWAY_BOTLINK_RELAY_ADDR`) which forwards
them down the same link and awaits the bot's ack (ARCHITECTURE.md §10/§12). When
`GATEWAY_VALIDATOR_ADDR` is unset Telegram auth is disabled; when `GATEWAY_BOTLINK_ADDR`
is unset the bot channel (out-of-app push + admin relay) is disabled.
The message-type catalog: `auth.telegram`, `auth.guest`,
`auth.email.request`, `auth.email.login`, `profile.get`, `game.submit_play`,
@@ -63,7 +67,7 @@ refetch). The social/account/history ops —
transcode pattern (`transcode_social.go`). Account linking & merge
`link.email.request/confirm/merge` and `link.telegram.confirm/merge`
(`transcode_link.go`); the telegram ops validate the **Login Widget** payload via the
connector (`ValidateLoginWidget`) and forward the trusted `external_id`. These
validator (`ValidateLoginWidget`) and forward the trusted `external_id`. These
**superseded** the former `email.bind.*` ops, which were removed.
## Configuration
@@ -76,7 +80,11 @@ connector (`ValidateLoginWidget`) and forward the trusted `external_id`. These
| `GATEWAY_BACKEND_GRPC_ADDR` | `localhost:9090` | backend push gRPC address |
| `GATEWAY_BACKEND_TIMEOUT` | `5s` | per backend REST call |
| `GATEWAY_ADMIN_USER` / `GATEWAY_ADMIN_PASSWORD` | unset | enable + guard the admin console at `/_gm` |
| `GATEWAY_CONNECTOR_ADDR` | unset | Telegram connector gRPC address (enables initData validation + out-of-app push) |
| `GATEWAY_VALIDATOR_ADDR` | unset | Telegram validator gRPC address (enables initData / Login Widget validation) |
| `GATEWAY_BOTLINK_ADDR` | unset | reverse mTLS bot-link listener the remote bot dials (enables out-of-app push + admin relay) |
| `GATEWAY_BOTLINK_RELAY_ADDR` | unset | plaintext internal listener serving the backend admin `SendToUser`/`SendToGameChannel` relay |
| `GATEWAY_BOTLINK_TLS_CERT` / `_KEY` / `_CA` | unset | gateway server cert, its key, and the CA that signs accepted bot client certs (required when `GATEWAY_BOTLINK_ADDR` is set) |
| `GATEWAY_BOTLINK_SEND_TIMEOUT` | `5s` | admin relay wait for the bot ack before reporting not-delivered |
| `GATEWAY_SESSION_TTL` | `10m` | cached session lifetime |
| `GATEWAY_SESSION_CACHE_MAX` | `50000` | cached session cap |
| `GATEWAY_PUSH_HEARTBEAT_INTERVAL` | `10s` | live-stream keep-alive (an immediate heartbeat also fires on open, under the ~15s edge idle timeout) |
+89 -18
View File
@@ -9,16 +9,23 @@ package main
import (
"context"
"errors"
"fmt"
"log"
"net"
"net/http"
"os/signal"
"syscall"
"time"
"go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"
"go.uber.org/zap"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials"
"google.golang.org/grpc/keepalive"
"scrabble/gateway/internal/admin"
"scrabble/gateway/internal/backendclient"
"scrabble/gateway/internal/botlink"
"scrabble/gateway/internal/config"
"scrabble/gateway/internal/connector"
"scrabble/gateway/internal/connectsrv"
@@ -26,9 +33,23 @@ import (
"scrabble/gateway/internal/ratelimit"
"scrabble/gateway/internal/session"
"scrabble/gateway/internal/transcode"
"scrabble/pkg/mtls"
botlinkv1 "scrabble/pkg/proto/botlink/v1"
telegramv1 "scrabble/pkg/proto/telegram/v1"
pkgtel "scrabble/pkg/telemetry"
)
const (
// botLinkKeepaliveTime is how often the gateway pings an idle bot-link stream
// to hold the WAN connection open and detect a dead bot.
botLinkKeepaliveTime = 30 * time.Second
// botLinkKeepaliveTimeout bounds the wait for a keepalive ping reply.
botLinkKeepaliveTimeout = 10 * time.Second
// botLinkMinPingInterval is the smallest client ping interval the gateway
// tolerates before treating it as abuse.
botLinkMinPingInterval = 10 * time.Second
)
const (
// shutdownTimeout bounds the graceful HTTP shutdown.
shutdownTimeout = 10 * time.Second
@@ -100,17 +121,47 @@ func run(ctx context.Context, cfg config.Config, logger *zap.Logger) error {
tracker := ratelimit.NewTracker()
hub := push.NewHub(0)
var conn *connector.Client
var validator transcode.TelegramValidator
if cfg.ConnectorAddr != "" {
conn, err = connector.New(cfg.ConnectorAddr)
if err != nil {
return err
if cfg.ValidatorAddr != "" {
conn, cerr := connector.New(cfg.ValidatorAddr)
if cerr != nil {
return cerr
}
defer func() { _ = conn.Close() }()
validator = conn
} else {
logger.Warn("telegram disabled (GATEWAY_CONNECTOR_ADDR unset)")
logger.Warn("telegram auth disabled (GATEWAY_VALIDATOR_ADDR unset)")
}
// The reverse bot-link: the remote Telegram bot dials this gateway over mTLS and
// the gateway pushes send commands down the stream. Out-of-app push is
// fire-and-forget; the backend admin relay (plaintext, internal) awaits the Ack.
var botHub *botlink.Hub
if cfg.BotLinkEnabled() {
botHub = botlink.NewHub(logger, tel.MeterProvider().Meter("scrabble/gateway/botlink"))
tlsCfg, terr := mtls.ServerConfig(cfg.BotLink.CertFile, cfg.BotLink.KeyFile, cfg.BotLink.CAFile)
if terr != nil {
return terr
}
botSrv := grpc.NewServer(
grpc.Creds(credentials.NewTLS(tlsCfg)),
grpc.StatsHandler(otelgrpc.NewServerHandler()),
grpc.KeepaliveParams(keepalive.ServerParameters{Time: botLinkKeepaliveTime, Timeout: botLinkKeepaliveTimeout}),
grpc.KeepaliveEnforcementPolicy(keepalive.EnforcementPolicy{MinTime: botLinkMinPingInterval, PermitWithoutStream: true}),
)
botlinkv1.RegisterBotLinkServer(botSrv, botHub)
if serr := serveGRPC(ctx, "botlink", cfg.BotLink.Addr, botSrv, logger); serr != nil {
return serr
}
if cfg.BotLink.RelayAddr != "" {
relaySrv := grpc.NewServer(grpc.StatsHandler(otelgrpc.NewServerHandler()))
telegramv1.RegisterTelegramServer(relaySrv, botlink.NewRelayServer(botHub, cfg.BotLink.SendTimeout))
if serr := serveGRPC(ctx, "botlink-relay", cfg.BotLink.RelayAddr, relaySrv, logger); serr != nil {
return serr
}
}
} else {
logger.Warn("telegram bot channel disabled (GATEWAY_BOTLINK_ADDR unset)")
}
// The admin console (backend /_gm) is fronted on the public listener behind
@@ -142,8 +193,8 @@ func run(ctx context.Context, cfg config.Config, logger *zap.Logger) error {
})
// Bridge the backend push stream into the fan-out hub (and the out-of-app
// channel via the connector).
go runPushPump(ctx, backend, hub, conn, logger)
// channel via the bot-link).
go runPushPump(ctx, backend, hub, botHub, logger)
// Periodically summarise rate-limiter rejections (Warn log + backend report).
go runThrottleReporter(ctx, tracker, backend, logger)
@@ -195,6 +246,27 @@ func runServers(ctx context.Context, cancel context.CancelFunc, servers []*named
return first
}
// serveGRPC starts a gRPC server on addr in the background and gracefully stops it
// when ctx is cancelled. It returns synchronously on a bind error so a
// misconfigured listener fails startup fast; a later Serve error is logged.
func serveGRPC(ctx context.Context, name, addr string, srv *grpc.Server, logger *zap.Logger) error {
lis, err := net.Listen("tcp", addr)
if err != nil {
return fmt.Errorf("gateway: listen %s (%s): %w", addr, name, err)
}
go func() {
<-ctx.Done()
srv.GracefulStop()
}()
go func() {
logger.Info("listener starting", zap.String("server", name), zap.String("addr", addr))
if err := srv.Serve(lis); err != nil && !errors.Is(err, grpc.ErrServerStopped) {
logger.Error("grpc listener failed", zap.String("server", name), zap.Error(err))
}
}()
return nil
}
// runThrottleReporter drains the rate-limiter rejection tracker on a fixed
// cadence, emits one Warn summary per throttled key and forwards the report to
// the backend (which feeds the admin throttled view and the high-rate
@@ -230,7 +302,7 @@ func runThrottleReporter(ctx context.Context, tracker *ratelimit.Tracker, backen
// the hub and re-subscribing after the stream ends, until the context is done. For
// the out-of-app push kinds it also routes events whose recipient has no live
// in-app stream to the platform connector (a nil connector disables that channel).
func runPushPump(ctx context.Context, backend *backendclient.Client, hub *push.Hub, conn *connector.Client, logger *zap.Logger) {
func runPushPump(ctx context.Context, backend *backendclient.Client, hub *push.Hub, bot *botlink.Hub, logger *zap.Logger) {
for ctx.Err() == nil {
stream, err := backend.SubscribePush(ctx, gatewayID)
if err != nil {
@@ -255,10 +327,10 @@ func runPushPump(ctx context.Context, backend *backendclient.Client, hub *push.H
EventID: ev.GetEventId(),
})
// Out-of-app fallback: when the recipient has no live in-app stream,
// deliver the event over the platform push channel. Done in a goroutine
// so a slow connector never stalls the in-app firehose.
if conn != nil && connector.OutOfAppKind(ev.GetKind()) && !hub.HasSubscribers(ev.GetUserId()) {
go deliverOutOfApp(ctx, backend, conn, ev.GetUserId(), ev.GetKind(), ev.GetPayload(), logger)
// deliver the event over the bot-link. Done in a goroutine so a slow
// target lookup never stalls the in-app firehose.
if bot != nil && connector.OutOfAppKind(ev.GetKind()) && !hub.HasSubscribers(ev.GetUserId()) {
go deliverOutOfApp(ctx, backend, bot, ev.GetUserId(), ev.GetKind(), ev.GetPayload(), logger)
}
}
if !sleep(ctx, pushReconnectDelay) {
@@ -271,7 +343,7 @@ func runPushPump(ctx context.Context, backend *backendclient.Client, hub *push.H
// Telegram identity and have not confined notifications to the app, asks the
// connector to deliver the event. It is best-effort: every failure is logged and
// dropped (the in-app stream remains the primary channel).
func deliverOutOfApp(ctx context.Context, backend *backendclient.Client, conn *connector.Client, userID, kind string, payload []byte, logger *zap.Logger) {
func deliverOutOfApp(ctx context.Context, backend *backendclient.Client, bot *botlink.Hub, userID, kind string, payload []byte, logger *zap.Logger) {
target, err := backend.PushTarget(ctx, userID)
if err != nil {
logger.Warn("push target lookup failed", zap.String("user_id", userID), zap.Error(err))
@@ -280,10 +352,9 @@ func deliverOutOfApp(ctx context.Context, backend *backendclient.Client, conn *c
if !connector.DeliverToTarget(target.ExternalID, target.NotificationsInAppOnly) {
return
}
// The single bot renders the message in the recipient's interface language.
if _, err := conn.Notify(ctx, target.ExternalID, kind, payload, target.Language); err != nil {
logger.Warn("out-of-app notify failed", zap.String("kind", kind), zap.Error(err))
}
// Fire-and-forget down the bot-link; the bot renders the message in the
// recipient's interface language and is dropped (best-effort) if no bot is up.
bot.Send(botlink.NotifyCommand(target.ExternalID, kind, payload, target.Language))
}
// sleep waits for d or until ctx is cancelled, reporting whether it waited the
+38
View File
@@ -0,0 +1,38 @@
package botlink
import (
botlinkv1 "scrabble/pkg/proto/botlink/v1"
telegramv1 "scrabble/pkg/proto/telegram/v1"
)
// NotifyCommand builds an out-of-app push command rendered by the bot in the
// recipient's interface language.
func NotifyCommand(externalID, kind string, payload []byte, language string) *botlinkv1.Command {
return &botlinkv1.Command{
Payload: &botlinkv1.Command_Notify{Notify: &telegramv1.NotifyRequest{
ExternalId: externalID,
Kind: kind,
Payload: payload,
Language: language,
}},
}
}
// SendToUserCommand builds an admin text message addressed to one user.
func SendToUserCommand(externalID, text string) *botlinkv1.Command {
return &botlinkv1.Command{
Payload: &botlinkv1.Command_SendToUser{SendToUser: &telegramv1.SendToUserRequest{
ExternalId: externalID,
Text: text,
}},
}
}
// SendToGameChannelCommand builds an admin text message for the bot's game channel.
func SendToGameChannelCommand(text string) *botlinkv1.Command {
return &botlinkv1.Command{
Payload: &botlinkv1.Command_SendToChannel{SendToChannel: &telegramv1.SendToGameChannelRequest{
Text: text,
}},
}
}
+259
View File
@@ -0,0 +1,259 @@
// Package botlink is the gateway side of the reverse Telegram bot channel: a remote
// bot dials the gateway and opens one long-lived mTLS gRPC stream
// (pkg/proto/botlink/v1), over which the gateway pushes send Commands and the bot
// returns an Ack per command. The Hub registers connected bots and routes commands:
// out-of-app push is fire-and-forget (Send), admin sends await the bot Ack
// (SendAwait). Delivery is best-effort, at-most-once — a command lost across a
// reconnect is not replayed. See docs/ARCHITECTURE.md.
package botlink
import (
"context"
"errors"
"strconv"
"sync"
"sync/atomic"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/metric"
"go.uber.org/zap"
"google.golang.org/grpc"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/status"
botlinkv1 "scrabble/pkg/proto/botlink/v1"
)
// ErrNoBot is returned by SendAwait when no bot is currently connected.
var ErrNoBot = errors.New("botlink: no bot connected")
// outboundBuffer is the per-link command queue depth; a full queue drops commands
// (at-most-once under backpressure).
const outboundBuffer = 64
// Hub registers connected bots and routes send commands to them. A single bot is
// expected today; the registry already holds a set so adding more later needs no
// rewrite.
type Hub struct {
botlinkv1.UnimplementedBotLinkServer
log *zap.Logger
mu sync.Mutex
links map[*link]struct{}
pending map[string]chan *botlinkv1.Ack
seq atomic.Uint64
connected metric.Int64UpDownCounter
commands metric.Int64Counter
}
// link is one connected bot's outbound queue and identity.
type link struct {
instanceID string
ownsUpdates bool
out chan *botlinkv1.ToBot
}
// NewHub builds a Hub. A nil meter disables metrics; a nil logger is tolerated.
func NewHub(log *zap.Logger, meter metric.Meter) *Hub {
if log == nil {
log = zap.NewNop()
}
h := &Hub{
log: log,
links: make(map[*link]struct{}),
pending: make(map[string]chan *botlinkv1.Ack),
}
if meter != nil {
h.connected, _ = meter.Int64UpDownCounter("botlink_connected_bots",
metric.WithDescription("Number of Telegram bots currently connected to the gateway bot-link."))
h.commands, _ = meter.Int64Counter("botlink_commands_total",
metric.WithDescription("Bot-link send commands by result (delivered, not_delivered, dropped, error)."))
}
return h
}
// Link implements botlinkv1.BotLinkServer: it registers the dialing bot, drains
// queued commands to it, and resolves Acks until the stream ends.
func (h *Hub) Link(stream grpc.BidiStreamingServer[botlinkv1.FromBot, botlinkv1.ToBot]) error {
first, err := stream.Recv()
if err != nil {
return err
}
hello := first.GetHello()
if hello == nil {
return status.Error(codes.InvalidArgument, "first bot-link message must be Hello")
}
l := &link{
instanceID: hello.GetInstanceId(),
ownsUpdates: hello.GetOwnsUpdates(),
out: make(chan *botlinkv1.ToBot, outboundBuffer),
}
h.register(l)
defer h.unregister(l)
ctx := stream.Context()
// A dedicated goroutine owns stream.Send; the Recv loop below owns stream.Recv.
go func() {
for {
select {
case <-ctx.Done():
return
case msg := <-l.out:
if err := stream.Send(msg); err != nil {
return
}
}
}
}()
for {
msg, err := stream.Recv()
if err != nil {
return err
}
if ack := msg.GetAck(); ack != nil {
h.resolve(ack)
}
}
}
// register adds a connected bot.
func (h *Hub) register(l *link) {
h.mu.Lock()
h.links[l] = struct{}{}
n := len(h.links)
h.mu.Unlock()
if h.connected != nil {
h.connected.Add(context.Background(), 1)
}
h.log.Info("bot connected",
zap.String("instance_id", l.instanceID),
zap.Bool("owns_updates", l.ownsUpdates),
zap.Int("connected", n))
}
// unregister removes a bot. l.out is left for the GC; its sender goroutine has
// already exited via the stream context, and stale enqueues simply never send
// (at-most-once).
func (h *Hub) unregister(l *link) {
h.mu.Lock()
delete(h.links, l)
n := len(h.links)
h.mu.Unlock()
if h.connected != nil {
h.connected.Add(context.Background(), -1)
}
h.log.Info("bot disconnected", zap.String("instance_id", l.instanceID), zap.Int("connected", n))
}
// pick returns one connected bot.
func (h *Hub) pick() (*link, bool) {
h.mu.Lock()
defer h.mu.Unlock()
for l := range h.links {
return l, true
}
return nil, false
}
// Send enqueues a fire-and-forget command to a connected bot, dropping it (with a
// warning) when no bot is connected or the queue is full.
func (h *Hub) Send(cmd *botlinkv1.Command) {
l, ok := h.pick()
if !ok {
h.count("dropped")
h.log.Warn("bot-link send dropped: no bot connected")
return
}
cmd.CommandId = h.nextID()
select {
case l.out <- &botlinkv1.ToBot{Command: cmd}:
default:
h.count("dropped")
h.log.Warn("bot-link send dropped: outbound queue full", zap.String("instance_id", l.instanceID))
}
}
// SendAwait enqueues a command and waits for the bot's Ack or until ctx is done.
// It returns ErrNoBot when no bot is connected, the delivered flag on a clean Ack,
// or (false, nil) when ctx (the deadline) fires before an Ack arrives.
func (h *Hub) SendAwait(ctx context.Context, cmd *botlinkv1.Command) (bool, error) {
l, ok := h.pick()
if !ok {
h.count("dropped")
return false, ErrNoBot
}
id := h.nextID()
cmd.CommandId = id
ackc := make(chan *botlinkv1.Ack, 1)
h.mu.Lock()
h.pending[id] = ackc
h.mu.Unlock()
defer func() {
h.mu.Lock()
delete(h.pending, id)
h.mu.Unlock()
}()
select {
case l.out <- &botlinkv1.ToBot{Command: cmd}:
case <-ctx.Done():
h.count("dropped")
return false, ctx.Err()
}
select {
case ack := <-ackc:
if e := ack.GetError(); e != "" {
h.count("error")
return false, errors.New(e)
}
h.count(deliveredLabel(ack.GetDelivered()))
return ack.GetDelivered(), nil
case <-ctx.Done():
h.count("error")
return false, nil
}
}
// resolve routes an Ack to its waiting SendAwait, if any.
func (h *Hub) resolve(ack *botlinkv1.Ack) {
h.mu.Lock()
c, ok := h.pending[ack.GetCommandId()]
h.mu.Unlock()
if ok {
select {
case c <- ack:
default:
}
}
}
// nextID returns a process-unique command id.
func (h *Hub) nextID() string {
return strconv.FormatUint(h.seq.Add(1), 10)
}
// count records one command result, when metrics are enabled.
func (h *Hub) count(result string) {
if h.commands == nil {
return
}
h.commands.Add(context.Background(), 1, metric.WithAttributes(resultAttr(result)))
}
// deliveredLabel maps the delivered flag to a metric result label.
func deliveredLabel(delivered bool) string {
if delivered {
return "delivered"
}
return "not_delivered"
}
// resultAttr is the metric attribute carrying a command result label.
func resultAttr(result string) attribute.KeyValue {
return attribute.String("result", result)
}
+171
View File
@@ -0,0 +1,171 @@
package botlink
import (
"context"
"errors"
"net"
"testing"
"time"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
"google.golang.org/grpc/test/bufconn"
botlinkv1 "scrabble/pkg/proto/botlink/v1"
telegramv1 "scrabble/pkg/proto/telegram/v1"
)
// fakeBot is a test bot that dials the hub, registers, and acks commands with a
// fixed delivered flag (or never, when ack is false).
type fakeBot struct {
delivered bool
ack bool
received chan *botlinkv1.Command
}
// startHub registers a Hub on an in-memory gRPC server and returns the hub plus a
// dialer for fake bots.
func startHub(t *testing.T) (*Hub, func(t *testing.T) botlinkv1.BotLinkClient) {
t.Helper()
lis := bufconn.Listen(1 << 20)
hub := NewHub(nil, nil)
srv := grpc.NewServer()
botlinkv1.RegisterBotLinkServer(srv, hub)
go func() { _ = srv.Serve(lis) }()
t.Cleanup(srv.Stop)
dial := func(t *testing.T) botlinkv1.BotLinkClient {
t.Helper()
conn, err := grpc.NewClient("passthrough:///bufnet",
grpc.WithContextDialer(func(context.Context, string) (net.Conn, error) { return lis.Dial() }),
grpc.WithTransportCredentials(insecure.NewCredentials()),
)
if err != nil {
t.Fatalf("dial: %v", err)
}
t.Cleanup(func() { _ = conn.Close() })
return botlinkv1.NewBotLinkClient(conn)
}
return hub, dial
}
// connect runs a fake bot against client until ctx is cancelled, returning once the
// hub has registered it.
func (f *fakeBot) connect(t *testing.T, ctx context.Context, hub *Hub, client botlinkv1.BotLinkClient) {
t.Helper()
f.received = make(chan *botlinkv1.Command, 8)
stream, err := client.Link(ctx)
if err != nil {
t.Fatalf("link: %v", err)
}
if err := stream.Send(&botlinkv1.FromBot{Msg: &botlinkv1.FromBot_Hello{Hello: &botlinkv1.Hello{InstanceId: "test"}}}); err != nil {
t.Fatalf("hello: %v", err)
}
go func() {
for {
msg, err := stream.Recv()
if err != nil {
return
}
cmd := msg.GetCommand()
f.received <- cmd
if !f.ack {
continue
}
_ = stream.Send(&botlinkv1.FromBot{Msg: &botlinkv1.FromBot_Ack{Ack: &botlinkv1.Ack{
CommandId: cmd.GetCommandId(),
Delivered: f.delivered,
}}})
}
}()
waitConnected(t, hub, 1)
}
// waitConnected blocks until the hub reports n connected bots.
func waitConnected(t *testing.T, hub *Hub, n int) {
t.Helper()
deadline := time.Now().Add(2 * time.Second)
for time.Now().Before(deadline) {
hub.mu.Lock()
got := len(hub.links)
hub.mu.Unlock()
if got == n {
return
}
time.Sleep(5 * time.Millisecond)
}
t.Fatalf("hub did not reach %d connected bots", n)
}
func TestHubSendAwaitDelivered(t *testing.T) {
hub, dial := startHub(t)
ctx := t.Context()
bot := &fakeBot{delivered: true, ack: true}
bot.connect(t, ctx, hub, dial(t))
delivered, err := hub.SendAwait(ctx, SendToUserCommand("42", "hi"))
if err != nil {
t.Fatalf("SendAwait: %v", err)
}
if !delivered {
t.Fatal("delivered = false, want true")
}
select {
case cmd := <-bot.received:
if cmd.GetSendToUser().GetExternalId() != "42" {
t.Errorf("received external_id = %q, want 42", cmd.GetSendToUser().GetExternalId())
}
case <-time.After(time.Second):
t.Fatal("bot received no command")
}
}
func TestHubSendAwaitNoBot(t *testing.T) {
hub, _ := startHub(t)
_, err := hub.SendAwait(context.Background(), SendToUserCommand("42", "hi"))
if !errors.Is(err, ErrNoBot) {
t.Errorf("err = %v, want ErrNoBot", err)
}
}
func TestHubSendAwaitDeadline(t *testing.T) {
hub, dial := startHub(t)
ctx := t.Context()
bot := &fakeBot{ack: false} // receives but never acks
bot.connect(t, ctx, hub, dial(t))
awaitCtx, awaitCancel := context.WithTimeout(ctx, 100*time.Millisecond)
defer awaitCancel()
delivered, err := hub.SendAwait(awaitCtx, SendToUserCommand("42", "hi"))
if err != nil {
t.Fatalf("SendAwait err = %v, want nil on deadline", err)
}
if delivered {
t.Error("delivered = true, want false on deadline")
}
}
func TestHubSendAsync(t *testing.T) {
hub, dial := startHub(t)
ctx := t.Context()
bot := &fakeBot{ack: false}
bot.connect(t, ctx, hub, dial(t))
hub.Send(NotifyCommand("42", "your_turn", nil, "en"))
select {
case cmd := <-bot.received:
if cmd.GetNotify().GetExternalId() != "42" {
t.Errorf("received external_id = %q, want 42", cmd.GetNotify().GetExternalId())
}
case <-time.After(time.Second):
t.Fatal("bot received no command")
}
}
func TestRelayServerNoBot(t *testing.T) {
hub, _ := startHub(t)
relay := NewRelayServer(hub, 200*time.Millisecond)
if _, err := relay.SendToUser(context.Background(), &telegramv1.SendToUserRequest{ExternalId: "42", Text: "hi"}); err == nil {
t.Fatal("expected an error with no bot connected")
}
}
+160
View File
@@ -0,0 +1,160 @@
package botlink
import (
"crypto/ecdsa"
"crypto/elliptic"
"crypto/rand"
"crypto/tls"
"crypto/x509"
"crypto/x509/pkix"
"encoding/pem"
"math/big"
"net"
"os"
"path/filepath"
"testing"
"time"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials"
"scrabble/pkg/mtls"
botlinkv1 "scrabble/pkg/proto/botlink/v1"
)
// mintCerts writes a CA, a server leaf (IP SAN 127.0.0.1) and a client leaf into a
// temp dir, returning their file paths. It exercises the real pkg/mtls loaders.
func mintCerts(t *testing.T) (caFile, srvCert, srvKey, cliCert, cliKey string) {
t.Helper()
dir := t.TempDir()
caKey, _ := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
caTmpl := &x509.Certificate{
SerialNumber: big.NewInt(1),
Subject: pkix.Name{CommonName: "test-ca"},
NotBefore: time.Now().Add(-time.Hour),
NotAfter: time.Now().Add(time.Hour),
IsCA: true,
KeyUsage: x509.KeyUsageCertSign,
BasicConstraintsValid: true,
}
caDER, err := x509.CreateCertificate(rand.Reader, caTmpl, caTmpl, &caKey.PublicKey, caKey)
if err != nil {
t.Fatalf("ca: %v", err)
}
caCert, _ := x509.ParseCertificate(caDER)
leaf := func(cn string, eku x509.ExtKeyUsage, ips []net.IP) (certPath, keyPath string) {
key, _ := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
tmpl := &x509.Certificate{
SerialNumber: big.NewInt(time.Now().UnixNano()),
Subject: pkix.Name{CommonName: cn},
NotBefore: time.Now().Add(-time.Hour),
NotAfter: time.Now().Add(time.Hour),
KeyUsage: x509.KeyUsageDigitalSignature,
ExtKeyUsage: []x509.ExtKeyUsage{eku},
IPAddresses: ips,
}
der, err := x509.CreateCertificate(rand.Reader, tmpl, caCert, &key.PublicKey, caKey)
if err != nil {
t.Fatalf("leaf %s: %v", cn, err)
}
certPath = filepath.Join(dir, cn+".crt")
keyPath = filepath.Join(dir, cn+".key")
writePEM(t, certPath, "CERTIFICATE", der)
keyDER, _ := x509.MarshalPKCS8PrivateKey(key)
writePEM(t, keyPath, "PRIVATE KEY", keyDER)
return certPath, keyPath
}
caFile = filepath.Join(dir, "ca.crt")
writePEM(t, caFile, "CERTIFICATE", caDER)
srvCert, srvKey = leaf("server", x509.ExtKeyUsageServerAuth, []net.IP{net.ParseIP("127.0.0.1")})
cliCert, cliKey = leaf("client", x509.ExtKeyUsageClientAuth, nil)
return caFile, srvCert, srvKey, cliCert, cliKey
}
func writePEM(t *testing.T, path, typ string, der []byte) {
t.Helper()
if err := os.WriteFile(path, pem.EncodeToMemory(&pem.Block{Type: typ, Bytes: der}), 0o600); err != nil {
t.Fatalf("write %s: %v", path, err)
}
}
// startMTLSHub starts the Hub behind a real mTLS gRPC listener and returns its
// address and the CA/client cert paths.
func startMTLSHub(t *testing.T) (hub *Hub, addr, caFile, cliCert, cliKey string) {
t.Helper()
caFile, srvCert, srvKey, cliCert, cliKey := mintCerts(t)
tlsCfg, err := mtls.ServerConfig(srvCert, srvKey, caFile)
if err != nil {
t.Fatalf("server config: %v", err)
}
lis, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatalf("listen: %v", err)
}
hub = NewHub(nil, nil)
srv := grpc.NewServer(grpc.Creds(credentials.NewTLS(tlsCfg)))
botlinkv1.RegisterBotLinkServer(srv, hub)
go func() { _ = srv.Serve(lis) }()
t.Cleanup(srv.Stop)
return hub, lis.Addr().String(), caFile, cliCert, cliKey
}
// TestMTLSValidClientDelivers verifies a CA-signed bot connects and receives a
// pushed command.
func TestMTLSValidClientDelivers(t *testing.T) {
hub, addr, caFile, cliCert, cliKey := startMTLSHub(t)
tlsCfg, err := mtls.ClientConfig(cliCert, cliKey, caFile, "127.0.0.1")
if err != nil {
t.Fatalf("client config: %v", err)
}
conn, err := grpc.NewClient(addr, grpc.WithTransportCredentials(credentials.NewTLS(tlsCfg)))
if err != nil {
t.Fatalf("dial: %v", err)
}
t.Cleanup(func() { _ = conn.Close() })
ctx := t.Context()
bot := &fakeBot{delivered: true, ack: true}
bot.connect(t, ctx, hub, botlinkv1.NewBotLinkClient(conn))
delivered, err := hub.SendAwait(ctx, SendToUserCommand("42", "hi"))
if err != nil || !delivered {
t.Fatalf("SendAwait = (%v, %v), want (true, nil)", delivered, err)
}
}
// TestMTLSRejectsClientWithoutCert verifies the gateway refuses a client that
// presents no CA-signed certificate — mTLS is the sole guard on the bot-link.
func TestMTLSRejectsClientWithoutCert(t *testing.T) {
hub, addr, caFile, _, _ := startMTLSHub(t)
caPEM, _ := os.ReadFile(caFile)
pool := x509.NewCertPool()
pool.AppendCertsFromPEM(caPEM)
// Trusts the server, but presents no client certificate.
noCert := credentials.NewTLS(&tls.Config{RootCAs: pool, ServerName: "127.0.0.1", MinVersion: tls.VersionTLS13})
conn, err := grpc.NewClient(addr, grpc.WithTransportCredentials(noCert))
if err != nil {
t.Fatalf("dial: %v", err)
}
t.Cleanup(func() { _ = conn.Close() })
stream, err := botlinkv1.NewBotLinkClient(conn).Link(t.Context())
if err == nil {
err = stream.Send(&botlinkv1.FromBot{Msg: &botlinkv1.FromBot_Hello{Hello: &botlinkv1.Hello{InstanceId: "rogue"}}})
if err == nil {
_, err = stream.Recv()
}
}
if err == nil {
t.Fatal("expected the handshake to be rejected without a client certificate")
}
hub.mu.Lock()
n := len(hub.links)
hub.mu.Unlock()
if n != 0 {
t.Errorf("connected bots = %d, want 0 (rogue must not register)", n)
}
}
+67
View File
@@ -0,0 +1,67 @@
package botlink
import (
"context"
"errors"
"time"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/status"
botlinkv1 "scrabble/pkg/proto/botlink/v1"
telegramv1 "scrabble/pkg/proto/telegram/v1"
)
// RelayServer lets the backend's admin console reach the remote bot through the
// gateway: it implements the two admin delivery methods of the Telegram service by
// forwarding them onto the bot-link and awaiting the bot's Ack. The validation and
// out-of-app push methods are intentionally unimplemented here — login validation
// runs against the home validator and out-of-app push goes straight to the Hub.
type RelayServer struct {
telegramv1.UnimplementedTelegramServer
hub *Hub
deadline time.Duration
}
// NewRelayServer builds the relay over hub. deadline bounds the wait for the bot's
// Ack before reporting the send as not delivered.
func NewRelayServer(hub *Hub, deadline time.Duration) *RelayServer {
return &RelayServer{hub: hub, deadline: deadline}
}
// SendToUser forwards an admin message for one user to the bot and reports whether
// it was delivered. A missing bot maps to Unavailable (parity with an unreachable
// connector); a deadline before the Ack reports delivered=false.
func (r *RelayServer) SendToUser(ctx context.Context, req *telegramv1.SendToUserRequest) (*telegramv1.SendResponse, error) {
delivered, err := r.await(ctx, SendToUserCommand(req.GetExternalId(), req.GetText()))
if err != nil {
return nil, err
}
return &telegramv1.SendResponse{Delivered: delivered}, nil
}
// SendToGameChannel forwards an admin message for the game channel to the bot and
// reports whether it was delivered.
func (r *RelayServer) SendToGameChannel(ctx context.Context, req *telegramv1.SendToGameChannelRequest) (*telegramv1.SendResponse, error) {
delivered, err := r.await(ctx, SendToGameChannelCommand(req.GetText()))
if err != nil {
return nil, err
}
return &telegramv1.SendResponse{Delivered: delivered}, nil
}
// await sends cmd with the relay deadline and translates the Hub outcome into the
// gRPC contract the backend client expects.
func (r *RelayServer) await(ctx context.Context, cmd *botlinkv1.Command) (bool, error) {
cctx, cancel := context.WithTimeout(ctx, r.deadline)
defer cancel()
delivered, err := r.hub.SendAwait(cctx, cmd)
switch {
case errors.Is(err, ErrNoBot):
return false, status.Error(codes.Unavailable, "no telegram bot connected")
case err != nil:
return false, status.Error(codes.Internal, err.Error())
default:
return delivered, nil
}
}
+50 -5
View File
@@ -28,10 +28,13 @@ type Config struct {
// checks before proxying admin traffic to the backend. Empty disables admin.
AdminUser string
AdminPassword string
// ConnectorAddr is the gRPC address of the Telegram connector side-service. The
// gateway calls it to validate Mini App initData and to deliver out-of-app push.
// Empty disables the telegram auth path and the out-of-app push channel.
ConnectorAddr string
// ValidatorAddr is the gRPC address of the Telegram validator side-service (home,
// plaintext, internal). The gateway calls it to validate Mini App initData and
// Login Widget data. Empty disables the telegram auth path.
ValidatorAddr string
// BotLink configures the reverse mTLS channel to the remote Telegram bot. An
// empty BotLink.Addr disables the bot channel (out-of-app push and admin relay).
BotLink BotLinkConfig
// SessionTTL bounds how long a resolved session stays cached; SessionCacheMax
// caps the number of cached sessions.
SessionTTL time.Duration
@@ -47,6 +50,29 @@ type Config struct {
Telemetry pkgtel.Config
}
// BotLinkConfig configures the gateway's reverse bot-link: the mTLS listener the
// remote Telegram bot dials, the plaintext listener the backend admin relay calls,
// and the mTLS material. The main host is already public, so exposing the bot-link
// listener on a dedicated port adds no static IP; the channel is guarded solely by
// mTLS (the bot has no fixed address to allow-list).
type BotLinkConfig struct {
// Addr is the mTLS gRPC listener the bot dials (e.g. ":9443"). Empty disables
// the whole bot channel.
Addr string
// RelayAddr is the plaintext internal gRPC listener that serves the backend
// admin SendToUser/SendToGameChannel relay (e.g. ":9092"). Empty disables it.
RelayAddr string
// CertFile, KeyFile and CAFile are the gateway server certificate, its key and
// the CA bundle that signs the accepted bot client certificates. Required when
// Addr is set.
CertFile string
KeyFile string
CAFile string
// SendTimeout bounds the admin relay's wait for the bot Ack before reporting
// the send as not delivered.
SendTimeout time.Duration
}
// RateLimitConfig holds the token-bucket limits per class. Public and admin are
// keyed per client IP; the authenticated class is keyed per user id; the email
// sub-limit guards the costly email-code path per IP.
@@ -72,6 +98,7 @@ const (
defaultSessionCacheMax = 50000
defaultPushHeartbeatInterval = 10 * time.Second // under the ~15 s edge idle timeout
defaultServiceName = "scrabble-gateway"
defaultBotLinkSendTimeout = 5 * time.Second
)
// DefaultMaxBodyBytes is the default request-body cap (GATEWAY_MAX_BODY_BYTES):
@@ -104,9 +131,16 @@ func Load() (Config, error) {
BackendGRPCAddr: envOr("GATEWAY_BACKEND_GRPC_ADDR", defaultBackendGRPCAddr),
AdminUser: os.Getenv("GATEWAY_ADMIN_USER"),
AdminPassword: os.Getenv("GATEWAY_ADMIN_PASSWORD"),
ConnectorAddr: os.Getenv("GATEWAY_CONNECTOR_ADDR"),
ValidatorAddr: os.Getenv("GATEWAY_VALIDATOR_ADDR"),
SessionCacheMax: defaultSessionCacheMax,
RateLimit: DefaultRateLimit(),
BotLink: BotLinkConfig{
Addr: os.Getenv("GATEWAY_BOTLINK_ADDR"),
RelayAddr: os.Getenv("GATEWAY_BOTLINK_RELAY_ADDR"),
CertFile: os.Getenv("GATEWAY_BOTLINK_TLS_CERT"),
KeyFile: os.Getenv("GATEWAY_BOTLINK_TLS_KEY"),
CAFile: os.Getenv("GATEWAY_BOTLINK_TLS_CA"),
},
}
tel := pkgtel.DefaultConfig(defaultServiceName)
tel.ServiceName = envOr("GATEWAY_SERVICE_NAME", tel.ServiceName)
@@ -128,12 +162,18 @@ func Load() (Config, error) {
if c.MaxBodyBytes, err = envInt("GATEWAY_MAX_BODY_BYTES", DefaultMaxBodyBytes); err != nil {
return Config{}, err
}
if c.BotLink.SendTimeout, err = envDuration("GATEWAY_BOTLINK_SEND_TIMEOUT", defaultBotLinkSendTimeout); err != nil {
return Config{}, err
}
if err := c.validate(); err != nil {
return Config{}, err
}
return c, nil
}
// BotLinkEnabled reports whether the reverse bot-link channel is configured.
func (c Config) BotLinkEnabled() bool { return c.BotLink.Addr != "" }
// AdminEnabled reports whether the admin console proxy should be mounted (both
// Basic-Auth credentials are configured).
func (c Config) AdminEnabled() bool {
@@ -159,6 +199,11 @@ func (c Config) validate() error {
if c.MaxBodyBytes <= 0 {
return fmt.Errorf("config: GATEWAY_MAX_BODY_BYTES must be positive")
}
if c.BotLink.Addr != "" {
if c.BotLink.CertFile == "" || c.BotLink.KeyFile == "" || c.BotLink.CAFile == "" {
return fmt.Errorf("config: GATEWAY_BOTLINK_ADDR requires GATEWAY_BOTLINK_TLS_CERT, _KEY and _CA")
}
}
if err := c.Telemetry.Validate(); err != nil {
return fmt.Errorf("config: %w", err)
}
+6 -19
View File
@@ -1,7 +1,9 @@
// Package connector is the gateway's gRPC client for the Telegram connector
// side-service: it validates Mini App initData and delivers out-of-app push. The
// connector lives on the trusted internal network, so the connection uses insecure
// (plaintext) transport credentials (ARCHITECTURE.md §12).
// Package connector is the gateway's gRPC client for the Telegram validator
// side-service: it validates Mini App initData and Login Widget data. The validator
// lives on the trusted internal network and holds the bot token only for HMAC, so
// the connection uses insecure (plaintext) transport credentials (ARCHITECTURE.md
// §12). Out-of-app push no longer goes through this client; it is delivered to the
// remote bot over the reverse mTLS bot-link (gateway/internal/botlink).
package connector
import (
@@ -92,18 +94,3 @@ func (c *Client) ValidateLoginWidget(ctx context.Context, data string) (User, er
FirstName: resp.GetFirstName(),
}, nil
}
// Notify delivers an out-of-app notification for a push event; delivered reports
// whether a message was actually sent.
func (c *Client) Notify(ctx context.Context, externalID, kind string, payload []byte, language string) (bool, error) {
resp, err := c.c.Notify(ctx, &telegramv1.NotifyRequest{
ExternalId: externalID,
Kind: kind,
Payload: payload,
Language: language,
})
if err != nil {
return false, err
}
return resp.GetDelivered(), nil
}