Bouncer server proxy with Redis Sentinel Error #16

Closed
opened 2024-02-05 15:47:57 +01:00 by pcaseiro · 2 comments
Owner

What append :
The redirections don't work with Redis Sentinel

How to reproduce:
1°) Setup a Kubernetes cluster (with kind or whatever)
2°) Checkout the feat/issue-10/add-k8s-kustomize branch
3°) Deploy the developpement overlay with the command : kubectl apply -k misc/k8s/overlay/dev
4°) Forward the port for the service bouncer-admin (8081) with kubectl port-forward or k9s
5°) Forward the port for the service boucner-proxy(8080) with kubectl port-forward or k9s
6°) Follow the bouncer doc "getting-started"
7°) Try to access the bouncer proxy after the redirection creation.

Error log from Bouncer

2024-02-05 14:41:01.102 [ERROR]    <./internal/proxy/director.go:191>    (*Server).run.func5.(*Director).Middleware.func1.1    could not rewrite request ...
  "error": ERR unknown command `scan`, with args beginning with: `0`, `match`, `proxy*`, 
           forge.cadoles.com/cadoles/bouncer/internal/store/redis.(*ProxyRepository).QueryProxy
               /src/internal/store/redis/proxy_repository.go:178
           forge.cadoles.com/cadoles/bouncer/internal/proxy/director.(*Director).getProxies
               /src/internal/proxy/director/director.go:79
           forge.cadoles.com/cadoles/bouncer/internal/proxy/director.(*Director).rewriteRequest
               /src/internal/proxy/director/director.go:26
           forge.cadoles.com/cadoles/bouncer/internal/proxy.(*Server).run.func5.(*Director).Middleware.func1.1
               /src/internal/proxy/director/director.go:189
           net/http.HandlerFunc.ServeHTTP
               /usr/local/go/src/net/http/server.go:2136
           github.com/go-chi/chi/v5.(*ChainHandler).ServeHTTP
               /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/chain.go:31
           github.com/go-chi/chi/v5.(*Mux).routeHTTP
               /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444
           net/http.HandlerFunc.ServeHTTP
               /usr/local/go/src/net/http/server.go:2136
           forge.cadoles.com/cadoles/bouncer/internal/proxy.(*Server).run.RequestLogger.func6.1
               /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/logger.go:54
           net/http.HandlerFunc.ServeHTTP
               /usr/local/go/src/net/http/server.go:2136
           github.com/go-chi/chi/v5/middleware.RealIP.func1
               /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/realip.go:36
           net/http.HandlerFunc.ServeHTTP
               /usr/local/go/src/net/http/server.go:2136
           github.com/go-chi/chi/v5.(*Mux).ServeHTTP
               /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90
           net/http.serverHandler.ServeHTTP
               /usr/local/go/src/net/http/server.go:2938
           net/http.(*conn).serve
               /usr/local/go/src/net/http/server.go:2009
           runtime.goexit
               /usr/local/go/src/runtime/asm_amd64.s:1650
           forge.cadoles.com/cadoles/bouncer/internal/proxy/director.(*Director).getProxies
               /src/internal/proxy/director/director.go:81
           forge.cadoles.com/cadoles/bouncer/internal/proxy/director.(*Director).rewriteRequest
               /src/internal/proxy/director/director.go:26
           forge.cadoles.com/cadoles/bouncer/internal/proxy.(*Server).run.func5.(*Director).Middleware.func1.1
               /src/internal/proxy/director/director.go:189
           net/http.HandlerFunc.ServeHTTP
               /usr/local/go/src/net/http/server.go:2136
           github.com/go-chi/chi/v5.(*ChainHandler).ServeHTTP
               /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/chain.go:31
           github.com/go-chi/chi/v5.(*Mux).routeHTTP
               /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444
           net/http.HandlerFunc.ServeHTTP
               /usr/local/go/src/net/http/server.go:2136
           forge.cadoles.com/cadoles/bouncer/internal/proxy.(*Server).run.RequestLogger.func6.1
               /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/logger.go:54
           net/http.HandlerFunc.ServeHTTP
               /usr/local/go/src/net/http/server.go:2136
           github.com/go-chi/chi/v5/middleware.RealIP.func1
               /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/realip.go:36
           net/http.HandlerFunc.ServeHTTP
               /usr/local/go/src/net/http/server.go:2136
           github.com/go-chi/chi/v5.(*Mux).ServeHTTP
               /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90
           net/http.serverHandler.ServeHTTP
               /usr/local/go/src/net/http/server.go:2938
           net/http.(*conn).serve
               /usr/local/go/src/net/http/server.go:2009
           runtime.goexit
               /usr/local/go/src/runtime/asm_amd64.s:1650
           forge.cadoles.com/cadoles/bouncer/internal/proxy/director.(*Director).rewriteRequest
               /src/internal/proxy/director/director.go:28
           forge.cadoles.com/cadoles/bouncer/internal/proxy.(*Server).run.func5.(*Director).Middleware.func1.1
               /src/internal/proxy/director/director.go:189
           net/http.HandlerFunc.ServeHTTP
               /usr/local/go/src/net/http/server.go:2136
           github.com/go-chi/chi/v5.(*ChainHandler).ServeHTTP
               /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/chain.go:31
           github.com/go-chi/chi/v5.(*Mux).routeHTTP
               /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444
           net/http.HandlerFunc.ServeHTTP
               /usr/local/go/src/net/http/server.go:2136
           forge.cadoles.com/cadoles/bouncer/internal/proxy.(*Server).run.RequestLogger.func6.1
               /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/logger.go:54
           net/http.HandlerFunc.ServeHTTP
               /usr/local/go/src/net/http/server.go:2136
           github.com/go-chi/chi/v5/middleware.RealIP.func1
               /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/realip.go:36
           net/http.HandlerFunc.ServeHTTP
               /usr/local/go/src/net/http/server.go:2136
           github.com/go-chi/chi/v5.(*Mux).ServeHTTP
               /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90
           net/http.serverHandler.ServeHTTP
               /usr/local/go/src/net/http/server.go:2938
           net/http.(*conn).serve
               /usr/local/go/src/net/http/server.go:2009
           runtime.goexit
               /usr/local/go/src/runtime/asm_amd64.s:1650
           forge.cadoles.com/cadoles/bouncer/internal/proxy.(*Server).run.func5.(*Director).Middleware.func1.1
               /src/internal/proxy/director/director.go:191
           net/http.HandlerFunc.ServeHTTP
               /usr/local/go/src/net/http/server.go:2136
           github.com/go-chi/chi/v5.(*ChainHandler).ServeHTTP
               /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/chain.go:31
           github.com/go-chi/chi/v5.(*Mux).routeHTTP
               /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444
           net/http.HandlerFunc.ServeHTTP
               /usr/local/go/src/net/http/server.go:2136
           forge.cadoles.com/cadoles/bouncer/internal/proxy.(*Server).run.RequestLogger.func6.1
               /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/logger.go:54
           net/http.HandlerFunc.ServeHTTP
               /usr/local/go/src/net/http/server.go:2136
           github.com/go-chi/chi/v5/middleware.RealIP.func1
               /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/realip.go:36
           net/http.HandlerFunc.ServeHTTP
               /usr/local/go/src/net/http/server.go:2136
           github.com/go-chi/chi/v5.(*Mux).ServeHTTP
               /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90
           net/http.serverHandler.ServeHTTP
               /usr/local/go/src/net/http/server.go:2938
           net/http.(*conn).serve
               /usr/local/go/src/net/http/server.go:2009
           runtime.goexit
               /usr/local/go/src/runtime/asm_amd64.s:1650
What append : The redirections don't work with Redis Sentinel How to reproduce: 1°) Setup a Kubernetes cluster (with kind or whatever) 2°) Checkout the feat/issue-10/add-k8s-kustomize branch 3°) Deploy the developpement overlay with the command : `kubectl apply -k misc/k8s/overlay/dev` 4°) Forward the port for the service bouncer-admin (8081) with kubectl port-forward or k9s 5°) Forward the port for the service boucner-proxy(8080) with kubectl port-forward or k9s 6°) Follow the bouncer doc "getting-started" 7°) Try to access the bouncer proxy after the redirection creation. Error log from Bouncer ``` 2024-02-05 14:41:01.102 [ERROR] <./internal/proxy/director.go:191> (*Server).run.func5.(*Director).Middleware.func1.1 could not rewrite request ... "error": ERR unknown command `scan`, with args beginning with: `0`, `match`, `proxy*`, forge.cadoles.com/cadoles/bouncer/internal/store/redis.(*ProxyRepository).QueryProxy /src/internal/store/redis/proxy_repository.go:178 forge.cadoles.com/cadoles/bouncer/internal/proxy/director.(*Director).getProxies /src/internal/proxy/director/director.go:79 forge.cadoles.com/cadoles/bouncer/internal/proxy/director.(*Director).rewriteRequest /src/internal/proxy/director/director.go:26 forge.cadoles.com/cadoles/bouncer/internal/proxy.(*Server).run.func5.(*Director).Middleware.func1.1 /src/internal/proxy/director/director.go:189 net/http.HandlerFunc.ServeHTTP /usr/local/go/src/net/http/server.go:2136 github.com/go-chi/chi/v5.(*ChainHandler).ServeHTTP /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/chain.go:31 github.com/go-chi/chi/v5.(*Mux).routeHTTP /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444 net/http.HandlerFunc.ServeHTTP /usr/local/go/src/net/http/server.go:2136 forge.cadoles.com/cadoles/bouncer/internal/proxy.(*Server).run.RequestLogger.func6.1 /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/logger.go:54 net/http.HandlerFunc.ServeHTTP /usr/local/go/src/net/http/server.go:2136 github.com/go-chi/chi/v5/middleware.RealIP.func1 /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/realip.go:36 net/http.HandlerFunc.ServeHTTP /usr/local/go/src/net/http/server.go:2136 github.com/go-chi/chi/v5.(*Mux).ServeHTTP /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90 net/http.serverHandler.ServeHTTP /usr/local/go/src/net/http/server.go:2938 net/http.(*conn).serve /usr/local/go/src/net/http/server.go:2009 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:1650 forge.cadoles.com/cadoles/bouncer/internal/proxy/director.(*Director).getProxies /src/internal/proxy/director/director.go:81 forge.cadoles.com/cadoles/bouncer/internal/proxy/director.(*Director).rewriteRequest /src/internal/proxy/director/director.go:26 forge.cadoles.com/cadoles/bouncer/internal/proxy.(*Server).run.func5.(*Director).Middleware.func1.1 /src/internal/proxy/director/director.go:189 net/http.HandlerFunc.ServeHTTP /usr/local/go/src/net/http/server.go:2136 github.com/go-chi/chi/v5.(*ChainHandler).ServeHTTP /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/chain.go:31 github.com/go-chi/chi/v5.(*Mux).routeHTTP /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444 net/http.HandlerFunc.ServeHTTP /usr/local/go/src/net/http/server.go:2136 forge.cadoles.com/cadoles/bouncer/internal/proxy.(*Server).run.RequestLogger.func6.1 /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/logger.go:54 net/http.HandlerFunc.ServeHTTP /usr/local/go/src/net/http/server.go:2136 github.com/go-chi/chi/v5/middleware.RealIP.func1 /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/realip.go:36 net/http.HandlerFunc.ServeHTTP /usr/local/go/src/net/http/server.go:2136 github.com/go-chi/chi/v5.(*Mux).ServeHTTP /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90 net/http.serverHandler.ServeHTTP /usr/local/go/src/net/http/server.go:2938 net/http.(*conn).serve /usr/local/go/src/net/http/server.go:2009 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:1650 forge.cadoles.com/cadoles/bouncer/internal/proxy/director.(*Director).rewriteRequest /src/internal/proxy/director/director.go:28 forge.cadoles.com/cadoles/bouncer/internal/proxy.(*Server).run.func5.(*Director).Middleware.func1.1 /src/internal/proxy/director/director.go:189 net/http.HandlerFunc.ServeHTTP /usr/local/go/src/net/http/server.go:2136 github.com/go-chi/chi/v5.(*ChainHandler).ServeHTTP /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/chain.go:31 github.com/go-chi/chi/v5.(*Mux).routeHTTP /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444 net/http.HandlerFunc.ServeHTTP /usr/local/go/src/net/http/server.go:2136 forge.cadoles.com/cadoles/bouncer/internal/proxy.(*Server).run.RequestLogger.func6.1 /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/logger.go:54 net/http.HandlerFunc.ServeHTTP /usr/local/go/src/net/http/server.go:2136 github.com/go-chi/chi/v5/middleware.RealIP.func1 /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/realip.go:36 net/http.HandlerFunc.ServeHTTP /usr/local/go/src/net/http/server.go:2136 github.com/go-chi/chi/v5.(*Mux).ServeHTTP /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90 net/http.serverHandler.ServeHTTP /usr/local/go/src/net/http/server.go:2938 net/http.(*conn).serve /usr/local/go/src/net/http/server.go:2009 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:1650 forge.cadoles.com/cadoles/bouncer/internal/proxy.(*Server).run.func5.(*Director).Middleware.func1.1 /src/internal/proxy/director/director.go:191 net/http.HandlerFunc.ServeHTTP /usr/local/go/src/net/http/server.go:2136 github.com/go-chi/chi/v5.(*ChainHandler).ServeHTTP /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/chain.go:31 github.com/go-chi/chi/v5.(*Mux).routeHTTP /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:444 net/http.HandlerFunc.ServeHTTP /usr/local/go/src/net/http/server.go:2136 forge.cadoles.com/cadoles/bouncer/internal/proxy.(*Server).run.RequestLogger.func6.1 /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/logger.go:54 net/http.HandlerFunc.ServeHTTP /usr/local/go/src/net/http/server.go:2136 github.com/go-chi/chi/v5/middleware.RealIP.func1 /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/middleware/realip.go:36 net/http.HandlerFunc.ServeHTTP /usr/local/go/src/net/http/server.go:2136 github.com/go-chi/chi/v5.(*Mux).ServeHTTP /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.8/mux.go:90 net/http.serverHandler.ServeHTTP /usr/local/go/src/net/http/server.go:2938 net/http.(*conn).serve /usr/local/go/src/net/http/server.go:2009 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:1650 ```
pcaseiro added the
Kind/Bug
Kind/Breaking
Priority
Critical
labels 2024-02-05 15:47:57 +01:00
Author
Owner

You also can use the command :

skaffold dev -p dev --cleanup=false --default-repo reg.cadoles.com/<YOUR_PERSONNAL_USER_NAME>

from feat/issue-10/add-k8s-kustomize branch

You also can use the command : ``` skaffold dev -p dev --cleanup=false --default-repo reg.cadoles.com/<YOUR_PERSONNAL_USER_NAME> ``` from `feat/issue-10/add-k8s-kustomize` branch
Owner

Le problème provenait d'une mauvaise configuration de Bouncer vis à vis de l'utilisation du mode "sentinel" de Redis. Pour faire court, la bonne configuration était la suivante:

redis:
  addresses:
  - rfs-bouncer-redis:${RFS_BOUNCER_REDIS_SERVICE_PORT}
  master: mymaster

cf. la documentation de l'opérateur Redis: https://github.com/spotahome/redis-operator?tab=readme-ov-file#connection-to-the-created-redis-failovers

J'ai également profité de l'occasion pour mettre en place une procédure de démarrage plus complète de l'environnement de test Kubernetes, disponible à l'adresse suivante: 6f61073d1f/misc/k8s

Le correctif de la configuration est poussé sur la branche.

Le problème provenait d'une mauvaise configuration de Bouncer vis à vis de l'utilisation du mode "sentinel" de Redis. Pour faire court, la bonne configuration était la suivante: ```yaml redis: addresses: - rfs-bouncer-redis:${RFS_BOUNCER_REDIS_SERVICE_PORT} master: mymaster ``` cf. la documentation de l'opérateur Redis: https://github.com/spotahome/redis-operator?tab=readme-ov-file#connection-to-the-created-redis-failovers J'ai également profité de l'occasion pour mettre en place une procédure de démarrage plus complète de l'environnement de test Kubernetes, disponible à l'adresse suivante: https://forge.cadoles.com/Cadoles/bouncer/src/commit/6f61073d1fd5f9ac201c8061f737e0a0a8edc7dc/misc/k8s Le correctif de la configuration est poussé sur la branche.
Sign in to join this conversation.
No Milestone
No project
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: Cadoles/bouncer#16
No description provided.