Add fetch by ID feature

This commit is contained in:
Vikram Rangnekar 2019-04-04 00:53:24 -04:00
parent 813540f257
commit 33c5ef0bf1
12 changed files with 138 additions and 482 deletions

View File

@ -3,3 +3,4 @@ tmp
*.md
web/build
docker-compose.*
docs

1
.gitignore vendored
View File

@ -26,3 +26,4 @@
.vscode
main
.DS_Store
.swp

351
README.md
View File

@ -4,7 +4,7 @@
Get an high-performance GraphQL API for your Rails app in seconds. Super Graph will auto-learn your database structure and relationships. Built in support for Rails authentication and JWT tokens.
![Super Graph Web UI](web/public/super-graph-web-ui.png?raw=true "Super Graph Web UI for web developers")
![Super Graph Web UI](docs/.vuepress/public/super-graph-web-ui.png?raw=true "Super Graph Web UI for web developers")
## Why I built Super Graph?
@ -29,355 +29,6 @@ And so after a lot of coffee and some avocado toasts Super Graph was born. An in
- High performance GO codebase
- Tiny docker image and low memory requirements
### GraphQL (GQL)
We currently support the `query` action which is used for fetching data. Support for `mutation` and `subscriptions` is work in progress. For example the below GraphQL query would fetch two products that belong to the current user where the price is greater than 10
#### GQL Query
```gql
query {
users {
id
email
picture : avatar
password
full_name
products(limit: 2, where: { price: { gt: 10 } }) {
id
name
description
price
}
}
}
```
The above GraphQL query returns the JSON result below. It handles all
kinds of complexity without you having to writing a line of code.
For example there is a while greater than `gt` and a limit clause on a child field. And the `avatar` field is renamed to `picture`. The `password` field is blocked and not returned. Finally the relationship between the `users` table and the `products` table is auto discovered and used.
#### JSON Result
```json
{
"data": {
"users": [
{
"id": 1,
"email": "odilia@west.info",
"picture": "https://robohash.org/simur.png?size=300x300",
"full_name": "Edwin Orn",
"products": [
{
"id": 16,
"name": "Sierra Nevada Style Ale",
"description": "Belgian Abbey, 92 IBU, 4.7%, 17.4°Blg",
"price": 16.47
},
...
]
}
]
}
}
```
## Try it out
```console
$ docker-compose run web rake db:create db:migrate db:seed
$ docker-compose -f docker-compose.image.yml up
$ open http://localhost:8080
```
The above command will download the latest docker image for Super Graph and use it to run an example that includes a Postgres DB and a simple Rails ecommerce store app.
If you want to build and run Super Graph from code then the below commands will build the web ui and launch Super Graph in developer mode with a watcher to rebuild on code changes.
```console
$ brew install yarn
$ (cd web && yarn install && yarn build)
$ go generate ./...
$ docker-compose up
```
#### Try with an authenticated user
In development mode you can use the `X-User-ID: 4` header to set a user id so you don't have to worries about cookies etc. This can be set using the *HTTP Headers* tab at the bottom of the web UI you'll see when you visit the above link. You can also directly run queries from the commandline like below.
#### Querying the GQL endpoint
```console
curl 'http://localhost:8080/api/v1/graphql' \
-H 'content-type: application/json' \
-H 'X-User-ID: 5' \
--data-binary '{"query":"{ products { name price users { email }}}"}'
```
## How to GraphQL
GraphQL (GQL) is a simple query syntax that's fast replacing REST APIs. GQL is great since it allows web developers to fetch the exact data that they need without depending on changes to backend code. Also if you squint hard enough it looks a little bit like JSON :smiley:
The below query will fetch an `users` name, email and avatar image (renamed as picture). If you also need the users `id` then just add it to the query.
```gql
query {
user {
full_name
email
picture : avatar
}
}
```
Super Graph support complex queries where you can add filters, ordering,offsets and limits on the query.
#### Logical Operators
Name | Example | Explained |
--- | --- | --- |
and | price : { and : { gt: 10.5, lt: 20 } | price > 10.5 AND price < 20
or | or : { price : { greater_than : 20 }, quantity: { gt : 0 } } | price >= 20 OR quantity > 0
not | not: { or : { quantity : { eq: 0 }, price : { eq: 0 } } } | NOT (quantity = 0 OR price = 0)
#### Other conditions
Name | Example | Explained |
--- | --- | --- |
eq, equals | id : { eq: 100 } | id = 100
neq, not_equals | id: { not_equals: 100 } | id != 100
gt, greater_than | id: { gt: 100 } | id > 100
lt, lesser_than | id: { gt: 100 } | id < 100
gte, greater_or_equals | id: { gte: 100 } | id >= 100
lte, lesser_or_equals | id: { lesser_or_equals: 100 } | id <= 100
in | status: { in: [ "A", "B", "C" ] } | status IN ('A', 'B', 'C)
nin, not_in | status: { in: [ "A", "B", "C" ] } | status IN ('A', 'B', 'C)
like | name: { like "phil%" } | Names starting with 'phil'
nlike, not_like | name: { nlike "v%m" } | Not names starting with 'v' and ending with 'm'
ilike | name: { ilike "%wOn" } | Names ending with 'won' case-insensitive
nilike, not_ilike | name: { nilike "%wOn" } | Not names ending with 'won' case-insensitive
similar | name: { similar: "%(b\|d)%" } | [Similar Docs](https://www.postgresql.org/docs/9/functions-matching.html#FUNCTIONS-SIMILARTO-REGEXP)
nsimilar, not_similar | name: { nsimilar: "%(b\|d)%" } | [Not Similar Docs](https://www.postgresql.org/docs/9/functions-matching.html#FUNCTIONS-SIMILARTO-REGEXP)
has_key | column: { has_key: 'b' } | Does JSON column contain this key
has_key_any | column: { has_key_any: [ a, b ] } | Does JSON column contain any of these keys
has_key_all | column: [ a, b ] | Does JSON column contain all of this keys
contains | column: { contains: [1, 2, 4] } | Is this array/json column a subset of value
contained_in | column: { contains: "{'a':1, 'b':2}" } | Is this array/json column a subset of these value
is_null | column: { is_null: true } | Is column value null or not
#### Aggregation
You will often find the need to fetch aggregated values from the database such as `count`, `max`, `min`, etc. This is simple to do with GraphQL, just prefix the aggregation name to the field name that you want to aggregrate like `count_id`. The below query will group products by name and find the minimum price for each group. Notice the `min_price` field we're adding `min_` to price.
```gql
query {
products {
name
min_price
}
}
```
Name | Explained |
--- | --- |
avg | Average value
count | Count the values
max | Maximum value
min | Minimum value
stddev | [Standard Deviation](https://en.wikipedia.org/wiki/Standard_deviation)
stddev_pop | Population Standard Deviation
stddev_samp | Sample Standard Deviation
variance | [Variance](https://en.wikipedia.org/wiki/Variance)
var_pop | Population Standard Variance
var_samp | Sample Standard variance
All kinds of queries are possible with GraphQL. Below is an example that uses a lot of the features available. Comments `# hello` are also valid within queries.
```javascript
query {
products(
# returns only 30 items
limit: 30,
# starts from item 10, commented out for now
# offset: 10,
# orders the response items by highest price
order_by: { price: desc },
# no duplicate prices returned
distinct: [ price ]
# only items with an id >= 30 and < 30 are returned
where: { id: { and: { greater_or_equals: 20, lt: 28 } } }) {
id
name
price
}
}
```
## It's easy to setup
Configuration files can either be in YAML or JSON their names are derived from the `GO_ENV` variable, for example `GO_ENV=prod` will cause the `prod.yaml` config file to be used. or `GO_ENV=dev` will use the `dev.yaml`. A path to look for the config files in can be specified using the `-path <folder>` command line argument.
```yaml
host_port: 0.0.0.0:8080
web_ui: true
debug_level: 1
enable_tracing: true
# When to throw a 401 on auth failure
# valid values: always, per_query, never
auth_fail_block: never
# Postgres related environment Variables
# SG_DATABASE_HOST
# SG_DATABASE_PORT
# SG_DATABASE_USER
# SG_DATABASE_PASSWORD
# Auth related environment Variables
# SG_AUTH_SECRET_KEY_BASE
# SG_AUTH_PUBLIC_KEY_FILE
# SG_AUTH_URL
# SG_AUTH_PASSWORD
# inflections:
# person: people
# sheep: sheep
auth:
type: header
field_name: X-User-ID
# auth:
# type: rails
# cookie: _app_session
# store: cookie
# secret_key_base: caf335bfcfdb04e50db5bb0a4d67ab9...
# auth:
# type: rails
# cookie: _app_session
# store: memcache
# host: 127.0.0.1
# auth:
# type: rails
# cookie: _app_session
# store: redis
# max_idle: 80,
# max_active: 12000,
# url: redis://127.0.0.1:6379
# password: ""
# auth:
# type: jwt
# cookie: _app_session
# secret: abc335bfcfdb04e50db5bb0a4d67ab9
# public_key_file: /secrets/public_key.pem
# public_key_type: ecdsa #rsa
database:
type: postgres
host: db
port: 5432
dbname: app_development
user: postgres
password: ''
#pool_size: 10
#max_retries: 0
#log_level: "debug"
# Define variables here that you want to use in filters
variables:
account_id: "select account_id from users where id = $user_id"
# Used to add access to tables
filters:
users: "{ id: { _eq: $user_id } }"
posts: "{ account_id: { _eq: $account_id } }"
# Fields and table names that you wish to block
blacklist:
- secret
- password
- encrypted
- token
```
If deploying into environments like Kubernetes it's useful to be able to configure things like secrets and hosts though environment variables therfore we expose the below environment variables. This is escpecially useful for secrets since they are usually injected in via a secrets management framework ie. Kubernetes Secrets
#### Postgres related environment Variables
```console
SG_DATABASE_HOST
SG_DATABASE_PORT
SG_DATABASE_USER
SG_DATABASE_PASSWORD
```
#### Auth related environment Variables
```console
SG_AUTH_SECRET_KEY_BASE
SG_AUTH_PUBLIC_KEY_FILE
SG_AUTH_URL
SG_AUTH_PASSWORD
```
## Authentication
You can only have one type of auth enabled. You can either pick Rails or JWT. Uncomment the one you use and leave the rest commented out.
#### JWT Tokens
```yaml
auth:
type: jwt
provider: auth0 #none
cookie: _app_session
secret: abc335bfcfdb04e50db5bb0a4d67ab9
public_key_file: /secrets/public_key.pem
public_key_type: ecdsa #rsa
```
For JWT tokens we currently support tokens from a provider like Auth0
or if you have a custom solution then we look for the `user_id` in the
`subject` claim of of the `id token`. If you pick Auth0 then we derive two variables from the token `user_id` and `user_id_provider` for to use in your filters.
We can get the JWT token either from the `authorization` header where we expect it to be a `bearer` token or if `cookie` is specified then we look there.
For validation a `secret` or a public key (ecdsa or rsa) is required. When using public keys they have to be in a PEM format file.
## Deploying Super Graph
How do I deploy the Super Graph service with my existing rails app? You have several options here. Esentially you need to ensure your app's session cookie will be passed to this service.
#### Custom Docker Image
Create a `Dockerfile` like the one below to roll your own
custom Super Graph docker image. And to build it `docker build -t my-super-graph .`
```dockerfile
FROM dosco/super-graph:latest
WORKDIR /app
COPY *.yml ./
```
#### Deploy under a subdomain
For this to work you have to ensure that the option `:domain => :all` is added to your rails app config `Application.config.session_store` this will cause your rails app to create session cookies that can be shared with sub-domains. More info here <http://excid3.com/blog/sharing-a-devise-user-session-across-subdomains-with-rails-3/>
#### With an NGINX loadbalancer
I'm sure you know how to configure it so that the Super Graph endpoint path `/api/v1/graphql` is routed to wherever you have this service installed within your architecture.
#### On Kubernetes
If your Rails app runs on Kubernetes then ensure you have an ingress config deployed that points the path to the service that you have deployed Super Graph under.
#### We use JWT tokens like those from Auth0
In that case deploy under a subdomain and configure this service to use JWT authentication. You will need the public key file or secret key. Ensure your web app passes the JWT token with every GQL request in the Authorize header as a `bearer` token.
## Contact me
[twitter.com/dosco](https://twitter.com/dosco)

View File

@ -2,7 +2,7 @@ title: Super Graph Development
host_port: 0.0.0.0:8080
web_ui: true
debug_level: 1
enable_tracing: true
enable_tracing: false
# Throw a 401 on auth failure for queries that need auth
# valid values: always, per_query, never

View File

@ -4,7 +4,7 @@ build_name: runner-build
build_log: runner-build-errors.log
valid_ext: .go, .tpl, .tmpl, .html, .yml
no_rebuild_ext: .tpl, .tmpl, .html
ignored: web, tmp, vendor, example
ignored: web, tmp, vendor, example, docs
build_delay: 600
colors: 1
log_color_main: cyan

View File

@ -263,7 +263,7 @@ func (v *selectBlock) renderJoinedColumns(w io.Writer, childIDs []int) error {
func (v *selectBlock) renderBaseSelect(w io.Writer, schema *DBSchema, childCols []*qcode.Column, childIDs []int) error {
var groupBy []int
isNotRoot := v.parent != nil
isRoot := v.parent == nil
isFil := v.sel.Where != nil
isAgg := false
@ -305,14 +305,20 @@ func (v *selectBlock) renderBaseSelect(w io.Writer, schema *DBSchema, childCols
fmt.Fprintf(w, ` FROM "%s"`, v.sel.Table)
if isNotRoot {
v.renderJoinTable(w, schema, childIDs)
if isRoot && isFil {
io.WriteString(w, ` WHERE (`)
if err := v.renderWhere(w); err != nil {
return err
}
io.WriteString(w, `)`)
}
switch {
case isNotRoot:
if !isRoot {
v.renderJoinTable(w, schema, childIDs)
io.WriteString(w, ` WHERE (`)
v.renderRelationship(w, schema)
if isFil {
io.WriteString(w, ` AND `)
if err := v.renderWhere(w); err != nil {
@ -320,16 +326,10 @@ func (v *selectBlock) renderBaseSelect(w io.Writer, schema *DBSchema, childCols
}
}
io.WriteString(w, `)`)
case isFil && !isAgg:
io.WriteString(w, ` WHERE (`)
if err := v.renderWhere(w); err != nil {
return err
}
io.WriteString(w, `)`)
}
if isAgg && len(groupBy) != 0 {
if isAgg {
if len(groupBy) != 0 {
fmt.Fprintf(w, ` GROUP BY `)
for i, id := range groupBy {
@ -339,13 +339,6 @@ func (v *selectBlock) renderBaseSelect(w io.Writer, schema *DBSchema, childCols
io.WriteString(w, ", ")
}
}
if isFil {
io.WriteString(w, ` HAVING (`)
if err := v.renderWhere(w); err != nil {
return err
}
io.WriteString(w, `)`)
}
}
@ -403,6 +396,16 @@ func (v *selectBlock) renderRelationship(w io.Writer, schema *DBSchema) {
}
func (v *selectBlock) renderWhere(w io.Writer) error {
if v.sel.Where.Op == qcode.OpEqID {
col, ok := v.schema.PCols[v.sel.Table]
if !ok {
return fmt.Errorf("no primary key defined for %s", v.sel.Table)
}
fmt.Fprintf(w, `(("%s") = ('%s'))`, col.Name, v.sel.Where.Val)
return nil
}
st := util.NewStack()
if v.sel.Where != nil {

View File

@ -317,6 +317,26 @@ func manyToManyReverse(t *testing.T) {
}
}
func fetchByID(t *testing.T) {
gql := `query {
product(id: 4) {
id
name
}
}`
sql := `SELECT json_object_agg('product', products) FROM (SELECT row_to_json((SELECT "sel_0" FROM (SELECT "products_0"."id" AS "id", "products_0"."name" AS "name") AS "sel_0")) AS "products" FROM (SELECT "products"."id", "products"."name" FROM "products" WHERE ((("id") = ('4'))) LIMIT ('1') :: integer) AS "products_0" LIMIT ('1') :: integer) AS "done_1337";`
resSQL, err := compileGQLToPSQL(gql)
if err != nil {
t.Fatal(err)
}
if resSQL != sql {
t.Fatal(errNotExpected)
}
}
func aggFunction(t *testing.T) {
gql := `query {
products {
@ -345,7 +365,7 @@ func aggFunctionWithFilter(t *testing.T) {
}
}`
sql := `SELECT json_object_agg('products', products) FROM (SELECT coalesce(json_agg("products"), '[]') AS "products" FROM (SELECT row_to_json((SELECT "sel_0" FROM (SELECT "products_0"."id" AS "id", "products_0"."max_price" AS "max_price") AS "sel_0")) AS "products" FROM (SELECT "products"."id", max("products"."price") AS max_price FROM "products" GROUP BY "products"."id" HAVING ((("products"."id") > (10))) LIMIT ('20') :: integer) AS "products_0" LIMIT ('20') :: integer) AS "products_0") AS "done_1337";`
sql := `SELECT json_object_agg('products', products) FROM (SELECT coalesce(json_agg("products"), '[]') AS "products" FROM (SELECT row_to_json((SELECT "sel_0" FROM (SELECT "products_0"."id" AS "id", "products_0"."max_price" AS "max_price") AS "sel_0")) AS "products" FROM (SELECT "products"."id", max("products"."price") AS max_price FROM "products" WHERE ((("products"."id") > (10))) GROUP BY "products"."id" LIMIT ('20') :: integer) AS "products_0" LIMIT ('20') :: integer) AS "products_0") AS "done_1337";`
resSQL, err := compileGQLToPSQL(gql)
if err != nil {
@ -362,6 +382,7 @@ func TestCompileGQL(t *testing.T) {
t.Run("withWhereAndList", withWhereAndList)
t.Run("withWhereIsNull", withWhereIsNull)
t.Run("withWhereMultiOr", withWhereMultiOr)
t.Run("fetchByID", fetchByID)
t.Run("belongsTo", belongsTo)
t.Run("oneToMany", oneToMany)
t.Run("manyToMany", manyToMany)

View File

@ -118,8 +118,17 @@ func updateSchema(schema *DBSchema, t *DBTable, cols []*DBColumn) {
// Below one-to-many relations use the current table as the
// join table aka through table.
if len(jcols) > 1 {
col1, col2 := jcols[0], jcols[1]
for i := range jcols {
for n := range jcols {
if n != i {
updateSchemaOTMT(schema, ct, jcols[i], jcols[n])
}
}
}
}
}
func updateSchemaOTMT(schema *DBSchema, ct string, col1, col2 *DBColumn) {
t1 := strings.ToLower(col1.FKeyTable)
t2 := strings.ToLower(col2.FKeyTable)
@ -143,7 +152,6 @@ func updateSchema(schema *DBSchema, t *DBTable, cols []*DBColumn) {
//rel2 := &DBRel{RelOneToManyThrough, ct, col2.Name, fc2.Name}
rel2 := &DBRel{RelOneToManyThrough, ct, col1.Name, fc1.Name, col2.Name}
schema.RelMap[TTKey{t2, t1}] = rel2
}
}
type DBTable struct {

View File

@ -24,7 +24,7 @@ type Column struct {
}
type Select struct {
ID int32
ID int16
AsList bool
Table string
Singular string
@ -84,6 +84,7 @@ const (
OpHasKeyAny
OpHasKeyAll
OpIsNull
OpEqID
)
func (t ExpOp) String() string {
@ -136,6 +137,8 @@ func (t ExpOp) String() string {
v = "op-has-key-all"
case OpIsNull:
v = "op-is-null"
case OpEqID:
v = "op-eq-id"
}
return fmt.Sprintf("<%s>", v)
}
@ -223,7 +226,7 @@ func (com *Compiler) compileQuery(op *Operation) (*Query, error) {
var selRoot *Select
st := util.NewStack()
id := int32(0)
id := int16(0)
fmap := make(map[*Field]*Select)
for i := range op.Fields {
@ -331,7 +334,13 @@ func (com *Compiler) compileArgs(sel *Select, args []*Arg) error {
if args[i] == nil {
return fmt.Errorf("[Args] unexpected nil argument found")
}
switch strings.ToLower(args[i].Name) {
an := strings.ToLower(args[i].Name)
switch an {
case "id":
if sel.ID == int16(0) {
err = com.compileArgID(sel, args[i])
}
case "where":
err = com.compileArgWhere(sel, args[i])
case "orderby", "order_by", "order":
@ -343,9 +352,13 @@ func (com *Compiler) compileArgs(sel *Select, args []*Arg) error {
case "offset":
err = com.compileArgOffset(sel, args[i])
}
if err != nil {
return err
}
}
return err
return nil
}
type expT struct {
@ -403,9 +416,35 @@ func (com *Compiler) compileArgNode(val *Node) (*Exp, error) {
return root, nil
}
func (com *Compiler) compileArgID(sel *Select, arg *Arg) error {
if sel.Where != nil && sel.Where.Op == OpEqID {
return nil
}
ex := &Exp{Op: OpEqID, Val: arg.Val.Val}
switch arg.Val.Type {
case nodeStr:
ex.Type = ValStr
case nodeInt:
ex.Type = ValInt
case nodeFloat:
ex.Type = ValFloat
default:
fmt.Errorf("expecting an string, int or float")
}
sel.Where = ex
return nil
}
func (com *Compiler) compileArgWhere(sel *Select, arg *Arg) error {
var err error
if sel.Where != nil {
return nil
}
sel.Where, err = com.compileArgObj(arg)
if err != nil {
return err

View File

@ -37,11 +37,11 @@ type gqlReq struct {
type gqlResp struct {
Error string `json:"error,omitempty"`
Data json.RawMessage `json:"data,omitempty"`
Extensions extensions `json:"extensions"`
Extensions *extensions `json:"extensions,omitempty"`
}
type extensions struct {
Tracing *trace `json:"tracing"`
Tracing *trace `json:"tracing,omitempty"`
}
type trace struct {
@ -141,7 +141,7 @@ func apiv1Http(w http.ResponseWriter, r *http.Request) {
resp := gqlResp{}
if tracing {
resp.Extensions = extensions{newTrace(st, et, qc)}
resp.Extensions = &extensions{newTrace(st, et, qc)}
}
resp.Data = json.RawMessage(root)

View File

@ -1,68 +0,0 @@
This project was bootstrapped with [Create React App](https://github.com/facebook/create-react-app).
## Available Scripts
In the project directory, you can run:
### `npm start`
Runs the app in the development mode.<br>
Open [http://localhost:3000](http://localhost:3000) to view it in the browser.
The page will reload if you make edits.<br>
You will also see any lint errors in the console.
### `npm test`
Launches the test runner in the interactive watch mode.<br>
See the section about [running tests](https://facebook.github.io/create-react-app/docs/running-tests) for more information.
### `npm run build`
Builds the app for production to the `build` folder.<br>
It correctly bundles React in production mode and optimizes the build for the best performance.
The build is minified and the filenames include the hashes.<br>
Your app is ready to be deployed!
See the section about [deployment](https://facebook.github.io/create-react-app/docs/deployment) for more information.
### `npm run eject`
**Note: this is a one-way operation. Once you `eject`, you cant go back!**
If you arent satisfied with the build tool and configuration choices, you can `eject` at any time. This command will remove the single build dependency from your project.
Instead, it will copy all the configuration files and the transitive dependencies (Webpack, Babel, ESLint, etc) right into your project so you have full control over them. All of the commands except `eject` will still work, but they will point to the copied scripts so you can tweak them. At this point youre on your own.
You dont have to ever use `eject`. The curated feature set is suitable for small and middle deployments, and you shouldnt feel obligated to use this feature. However we understand that this tool wouldnt be useful if you couldnt customize it when you are ready for it.
## Learn More
You can learn more in the [Create React App documentation](https://facebook.github.io/create-react-app/docs/getting-started).
To learn React, check out the [React documentation](https://reactjs.org/).
### Code Splitting
This section has moved here: https://facebook.github.io/create-react-app/docs/code-splitting
### Analyzing the Bundle Size
This section has moved here: https://facebook.github.io/create-react-app/docs/analyzing-the-bundle-size
### Making a Progressive Web App
This section has moved here: https://facebook.github.io/create-react-app/docs/making-a-progressive-web-app
### Advanced Configuration
This section has moved here: https://facebook.github.io/create-react-app/docs/advanced-configuration
### Deployment
This section has moved here: https://facebook.github.io/create-react-app/docs/deployment
### `npm run build` fails to minify
This section has moved here: https://facebook.github.io/create-react-app/docs/troubleshooting#npm-run-build-fails-to-minify

Binary file not shown.

Before

Width:  |  Height:  |  Size: 911 KiB