Introducing LAMS, a LookML style guide and linter

[fabio1]

The Problem

A frequently asked question by those that develop in Looker is around how to write code that is reusable, self-explanatory and reliable. This is often because they have been working on the same model for a significant period of time, very often with multiple contributors, and it has become bloated and very hard to maintain and build upon.

The Why

Look At Me Sideways (LAMS) seeks to address this problem by:

1. Helping developer teams adhere to best practices
2. Aiding anyone interested in cleaning and refactoring their LookML model
3. Ensuring consistency and a certain level of code quality
4. Automating the pull request review process for those that enforce them
5. Maintaining a clean master branch at all times

The How

LAMS is a linter and a style guide that picks up and flags anti-patterns or anything that deviates from recommended LookML constructs. It also explains exactly what’s wrong as well as provides common best-practice solutions and/or alternatives. The style guide itself is intended to evolve over time as new conventions are identified, suggested and/or potentially contributed by the community it supports.

In order to get the most of out it, it is recommended to run it in your CI environment. When set up like this, the workflow involves four steps. It all starts when a developer commits new code from the Looker IDE (Step 1). GitHub then sends off a webhook to the server hosting LAMS asking it to check the new code (Step 2). Once LAMS does this, it adds the result files to the trigger branch (Step 3) and the user will be prompted to pull the files from his remote branch (Step 4). The entire process takes a few seconds at most and is entirely seamless, providing everything that the developer needs from within the platform.

See a video of LAMS in action:

https://vimeo.com/313492643/30c4e0175c

The Setup

LAMS can be installed either as a standalone npm package using these instructions or deployed in your CI environment. The latter is highly recommended but since it can be quite an involved process, we are also shipping it with a dockerized version that contains a fully functional Jenkins server with a pre-configured job. This will allow you to get setup in just a few minutes and hit the ground running. If you want to contribute to the project, check it out on github.

LAMS is developed and maintained by Fabio Beltramini (aka @fabio1 ) and Joseph Axisa (aka @jax1 )

[anonymous]

Hi @fabio1 ,

Please help me with the syntax to implement multiple exceptions in manifest file.

I have tried two different syntaxes, but I get parsing errors for them as mentioned below.

Syntax 1:

LAMS

rule_exemptions: {

F2: “Rule not Required”

}

LAMS

rule_exemptions: {

F3: “Rule not Required”

}

Error:

{
error: {
toString: [Function: toString],
message: ‘Expected “#”, “:”, or [ \t\n\r] but “r” found.’,
expected: [Array],
found: ‘r’,
location: [Object],
name: ‘SyntaxError’,
context: ‘3:\t F2: “Rule not Required”\n’ +
‘4:\t }\n’ +
‘5:\t LAMS\n’ +
‘6:\t rule_exemptions: {\n’ +
‘7:\t F3: “Rule not Required”\n’ +
‘8:\t }’
},
_file_path: ‘manifest.lkml’,
_file_rel: ‘’,
_file_name: ‘’,
_file_type: ‘manifest’
},

Syntax 2:

LAMS

rule_exemptions: {

F2, F3 “Rule not Required”

}

Error:

{
error: {
toString: [Function: toString],
message: ‘Expected “#”, “:”, [ \t\n\r], or [\-+_a-zA-Z0-9.] but “,” found.’,
expected: [Array],
found: ‘,’,
location: [Object],
name: ‘SyntaxError’,
context: ‘’
},
_file_path: ‘manifest.lkml’,
_file_rel: ‘’,
_file_name: ‘’,
_file_type: ‘manifest’
},

Please let me know the right way to implement multiple exceptions in a manifest file

[fabio1]

Hi @lokhandepriyank , thanks for the question

Please help me with the syntax to implement multiple exceptions in manifest file.

# LAMS
# rule_exemptions: {
#  F2: "Reason for F2"
#  F3: "Reason for F3"
# }

[anonymous]

Thanks @fabio1 , That helped me

[anonymous]

Hi @fabio1 ,

Is there a way while implementing LAMS, we can exclude a particular folder from Looker project that won’t be checked for LAMS rules.

[dylangbaker]

https://community.looker.com/lookml-5/introducing-the-style-validator-a-new-lookml-linter-32134

Just wanted to drop a note that the team at Spectacles have released their new linter, which might be of interest to people visiting this thread.

[anique_trip-163]

Hey @fabio1 this is super helpful! We are trying to implement this with some custom rules. I was using your example

($boolean ($match
    "^Is |^Has "
    ::match:label
))

and trying something similar, namely to setup a rule that checks whether booleans (dimensions of type: yesno) start with “is” or “has” in the name. I used the following syntax, but it doesn’t seem to recognize boolean dimensions, the output shows errors of all dimensions of all types, but I only want to apply it on dimensions that have type: yesno (so booleans). So not sure what $boolean does in the example syntax.

LAMS

rule: boolean_name {

description: “Boolean dimensions must start with is or has”

match: “$.model..view.[dimension].*”

expr_rule: ($boolean ($match "^is |^has "::match:dimension)) ;;

}

Could you help me with the correct syntax for this ? Thanks a lot!

[fabio1]

Hi @anique_trip-163

To start, I’ll clarify the existing example. The expressions translate to JavaScript for execution and use its underlying type system. $match corresponds to the String.prototype.match method, which can return an array for a successful match or null for a failed match. The purpose here of the $boolean function is to coerce these values into true/false, which are acceptable results for a LAMS custom rule expression. (A few other types are acceptable, including strings for custom error messages)

Here are three different ways you could apply this expression to yesno dimensions:

#rule: F_yn1 {

description: “Dimensons of type:yesno must be labeled starting with Is or Has”

match: “$.model..view..dimension[?(@.type===‘yesno’)]”

expr_rule: ($boolean ($match "^is |^has "::match:dimension)) ;;

#}

#rule: F_yn2 {

description: “Dimensions must either be labeled starting with Is or Has, or not be type:yesno”

match: “$.model..view..dimension.*”

expr_rule: ($any

(!== ::match:type “yesno”)

($boolean ($match "^is |^has "::match:dimension))

);;

#}

#rule: F_yn3 {

description: “Dimensions must be labeled starting with Is or Has if they are type:yesno”

match: “$.model..view..dimension.*”

expr_rule: ($if (=== ::match:type “yesno”)

($boolean ($match "^is |^has "::match:dimension))

true

);;

#}

And the corresponding output:

The first approach is nice in that is actually restricts the number of matches, which can make the summary statistics seem more representative. However, the property-based selection can be limiting, so sometimes you have to fall back to a more general approach of matching all fields and applying an $if statement inside the rule expression.

PS. In case you missed it, one convenient way to test things out is the Rule Sandbox

[anique_trip-163]

Hey @fabio1 thanks a lot for your reply, super helpful! I understand the purpose of the $boolean now. I tried to use the example but strange thing is it doesn’t seem to work if I use the dimension parameter. Let’s say I have this example dimension in a view:

dimension: is_deleted {
label: “Is Deleted”
type: yesno
sql: ${TABLE}.is_deleted ;;
}

Then I used this rule syntax (as you shared):

LAMS

rule: F_yn1 {

description: “Dimensons of type:yesno must be labeled starting with Is or Has”

match: “$.model..view..dimension[?(@.type===‘yesno’)]”

expr_rule: ($boolean ($match "^is |^has "::match:dimension)) ;;

#}

It provides this output, however the dimension is called dimension: is_deleted, so that should not give an error right..

If I try it on a different parameter, for example on label, in one example view like this:

LAMS

rule: F_yn {

description: “Dimensons of type:yesno must have a label starting with Is or Has”

match: “$.model.core.view.*.dimension[?(@.type===‘yesno’)]”

expr_rule: ($boolean ($match "^Is |^Has " ::match:label)) ;;

}

It provides the correct output

I’m not sure if I’m doing something wrong or how it does work for the label parameter but not for the dimension parameter, would you have any idea? Thanks in advance

It

[fabio1]

Hey @anique_trip-163 , sorry for the confusion, when I wrote the examples, I just tested them partially.. they will always error for type yesno dimensions, because I left the reference to ::match:dimension unchanged. This would only make sense if there were a property named dimension under the matched dimension.

You have figured it out well with the label property, and often that is what people want because Is/Has dimension labels should end with a “?”, so an explicit label is required.

However, if you wanted to accept an is_/has_ field name, with no label, you can use the $name property that is added to the dimension, referred to in the expression by ::match:$name

You could accept either one like this… I didn’t test this, but hopefully I wrote it correctly

LAMS

rule: F_yn {

description: “Dimensions of type:yesno must have a label starting with Is or Has”

match: “$.model.core.view.*.dimension[?(@.type===‘yesno’)]”

expr_rule: ($any

($boolean ($match "^Is |^Has " ::match:label))

($boolean ($match “^is_|^has_” ::match:$name))

);;

}

Or maybe you want the field to always be named this way, not just labeled. It’s up to you

[anique_trip-163]

Hey @fabio1 ah great this worked! I needed the $name property for the actual name of the dimension or measure. I tested it and it works now, so thanks a lot for the explanation!

[anique_trip-163]

Hey @fabio1 nice stuff! We are trying to set this up in our Gitlab CI. I’m using the below .gitlab-ci.yml file and added this in our Looker project. It seems like this checks the rules on all the existing LookML files in the project, however we only want to apply it to files that are being changed in the commit/branch, and not on all existing/previously built LookML code. Would you know how we could set this up so that it only checks for actual changed files in the commit/branch? That would be super helpful!

image: alpine:edge

before_script:

• apk --no-cache --repository http://dl-cdn.alpinelinux.org/alpine/edge/testing/ upgrade
• apk --no-cache --repository http://dl-cdn.alpinelinux.org/alpine/edge/testing/ add nodejs npm
• npm i -g @looker/look-at-me-sideways@3 --unsafe-perm

stages:

• run-lams

lams:
stage: run-lams
when: always
allow_failure: true
script:

• lams --reporting=no --output=lines,markdown --on-parser-error=info

after_script:

- git add . && git commit -m’update issues.md’

artifacts:
expire_in: 2 weeks
when: always
paths:

• issues.md

[fabio1]

Hi @anique_trip-163

First, a bit of background about our Gitlab CI example configuration. It was contributed by members of the community while LAMS was in v1. Between v1 → v2, LAMS’ default output mode changed. It seems like you are already adapting the commands a bit, so hopefully it works for you. If so, perhaps consider sending along a PR to update our Gitlab CI example doc, as I am not familiar enough with the conventions and best practices around the artifacts concept used by Gitlab CI.

• Our current (v1) Gitlab CI example config
• V2 release notes
• Sample of change to our Github Actions example

As for your question about linting changed files… a lot of LAMS users have ended up doing something like this over the years, using git to identify changed files, and passing this to LAMS’ source argument. However, I always found the approach of diff-then-lint to be lacking. For one, file-level granularity is very coarse, but more importantly it fails to handle things like extensions/refinements/includes which may compose LookML across multiple files, some of which may be unmodified. That’s why in v3.1-,v3.1,-This%20update%20introduces), released last week, I introduced incremental linting as a native feature (in beta). Take a look at that, and let me know whether it works for your needs/use case. The feature is still open to changes based on community feedback!

[anique_trip-163]

Hey @fabio1 thanks for your reply! What do you mean with incremental linting was released in v3.1, how would incremental linting work exactly? Would that work similar to changed files? And when could this feature be live in production to start using? Curious to hear more! Thanks, Anique

[fabio1]

Hi Anique! The idea behind this feature is that if at any point (for example, once manually, or upon merging a set of changes), you wish to exempt all current errors in future runs, you can run LAMS with --output=add-exemptions and check in the resulting/updated lams-exemptions.ndjson file to your repo. On subsequent runs, all those errors would be exempt, and so the reported errors would just be the new/incremental ones.

I would love to chat with you to understand if this works for your needs, or to better understand how you would configure this in Gitlab CI, so if you’re interested in setting up a call net week, shoot me an email at (PII Removed by Staff)

[borysarkhypenko]

Hi Fabio,

thank you for implementing this!

could you tell me please:

• where do you store lams-exemptions.ndjson, in workflows folder or in the root directory of Looker?
• What could be the reason that no additions happen to the lams-exemptions.ndjson after running LAMS? here is my code

name: LAMS Looker Linter Push validation

on: [push]

jobs:
lams_job:
runs-on: ubuntu-latest
name: LAMS LookML Linter Job
steps:
- name: Checkout your LookML
uses: actions/checkout@v1
- name: Setup Node
uses: actions/setup-node@v1
with:
node-version: '16.x'
- name: Install LAMS
run: npm install -g <li-user uid="413118" login="Looker"></li-user>/look-at-me-sideways@3
- name: Run LAMS
# See [PRIVACY.md](https://github.com/looker-open-source/look-at-me-sideways/blob/master/PRIVACY.md)
run: lams --reporting=no -output=add-exemptions

I do get the errors after execution but they arent getting appended to the ndjson file.

Many thanks in advance!

[fabio1]

Hi @borysarkhypenko - I’m glad you’re finding value in LAMS. I’m not 100% sure of the cause, but for starters, I notice that you only have a single hyphen to start this argument, whereas the convention is two hyphens.

-output=add-exemptions

If you are still having trouble, feel free to open an issue on the Github repo’s issue tracker, as I follow that as well, and it can be a bit more organized to keep things in one issue thread.

[fabio1]

Also, lams-exemptions.nsdjon is stored/found in the working directory where you executed LAMS. This is by default the same as the LookML project root, though it may be different if you invoke LAMS to intentionally change that.

https://github.com/looker-open-source/look-at-me-sideways/blob/master/lib/loaders/lams-exemptions.js#L21