Sense is cool.    
So is DevOps.

Meet SenseOps


Thoughts on best practices for enterprise grade use of Qlik Sense

SenseOps.rocks is a sister site of Ptarmigan Labs. Neither are affiliated with or endorsed by Qlik in any way. Views expressed by me are my own.

DevOps

"DevOps is a term used to refer to a set of practices that emphasize the collaboration and communication of both software developers and information technology (IT) professionals while automating the process of software delivery and infrastructure changes.
It aims at establishing a culture and environment where building, testing, and releasing software can happen rapidly, frequently, and more reliably."

- Wikipedia 

By Kharnagy (Own work), via Wikimedia Commons

DevOps is an established way of working in the IT industry at large, but how do we apply it to Sense? Specifically, what tools do we use?

Some of the concepts mentioned on this site are common sense (no pun intended) also in Sense development. But other concepts, that are widely adopted in most other development environments, are entirely missing in the Sense ecosystem.

To put it differently: So far much Sense development has been done in rather ad-hoc ways, not using best practices and learnings from the wider software development community.
I argue that we can and should change this - let us be the professionals we after all are.

This site discuss some of the available options. Some are battle proven and some on the to-try-out list. 
Feel free to join the discussion, sharing your experiences!

Plan

“Plans are nothing; planning is everything.” - Dwight D. Eisenhower

No matter if you use agile concepts or six month plans - thinking through a coming project is essential to success. My experience is that Sense projects tend to include lots of small(ish), distinct tasks that need to be tracked. Maybe as parts of larger deliveries or features. 
Both Jira and Trello work well for this - with the latter having a reasonably generous free tier if you want to try it out.
 
There are of course also many other options for planning - the key thing is that you track what needs to be done.

Not much new here - this is standard best practices in most software development projects.

Create

"Creativity takes courage.” - Henri Matisse

This is the programming phase.
The most obvious choice here is to use Sense's built-in, web based edtor. If you have developed software on other platforms, you will however quickly realise that there is no revision control, it takes time opening the web editor, it is slow, has rather primitive search and replace etc.
The web based editor is very useful for learnng Sense and prototyping new ideas, once you know what needs to be done you are however better off using a real text editor.
 
Both Visual Studio Code and Sublime have reasonably good syntax highlightning for Sense load scripts (in the form of .qvs files). Especially VS Code is very, very nice. It works the same across all major OSs, is intuitive to learn, lightning quick and very extensible. Highly recommended!
Atom is a cross-platform editor from GitHub - quite similar to VS Code in some aspects, and some people prefer Atom to VS Code. No Sense syntax highlighting though.

Finally, if using a separate text editor for creating .qvs files, those files need to be stored somewhere. The good news is that all major text editors have good integrations with GitHub  many also work with GitHub Enterprise and GitLab  Storing Sense load scripts in git is nothing short of transformative. It allows for experimenting on different branches, developers can collaborate on load scripts, you can roll back to earlier releases if needed. 
All things that simply are must-haves in any serious systems development culture.

Verify & test

“Testing leads to failure, and failure leads to understanding” - Burt Rutan

By now we have some code/load script. We need to verify that it works as intended.
Once again, Qlik’s standard web editor is the natural choice for this. If you have used some other editor, you will need to copy the script code over to the web editor, then do a reload, and afterwards verify that you get the desired results. 

Given the tight coupling between the load script editor, UI designer and data model viewer (all browser based), this is the environment where most testing will be done. 
That said, I think there are very interesting opportunities to explore by checking in all script code into git, then let Jenkins take over.

Jenkins is an automation server. Just as it sounds, it automates otherwise manual tasks.
That is a gross simplification though, as Jenkins can be configured to automate almost anything. In a testing context, I would like to explore things such as 

- Verify that mandatory code segments are really included in the scripts checked into git.
- Automatically reload the script by injecting it into a temporary test app, and monitor the results from the reload.   
- Notify developers and other stakeholders via email, Slack etc when tests fail and/or pass. 
- Adapt existing testing frameworks to work with Sense. Unit testing is a very powerful concept, but so far largely missing from Sense development. 
- Use web testing tools like Selenium to test the user interface of Sense apps.

The automatic tests above could be triggered when code is checked into certain branches in git. That way we could even get something quite close to continuous testing for Sense development - very cool. 

Package

“No matter where you go or why - pack light” - Unknown

Packing might not seem like an important part of a Sense development workflow. 
Still, if implemented well it can provide significant value. 

Consider the time when a Sense app is ready for deployment. The user interface is done and you check in the final load script to the master branch in a git repository.
This triggers Jenkins, who copies the final load script from git to a folder hierarchy on the Sense server where .qvs files are stored.

The Qlik Deployment Framework (QDF) is an interesting entry in this field. It uses a container concept to compartmentalize code and resources. From my (admittedly limited) experience of QDF, it seems that it requires some up-front investment to get started, but the rewards are very clear once the containers are reused between multiple Sense apps.
QDF is actually kind of hard to categorise, it really spans multople phases of SenseOps. 
 
Jenkins can also call Butler to get JSON formatted metadata (incl user interface definitions) for the app, combines this with the load script from git, package all the files in a ZIP file and copies this file to an archive location.

(Note: Butler was created by myself. The project is open source and available on GitHub)

Release

The tools for release management will largely be the same as for packaging. The load script and possibly also the complete app metadata (incl user interface) is stored in GitHub, GitHub Enterprise, GitLab or similar. 

Jenkins then copies relevant files to the Sense server, where the files will be used during the next reload. 
Using Butler, it is trivial for Jenkins to also start reloads of the just deployed apps. 

Having a standard way of doing release notes has proven very useful. One way of doing this is to have a standardised sheet called “Release notes” in all Sense apps. This sheet can get its data from a Google sheet using Sense’s excellent "Web Connectors”. The release note would contain information on what features/bugs were added/fixed in what app version, together with links to the associated Jira or Trello tickets. 
This way all users of an app have immediate access to information on what has changed in apps, from within the app itself.

Configure

“You cannot always control what goes on outside. But you can always control what goes on inside.” - Wayne Dyer

Many QlikView and Sense apps tend to use static configuration tables defined in the load scripts. While this works and is perfectly fine in many cases, it has some serious drawbacks when it comes to flexibility. 
For example, if someone else, other than the Sense developer has the domain knowledge needed to create a mapping table of some kind, that person should be the one maintaining the mapping table.
The mapping table can be placed in an Excel or text file on a shared directory, true. But then there is no or very little control over what changes are made to the files. Rolling back to previous versions is hard.
A much better solution is to store configuration files in GitHubGitLab or other revision control software. By doing so you suddenly get benefits such as peer review for all changes,  easy rollback to previous versions, strong access control (not for public GitHub respositories, obviously..). Storing configuration data in the same place as load scripts also makes very good sense.

Another option is to store configuration data in Google Sheets  Very easy to work with using a web browser, strong access control and possible to see who made what changes. You can even collaborate on config files in real time, watching each others’ changes as they happen!

Monitor

“Who monitors the Monitor?” - Halo

This is, in my personal view, the most underdeveloped area of Qlik Sense right now, at least as Sense works out of the box.
Or rather, it is possible to create good monitoring tools for Sense - most of the pieces are actually there in the form of various very good Sense APIs. It takes some work, but the results are very rewarding.

As for tools, you are faced with a whole palette of options. 

Instant messaging tools such as Slack are simply awesome for sending status messages from Sense load scripts.
Want to notify a stakeholder that an app has reloaded with fresh data? Just ping them at the end of the load script, including a direct link to the app.
A data source does not contain the expected data? Send a message to the app’s developer. 
Slack has a free tier that allows you to fully test this concept 

Sense uses the log4net logging framework. This framework has a feature where you can add your own hooks and get notified when certain log events happen. 
The Butler microservice was to a large degree created to tap into this source of events, and make them available through more structured and easy to consume channels.
Butler gets notifications when users start new sessions and connections, and when users end their sessions and connections. Butler also monitors failing task reloads. 

Butler forwards all these events as MQTT messages. MQTT is a publish-subscribe protocol, meaning that any application can subscribe to exactly the MQTT topics that are of interest. This in turn results in a very clear de-coupling of responsibilities. Sense do the data reloads. Butler forwards the messages, and other systems can act as needed on the results and messages coming from the data reloads.

A fun, but surprisingly useful use of the MQTT messages is to feed them into small wifi equipped gadgets and displays that show various health aspects of the Sense environment. As an example, a pacman night lamp with colour changing lights in it was converted to blink green when a user logs into Sense, and red when a user leaves Sense  Failing reload tasks could easily be included too, maybe with some animated light effects..

Selenium is a tool intended for testing of web apps. It can however also be used to continuously test websites, making sure that they are accessible to the intended users. In the context of Sense apps, Selenium will imitate an end user accessing one or more apps. If an app does not load within some well defined timeout, an alert can be sent to the Sense administrator(s). This concept effectively handles the scenario where sysadmins think all is well, but Sense for some reason is still not available to end users.

Back to MQTT though. Another easy, but powerful possibility enabled by MQTT is to use tools like Node-RED to very easily build dashboards that show Sense stats. This real-time chart showing active users was for example created using Node-RED.

While Node-RED offers very good connectivity to all kinds of systems and tools, it is by design more of a prototyping tool. I have used it for monitoring tasks 24/7 for months without issues, but I would also like to try out better, more enterprise grade options.

Enter InfluxDB and Grafana.
Both can be run as Docker containers, minimising the impact on existing systems. 
Feeding the user logon/logoff and task failure events into InfluxDB, they can then be visualised using Grafana. Grafana is IMHO one of the very best options for visualisation of real-time data - its visuals are very, very slick.. As much as I’d like to use Sense for this, real-time data is not a strongpoint of Sense.

The good news here is that there is a Butler family member taking care of this: Butler SOS can be set up and configured in less than 15 minutes, then providing real-time monitoring of any Sense Enterprise environemnt. A sample screen shot can be found to the right.

Thoughts or ideas?
Used any concepts or tools that worked well together with Sense?

Feel free to join the discussion below!