Jsonnet
Author: h | 2025-04-24
drone jsonnet. This subcommand converts a jsonnet configuration file to a yaml configuration file. $ drone jsonnet -help NAME: drone jsonnet - generate .drone.yml from jsonnet USAGE: Jsonnet. Jsonnet language support in Zed is provided by the community-maintained Jsonnet extension. Tree-sitter: sourcegraph/tree-sitter-jsonnet Language Server
jsonnet-bundler/jsonnet-bundler: A jsonnet package manager.
What you'll learnUnderstand the benefits of using JsonnetLearn about all Jsonnet featuresStructure you Jsonnet code for flexibilityManage your jsonnet code and its dependencies in your source versioning systemUnderstand how Jsonnet is executedRequirementsProgramming experience is beneficialUnderstand how Jsonnet works, and be productive writing Jsonnet code. Based on real worl experience.Jsonnet is a programming language used to generate JSON and YAML content efficiently and with flexibility. JSON and YAML configuration formats are used pervasilvely. From docker-compose to Hashicorp tools like Vagrant and Terraform. Jsonnet is a tool of interest for anyone needing to generate JSON or YAML configuration. As such it is no surprise to see it is used by some tools helping to manage complex Kubernetes configurations.This training helps you get productive rapidly with the Jsonnet language. It covers from the basics to the advanced techniques:Syntax and data types: The Jsonnet syntax is close to JSON. Actually a JSON document is a valid Jsonnet program! Discover the syntax along with the data types supported by Jsonnet.Understanding the Jsonnet execution: When you know the syntax, it is tume to understand how Jsonnet code is executed. From Jsonnet expressions to JSON manifestationJsonnet functions: Learn to define and use function to avoid repetitions and modularize your codeJsonnet objects: Discover the power of combining objects, and more importantly, understand how it works. Learn about inheritances, and how to augment an object’s existing valueComprehension: Master array and object comprehension, allowing you to transform arrays to other arrays and even objectsParameterised runs: See how you can pass parameters to a Jsonnet programLibraries: Organise your code by authoring and using libraries. Manage your project’s dependencies with the Jsonnet Bundler, the package manager. Get tips on how to manage your Jsonnet with your source code versioning system.Generate errors: Learn how to interrupt the Jsonnet execution earlyWho this course is
GitHub - jsonnet-bundler/jsonnet-bundler: A jsonnet package
Jsonnet best practices we've found: For large templates (>10 parameters), avoid directly overriding internal fields when subclassing. Rather, define explicit constructors for templates using Jsonnet functions, which helps with readability and encourages modularity. Check your Jsonnet configurations into source control. Add pre-commit tests to make sure all checked in templates compile. This avoids inadvertent breakages when common templates are updated. You can also create "unit tests" using Jsonnet assert expressions that assert invariants over template variables. Consider also checking in the Jsonnet materialized JSON / YAML to make changes more visible during code review. Because common Jsonnet templates may be imported by many files, changing one file can affect the output of many. Fortunately, unintended changes can be detected at compile time if the materialized YAML is also included with the Jsonnet changes. Refactor often -- there's no risk of breakage. Since Jsonnet templates compile to concrete JSON / YAML files, it's possible to check the before and after outputs to ensure the refactoring was correct (i.e. zero net materialized diff). Push as much configuration to Jsonnet as possible. It may be tempting to introduce logic into services based on some environment flag (e.g. dev vs prod). We've found this to be an anti-pattern -- such configuration should be determined at template compile time rather than at runtime. This principle of hermetic configuration helps prevent surprises when services are deployed to new environments.For more detailed recommendations, check out our newly posted Jsonnet style guide. We also want to welcome any comments or contributions there from the community.ConclusionIn this blog post, we've shared our experience at Databricks using Jsonnet to simplify infrastructure management. As a unifying templating language that enables composition across a broad range of infrastructure and services, Jsonnet brings us one step closer to a totally declarative infrastructure. We've found Jsonnet easy to use and flexible enough to serve as our primary configuration language, and suggest you try it too!If you found this topic interesting, watch out for future blog posts on the tooling we've built around Jsonnet and Kubernetes. We're also hiring for Cloud platform, infrastructure, and Databricks Serverless.java-jsonnet/java-jsonnet - GitHub
{ or: [ { to: 'pippo@gmail.com' }, { to: 'pippo@hotmail.com' }, ],};// The exported configuration starts here{ version: 'v1alpha3', // Optional author information (used in exports). author: { name: 'Pippo Pluto', email: 'pippo@gmail.com' }, rules: [ { filter: { and: [ { list: 'geeks@newsletter.com' }, { not: me }, // Reference to the local variable 'me' ], }, actions: { archive: true, labels: ['news'], }, }, ],}The Jsonnet configuration file contains mandatory version information, optionalauthor metadata and a list of rules. Rules specify a filter expression and a setof actions that will be applied if the filter matches.Filter operators are prefix of the operands they apply to. In the example above,the filter applies to emails that come from the mail list 'geeks@newsletter.com'AND the recipient is not 'me' (which can be 'pippo@gmail.com' OR'pippo@hotmail.com').We will see all the features of the configuration file in the followingsections.Search operatorsSearch operators are the same as the ones you find in the Gmail filterinterface:from: the mail comes from the given addressto: the mail is delivered to the given addresssubject: the subject contains the given wordshas: the mail contains the given wordsIn addition to those visible in the Gmail interface, you can specify nativelythe following common operators:list: the mail is directed to the given mail listcc: the mail has the given address as CC destinationbcc: the mail has the given address as BCC destinationreplyto: the mail has the given address as Reply-To destinationOne more special function is given if you need to use less common operators1, or want to compose your query manually:query: passes the given contents verbatim to the Gmail filter, withoutescaping or interpreting the contents in any way.Example:{ version: 'v1alpha3', rules: [ { filter: { subject: 'important mail' }, actions: { markImportant: true, }, }, { filter: { query: 'dinner AROUND 5 friday has:spreadsheet', }, actions: { delete: true, }, }, ],}Logic operatorsFilters can contain only one expression. If you want to combine multiple of themin the same rule, you have to use logic operators (and, or, not). Theseoperators do what you expect:and: is true only if all the sub-expressions are also trueor: is true if one or more sub-expressions are truenot: is true if the sub-expression is false.Example:{ version: 'v1alpha3', rules: [ { filter: { or: [ { from: 'foo' }, { and: [ { list: 'bar' }, { not: { to: 'baz' } }, ], }, ], }, actions: { markImportant: true, }, }, ],}This composite filter marks the incoming mail as important if:the message comes from "foo", orit is coming from the mailing list "bar" and not directed to "baz"Reusing filtersFilters can be named and referenced in other filters. This allows reusingconcepts and so avoid repetition. Note that this is not a. drone jsonnet. This subcommand converts a jsonnet configuration file to a yaml configuration file. $ drone jsonnet -help NAME: drone jsonnet - generate .drone.yml from jsonnet USAGE: Jsonnet. Jsonnet language support in Zed is provided by the community-maintained Jsonnet extension. Tree-sitter: sourcegraph/tree-sitter-jsonnet Language Serverjsonnet/python/_jsonnet_test.py at master google/jsonnet
As part of the effort for Databricks Community Edition, our free tier Apache Spark service for education. Since then, Jsonnet has exploded in popularity within Databricks engineering, with over 40,000 lines of Jsonnet in 1000+ distinct files checked into our main development repository. These templates expand to hundreds of thousands of lines of raw YAML once materialized.Teams at Databricks currently use Jsonnet to manage configurations for Kubernetes resources (including internal configurations of services running on Kubernetes), AWS CloudFormation, Terraform, Databricks Jobs, and also for tasks such as TLS cert management and defining Prometheus alerts. Over the past two years, Jsonnet has grown to become the de-facto standard configuration language within engineering.Jsonnet BasicsJsonnet is a configuration language that helps you define JSON data. The basic idea is that some JSON fields may be left as variables or expressions that are evaluated at compilation time. For example, the JSON object {"count": 4} may be expressed as {count: 2 + 2} in Jsonnet. You can also declare hidden fields with "::" that can be referenced during compilation, e.g. {x:: 2, y:: 2, count: $.x + $.y} also evaluates to {"count": 4}.Jsonnet objects may be subclassed by concatenating ("+") objects together to override field values, e.g. suppose we define the following.Then, the Jsonnet expression (base + {x:: 10}) will compile to {"count": 12}. In fact, the Jsonnet compiler requires you to override x since the default value raises an error. You can, therefore, think of base as defining an abstract base class in Jsonnet.Jsonnet compilation is completely deterministic and cannot perform external I/O, making it ideal for defining configuration. We've found that Jsonnet strikes the right balance between restrictiveness and flexibility -- previously we generated configurations using Scala code, which erred too much on the side of flexibility, leading to many headaches.At Databricks, we extended our kubectl tool (dubbed kubecfg) so that it can take Jsonnet files directly as arguments. Internally it compiles the Jsonnet to plain JSON / YAML before sending the data to Kubernetes. Kubernetes then creates or updates objects as needed based on the uploaded configuration.Composing Kubernetes Objects with JsonnetTo better understand how Jsonnet can be used with Kubernetes, let's consider the task of deploying an idealized [1] single-tenant "Databricks platform" for an enterprise customer. Here we want to create two distinct but related service deployments: a Webapp (for the interactive workspace) and Cluster manager (for managing Spark clusters). In addition,google/jsonnet: Jsonnet - The data templating language - GitHub
The services need access to an AWS RDS database to store persistent data.[1]: Note that we don't actually create individual Jsonnet files for each customer in reality -- that would create a frightening number of Jsonnet files and would in itself be a problem.Template for Defining Kubernetes DeploymentsFor these examples, we'll be using service-deployment.jsonnet.TEMPLATE, a simplified version of one of our internal base templates that define a Kubernetes service and deployment together as a pair. The instantiated service and deployment together constitute a standalone "production service" in Kubernetes that can receive network traffic. Note that the template has two required arguments in addition to several optional arguments, including optional config passed to the service binary itself:service-deployment.jsonnet.TEMPLATE:Example 1: One File for Each Service DeploymentGiven service-deployment.jsonnet.TEMPLATE, the simplest option we have is to create separate Jsonnet files for the Webapp and Cluster manager. In each file we must import the base template, subclass it, and fill in the required parameters.We specify the service name, the Docker image containing the service binary, and some service specific configurations including the RDS address. In this example, the RDS address is hard-coded, but later on we'll show how to import that metadata from e.g. the output file of running a CloudFormation script.The following file describes the manager service (here service takes on its standard meaning, we'll refer to Kubernetes services explicitly as such going on). We also construct service-specific configurations within the Jsonnet template via the serviceConf:: field, which the template passes to the pod as an environment variable. We've found it useful to unify service and Kubernetes configs in this way:simple/foocorp-manager.jsonnet:For the webapp service to create clusters, it must specify the cluster manager’s Kubernetes DNS address, which can be determined ahead of time from the Kubernetes service name:simple/foocorp-webapp.jsonnet:If you have the example code repo cloned, you can view the materialized outputs of these templates by running the jsonnet compiler on the files, e.g.:$ jsonnet examples/databricks/simple/foocorp-webapp.jsonnetSo what has this gotten us? We've managed to at least remove some of the standard Kubernetes boilerplate around defining services, reducing each deployment definition from over 100 lines to about 10. However, we can do better -- there are still duplicated parameters which would make this pattern hard to maintain if there were many distinct customers or more services required per customer.Example 2: Composing Together Deployments in a Single FileSince both the Webapp and Cluster manager services are deployed togetherGitHub - rolehippie/jsonnet: Ansible role to install jsonnet
In [ 'manual-label1', 'priority', 'priority/p1', ]],}The resulting list of labels will be:labels: [ // Automatically added { name: 'directed' }, { name: 'lists' }, // Implied parent label { name: 'lists/baz' }, { name: 'lists/foobar' }, { name: 'wow' }, // Manually added { name: 'manual-label1' }, { name: 'priority' }, { name: 'priority/p1' },]Note that there's no need to specify the label lists, because even if it's notused in any filter, it's the parent of a label that is used.Things to keep in mind / gotchas:Removing the last filter referencing a label will delete the label.The only thing managed by the function is the list of labels names. You needto apply some transformations yourself if you want other properties (e.g. thecolor).If you have labels that are not referenced by any filters (maybe archivelabels, or labels applied manually). You have to remember to specify themmanually in the list.Thanks to legeana for the idea!Multiple Gmail accountsIf you need to manage two or more accounts, it's useful to setup bash aliasesthis way:alias gmailctlu1='gmailctl --config=$HOME/.gmailctlu1'alias gmailctlu2='gmailctl --config=$HOME/.gmailctlu2'You will then be able to configure both accounts separately by using one orthe other alias.Known issuesApply filters to existing emailsgmailctl doesn't support this functionality for security reasons. The projectcurrently needs only very basic permissisons, and applying filters to existingemails requires full Gmail access. Bugs in gmailctl or in your configurationwon't screw up your old emails in any way, so this is an important safetyfeature. If you really want to do this, you can manually export your rules withgmailctl export > filters.xml, upload them by using the Gmail Settings UI andselect the "apply new filters to existing email" checkbox.OAuth2 authentication errorsGmail APIs require strict controls, even if you are only accessing your owndata. If you're getting errors similar to:oauth2: cannot fetch token: 400 Bad RequestResponse: { "error": "invalid_grant", "error_description": "Bad Request"}it's likely your auth token expired. Try refreshing it with:$ gmailctl init --refresh-expiredand follow the instructions on screen.If this doesn't help, retry the authorization workflow from the start:$ gmailctl init --reset$ gmailctl initYAML config is unsupportedgmailctl recently deprecated older config versions (v1alpha1, v1alpha2).There's however a migration tool to port those into the latest Jsonnet format.To convert your config: /tmp/gmailctl-config.jsonnet">$ go run github.com/mbrt/gmailctl/cmd/gmailctl-config-migrate \ ~/.gmailct/config.yaml > /tmp/gmailctl-config.jsonnetNote: Adjust your paths if you're not keeping your config file in thedefault directory.Confirm that the new config file doesn't have errors, nor shows diffs with yourremote filters.$ gmailctl diff -f /tmp/gmailctl-config.jsonnetIf everything looks good, replace the old with the new config:$ mv /tmp/gmailctl-config.jsonnet ~/.gmailctl/config.jsonnet$ rm ~/.gmailctl/config.yamlComparison with existing projectsgmail-britta has similarmotivations and is quite popular. The difference between that project andthis one are:gmail-britta uses a custom DSL (versus Jsonnet in gmailctl)gmail-britta is imperative because it allows you to write arbitrary RubycodeGitHub - google/jsonnet: Jsonnet - The data templating language
This blog post is part of our series of internal engineering blogs on Databricks platform, infrastructure management, integration, tooling, monitoring, and provisioning.At Databricks engineering, we are avid fans of Kubernetes. Much of our platform infrastructure runs within Kubernetes, whether in AWS cloud or more regulated environments.However, we have found that Kubernetes alone is not enough for managing a complex service infrastructure, which may encompass both resources created in Kubernetes (e.g., pods, services) and external resources such as IAM roles. Management complexity comes from (1) need for visibility into the current state of the infrastructure (2) reasoning about how to make changes to the infrastructure.To help reduce management complexity, it is desirable for infrastructure to be declarative (i.e., described by a set of configuration files or templates). The first advantage is that one can easily inspect the target state of infrastructure by reading the configuration files. Second, since the infrastructure is entirely described by the files, changes can be proposed, reviewed, and applied as part of a standard software development workflow.What's Missing?Kubernetes YAML configuration files already implement declarative updates to objects in Kubernetes. The user only need edit an object's YAML file and then run $ kubectl apply -f theobject.yaml to sync the changes to Kubernetes. These YAML files may be checked into source control, and if needed the user can query Kubernetes to inspect the difference between the live version and the local file.However, one runs into pain points when attempting to apply this methodology to a larger production environment: Common structures cannot be shared across YAML files. You typically set up a service not once but several times, perhaps in {dev, staging, prod} environments, across different geographic regions such as {us-west, us-east, asia-pacific}, and Cloud providers {AWS, Azure, GCE}. YAML files must often reference metadata about externally defined entities such as relational databases, networking configurations, or SSL certificates. Complex multi-tier service deployments may require the composition of many distinct resources or YAML files. Updating a deployment composed of many such files becomes burdensome. In this blog post, we describe how we use Google's Jsonnet configuration language to solve these problems, walk through an example of templatizing a service deployment using Jsonnet, and propose a Jsonnet style guide for the infrastructure templating use case. We've found that Jsonnet is easy to get started with and scales well to complex use cases.Jsonnet Usage at DatabricksLate 2015 we started experimenting with Jsonnet. drone jsonnet. This subcommand converts a jsonnet configuration file to a yaml configuration file. $ drone jsonnet -help NAME: drone jsonnet - generate .drone.yml from jsonnet USAGE: Jsonnet. Jsonnet language support in Zed is provided by the community-maintained Jsonnet extension. Tree-sitter: sourcegraph/tree-sitter-jsonnet Language Server
jsonnet/doc/learning/tutorial.html at master google/jsonnet
As a unit or shard per customer, it makes sense that there should just be one file per customer describing their unique requirements. It is indeed possible to define a single template, shard.jsonnet.TEMPLATE, that composes together both the Webapp and Cluster manager deployments without duplication of params.The trick here is that the template defines a Kubernetes "List" object, which can include multiple Kubernetes resources within one JSON object. We merge the sub-lists produced by the service-deployment templates using the Jsonnet standard library function std.flattenArrays:shard-v1/shard.jsonnet.TEMPLATE:Now we can conveniently define FooCorp's entire deployment without any duplicated data, and using only one file:shard-v1/foocorp-shard.jsonnet:Example 3: Subclassing Templates for Multiple EnvironmentsTo see the flexibility of Jsonnet, consider how you might subclass the templates defined in the above examples to create shards that run in the development environment. In dev, deployments should use the dev database instead of the production one, and we also want to run with the latest bleeding-edge Docker images. To do this, we can further subclass the shard.jsonnet.TEMPLATE to different environments. This makes it possible to create specific production or development shards from these specialized templates:To make the interface exposed by shard.jsonnet.TEMPLATE more explicit, instead of exporting an object we will export a function newShard(name, release, env), which can be called to construct the shard object. The env object encapsulates the differences between environments (e.g. database URL).shard-v2/shard.jsonnet.TEMPLATE:The dev shard template now just needs to fill in the required env for dev and specify the bleeding-edge release by default. It obtains the env by importing a plain JSON file containing the needed database metadata:shard-v2/dev-shard.jsonnet.TEMPLATE:The full example code for the prod and dev shard templates can be found here: that the use of functions for subclassing is optional -- we could have instead derived dev-shard.jsonnet.TEMPLATE and prod-shard.jsonnet.TEMPLATE using only field overrides. However, in our experience, the use of ad-hoc field overrides, though powerful, tends to lead to more brittle templates. We've found it useful to establish best practices to limit the use of such constructs in larger templates.Jsonnet Style GuideIn the above examples, we saw how Jsonnet templates can be used to remove duplication of configuration data between Kubernetes objects, compose multiple deployments together, and reference external entities. This greatly simplifies infrastructure management. In large projects, however, templates themselves may become a source of complexity. We've found this to be manageable with some best practices and investment in tooling around Jsonnet:Here are a fewjsonnet/examples/conditionals.jsonnet at master google/jsonnet - GitHub
Gmailctl functionalitybut comes directly from the fact that we rely on Jsonnet.Example:local toMe = { or: [ { to: 'myself@gmail.com' }, { to: 'myself@yahoo.com' }, ],};local notToMe = { not: toMe };{ version: 'v1alpha3', rules: [ { filter: { and: [ { from: 'foobar' }, notToMe, ], }, actions: { delete: true, }, }, { filter: toMe, actions: { labels: ['directed'], }, }, ],}In this example, two named filters are defined. The toMe filter gives a nameto emails directed to 'myself@gmail.com' or to 'myself@yahoo.com'. The notToMefilter negates the toMe filter, with a not operator. Similarly, the tworules reference the two named filters above. The name reference is basicallycopying the definition of the filter in place.The example is effectively equivalent to this one:{ version: 'v1alpha3', rules: [ { filter: { and: [ { from: 'foobar' }, { not: { or: [ { to: 'myself@gmail.com' }, { to: 'myself@yahoo.com' }, ], }, }, ], }, actions: { delete: true, }, }, { filter: { or: [ { to: 'myself@gmail.com' }, { to: 'myself@yahoo.com' }, ], }, actions: { labels: ['directed'], }, }, ],}Relying on Jsonnet also allows importing code and raw data from other files3.ActionsEvery rule is a composition of a filter and a set of actions. Those actions willbe applied to all the incoming emails that pass the rule's filter. These actionsare the same as the ones in the Gmail interface:archive: true: the message will skip the inbox;delete: true: the message will go directly to the trash can;markRead: true: the message will be mark as read automatically;star: true: star the message;markSpam: false: do never mark these messages as spam. Note that setting thisfield to true is not supported by Gmail (I don't know why);markImportant: true: always mark the message as important, overriding Gmailheuristics;markImportant: false: do never mark the message as important, overridingGmail heuristics;category: : force the message into a specific category (supportedcategories are "personal", "social", "updates", "forums", "promotions");labels: [list, of, labels]: an array of labels to apply to the message. Notethat these labels have to be already present in your settings (they won't becreated automatically), and you can specify multiple labels (normally Gmailallows only one label per filter).forward: 'forward@to.com': forward the message to another email address. Theforwarding address must be already in your settings (Forwarding and POP/IMAP >Add a forwarding address). Gmail allows no more than 20 forwarding filters.Only one address can be specified for one filter.Example:{ version: 'v1alpha3', rules: [ { filter: { from: 'love@gmail.com' }, actions: { markImportant: true, category: 'personal', labels: ['family', 'P1'], }, }, ],}LabelsYou can optionally manage your labels with gmailctl. The config contains alabels section. Adding labels in there will opt you in to full labelmanagement as well. If you prefer to manage your labels. drone jsonnet. This subcommand converts a jsonnet configuration file to a yaml configuration file. $ drone jsonnet -help NAME: drone jsonnet - generate .drone.yml from jsonnet USAGE: Jsonnet. Jsonnet language support in Zed is provided by the community-maintained Jsonnet extension. Tree-sitter: sourcegraph/tree-sitter-jsonnet Language Servergoogle/vim-jsonnet: Jsonnet filetype plugin for Vim.
Preferred way if you want to startyour filters from scratch.NOTE: It's recommended to backup your current configuration before you applythe generated one for the first time. Your current filters will be wiped andreplaced with the ones specified in the config file. The diff you'll get duringthe first run will probably be pretty big, but from that point on, all changesshould generate a small and simple to review diff.Migrate from another solutionIf you want to preserve your current filters and migrate to a more saneconfiguration gradually, you can try to use the download command. This willlook up at your currently configured filters in Gmail and try to create aconfiguration file matching the current state.NOTE: This functionality is experimental. It's recommended to download thefilters and check that they correspond to the remote ones before making anychanges, to avoid surprises. Also note that the configuration file will be quiteugly, as expressions won't be reconstructed properly, but it should serve as astarting point if you are migrating from other systems.Example of usage: ~/.gmailctl/config.jsonnet# check that the diff is empty and no errors are presentgmailctl diff# happy editing!gmailctl edit"># download the filters to the default configuration filegmailctl download > ~/.gmailctl/config.jsonnet# check that the diff is empty and no errors are presentgmailctl diff# happy editing!gmailctl editOften you'll see imported filters with the isEscaped: true marker. This tellsgmailctl to not escape or quote the expression, as it might contain operatorsthat have to be interpreted as-is by Gmail. This happens when the downloadcommand was unable to map the filter to native gmailctl expressions. It'srecommended to manually port the filter to regular gmailctl operators beforedoing any changes, to avoid unexpected results. Example of such conversion:{ from: "{foo bar baz}", isEscaped: true,}Can be translated into:{ or: [ {from: "foo"}, {from: "bar"}, {from: "baz"}, ],}Other commandsAll the available commands (you can also check with gmailctl help): apply Apply a configuration file to Gmail settings debug Shows an annotated version of the configuration diff Shows a diff between the local configuration and Gmail settings download Download filters from Gmail to a local config file edit Edit the configuration and apply it to Gmail export Export filters into the Gmail XML format help Help about any command init Initialize the Gmail configuration test Execute config testsConfigurationNOTE: Despite the name, the configuration format is stable at v1alpha3.If you are looking for the deprecated versions v1alpha1, or v1alpha2,please refer to docs/v1alpha1.md anddocs/v1alpha2.md.The configuration file is written in Jsonnet, that is a very powerfulconfiguration language, derived from JSON. It adds functionality such ascomments, variables, references, arithmetic and logic operations, functions,conditionals, importing other files, parameterizations and so on. For moredetails on the language, please refer to the officialtutorial.Simple example:// Local variables help reuse config fragmentslocal me =Comments
What you'll learnUnderstand the benefits of using JsonnetLearn about all Jsonnet featuresStructure you Jsonnet code for flexibilityManage your jsonnet code and its dependencies in your source versioning systemUnderstand how Jsonnet is executedRequirementsProgramming experience is beneficialUnderstand how Jsonnet works, and be productive writing Jsonnet code. Based on real worl experience.Jsonnet is a programming language used to generate JSON and YAML content efficiently and with flexibility. JSON and YAML configuration formats are used pervasilvely. From docker-compose to Hashicorp tools like Vagrant and Terraform. Jsonnet is a tool of interest for anyone needing to generate JSON or YAML configuration. As such it is no surprise to see it is used by some tools helping to manage complex Kubernetes configurations.This training helps you get productive rapidly with the Jsonnet language. It covers from the basics to the advanced techniques:Syntax and data types: The Jsonnet syntax is close to JSON. Actually a JSON document is a valid Jsonnet program! Discover the syntax along with the data types supported by Jsonnet.Understanding the Jsonnet execution: When you know the syntax, it is tume to understand how Jsonnet code is executed. From Jsonnet expressions to JSON manifestationJsonnet functions: Learn to define and use function to avoid repetitions and modularize your codeJsonnet objects: Discover the power of combining objects, and more importantly, understand how it works. Learn about inheritances, and how to augment an object’s existing valueComprehension: Master array and object comprehension, allowing you to transform arrays to other arrays and even objectsParameterised runs: See how you can pass parameters to a Jsonnet programLibraries: Organise your code by authoring and using libraries. Manage your project’s dependencies with the Jsonnet Bundler, the package manager. Get tips on how to manage your Jsonnet with your source code versioning system.Generate errors: Learn how to interrupt the Jsonnet execution earlyWho this course is
2025-03-28Jsonnet best practices we've found: For large templates (>10 parameters), avoid directly overriding internal fields when subclassing. Rather, define explicit constructors for templates using Jsonnet functions, which helps with readability and encourages modularity. Check your Jsonnet configurations into source control. Add pre-commit tests to make sure all checked in templates compile. This avoids inadvertent breakages when common templates are updated. You can also create "unit tests" using Jsonnet assert expressions that assert invariants over template variables. Consider also checking in the Jsonnet materialized JSON / YAML to make changes more visible during code review. Because common Jsonnet templates may be imported by many files, changing one file can affect the output of many. Fortunately, unintended changes can be detected at compile time if the materialized YAML is also included with the Jsonnet changes. Refactor often -- there's no risk of breakage. Since Jsonnet templates compile to concrete JSON / YAML files, it's possible to check the before and after outputs to ensure the refactoring was correct (i.e. zero net materialized diff). Push as much configuration to Jsonnet as possible. It may be tempting to introduce logic into services based on some environment flag (e.g. dev vs prod). We've found this to be an anti-pattern -- such configuration should be determined at template compile time rather than at runtime. This principle of hermetic configuration helps prevent surprises when services are deployed to new environments.For more detailed recommendations, check out our newly posted Jsonnet style guide. We also want to welcome any comments or contributions there from the community.ConclusionIn this blog post, we've shared our experience at Databricks using Jsonnet to simplify infrastructure management. As a unifying templating language that enables composition across a broad range of infrastructure and services, Jsonnet brings us one step closer to a totally declarative infrastructure. We've found Jsonnet easy to use and flexible enough to serve as our primary configuration language, and suggest you try it too!If you found this topic interesting, watch out for future blog posts on the tooling we've built around Jsonnet and Kubernetes. We're also hiring for Cloud platform, infrastructure, and Databricks Serverless.
2025-04-24As part of the effort for Databricks Community Edition, our free tier Apache Spark service for education. Since then, Jsonnet has exploded in popularity within Databricks engineering, with over 40,000 lines of Jsonnet in 1000+ distinct files checked into our main development repository. These templates expand to hundreds of thousands of lines of raw YAML once materialized.Teams at Databricks currently use Jsonnet to manage configurations for Kubernetes resources (including internal configurations of services running on Kubernetes), AWS CloudFormation, Terraform, Databricks Jobs, and also for tasks such as TLS cert management and defining Prometheus alerts. Over the past two years, Jsonnet has grown to become the de-facto standard configuration language within engineering.Jsonnet BasicsJsonnet is a configuration language that helps you define JSON data. The basic idea is that some JSON fields may be left as variables or expressions that are evaluated at compilation time. For example, the JSON object {"count": 4} may be expressed as {count: 2 + 2} in Jsonnet. You can also declare hidden fields with "::" that can be referenced during compilation, e.g. {x:: 2, y:: 2, count: $.x + $.y} also evaluates to {"count": 4}.Jsonnet objects may be subclassed by concatenating ("+") objects together to override field values, e.g. suppose we define the following.Then, the Jsonnet expression (base + {x:: 10}) will compile to {"count": 12}. In fact, the Jsonnet compiler requires you to override x since the default value raises an error. You can, therefore, think of base as defining an abstract base class in Jsonnet.Jsonnet compilation is completely deterministic and cannot perform external I/O, making it ideal for defining configuration. We've found that Jsonnet strikes the right balance between restrictiveness and flexibility -- previously we generated configurations using Scala code, which erred too much on the side of flexibility, leading to many headaches.At Databricks, we extended our kubectl tool (dubbed kubecfg) so that it can take Jsonnet files directly as arguments. Internally it compiles the Jsonnet to plain JSON / YAML before sending the data to Kubernetes. Kubernetes then creates or updates objects as needed based on the uploaded configuration.Composing Kubernetes Objects with JsonnetTo better understand how Jsonnet can be used with Kubernetes, let's consider the task of deploying an idealized [1] single-tenant "Databricks platform" for an enterprise customer. Here we want to create two distinct but related service deployments: a Webapp (for the interactive workspace) and Cluster manager (for managing Spark clusters). In addition,
2025-03-27