Download CrateDB

Happy New Year and happy new CrateDB…We just released a testing release of CrateDB 2.3, and here’s a short list of some of the major improvements:SQL EnhancementsUNION ALL - combine the results of multiple queriesselect id1 from t1 UNION ALL select id2 from t2Removed the 32K string size limitation in STRING columnsHyperLogLog_distinct - approximate distinct count aggregation using the HyperLogLog++ algorithm - good for quick, albeit approximate, counts on high-cardinality fieldsselect hyperloglog_distinct(col1) from t1Subqueries now supported in UPDATE, DELETE, INSERT statementsupdate t1 set name = ‘hello’ where id in (select id from t2)ORDER BY, GROUP BY - global aggregates can operate on unindexed columnsSubqueries that filter on primary keys run fasterAdministration and ConnectivityNew system monitoring metrics, including container environment (cgroup) metricsAutomatic shard rebalancing can be disabled and controlled manuallyPostgreSQL wire protocol compatibility improvementsSecurity User/Password AuthenticationIn 2017, we introduced host- and certificate-based authentication and in-flight data encryption.In 2.3, we have added secure user/password authentication to control access via the CrateDB REST and PostgreSQL wire protocol interfaces and the CrateDB Admin UI and command line tools. Passwords are never leaked, nor stored in clear text.Beautiful new documentationThe organization of the CrateDB documentation was overhauled completely to make it easier to navigate and search.A Getting Started section was added for people who’s new year’s resolution is to learn CrateDBThe rest of what’s new in 2.3…You can read about all the rest of the 2.3 enhancements in the release notes. And if you haven’t done so already, you can download CrateDB 2.3 (testing release) from here.Have fun with 2.3, and we wish you an excellent 2018!

Install CrateDB deb packages using the apt package manager.This installation method is suitable for Debian systems and derivateslike Ubuntu.Configure package repository¶You will need to configure your system to register with and trust packages fromthe CrateDB package repository:# Install prerequisites.sudo apt updatesudo apt install --yes apt-transport-https apt-utils curl gnupg lsb-release# Import the public GPG key for verifying the package signatures.curl -sS | \ sudo tee /etc/apt/trusted.gpg.d/cratedb.asc# Add CrateDB repository to Aptecho "deb default main" | \ sudo tee /etc/apt/sources.list.d/crate-stable.listNoteCrateDB provides two repositories. A stable and a testing repository. To usethe testing repository, replace stable with testing in the commandabove. You can read more about our release workflow.Now, update the package sources:You should see a success message. This indicates that the CrateDB packagerepository is correctly registered.Install CrateDB¶With everything set up, you can install CrateDB:sh$ sudo apt install crateAfter the installation is finished, you can start the crate service:sh$ sudo systemctl start crateOnce the service is up and running, you can access CrateDB by visiting:Configure CrateDB¶Please visit the Configuration Settings documentation section to learnabout the location and meaning of CrateDB’s configuration files.Control CrateDB on Linux¶You can control the crate service with the systemctl utility program:sudo systemctl COMMAND crateReplace COMMAND with start, stop, restart, status andso on.Notes¶After the installation is finished, the crate service should be installed,but may not be configured to start automatically. Use the following command tostart CrateDB:sudo systemctl start crateIn order to make the service reboot-safe, invoke:sudo systemctl enable cratePost-install notes¶After successfully installing CrateDB, for example on your workstation, the web-basedAdmin UI can be visited at:Also, let us outline those information entrypoints as suggestions to explore next:Read more details about the Configuration.The background about Bootstrap checks.Multi-node configuration within the section about Clusteringand Going into production.When operating a CrateDB cluster in production, please also takeperformance tuning into consideration.NoteThis kind of installation flavor

Getting CrateDB working on Azure with Linux or Windows is a simple process. Youcan use Azure’s management console or CLI interface (Learn how to installhere).Table of contentsAzure and LinuxCreate a resource groupCreate a network security groupCreate a virtual networkCreate virtual machinesInstall CrateDBConfigure CrateDBAzure and WindowsInitial setupCreate virtual machinesInstall CrateDBConfigure CrateDB and WindowsAzure and Linux¶Create a resource group¶Azure uses ‘Resource Groups’ to group together related services and resourcesfor easier management.Create a resource group for the CrateDB cluster by selecting Resource groupsunder the new left hand panel of the Azure portal.Create a network security group¶CrateDB uses two ports, one for inter-node communication (4300) and one forit’s http endpoint (4200), so access to these needs to be opened.Create a New Security Group, giving it a name and assigning it to the‘Resource Group’ just created.Find that security group in your resources list and open it’s settings,navigating to the Inbound security rules section.Add a rule for each port:Create a virtual network¶To create a cluster of CrateDB nodes on some cloud hosting providers, CrateDBrelies on unicast for inter-node communication.The easiest way to get Unicast communication working with Azure is to create aVirtual Network (+ -> Networking -> Virtual Network) so that all the clusternodes exist on the same IP range. Give the network a name, a region and letAzure handle all the remaining settings by clicking the next arrow on eachscreen.Once the Virtual Network has been created, find it in your resources list, openthe edit screen and the Subnets setting. Add the security group createdearlier to the subnet.Create virtual machines¶Next create virtual machines to act as your CrateDB nodes. In this tutorial, Ichose two low-specification Ubuntu 14.04 servers, but you likely have your ownpreferred configurations.Most importantly, make sure you select the Virtual Network created earlier.Install CrateDB¶Note that these instructions should be followed on each VM in your cluster.To Install CrateDB, ssh into your VMs and follow the standard process forLinux installation, this will automatically start an instance of CrateDB,which we will need to restart after the next step.Configure CrateDB¶Note that these instructions should be followed on each VM in your cluster.To set the Unicast hosts for the CrateDB cluster we change the defaultconfiguration file at /etc/crate/crate.yml.Uncomment / add these lines:CrateDB VersionReferenceConfiguration Examplelatestdiscovery.seed_hosts: - node1.example.com:4300 - node2.example.com:4300 - 10.0.1.102:4300 - 10.0.1.103:43003.3discovery.zen.ping.unicast.hosts: - node1.example.com:4300 - node2.example.com:4300 - 10.0.1.102:4300 - 10.0.1.103:4300Note: You might want to try DNS based discovery for inter-node communication.Uncomment and set the cluster nameRestart CrateDB service crate restart.Azure and Windows¶Initial setup¶To create a Resource Group, Network security group and virtual network, followthe same steps as for Azure and Linux.Create virtual machines¶Similar steps to creating Virtual Machines for Azure and Linux, but create theVM based on the ‘Windows Server 2012 R2 Datacenter’ image.Install CrateDB¶Note that these instructions

Use cases Database for Log & Event Analysis CrateDB transforms log and event analysis by providing real-time insights from vast volumes of log data generated by applications and systems. Its scalable ingestion engine and automatic indexing enable rapid querying, full-text search and semantic search, allowing businesses to detect anomalies, troubleshoot issues, and optimize performance instantly. With CrateDB, unlock powerful log analytics that drive informed decision-making and enhance operational efficiency. Millions of data points per second CrateDB allows to ingest huge volumes of log data at very high speed on clusters of inexpensive servers. Intuitive data modeling CrateDB can collect any type of data (structured, unstructured, semi-structured, time-series, geospatial, BLOB) and can accept any type of log without any specific development needed. Data indexing on the fly CrateDB indexes all data on the fly right upon ingestion, which enables immediate analysis. It also indexes all fields by default, with no impact on performance, thanks to columnar indexing. This is very useful to perform multiple types of ad-hoc queries. Native SQL CrateDB implements SQL natively, making it easy for any developer to use and integrate. No additional knowledge is required. Real-time query performance CrateDB enables in-memory SQL query performance thanks to parallel query processing and distributed columnar field caches. Always on Real-time analytics for video streaming Real-time analytics for video streaming Learn how Bitmovin improves the streaming experience with real-time analytics. User stories Log & event analytics Bitmovin is a leading video streaming company. They use CrateDB to store 140 terabytes of storage, both user events and user interactions. Every day, there is one billion of new lines of data, with the largest tables containing around 60 billion playback events. "It is through the use of CrateDB that we are able to offer our large-scale video analytics component in the first place. Comparable products are either not capable of handling the large flood of data or they are simply too expensive."Daniel Hölbling-InzkoSenior Director of Engineering - AnalyticsBitmovin Log & event analytics DriveNow helps travelers find and easily compare the best online rates for car and campervan rentals in real time. They use

CrateDB supports two kinds of logging:Application logging with Log4jJava Virtual Machine (JVM) garbage collection loggingWe use “application” here to distinguish between CrateDB running as a Javaapplication and the JVM itself, which runs CrateDB.Because garbage collection logging is a native feature of the JVM it behavesdifferently and is configured differently.Table of contentsApplication loggingLog4jConfiguration fileLog levelsRun-time configurationJVM loggingGarbage collectionEnvironment variablesApplication logging¶Log4j¶CrateDB uses Log4j.Configuration file¶You can configure Log4j with the log4j2.properties file in the CrateDBconfiguration directory.The log4j2.properties file is formatted using YAML and simplifies Log4jconfiguration by allowing you to use the PropertyConfigurator but withouthaving to tediously repeat the log4j prefix.Here’s one example:rootLogger.level = inforootLogger.appenderRef.console.ref = console# log query execution errors for easier debugginglogger.action.name = org.crate.action.sqllogger.action.level = debugappender.console.type = Consoleappender.console.name = consoleappender.console.layout.type = PatternLayoutappender.console.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] %marker%m%nAnd here is a snippet of the generated properties ready for use with log4j.You get the point.Log levels¶Possible log levels are the same as for Log4j, in order of increasingimportance:TRACEDEBUGINFOWARNERRORLog levels must be provided as string literals in the SET statement.NoteBe careful using the TRACE log level because it’s extremely verbose,can obscure other important log messages and even fill up entire data disksin some cases.Run-time configuration¶It’s possible to set the log level of loggers at runtime using SET, like so:SET GLOBAL TRANSIENT "logger.action" = 'INFO';In this example, the log level INFO is applied to the action logger.In addition to being able to configure any of the standard loggers, you can configure the root (i.e. default) loggerusing logger._root.As with any setting, you can inspect the current configuration by querying thesys.cluster table.TipRun-time logging configuration is particularly useful if you are debugginga problem and you want to increase the log level without restarting nodes.Run-time logging configuration is applied across the whole cluster, andoverrides the start-up configuration defined in each respectivelog4j2.properties file.CautionThe RESET statement is supported but logging configurationis only reset when the whole cluster is restarted.JVM logging¶CrateDB exposes some native JVM logging functionality.Garbage collection¶CrateDB logs JVM garbage collection times using the built-in garbagecollection logging of the JVM.Environment variables¶The following environment variables can be used to configuregarbage collection logging.CRATE_DISABLE_GC_LOGGING: boolean integer (default: 0)Whether to disable garbage collection logging.Set to 1 to disable.NoteSince CrateDB 3.0, Garbage collection logging is enabled by default.CRATE_GC_LOG_DIR: path to logs directory (default: varies)The log file directory.For a basic installation, the logs directoryin the CRATE_HOME directory is the default.f you have installed a CrateDB Linux package, thedefault directory is /var/log/crate instead.CRATE_GC_LOG_SIZE: file size (default: 64m)Maximum file size of log